An environmental monitoring method and apparatus

Abstract

The application discloses an environment monitoring method and device, which is used for realizing accurate counting of river floating objects in a river monitoring image, so that a more accurate and real-time monitoring result of the river environment quality is obtained. The environment monitoring method provided by the application comprises the following steps: determining a river monitoring image; performing Gaussian filtering and smoothing processing on the river monitoring image, and performing down-sampling on the image obtained through the Gaussian filtering and smoothing processing for a preset number of times; constructing a Gaussian pyramid model by using the image obtained through the Gaussian filtering and smoothing processing and the image obtained through each down-sampling; the Gaussian pyramid model comprises multiple layers of images, wherein the bottom layer of image is the image obtained through the Gaussian filtering and smoothing processing, and each layer of image other than the bottom layer is the image obtained through one down-sampling; determining a region of interest of the river monitoring image by using each layer of image of the Gaussian pyramid model; counting the river floating objects in the region of interest; and determining the river environment quality based on the counting result.

Description

An environmental monitoring method and device Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an environmental monitoring method and apparatus. Background Technology

[0002] Rivers are an important part of the Earth's environment and the foundation for human survival and development. With economic and industrial development, river pollution incidents have become increasingly frequent. Among these, floating debris is a major concern, including discarded household waste and dead plants and animals. If not cleaned up promptly, it will damage the aquatic ecosystem.

[0003] Most existing technologies only use deep learning networks to detect whether there are floating objects in river images. However, river images are subject to many interfering factors that can affect the detection results, making it impossible to accurately and in real time assess the quality of the river environment and thus preventing the timely implementation of appropriate protection strategies. Summary of the Invention

[0004] This application provides an environmental monitoring method and apparatus to accurately count floating objects in river monitoring images, thereby obtaining more accurate and real-time monitoring results of river environmental quality.

[0005] An environmental monitoring method provided in this application includes:

[0006] Identify river monitoring images;

[0007] The river monitoring image is smoothed using Gaussian filtering, and the image obtained by Gaussian filtering is downsampled a preset number of times. A Gaussian pyramid model is constructed using the image obtained by Gaussian filtering and the image obtained by each downsampling. The Gaussian pyramid model includes multiple layers of images, where the bottom layer image is the image obtained by Gaussian filtering, and each other layer image is the image obtained by one downsampling. The region of interest of the river monitoring image is determined using each layer of the Gaussian pyramid model.

[0008] Count the floating debris in the river within the region of interest, and determine the river's environmental quality based on the counting results.

[0009] This method involves determining river monitoring images; performing Gaussian filtering smoothing on the river monitoring images; and downsampling the smoothed images a predetermined number of times. A Gaussian pyramid model is constructed using the smoothed images and the images obtained from each downsampling. The Gaussian pyramid model comprises multiple layers of images, where the bottom layer is the smoothed image obtained through Gaussian filtering, and each other layer is the image obtained from one downsampling. The region of interest (ROI) of the river monitoring image is determined using each layer of the Gaussian pyramid model. Floating debris within the ROI is counted, and the river environmental quality is determined based on the counting results. By accurately counting floating debris in the river monitoring images, an accurate and real-time assessment of the river environmental quality can be achieved, enabling timely implementation of appropriate protection strategies.

[0010] In some embodiments, determining the region of interest (ROI) of the river monitoring image using each layer of the Gaussian pyramid model includes:

[0011] By sliding a preset sliding window across each layer of the Gaussian pyramid model image, and using the directional gradient histogram of the image region where the sliding window slides each time, the region of interest in the river monitoring image is determined.

[0012] This method enables more precise screening of key areas in river monitoring images using image information of different resolutions, thereby improving the accuracy of subsequent river debris counting.

[0013] In some embodiments, the method includes:

[0014] For each layer of the Gaussian pyramid model, the image region where the sliding window slides each time is determined using the gradient direction and gradient magnitude of each pixel in the image region. The histogram of directional gradients (HCG) of the image region is then used to obtain the HCG feature vector of the image region. For each layer of the Gaussian pyramid model, the HCG feature vectors of the image regions where the sliding window slides each time are concatenated to obtain the gradient image corresponding to that layer.

[0015] The gradient images corresponding to each layer of the Gaussian pyramid model are scaled to the same size and then added together. The average of the sums is then calculated to obtain the region of interest of the river monitoring image.

[0016] This method enables precise location of floating debris in river monitoring images, thereby improving the accuracy of subsequent generation of river debris density maps and ultimately enhancing the accuracy of river debris counting.

[0017] In some embodiments, before performing Gaussian filtering smoothing on the river monitoring image, the method further includes:

[0018] The river monitoring images were processed using gamma correction.

[0019] This method allows for the adjustment of the contrast between light and dark areas in river monitoring images, reducing the impact of local shadows and changes in lighting, while also suppressing noise interference.

[0020] In some embodiments, counting river debris within the region of interest includes:

[0021] The image within the region of interest is convolved using a backbone network consisting of multiple convolutional layers to obtain a first feature map.

[0022] The first feature map is convolved through multiple parallel branches with different receptive fields to obtain feature maps of various granularities. Each feature map of a certain granularity corresponds to the processing result of one of the branches. The branch is a convolutional layer of a convolutional neural network.

[0023] Based on the first feature map and the feature maps of various granularities, a second feature map is obtained;

[0024] The second feature map is convolved by a decoder consisting of multiple convolutional layers to obtain a river floating object density map. The river floating object density map is then used to count the river floating objects to obtain the river floating object count result corresponding to the river monitoring image.

[0025] This method obtains a first feature map based on the region of interest, and then obtains a second feature map based on the first feature map. This enables the acquisition of richer image features in complex scenes, reduces the impact of image granularity changes on the accuracy of river floating object detection, and thus obtains a more accurate river floating object density map, improving the accuracy of river floating object counting.

[0026] In some embodiments, obtaining the second feature map based on the first feature map and the feature maps of multiple granularities includes:

[0027] For each feature map obtained from the aforementioned branch, the matrix corresponding to the first feature map is subtracted from the matrix corresponding to the feature map obtained from that branch. The feature map obtained from the subtraction is then passed through a preset activation function to obtain a floating object correlation weight map. The probability that the image within the region of interest contains floating objects is determined based on the floating object correlation weight map. The matrix corresponding to the floating object correlation weight map is multiplied from the matrix corresponding to the feature map obtained from that branch to obtain a feature map for enhanced display of the area where the river floating objects are located.

[0028] The enhanced feature map of each of the river floating objects is used to fuse image features with the first feature map to obtain a second feature map.

[0029] This method enhances the display of the area where floating objects are located in the river and suppresses the display of the river background, thereby further improving the detection accuracy of floating objects in the river.

[0030] In some embodiments, the step of using the river debris density map to count river debris and obtaining the river debris count result corresponding to the river monitoring image includes:

[0031] The river floating debris density map was subjected to image morphological processing;

[0032] An integration operation is performed on the river floating debris density map after the image morphological processing to obtain the river floating debris count result corresponding to the river monitoring image.

[0033] This method eliminates interference areas in the river debris density map, further enhances the display of areas where river debris is located, reduces misjudgments, and improves the accuracy of river debris counting.

[0034] Another embodiment of this application provides an environmental monitoring device, which includes a memory and a processor, wherein the memory is used to store program instructions, and the processor is used to call the program instructions stored in the memory and execute any of the methods described above according to the obtained program.

[0035] Furthermore, according to embodiments, for example, a computer program product for a computer is provided, which includes software code portions that, when the product is run on the computer, perform the steps of the methods defined above. The computer program product may include a computer-readable medium on which the software code portions are stored. Furthermore, the computer program product may be directly loaded into the computer's internal memory and / or sent via a network through at least one of an upload process, a download process, and a push process.

[0036] Another embodiment of this application provides a computer-readable storage medium storing computer-executable instructions for causing the computer to perform any of the methods described above. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 is a schematic flowchart of a method for determining a region of interest in an image according to an embodiment of this application;

[0039] Figures 2 to 4 are schematic diagrams of the directional gradient histograms provided in the embodiments of this application;

[0040] Figure 5 is a schematic diagram of a HOG feature vector provided in an embodiment of this application;

[0041] Figure 6a is a schematic diagram of a river monitoring image provided in an embodiment of this application;

[0042] Figure 6b is a schematic diagram of a horizontal gradient image of floating objects in a river provided in an embodiment of this application;

[0043] Figure 6c is a schematic diagram of a vertical gradient image of river floating objects provided in an embodiment of this application;

[0044] Figure 6d is a schematic diagram of a gradient image of floating objects in a river provided in an embodiment of this application;

[0045] Figure 7 is a schematic flowchart of a method for counting floating objects in a river according to an embodiment of this application;

[0046] Figure 8 is a schematic diagram of the specific process for obtaining multi-granularity fusion features provided in the embodiments of this application;

[0047] Figure 9 is a structural schematic diagram of a river floating debris counting model provided in an embodiment of this application;

[0048] Figure 10 is a schematic diagram of a specific process for generating a river floating debris density map according to an embodiment of this application;

[0049] Figure 11 is a schematic diagram of the training process of a river floating object counting model provided in an embodiment of this application;

[0050] Figure 12 is a schematic diagram of obtaining an input image by cropping a training set according to an embodiment of this application;

[0051] Figure 13 is a schematic diagram of a process for obtaining a river floating debris density map using a river floating debris counting model according to an embodiment of this application;

[0052] Figure 14 is a schematic flowchart of an environmental monitoring method provided in an embodiment of this application;

[0053] Figure 15 is a schematic diagram of an environmental monitoring system provided in an embodiment of this application;

[0054] Figure 16 is a schematic diagram of the overall process of an environmental monitoring method provided in an embodiment of this application;

[0055] Figure 17 is a schematic diagram of the structure of an environmental monitoring device provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0057] This application provides an environmental monitoring method and apparatus to accurately count floating objects in river monitoring images, thereby obtaining more accurate and real-time monitoring results of river environmental quality.

[0058] The method and apparatus are based on the same concept of the application. Since the methods and apparatus solve problems in similar ways, the implementation of the apparatus and methods can refer to each other, and the repeated parts will not be described again.

[0059] The terms "first," "second," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0060] The following examples and embodiments are to be understood as illustrative only. While this specification may refer to "a," "an," or "some" examples or embodiments in several places, this does not mean that every such reference relates to the same example or embodiment, nor does it mean that the feature applies only to a single example or embodiment. Individual features of different embodiments may also be combined to provide other embodiments. Furthermore, terms such as "comprising" and "including" should be understood not to limit the described embodiments to consisting only of those features mentioned; such examples and embodiments may also include features, structures, units, modules, etc., not specifically mentioned.

[0061] The various embodiments of this application will now be described in detail with reference to the accompanying drawings. It should be noted that the order in which the embodiments are presented in this application represents only a chronological order and does not represent the superiority or inferiority of the technical solutions provided by the embodiments.

[0062] It should be noted that the technical solutions provided in this application embodiment are illustrated using environmental monitoring of river monitoring images collected by monitoring equipment as an example, but are not limited thereto.

[0063] 1. In the embodiments of this application, the term "receptive field" refers to the size of the region on the input image that the pixels on the feature map output by each layer of the deep learning network are mapped to.

[0064] 2. In the embodiments of this application, the term "floating object correlation weight map" represents the probability that a certain region in the image is a floating object in the river. The closer the probability is to 1, the greater the probability that the region is a floating object in the river. The closer the probability is to 0, the smaller the probability that the region is a floating object in the river.

[0065] 3. In the embodiments of this application, the term "area ratio" refers to the ratio obtained by dividing the number of pixels in the area where the floating objects in the river are located by the number of pixels in the river monitoring image.

[0066] Rivers are an important part of the Earth's environment and the foundation for human survival and development. With economic and industrial development, river pollution incidents are frequent. Among these, floating debris is a major concern, including discarded household waste and dead plants and animals. If not cleaned up promptly, it will damage the aquatic ecosystem. Current technologies mostly use deep learning networks to detect the presence of floating debris in river images. However, many interfering factors exist in river imagery, easily affecting the detection results and hindering accurate, real-time assessment of river environmental quality, thus preventing timely implementation of appropriate protection strategies.

[0067] To address the aforementioned issues, this application proposes an environmental monitoring method that can be applied to monitoring equipment. This method determines the region of interest (ROI) in a river monitoring image using an oriented gradient histogram and counts the floating objects within that ROI using a trained river debris counting model, thereby achieving dynamic assessment of river environmental quality.

[0068] Referring to Figure 1, an embodiment of this application provides a method for determining the region of interest in an image, which includes, for example, the following steps:

[0069] Step S101: Obtain river monitoring images;

[0070] The river monitoring images in this step are obtained, for example, by acquiring river monitoring video through monitoring equipment, segmenting the video frame by frame using video processing tools to obtain multiple frames of river monitoring images, and using any one of these frames as the river monitoring image.

[0071] Step S102: The color space of the river monitoring image obtained in step S101 is standardized using the Gamma correction method;

[0072] The main purpose of this step is to adjust the contrast of the river monitoring images, reduce the impact of local shadows and changes in lighting, and suppress noise interference.

[0073] Step S103: Perform Gaussian filtering smoothing on the image obtained by Gamma correction, and perform multiple downsampling operations on the image obtained by Gaussian filtering smoothing. Use the image obtained by each downsampling operation and the image obtained by Gaussian filtering smoothing to construct a Gaussian pyramid model.

[0074] For example, this step constructs a three-layer Gaussian pyramid model based on the image obtained through Gaussian filtering and smoothing, as well as images obtained through multiple consecutive downsampling processes. The bottom layer image of the Gaussian pyramid model is the image obtained through Gaussian filtering and smoothing; the middle layer images are obtained by downsampling the bottom layer image, for example, by 1 / 4; and the top layer image is obtained by downsampling the middle layer images in the same way. From the bottom layer to the top layer of the Gaussian pyramid model, the image size gradually decreases, and the image resolution gradually decreases. The Gaussian pyramid model can effectively describe image information in a multi-scale manner, facilitating subsequent processing.

[0075] Step S104: Slide a preset sliding window across each layer of the Gaussian pyramid model image, and use the directional gradient histogram of the image region where the sliding window slides each time to determine the region of interest in the river monitoring image.

[0076] This step involves sliding a preset sliding window across each layer of the Gaussian pyramid model with a preset step size. For each sliding window in the image region, the horizontal and vertical gradients of each pixel in that region are calculated using the following formulas one and two.

[0077] G x (x,y)=H(x+1,y)-H(x-1,y)Formula 1

[0078] G y (x,y)=H(x,y+1)-H(x,y-1)Formula 2

[0079] Among them, G x (x,y), G y (x,y) represent the horizontal and vertical gradients at the image pixel (x,y), respectively; H(x+1,y), H(x-1,y), H(x,y+1), and H(x,y-1) represent the pixel values ​​at the image pixels at (x+1,y), (x-1,y), (x,y+1), and (x,y-1), respectively.

[0080] Based on the calculated horizontal and vertical gradients of each pixel, the gradient magnitude and gradient direction of each pixel are calculated using the following formulas 3 and 4.

[0081]

[0082]

[0083] Where G(x,y) represents the gradient magnitude of the pixel at (x,y); α(x,y) represents the gradient direction of the pixel at (x,y) (its value is any angle within the range of 0 to 360 degrees).

[0084] In some embodiments, the gradient direction of a pixel can be pre-divided into intervals using angles. For example, the angle of the gradient direction is a radian value ranging from 0 to 360 degrees. Since two completely opposite directions are considered the same, the absolute value of the gradient direction is taken for simplicity of calculation. Therefore, the angle range of the gradient direction is 0 to 180 degrees. For example, this angle range of 0 to 180 degrees can be divided into 9 equal parts, i.e., 9 intervals, each interval being 20 degrees, from the 1st interval to the 9th interval. These 9 intervals are assigned numbers, for example, interval numbers are 0, 1, 2, 3, 4, 5, 6, 7, 8, and the angles corresponding to each interval are [0, 20], (20, 40], ... (160, 180), respectively. For example, 20 degrees corresponds to interval 0.

[0085] Therefore, in some embodiments, an initial value of 0 is set for each interval. For the gradient direction (i.e., angle) of each pixel within the image region that the sliding window slides to each time, the gradient magnitude of that pixel can be projected onto the interval corresponding to its gradient direction. For example, if the gradient direction of a pixel is 20 degrees (corresponding to interval 0) and the gradient magnitude is 2, then the value corresponding to interval 0 is increased by 2; as another example, if the gradient direction of a pixel is 110 degrees (corresponding to interval 5) and the gradient magnitude is 3, then the value corresponding to interval 5 is increased by 3. And so on. For any image region that the sliding window slides to each time, the gradient magnitude of each pixel within that image region can be projected onto the interval corresponding to its gradient direction, thus obtaining the values ​​corresponding to each of the 9 intervals, i.e., 9 values. Then, using these 9 values, a histogram with 9 intervals is obtained (for example, as shown in Figures 2-4, the larger the value, the higher the corresponding height), which is the directional gradient histogram of that image region (also called the current sliding window).

[0086] Besides the method described above of projecting the gradient magnitude of a pixel to the interval corresponding to the gradient direction of that pixel, other methods can also be used for projection. In some embodiments, for example, on any layer of an image, when the gradient direction of any pixel within a sliding window at any region position on that layer of the image is 70 degrees and the gradient magnitude is 2, since 70 degrees is between 60 and 80 degrees, the gradient magnitude corresponding to this pixel needs to be distributed to the third interval (i.e., interval 2) and the fourth interval (i.e., interval 3) according to a linear ratio. 70 degrees differs from 60 degrees by 10 degrees and from 80 degrees by 10 degrees. The ratio of the two differences is 1:1. Therefore, the gradient magnitude is distributed to the intervals corresponding to 60 degrees and 80 degrees in a 1:1 ratio. For example, the value in interval 2 is increased by 1, and the value in interval 3 is increased by 1.

[0087] For each sliding window, the histogram of directional gradients is obtained for the image region it occupies. The magnitude of the gradients in each interval is counted to obtain the feature vector of the histogram of directional gradients for the image region where the sliding window is located. This feature vector of the histogram of directional gradients is 9-dimensional. For example, if the gradient magnitudes corresponding to the 1st to 9th intervals are 10, 8, 12, 6, 8, 10, 12, 10, 12, then the feature vector of the histogram of directional gradients for the sliding window is (10, 8, 12, 6, 8, 10, 12, 10, 12).

[0088] For each pixel, the gradient magnitude and gradient direction can be calculated. It can be regarded as a directional vector, as shown in Figure 5. That is to say, each pixel can be regarded as a directional arrow. The length of the arrow corresponds to the gradient magnitude, and the angle of the arrow corresponds to the gradient direction.

[0089] By using histograms of 9 intervals to statistically analyze the pixel gradient information of each sliding window, we can obtain the gradient information statistics chart of that sliding window, namely the directional gradient histogram, as shown in Figures 2-4. The horizontal axis represents the gradient direction, and the vertical axis represents the gradient magnitude.

[0090] As shown by the histogram of oriented gradients (OARs), the gradient magnitude changes significantly in a certain gradient direction at the edges of floating objects in the river. Considering the significant difference between the OARs of the floating objects and the background in the river monitoring image, and given the similar characteristics of the river background (which is generally periodic), the difference in gradient magnitude between the river and background is small. For example, as shown in Figures 6a-6d, the distribution patterns of image gradients at the floating objects and the river background are clearly different. When a sliding window moves across the image, sliding windows within the same river background area show similar gradient magnitude and direction distributions. Converting these into OARs yields similarly shaped OARs. If the OAR obtained within the current sliding window is similar to that of an adjacent sliding window, the probability of the area containing floating objects is low. If it differs significantly from adjacent sliding windows, the image area where the sliding window is located is a region of interest (ROI).

[0091] At the edge of floating objects in the river, the directional gradient histogram will show a large variation in a certain gradient direction. For example, as shown in Figure 2, in the directional gradient histogram of the image region where the sliding window is located, the height (i.e., gradient magnitude, i.e., the value corresponding to the interval) of each interval does not change abruptly, that is, they are not much different (within a preset range, which can be determined according to actual needs). This indicates that the image region where the sliding window is located does not contain the edge information of floating objects in the river. However, the gradient magnitudes of each interval in Figures 3 and 4 differ significantly. The gradient magnitude of interval 4 in Figure 3 changes abruptly, and the gradient magnitudes of intervals 1 and 8 in Figure 4 change abruptly. This indicates that the image regions where the sliding windows in Figures 3 and 4 are located contain the edge of floating objects in the river.

[0092] As the sliding window moves across each layer of the Gaussian pyramid model, the histogram of directional gradients (HCGs) obtained at each slide yields the HCG feature vector of the image region at each slide location. These HCG feature vectors from each layer are concatenated, and the concatenation result is visualized, for example, using Python's OpenCV, to obtain the gradient image corresponding to that layer. This gradient image identifies the areas where floating debris is located, as shown in Figures 6a-6d. Figure 6a represents a river monitoring image, Figure 6b represents the horizontal gradient image of floating debris, Figure 6c represents the vertical gradient image of floating debris, and Figure 6d represents the gradient image of floating debris. The white box in Figure 6d marks the marked areas where floating debris is located. By summing the pixel values ​​of the gradient images corresponding to each layer of the Gaussian pyramid model and averaging the sum, the region of interest (i.e., the area where floating debris is located) of the river monitoring image is obtained. For example, the sizes of the three layers of the Gaussian pyramid model are N*N, 1 / 4N*1 / 4N, and 1 / 16N*1 / 16N, respectively. This will result in gradient images of N*N, 1 / 4N*1 / 4N, and 1 / 16N*1 / 16N. These gradient images are scaled back to N*N (the original size), and the pixel values ​​of the scaled gradient images are summed. The sum is then averaged (or a weighted sum of the pixel values ​​of each scaled gradient image, for example, each gradient image has a weight of 0.33, so the pixel values ​​are multiplied by 0.33 before summing). This yields an N*N gradient image on which the region of interest for the river monitoring image has been determined, as shown by the white boxes in Figure 6d.

[0093] Referring to Figure 7, an embodiment of this application provides a method for counting river debris. Taking the counting of river debris in the region of interest using a river debris counting model as an example, the structure of this river debris counting model includes, for example, a backbone network (network front-end), a multi-granularity feature extraction module, a multi-granularity feature fusion module, and a regression module. The specific steps include:

[0094] Step S201: Input the region of interest of the river monitoring image into a backbone network consisting of the first 13 convolutional layers of, for example, VGG-16 (including 13 convolutional layers and 3 fully connected layers) for convolutional processing to obtain the basic feature map.

[0095] In this step, the basic feature map is represented by F. v If F represents F, then v It can be defined as:

[0096] F v =f νgg (I) Formula 5

[0097] Among them, f νggF represents the backbone network. v This represents the basic feature map obtained through backbone network processing.

[0098] Step S202: The basic feature map obtained in step S201 is convolved by the Multi-granularity Feature Extraction Module (MFEM) to obtain feature maps of multiple granularities.

[0099] The multi-granularity feature extraction module in this step consists of multiple parallel branches with different receptive fields. Each branch is, for example, a convolutional layer of a convolutional neural network. This module performs convolutional processing on the basic feature map to obtain feature maps of different granularities, thereby acquiring richer image features in complex scenes. This improves the generalization performance of the river debris counting model, enhances the accuracy of the river debris density map generated by the model, and ultimately achieves the goal of accurate river debris counting.

[0100] Specifically, the basic feature map F v The input is fed into an MFEM module consisting of four parallel branches with different receptive fields. For example, the first branch of the MFEM module is a convolutional layer with a kernel size of 3×3 and a dilation rate of 6, the second branch is a convolutional layer with a kernel size of 3×3 and a dilation rate of 12, the third branch is a convolutional layer with a kernel size of 3×3 and a dilation rate of 18, and the fourth branch is a global adaptive average pooling layer with an output feature map size of 1×1.

[0101] It should be noted that the kernel sizes of the convolutional layers from the first to the third branch can also be different. A larger kernel means a larger receptive field for the MFEM module, while a smaller kernel means a smaller receptive field. In some embodiments, the first to the third branch use small convolutional kernels of the same size, such as 3×3, with increasing dilatation rates to provide different receptive fields, thereby obtaining feature maps of different granularities (i.e., different scales). The number of parallel branches used by the MFEM module, as well as the kernel size and dilatation rate used in each branch, are set according to actual needs, and no restrictions are imposed in this embodiment.

[0102] Based on F v The MFEM module utilizes the different receptive fields of different branches to obtain feature maps of various granularities. The expression is: where s represents the granularity of the feature map; and i takes values ​​of 1, 2, 3, and 4, corresponding to the four different branches of the MFEM module. These represent the first to fourth branches of the MFEM module, respectively, for F. v The resulting feature map is obtained through processing.

[0103] Step S203: Based on the basic feature map and feature maps of multiple granularities, the multi-granularity feature map is obtained by processing through the Multi-granularity Feature Integration Module (MFIM).

[0104] Specifically, the MFIM module performs pixel-level subtraction operations between the basic feature maps obtained from the backbone network and the feature maps of different granularities processed by the MFEM module. The results are then subjected to dimensionality reduction and delinearization using an activation function to obtain a correlation weight map of river floating debris features, representing the probability that a feature in a certain region of the image is a river floating debris feature. This correlation weight map is then multiplied pixel-wise by the corresponding granular feature map, for example, by... The obtained floating object correlation weight map is then compared with... Pixel multiplication is performed, and the resulting feature maps are fused to obtain a multi-granularity fused feature map.

[0105] Step S204: Input the multi-granularity fusion feature map obtained in step S203 into the decoder composed of different convolutional layers to generate a river floating object density map, and perform an integration operation on the river floating object density map to obtain the number of river floating objects corresponding to the river monitoring image.

[0106] Referring to Figure 8, the specific steps for obtaining a multi-granularity fused feature map based on the basic feature map and feature maps of various granularities through the multi-granularity feature fusion module include:

[0107] Step S301: Transfer the basic feature map F v The input is fed into a convolutional layer with a kernel size of, for example, 1×1, for dimensionality reduction;

[0108] The purpose of dimensionality reduction is to make F v The result obtained in step S202 above Same dimensions;

[0109] Step S302: Convert the dimensionality-reduced base feature map F v Compare these features with the four feature maps of different granularities obtained in step S202 above. Performing an image subtraction operation yields the lost detail feature map, which is called the contrast feature map.

[0110] In the feature extraction stage of the river floating debris counting model, the region of interest is processed by the network front-end (i.e., backbone network) and MFEM module as shown in Figure 9, which will result in the loss of some detailed features. If these detailed features can be recovered, the quality of the river floating debris density map generated by the river floating debris counting model and the counting performance of the river floating debris counting model will be improved.

[0111] Step S303: Dimensionality reduction of the contrast feature map obtained in step S302 is performed using a convolutional layer with a kernel size of, for example, 1×1.

[0112] The purpose of dimensionality reduction here is to reduce the number of parameters, reduce the computational cost of the river floating debris counting model, and prepare for subsequent steps.

[0113] Step S304: Delinearize the dimensionality-reduced contrast feature map, for example, by using a Sigmoid activation function, to obtain a floating object correlation weight map M. i The definition is as follows:

[0114]

[0115] in, F represents the four feature maps with different granularities obtained in step S202 above. v Represents the basic feature map, C 1×1 This indicates a convolutional layer with a kernel size of 1×1, and Sigmoid represents the activation function.

[0116] The pixel values ​​in the floating object correlation weight map are normalized. After normalization, the closer the pixel value of a certain area is to 1, the greater the probability that the area is a floating object in the river. The closer the pixel value is to 0, the smaller the probability that the area is a floating object in the river.

[0117] Step S305: Calculate the floating object correlation weight map M obtained in step S304. i Compare these features with the four feature maps of different granularities obtained in step S202 above. Perform pixel-by-pixel multiplication, and then fuse the feature map obtained by pixel-by-pixel multiplication (i.e., the feature map that enhances the display of the area where the floating objects in the river are located) with the basic feature map (i.e., the first feature map) to obtain a multi-granularity fused feature map (i.e., the second feature map);

[0118] M i and Pixel-wise multiplication is performed to optimize feature maps of various granularities by utilizing the lost detail information weight map. The optimized feature maps of various granularities are then compared with the base feature map F. vChannel-wise fusion is performed (features of the same size in different dimensions are superimposed together; for example, as shown in Figure 9, the MFIM module overlaps the four squares on the left and right to obtain a multi-granularity fused feature map) to ensure the completeness of feature extraction. Multi-granularity fused feature map F mfim The definition is as follows:

[0119]

[0120] Where * indicates pixel-by-pixel multiplication, and [.|.] indicates pixel-direction fusion operation (i.e., the process of pixel-by-pixel multiplication).

[0121] M i and Pixel-by-pixel multiplication is performed to enhance regions in the image with a high probability of being river debris and suppress regions with a low probability of being river debris.

[0122] Referring to Figure 10, the specific steps for generating a river debris density map using multi-granularity fusion feature maps and counting the river debris in the river debris density map include:

[0123] Step S401: Input the multi-granularity fused feature map into the decoder (i.e., the regression module) composed of different convolutional layers to obtain the river floating debris density map;

[0124] This decoder, for example, consists of six dilated convolutional layers and one convolutional layer with a kernel size of 1×1. Each dilated convolutional layer has a 3×3 kernel size and a dilation rate of 2, but the number of channels decreases progressively: 512, 512, 512, 256, 128, and 64. Finally, the feature map output from the 64-channel convolutional layer is reduced to 1 dimension by the 1×1 kernel layer, yielding the final river debris density map.

[0125] It should be noted that the dilated convolutional layer structure of the decoder can be adjusted according to actual needs, as long as the dimension of the river floating debris density map output by the decoder is 1, that is, the size of the river floating debris density map is consistent with the size of the image input to the river floating debris counting model.

[0126] Step S402: Perform image morphological processing on the river floating debris density map;

[0127] Applying techniques such as expansion erosion to the river debris density map, smoothing the edges of river debris, and breaking up small connected areas can eliminate some interference areas. Furthermore, enhancing the display of the area where river debris is located can reduce misjudgments and thus improve the counting accuracy of river debris counting models.

[0128] Step S403: Perform an integration operation on the river floating object density map obtained from image morphological processing to obtain the number of river floating objects corresponding to the river monitoring image, i.e., the river floating object count result;

[0129] The integration operation in this step involves summing the pixel values ​​of the river debris density map. The resulting value is the number of river debris in the river monitoring image. For example, if the sum of all pixel values ​​in a river debris density map is 10.63, then the number of river debris in the river monitoring image is 10.

[0130] In some embodiments, the river debris density map obtained through morphological processing can be binarized, and the number of pixels in the region containing river debris in the binarized image can be counted. The ratio of this number of pixels to the total number of pixels in the river monitoring image is the area proportion of the river debris region. Changes in the area proportion reflect changes in the number of river debris; a larger area proportion indicates a greater number of river debris in the river monitoring image.

[0131] The quality of the river environment can be assessed by evaluating the quantity and / or area of ​​floating debris, thereby enabling more accurate real-time judgments on the river environment.

[0132] The aforementioned river floating object counting model is obtained by training a deep learning network. During the training process, as shown in Figure 11, the first step is to construct a dataset for training the river floating object counting model, including a training set and a test set. The river monitoring video collected by the monitoring equipment is segmented frame by frame using a video processing tool to obtain multiple frames of river monitoring images. These multiple frames of river monitoring images are then processed, and the resulting images are used to construct a river floating object image dataset for the river floating object counting model. 80% of the images in this dataset are used as the training set for the river floating object counting model, and 20% are used as the test set.

[0133] The image processing described above mainly involves data augmentation of image data, using limited image data to obtain more image data, thereby increasing the number and diversity of training samples for the river floating object counting model. This reduces computational overhead, improves the training speed and generalization ability of the river floating object counting model, enables the training process to converge faster, and makes the river floating object counting model more robust.

[0134] In some embodiments, data augmentation includes image cropping, mirroring, etc., and manually annotating the acquired river monitoring images to obtain ground truth images. The river monitoring images are randomly cropped, and the ground truth images obtained from the cropped images are processed in the same way according to the cropping location; or, the river monitoring images are randomly mirrored, and the ground truth images obtained from the river monitoring images are processed in the same way.

[0135] Using the above method, the region of interest in the training set image is determined, and the training set is randomly cropped according to the region of interest, for example, cropped to a size of 576*768, to obtain the input image for training the river floating object counting model, as shown in Figure 12.

[0136] The river floating debris counting model is trained by inputting images. The model first predicts the river floating debris density map, and then counts the river floating debris based on the predicted density map. The predicted number of river floating debris and / or the area ratio of river floating debris are obtained by the model. The results are compared with the river floating debris count results in the artificially created ground truth density map to determine the accuracy of the river floating debris counting model in predicting the number of river floating debris.

[0137] During training, a loss function is added to the river floating debris counting model to determine whether the model has converged (i.e., the value calculated by the loss function is infinitely close to 0). If it has not converged, the parameters of the river floating debris counting model are adjusted (i.e., parameter optimization), and then the model is trained again until it converges. The training ends, and a trained river floating debris counting model is obtained, as shown in Figure 13. The trained model is then tested using a test set to verify its counting effect, resulting in the river floating debris counting model described above for counting river floating debris in a region of interest.

[0138] Referring to Figure 14, an environmental monitoring method provided in this application includes the following steps:

[0139] Step S501: Obtain river monitoring images;

[0140] Step S502: Perform Gaussian filtering smoothing on the river monitoring image, and downsample the image obtained by Gaussian filtering smoothing a preset number of times. Construct a Gaussian pyramid model using the image obtained by Gaussian filtering smoothing and the image obtained by each downsampling. Use each layer of the Gaussian pyramid model to determine the region of interest of the river monitoring image.

[0141] Step S503: Input the region of interest obtained in step S502 into the river floating debris counting model to obtain the river floating debris density map;

[0142] Step S504: Perform image morphological processing on the river floating debris density map;

[0143] Step S505: Calculate the area ratio of river floating objects in the river floating object density map after image morphological processing;

[0144] Step S506: Based on the area ratio of floating debris in the river obtained in step S505, assess whether the river environmental quality is up to standard. If it is up to standard, repeat steps S501 to S505; if it is not up to standard, proceed to step S507.

[0145] In this step, by setting an area percentage threshold, when the area percentage of floating debris in the river is greater than or equal to the threshold, it indicates that there is a lot of floating debris in the river and the river's environmental quality does not meet the requirements, so the floating debris needs to be cleaned up; when the area percentage of floating debris in the river is less than the threshold, it indicates that there is a little floating debris in the river and the river's environmental quality meets the requirements, so the floating debris does not need to be cleaned up.

[0146] Step S507: Issue an early warning for substandard river environmental quality and remind relevant staff to clean up floating debris in the river in a timely manner.

[0147] Referring to Figure 15, an environmental monitoring system provided in this application embodiment includes:

[0148] Image determination module 100 is used to determine river monitoring images;

[0149] The region of interest determination module 200 is used to perform Gaussian filtering smoothing on the river monitoring image, and to downsample the image obtained by Gaussian filtering smoothing a preset number of times. The Gaussian pyramid model is constructed using the image obtained by Gaussian filtering smoothing and the image obtained by each downsampling. The region of interest of the river monitoring image is determined using the image of each layer of the Gaussian pyramid model.

[0150] The river floating object counting module 300 counts the floating objects in the river within the area of ​​interest and determines the river environmental quality based on the counting results.

[0151] Among them, the river floating object counting module 300 (i.e. the river floating object counting model mentioned above) includes, for example, the backbone network, multi-granularity feature extraction module, multi-granularity feature fusion module, and regression module mentioned above.

[0152] In summary, referring to Figure 16, an environmental monitoring method provided in this application includes:

[0153] Step S601: Determine the river monitoring image;

[0154] In this step, the river monitoring images are processed by segmenting the river monitoring video collected by the monitoring equipment into frames using video processing tools to obtain multiple frames of river monitoring images. Then, any one of these frames is used as the river monitoring image for subsequent steps.

[0155] Step S602: Perform Gaussian filtering smoothing on the river monitoring image, and downsample the image obtained by Gaussian filtering smoothing a preset number of times (e.g., the above-mentioned 1 / 4 downsampling). Construct a Gaussian pyramid model using the image obtained by Gaussian filtering smoothing and the image obtained by each downsampling. The Gaussian pyramid model includes multiple layers of images, where the bottom layer image is the image obtained by Gaussian filtering smoothing, and each other layer image is the image obtained by one downsampling. Use each layer of the Gaussian pyramid model to determine the region of interest of the river monitoring image.

[0156] This step utilizes the gradient difference information between river floating objects and the river background to provide an oriented gradient histogram for accurate localization of river floating objects in river monitoring images, thereby improving the detection accuracy of subsequent river floating object counting models.

[0157] Step S603: Count the floating objects in the river within the area of ​​interest, and determine the river environmental quality based on the counting results;

[0158] The counting can be either counting the number of floating objects in the river within the area of ​​interest, or counting the area percentage of floating objects in the river.

[0159] The river environmental quality is determined by the counting results obtained in step S603, thereby enabling a more accurate and real-time assessment of the river environment.

[0160] To more accurately filter out areas of interest in river monitoring images and improve the accuracy of subsequent floating debris counting, in some embodiments, determining the region of interest in the river monitoring image using each layer of the Gaussian pyramid model includes:

[0161] By sliding a preset sliding window across each layer of the Gaussian pyramid model image, and using the directional gradient histogram of the image region where the sliding window slides each time, the region of interest in the river monitoring image is determined.

[0162] To achieve precise location of river debris in river monitoring images and improve the accuracy of subsequent river debris density maps, in some embodiments, the method includes:

[0163] For each layer of the Gaussian pyramid model, the image region where the sliding window slides each time is determined using the gradient direction and gradient magnitude of each pixel in the image region. The feature vector of the histogram ...

[0164] The gradient images corresponding to each layer of the Gaussian pyramid model are scaled to the same size and then added together. The average of the sums is then calculated to obtain the region of interest of the river monitoring image.

[0165] To adjust the contrast of river monitoring images, reduce the impact of local shadows and lighting variations, and suppress noise interference, in some embodiments, before performing Gaussian filtering smoothing on the river monitoring images, the following steps are also included:

[0166] The river monitoring images were processed using gamma correction.

[0167] To obtain a more accurate river debris density map and improve the accuracy of river debris counting, in some embodiments, the counting of river debris within the region of interest includes:

[0168] The image within the region of interest is convolved using a backbone network consisting of multiple convolutional layers (e.g., the backbone network consisting of the first 13 convolutional layers of VGG-16 mentioned above) to obtain a first feature map (e.g., the basic feature map F mentioned above). υ );

[0169] The first feature map is convolved using multiple parallel branches with different receptive fields (e.g., the multi-granularity feature extraction module consisting of four parallel branches with different receptive fields as described above) to obtain feature maps of various granularities (e.g., the aforementioned...). Each feature map at each granularity corresponds to the processing result of one of the branches; wherein, the branch is a convolutional layer of a convolutional neural network (such as the dilated convolutional layer and pooling layer mentioned above).

[0170] Based on the first feature map and the feature maps of various granularities, a second feature map is obtained (e.g., the multi-granularity fusion feature map F mentioned above). mfim );

[0171] The second feature map is convolved by a decoder composed of different convolutional layers to obtain a river floating object density map. The river floating object density map is then used to count the river floating objects to obtain the river floating object count result corresponding to the river monitoring image.

[0172] To enhance the display of areas containing floating debris in rivers, suppress the display of river background, and improve the detection accuracy of floating debris, in some embodiments, the second feature map is obtained based on the first feature map and the feature maps of multiple granularities, including:

[0173] For each feature map obtained from the aforementioned branch, the matrix corresponding to the first feature map is used to perform a subtraction operation (i.e., the pixel-level subtraction operation described above) with the matrix corresponding to the feature map obtained from that branch. The feature map obtained from the subtraction operation is then passed through a preset activation function (e.g., the sigmoid function described above) to obtain a floating object correlation weight map (e.g., the M-axis described above). i The probability that an image within the region of interest contains floating objects is determined based on the floating object correlation weight map. The matrix corresponding to the floating object correlation weight map is multiplied by the matrix corresponding to the feature map obtained from this branch (i.e., the pixel-by-pixel multiplication mentioned above) to obtain the feature map for enhanced display of the area where the river floating objects are located (e.g., the contrast feature map mentioned above).

[0174] The enhanced feature map of each of the river floating objects is obtained and then fused with the first feature map (e.g., fusion in the channel direction as described above) to obtain the second feature map.

[0175] To further enhance the display of areas where floating objects are located, reduce false positives, and improve the accuracy of river floating object counting, in some embodiments, the process of using the river floating object density map to count river floating objects and obtain the river floating object counting result corresponding to the river monitoring image includes:

[0176] The density map of floating debris in the river is subjected to image morphological processing (e.g., the aforementioned expansion erosion);

[0177] The river debris density map after morphological processing of the image is integrated to obtain the river debris count result corresponding to the river monitoring image.

[0178] The counting result can be either the number of floating objects in the river in the aforementioned river monitoring image or the area percentage of floating objects in the aforementioned river monitoring image.

[0179] The following describes the device or apparatus provided in the embodiments of this application, and the explanations or examples of the same or corresponding technical features as those described in the above methods will not be repeated hereafter.

[0180] Referring to Figure 17, an environmental monitoring device is provided in an embodiment of this application. This device can be built into a monitoring device or can be independent of the monitoring device, including:

[0181] Processor 600 is used to read the program from memory 620 and execute the following procedures:

[0182] Identify river monitoring images;

[0183] The river monitoring image is smoothed using Gaussian filtering, and the image obtained by Gaussian filtering is downsampled a preset number of times. A Gaussian pyramid model is constructed using the image obtained by Gaussian filtering and the image obtained by each downsampling. The Gaussian pyramid model includes multiple layers of images, where the bottom layer image is the image obtained by Gaussian filtering, and each other layer image is the image obtained by one downsampling. The region of interest of the river monitoring image is determined using each layer of the Gaussian pyramid model.

[0184] Count the floating debris in the river within the region of interest, and determine the river's environmental quality based on the counting results.

[0185] In some embodiments, determining the region of interest (ROI) of the river monitoring image using each layer of the Gaussian pyramid model includes:

[0186] By sliding a preset sliding window across each layer of the Gaussian pyramid model image, and using the directional gradient histogram of the image region where the sliding window slides each time, the region of interest in the river monitoring image is determined.

[0187] In some embodiments, the processor 600 is further configured to read a program from the memory 620 and execute it:

[0188] For each layer of the Gaussian pyramid model, the image region where the sliding window slides each time is determined using the gradient direction and gradient magnitude of each pixel in the image region. The histogram of directional gradients (HCG) of the image region is then used to obtain the HCG feature vector of the image region. For each layer of the Gaussian pyramid model, the HCG feature vectors of the image regions where the sliding window slides each time are concatenated to obtain the gradient image corresponding to that layer.

[0189] The gradient images corresponding to each layer of the Gaussian pyramid model are scaled to the same size and then added together. The average of the sums is then calculated to obtain the region of interest of the river monitoring image.

[0190] In some embodiments, before performing Gaussian filtering smoothing on the river monitoring image, the method further includes:

[0191] The river monitoring images were processed using gamma correction.

[0192] In some embodiments, counting river debris within the region of interest includes:

[0193] The image within the region of interest is convolved using a backbone network consisting of multiple convolutional layers to obtain a first feature map.

[0194] The first feature map is convolved through multiple parallel branches with different receptive fields to obtain feature maps of various granularities. Each feature map of a certain granularity corresponds to the processing result of one of the branches. The branch is a convolutional layer of a convolutional neural network.

[0195] Based on the first feature map and the feature maps of various granularities, a second feature map is obtained;

[0196] The second feature map is convolved by a decoder composed of different convolutional layers to obtain a river floating object density map. The river floating object density map is then used to count the river floating objects to obtain the river floating object count result corresponding to the river monitoring image.

[0197] In some embodiments, obtaining the second feature map based on the first feature map and the feature maps of multiple granularities includes:

[0198] For each feature map obtained from the aforementioned branch, the matrix corresponding to the first feature map is subtracted from the matrix corresponding to the feature map obtained from that branch. The feature map obtained from the subtraction is then passed through a preset activation function to obtain a floating object correlation weight map. The probability that the image within the region of interest contains floating objects is determined based on the floating object correlation weight map. The matrix corresponding to the floating object correlation weight map is multiplied from the matrix corresponding to the feature map obtained from that branch to obtain a feature map for enhanced display of the area where the river floating objects are located.

[0199] The enhanced feature map of each of the river floating objects is used to fuse image features with the first feature map to obtain a second feature map.

[0200] In some embodiments, the step of using the river debris density map to count river debris and obtaining the river debris count result corresponding to the river monitoring image includes:

[0201] The river floating debris density map was subjected to image morphological processing;

[0202] An integration operation is performed on the river floating debris density map after the image morphological processing to obtain the river floating debris count result corresponding to the river monitoring image.

[0203] In some embodiments, the environmental monitoring device provided in this application further includes a transceiver 610 for receiving and sending data under the control of a processor 600.

[0204] In Figure 17, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 600 and memory represented by memory 620. The bus architecture may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. Transceiver 610 may be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium.

[0205] In some embodiments, the environmental monitoring device provided in this application further includes a user interface 630. The user interface 630 may be an interface that can connect to external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.

[0206] The processor 600 is responsible for managing the bus architecture and general processing, while the memory 620 can store the data used by the processor 600 when performing operations.

[0207] In some embodiments, the processor 600 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a CPLD (Complex Programmable Logic Device).

[0208] This application provides a computing device, which may specifically be a desktop computer, portable computer, smartphone, tablet computer, personal digital assistant (PDA), etc. The computing device may include a central processing unit (CPU), memory, input / output devices, etc. Input devices may include a keyboard, mouse, touchscreen, etc., and output devices may include display devices, such as a liquid crystal display (LCD) or a cathode ray tube (CRT).

[0209] The memory may include read-only memory (ROM) and random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In the embodiments of this application, the memory may be used to store the program of any of the methods provided in the embodiments of this application.

[0210] The processor executes any of the methods described in the embodiments of this application according to the program instructions stored in the memory.

[0211] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the methods described in the above embodiments. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0212] This application provides a computer-readable storage medium for storing computer program instructions used in the apparatus provided in the above-described embodiments, including a program for performing any of the methods provided in the above-described embodiments. The computer-readable storage medium may be a non-transitory computer-readable medium.

[0213] The computer-readable storage medium can be any available medium or data storage device that a computer can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0214] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0215] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0216] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0217] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0218] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An environmental monitoring method, characterized in that, The method includes: determining a river monitoring image; performing Gaussian filtering smoothing on the river monitoring image, and downsampling the image obtained by Gaussian filtering smoothing a preset number of times; constructing a Gaussian pyramid model using the image obtained by Gaussian filtering smoothing and the image obtained from each downsampling, wherein the Gaussian pyramid model includes multiple layers of images, wherein the bottom layer image is the image obtained by Gaussian filtering smoothing, and each other layer image is the image obtained from one downsampling; determining the region of interest (ROI) of the river monitoring image using each layer of the Gaussian pyramid model; counting river floating objects within the ROI, and determining the river environmental quality based on the counting results; wherein counting river floating objects within the ROI includes: performing convolution processing on the image within the ROI through a backbone network composed of multiple convolutional layers to obtain a first feature map; performing convolution processing on the first feature map through multiple parallel branches with different receptive fields to obtain feature maps of multiple granularities, each feature map corresponding to the processing result of one branch; wherein the branch is a convolutional layer of a convolutional neural network; based on The first feature map and the feature maps of various granularities are used to obtain a second feature map. A decoder composed of multiple convolutional layers is used to perform convolution processing on the second feature map to obtain a river floating debris density map. The river floating debris density map is then used to count river floating debris to obtain the river floating debris count result corresponding to the river monitoring image. The process of obtaining the second feature map based on the first feature map and the feature maps of various granularities includes: for each branch of the feature map, subtracting the matrix corresponding to the first feature map from the matrix corresponding to that branch of the feature map, and passing the subtracted feature map through a preset activation function to obtain a floating debris correlation weight map. The probability that an image within a region of interest contains floating debris is determined based on the floating debris correlation weight map. The matrix corresponding to the floating debris correlation weight map is multiplied by the matrix corresponding to the feature map of that branch to obtain a feature map for enhanced display of the river floating debris area. Finally, each of the obtained enhanced display feature maps for the river floating debris area is fused with the first feature map to obtain the second feature map.

2. The method according to claim 1, characterized in that, The step of determining the region of interest (ROI) of the river monitoring image using each layer of the Gaussian pyramid model includes: sliding a preset sliding window across each layer of the Gaussian pyramid model and using the directional gradient histogram of the image region where the sliding window slides at each step to determine the ROI of the river monitoring image.

3. The method according to claim 2, characterized in that, The method includes: for each layer of the Gaussian pyramid model, determining the histogram of directional gradients (HCG) of the image region where the sliding window slides each time, using the gradient direction and gradient magnitude of each pixel in the image region, and obtaining the HCG feature vector of the image region using the HCG; for each layer of the Gaussian pyramid model, concatenating the HCG feature vectors of the image region where the sliding window slides each time, to obtain the gradient image corresponding to that layer; scaling the gradient images corresponding to each layer of the Gaussian pyramid model to the same size, adding them together, and averaging the results to obtain the region of interest of the river monitoring image.

4. The method according to claim 1, characterized in that, Before performing Gaussian filtering smoothing on the river monitoring image, the method further includes processing the river monitoring image using gamma correction.

5. The method according to claim 1, characterized in that, The step of using the river debris density map to count river debris and obtain the river debris count result corresponding to the river monitoring image includes: performing image morphological processing on the river debris density map; and performing an integration operation on the image morphologically processed river debris density map to obtain the river debris count result corresponding to the river monitoring image.

6. An environmental monitoring device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method according to any one of claims 1 to 5.

7. A computer program product for use in a computer, characterized in that, Includes a software code portion that, when the product is run on the computer, is used to perform the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing the computer to perform the method according to any one of claims 1 to 5.

Images