River garbage recognition and positioning method, system and device based on deep learning

Abstract

The present application belongs to the field of computer vision, and specifically relates to a river garbage recognition and positioning method, system and device based on deep learning, aiming to solve the problems of low efficiency, high cost, inaccurate positioning and low intelligent level, etc. The present application includes: collecting multi-source image data of fixed monitoring and unmanned aerial vehicle inspection, and extracting physical characteristics such as water surface flow rate; parallel inputting the preprocessed data into a double-track deep learning model, fusing the detection results and feature vectors through a space-time correlation and coordinate mapping mechanism, and combining the water surface flow field to output the accurate geographic coordinates of the garbage, the refined classification attributes and the continuous drift trajectory. The present application realizes all-weather, automatic and high-precision monitoring and tracking of river floating garbage, and significantly improves the intelligent level and decision efficiency of water environment governance.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and specifically relates to a method, system and device for river garbage identification and localization based on deep learning. Background Technology

[0002] As an important component of urban water systems, the water quality of rivers directly affects urban ecology and residents' lives. Floating garbage in rivers, such as plastic bottles, foam boards, and dead branches and leaves, not only seriously damages the aquatic landscape and pollutes the water, but may also block waterways and sluice gates, posing safety hazards to flood control and drainage.

[0003] Currently, river debris cleanup mainly relies on manual patrols and retrieval. This method suffers from limited patrol coverage and low efficiency, making it difficult to monitor vast or remote river sections in real time, especially at night or in inclement weather conditions where blind spots exist. Furthermore, because floating debris moves with the current, manually reported location information is often delayed and inaccurate, leading to low retrieval efficiency. In addition, manual patrols have limited ability to determine the type, quantity, and density of debris, lacking systematic data support, making it difficult to make scientific cleanup decisions, trace pollution sources, and evaluate treatment effectiveness, and incurring high labor costs.

[0004] To address these issues, video surveillance systems have been deployed in some areas. However, most existing surveillance systems are designed for security purposes and lack intelligent analysis capabilities for floating objects on the water surface. The few methods that incorporate image recognition technology also have shortcomings.

[0005] On the one hand, monitoring systems based on fixed cameras have a single perspective, limited coverage, and numerous blind spots. On the other hand, these systems typically process data from fixed cameras and mobile inspection devices such as drones in isolation, failing to form a collaborative monitoring system. Furthermore, they have weak capabilities for fine-grained waste classification, making it difficult to distinguish the types and levels of harm of different types of waste, thus affecting the targeted nature of subsequent treatment. Therefore, there is an urgent need for a technological solution capable of automating, achieving high precision, and locating waste in river channels over a wide area. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, namely the low efficiency, high cost, inaccurate positioning, and low level of intelligence in current river waste monitoring and cleaning methods, this invention provides a method, system, and device for river waste identification and positioning based on deep learning.

[0007] The first aspect of this invention proposes a method for identifying and locating river debris based on deep learning, comprising the following steps:

[0008] Multi-source image data of the river channel is collected and preprocessed to generate preprocessed image data and environmental feature data containing water surface velocity and disturbance area information.

[0009] The preprocessed image data is input in parallel into the target detection track and feature analysis track of the deep learning visual processing model;

[0010] In the target detection trajectory, the preprocessed image data is processed by a first convolutional neural network to output the bounding box coordinates and preliminary category probabilities of the floating objects in the river.

[0011] In the feature analysis track, the target region in the preprocessed image data is processed by a second convolutional neural network to output a multidimensional fine-grained feature vector of the target region;

[0012] Based on the spatiotemporal correlation and coordinate mapping mechanism, the bounding box coordinates, the preliminary category probability, and the multidimensional fine-grained feature vector are fused and processed, and combined with the water surface velocity vector field, the geographic coordinates, classification attributes, and drift trajectory of river garbage are output.

[0013] Furthermore, the preprocessing step also includes:

[0014] If the scene is determined to be a strong light scene, an image enhancement algorithm based on multi-scale decomposition is executed to separate and reconstruct the reflection component of the image in the logarithmic domain.

[0015] If the scene is determined to be in low light, an adaptive histogram equalization algorithm with limited contrast is executed to perform block processing and interpolation on the image brightness channel.

[0016] If the weather is determined to be rainy or foggy, a defogging algorithm based on dark channel priors is executed to reconstruct the image by constructing an atmospheric scattering model and estimating transmittance.

[0017] The image processed by the adaptively selected algorithm is used as the preprocessed image data.

[0018] Furthermore, the processing procedure of the first convolutional neural network includes:

[0019] Multi-scale features of the preprocessed image data are extracted using the backbone network;

[0020] The multi-scale features are fused using a feature pyramid structure;

[0021] The decoupled detection heads predict the target's category and bounding box location based on the fused features.

[0022] For partially submerged targets, the bounding box is expanded to cover the inferred complete area by combining edge detection results and water surface boundary information, and the bounding box coordinates are obtained.

[0023] Furthermore, the second convolutional neural network has a multi-branch structure, and its processing includes:

[0024] Deep semantic features of the target region are extracted using the first branch network;

[0025] Surface texture features of the target region are extracted using a multi-scale filter bank of the second branch network;

[0026] The shape invariant moments and Fourier descriptors of the target region contour are extracted using the third branch network.

[0027] The color histogram and dominant hue of the target region are extracted in the HSV color space using the fourth branch network.

[0028] The deep semantic features, surface texture features, shape features, and color features are concatenated and then weighted and dimensionality reduced using an attention mechanism to generate the multidimensional fine-grained feature vector.

[0029] Furthermore, the geographic coordinates of the output river waste include:

[0030] For fixed surveillance images, the image pixel coordinates are back-projected to three-dimensional space using pre-calibrated camera imaging model parameters, and the plane coordinates of the target in the world coordinate system are calculated by introducing the water surface elevation plane constraint, and then converted into standard geographic coordinates.

[0031] For UAV inspection images, the image pixel coordinates are back-projected to the object space based on the collinearity equation, and the three-dimensional geographic coordinates of the target in the standard geodetic coordinate system are solved by combining the sensor pose parameters acquired in real time.

[0032] Furthermore, the drift trajectory of the exported river debris includes:

[0033] Between consecutive frames, a multi-target tracking algorithm is used to integrate motion information and appearance features, and associate the detected bounding box of the current frame with the existing tracking trajectory;

[0034] The filtering algorithm is used to predict and update the status of the successfully associated trajectories to form a continuous motion trajectory;

[0035] For targets across different monitoring sources, spatiotemporal constraints are constructed based on the water surface velocity vector field and time difference, and the similarity of the multidimensional fine-grained feature vectors between targets is used to determine the identity of the targets, thereby realizing cross-domain trajectory association and continuation.

[0036] Furthermore, the classification attributes of the output river waste include:

[0037] The preliminary category probability vector is concatenated with the multidimensional fine-grained feature vector to form a fused feature vector;

[0038] The fused feature vector is input into a classifier network for processing, and the refined waste category is output as the classification attribute.

[0039] Furthermore, the preprocessing step of generating environmental feature data includes:

[0040] The optical flow field between consecutive frames is calculated using a dense optical flow algorithm. After motion compensation and statistical filtering, the water surface velocity vector field is estimated.

[0041] A single frame image is converted to the HSV color space, and a mask for highly reflective areas is generated based on threshold segmentation and morphological operations of the luminance and saturation channels.

[0042] A two-dimensional frequency domain transformation is performed on the water surface region in a single frame image. Periodic texture features are extracted by a ring bandpass filter. After inverse transformation and threshold segmentation, a ripple interference region mask is generated.

[0043] The highly reflective area mask and the ripple interference area mask are combined to form the optical interference area mask.

[0044] A second aspect of this invention proposes a deep learning-based system for identifying and locating river debris, and a deep learning-based method for identifying and locating river debris. The system includes:

[0045] The data processing module is configured to collect multi-source image data of the river channel and preprocess the multi-source image data to generate preprocessed image data and environmental feature data containing water surface velocity and disturbance area information.

[0046] The target detection module is configured to input the preprocessed image data in parallel into the target detection track and feature analysis track of the deep learning visual processing model;

[0047] In the target detection trajectory, the preprocessed image data is processed by a first convolutional neural network to output the bounding box coordinates and preliminary category probabilities of the floating objects in the river.

[0048] In the feature analysis track, the target region in the preprocessed image data is processed by a second convolutional neural network to output a multidimensional fine-grained feature vector of the target region;

[0049] The fusion output module is configured to fuse the bounding box coordinates, the preliminary category probability, and the multidimensional fine-grained feature vector based on a spatiotemporal correlation and coordinate mapping mechanism, and combine them with the water surface velocity vector field to output the geographic coordinates, classification attributes, and drift trajectory of river garbage.

[0050] In a third aspect, the present invention provides an apparatus comprising:

[0051] At least one processor;

[0052] and a memory communicatively connected to at least one of the processors;

[0053] The memory stores instructions that can be executed by the processor to implement a deep learning-based method for identifying and locating river debris.

[0054] The beneficial effects of this invention are:

[0055] This invention extracts environmental feature data such as water surface velocity, highly reflective areas, and ripple interference areas through a preprocessing stage. This enables subsequent deep learning models to proactively avoid or compensate for common interferences such as highly reflective surfaces and water ripples, preventing false detections and missed detections under adverse imaging conditions. Simultaneously, the invention employs a dual-track processing architecture of target detection and feature analysis, improving the overall recognition accuracy for different types and forms of waste, such as effectively identifying targets partially submerged underwater.

[0056] This invention uses a coordinate mapping mechanism to accurately convert pixel coordinates in an image into latitude and longitude coordinates. Specifically, it converts them into coordinates in geodetic coordinate systems such as WGS84 or CGCS2000. This completely solves the pain points of vague and delayed location reporting in traditional manual reporting, providing backend waste retrieval systems or manual teams with clear and specific target locations, enabling precise dispatching and efficient cleanup, and significantly improving the operational efficiency of waste disposal.

[0057] To address the characteristic of debris drifting with water currents, this invention uses pre-processed water surface velocity information as a key physical constraint, applying it to multi-target tracking and cross-camera target re-identification tasks. This method, which integrates physical environment information and visual features, significantly improves the success rate of associating the same target across different camera views or different inspection segments of a drone. It can generate continuous and reliable debris drift trajectories and provides data support for predicting potential future accumulation areas.

[0058] This invention utilizes a multi-branch network within a feature analysis orbital to deeply mine fine-grained features of targets, including texture, color, and shape, thereby outputting highly refined categories far exceeding traditional coarse classifications. For example, the system no longer simply identifies targets as garbage, but can specifically distinguish between different types of floating debris such as plastic bottles, foam boards, and dead branches. This high-value classification data provides unprecedented data support for waste sorting and treatment, pollution source analysis, and the quantitative assessment of water environment management effectiveness.

[0059] This invention integrates multi-source data from fixed cameras and drones to construct an all-weather, large-scale automated monitoring system. The entire process, from image acquisition to final output of structured data, requires no manual intervention, significantly reducing reliance on human patrols and saving operating costs. Through systematic data fusion, it achieves a complementary advantage between the continuity of fixed monitoring and the flexibility of drone inspections, forming a highly efficient collaborative monitoring network. Attached Figure Description

[0060] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0061] Figure 1 This is a flowchart of a deep learning-based method for identifying and locating river debris according to the present invention.

[0062] Figure 2 This is a structural diagram of a deep learning-based river debris identification and localization system according to the present invention.

[0063] Figure 3 This is a schematic diagram of the structure of a computer system used to implement the methods, systems, and electronic devices of this application. Detailed Implementation

[0064] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0065] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0066] The first embodiment of the present invention provides a method for river debris identification and localization based on deep learning, comprising the following steps:

[0067] Step S10: Collect multi-source image data of the river channel and preprocess the multi-source image data to generate preprocessed image data and environmental feature data containing water surface velocity and interference area information.

[0068] Step S20: Input the preprocessed image data in parallel into the target detection track and feature analysis track of the deep learning visual processing model;

[0069] In the target detection trajectory, the preprocessed image data is processed by a first convolutional neural network to output the bounding box coordinates and preliminary category probabilities of the floating objects in the river.

[0070] In the feature analysis track, the target region in the preprocessed image data is processed by a second convolutional neural network to output a multidimensional fine-grained feature vector of the target region;

[0071] Step S30: Based on the spatiotemporal correlation and coordinate mapping mechanism, the bounding box coordinates, the preliminary category probability and the multidimensional fine-grained feature vector are fused and processed, and combined with the water surface velocity vector field, the geographical coordinates, classification attributes and drift trajectory of river garbage are output.

[0072] To more clearly illustrate the deep learning-based method for river debris identification and localization of this invention, the following will be combined with... Figure 1 The steps in the embodiments of the present invention are described in detail below:

[0073] Step S10: Collect multi-source image data of the river channel and preprocess the multi-source image data to generate preprocessed image data and environmental feature data containing water surface velocity and interference area information.

[0074] The preprocessing step generates environmental feature data including:

[0075] Step S11: Calculate the optical flow field between consecutive frames using the dense optical flow algorithm. After motion compensation and statistical filtering, estimate the water surface velocity vector field.

[0076] Step S12: Convert the single-frame image to the HSV color space, and generate a mask for highly reflective areas based on threshold segmentation and morphological operations of the brightness and saturation channels.

[0077] Step S13: Perform two-dimensional frequency domain transformation on the water surface area in a single frame image, extract periodic texture features through a ring bandpass filter, and generate a ripple interference region mask after inverse transformation and threshold segmentation.

[0078] Step S14: The strong reflective area mask and the ripple interference area mask are merged to form the optical interference area mask.

[0079] The preprocessing steps in this embodiment also include:

[0080] Determine the lighting and weather category based on the statistical characteristics of the image;

[0081] If the scene is determined to be a strong light scene, an image enhancement algorithm based on multi-scale decomposition is executed to separate and reconstruct the reflection component of the image in the logarithmic domain.

[0082] If the scene is determined to be in low light, an adaptive histogram equalization algorithm with limited contrast is executed to perform block processing and interpolation on the image brightness channel.

[0083] If the weather is determined to be rainy or foggy, a defogging algorithm based on dark channel priors is executed to reconstruct the image by constructing an atmospheric scattering model and estimating transmittance.

[0084] The image processed by the adaptively selected algorithm is used as the preprocessed image data.

[0085] The specific implementation of this embodiment is as follows. The system first performs multi-source image data acquisition, obtaining a continuous video stream through fixed monitoring cameras deployed on the riverbank, and controlling a drone to conduct inspection and shooting along a preset route to obtain a sequence of images. The raw data enters the preprocessing stage, which performs two core tasks in parallel: generating environmental feature data containing information on water surface velocity and interference areas; and adaptively performing image enhancement based on the image content to generate high-quality input images.

[0086] The generation of environmental feature data begins with optical flow analysis of consecutive video frames. For a fixed camera, two consecutive grayscale images are taken. First, a Gaussian kernel is used for convolution smoothing to suppress noise, resulting in a smooth frame. ,in This represents the convolution operation. The standard deviation of the Gaussian kernel controls the smoothing degree. Then, the Farneback dense optical flow algorithm is applied. This algorithm targets the midpoint of the image. The gray-level distribution of the neighborhood is modeled using a quadratic polynomial: ,in Represents the coordinate vector of a pixel within its neighborhood. It is a 2×2 symmetric matrix. It is a 2×1 vector. It is a scalar that collectively describes the grayscale distribution of the neighborhood. By assuming constant brightness and consistency with local motion, the displacement vector of the pixel from the previous frame to the next frame is solved. After traversing all pixels, the initial dense optical flow field is obtained. .

[0087] To compensate for camera shake, a global affine transformation model H is estimated using feature point matching and the RANSAC algorithm, and the relative motion field is calculated. Set an amplitude threshold. Filter out those that meet the requirements The stationary vector. Statistical analysis of the remaining effective vectors, such as calculating the median, ultimately yields an estimated value representing the main water flow motion. , which serves as the vector field of water surface velocity.

[0088] Simultaneously, the system performs optical interference region identification on single-frame images. Strong reflective area detection is performed in the HSV color space. The original RGB image I... RGB Convert to HSV space to obtain components Where H represents hue, S represents saturation, and V represents brightness. A high brightness threshold is set. (For example and low saturation threshold (For example ), Filter pixels to generate initial mask A morphological closing operation (dilation followed by erosion) is performed on this binary mask to connect adjacent bright spots and fill voids, generating a coherent, highly reflective region mask. Glare .

[0089] Ripple interference zone detection relies on frequency domain analysis. First, the water surface area is initially segmented using a color threshold and converted into a grayscale image G. water Perform a two-dimensional discrete Fourier transform: Where F represents the Fourier transform operator, and u and v are frequency domain coordinates. The logarithmic amplitude spectrum is calculated after centering the spectrum. Design a ring bandpass filter. Its value is 1 within the frequency band corresponding to the regular ripples, and 0 otherwise. Frequency domain filtering is then performed. ,in This indicates element-wise multiplication. Performing an inverse Fourier transform on the filtered result yields the ripple response map in the spatial domain. An adaptive thresholding method (such as the Otsu algorithm) is used to determine the segmentation threshold T. ripple The binary image is obtained. Then, the shape is optimized through morphological opening and closing operations to generate a ripple interference region mask. Ripple Finally, the two types of masks are merged to obtain a complete optical interference region mask. Together they constitute environmental characteristic data.

[0090] Another core task in the preprocessing stage is adaptive image enhancement. The system analyzes the global statistical features of the input image (such as average brightness and contrast) and automatically determines its category: overexposed in strong light, dim in weak light, or hazy in rain or fog.

[0091] If the scene is determined to be in strong light, the multi-scale Retinex algorithm is activated. This algorithm models the image I(x,y) as the product of the illumination component L(x,y) and the reflection component R(x,y), i.e., I = L⋅R. During processing, each channel of the RGB image is... Transform to the logarithmic domain. Use K (usually K=3) Gaussian kernel functions with different scales. Perform convolution, where This is the square of the standard deviation of the Gaussian kernel at the k-th scale. The convolution result is... The reflection components are estimated by difference weighting in the number domain: ,in The weight for the k-th scale is usually taken as an equal value of 1 / K. Convert back to the real number domain. To restore color, a color restoration factor is introduced. ,For example ,in To adjust the parameters (usually 0 < ≤1). The final output is After merging the channels, they are linearly mapped to the display area.

[0092] If the scene is determined to be low-light, then contrast-limited adaptive histogram equalization is used, processing only the luminance channel L in the Lab color space. The luminance image I... L Divide the area into M×N non-overlapping rectangular sub-blocks. For each sub-block, calculate its grayscale histogram H. local (i), where i is the grayscale index. Perform contrast limiting: Set the cropping threshold. Where P is the total number of pixels in the sub-block, L is the total number of gray levels (e.g., 256), and clipLimit is the preset clipping limiting factor (usually 2.0 to 4.0). Any histogram bar counts higher than β are clipped to β, and the clipped pixels are evenly redistributed across all gray levels. The clipped histogram is then equalized to obtain the gray-level transformation function T for the sub-block. mn For any pixel (x, y) in the image, its enhanced brightness value It is calculated from the transformation function of its four adjacent sub-blocks using bilinear interpolation: ,in The weights are based on the distance from each pixel to the center of each sub-block. Finally, the processed L channel is merged with the original a and b channels and converted back to RGB space.

[0093] If the weather is determined to be rainy or foggy, a dehazing algorithm based on dark channel priors is applied. For a foggy image I, its dark channel... Defined as: Ω(x,y) is a local window centered at (x,y). The brightest 0.1% of pixels are selected from the dark channel image, and the maximum RGB value of the corresponding position of these pixels in the original image I is taken as the global atmospheric light value A = (A0 - A1)2. r A g A b Estimated coarse transmittance map: ,in To preserve the sense of haze, a factor of 0.95 is typically used. Guided filtering is applied using the original grayscale image as a guide. Edge-preserving filtering is performed to obtain a refined transmittance map t(x,y). Finally, the haze-free image J is recovered by inverse solving an atmospheric scattering model. ,in This is set as the lower limit threshold for transmittance (usually 0.1) to prevent the denominator from being too small. The restored image can then undergo contrast stretching.

[0094] After the above preprocessing, the system obtains enhanced image data and data including the water surface velocity vector field V. water and optical interference mask Disturbance The environmental feature data, together with the input of the subsequent deep learning visual processing model, provide a reliable data foundation and physical prior for the accurate identification, location and tracking of river garbage.

[0095] Step S20: Input the preprocessed image data in parallel into the target detection track and feature analysis track of the deep learning visual processing model;

[0096] In the target detection trajectory, the preprocessed image data is processed by a first convolutional neural network to output the bounding box coordinates and preliminary category probabilities of the floating objects in the river.

[0097] In the feature analysis track, the target region in the preprocessed image data is processed by a second convolutional neural network to output a multidimensional fine-grained feature vector of the target region;

[0098] In this embodiment, the processing procedure of the first convolutional neural network includes:

[0099] Multi-scale features of the preprocessed image data are extracted using the backbone network;

[0100] The multi-scale features are fused using a feature pyramid structure;

[0101] The decoupled detection heads predict the target's category and bounding box location based on the fused features.

[0102] For partially submerged targets, the bounding box is expanded to cover the inferred complete area by combining edge detection results and water surface boundary information, and the bounding box coordinates are obtained.

[0103] The second convolutional neural network has a multi-branch structure, and its processing includes:

[0104] Deep semantic features of the target region are extracted using the first branch network;

[0105] Surface texture features of the target region are extracted using a multi-scale filter bank of the second branch network;

[0106] The shape invariant moments and Fourier descriptors of the target region contour are extracted using the third branch network.

[0107] The color histogram and dominant hue of the target region are extracted in the HSV color space using the fourth branch network.

[0108] The deep semantic features, surface texture features, shape features, and color features are concatenated and then weighted and dimensionality reduced using an attention mechanism to generate the multidimensional fine-grained feature vector.

[0109] In the object detection track, the first convolutional neural network is built based on an improved YOLOv8 architecture. Its processing begins with the backbone network CSPDarknet53, used to extract multi-scale features from the input image. To enhance the detection capability for small-sized floating debris, an additional high-resolution feature layer is added to the multi-scale features output by the original network. Subsequently, the multi-scale features are fused through an enhanced path aggregation network structure. This structure includes bottom-up and top-down paths, integrating deep semantic information and shallow detail information at different scales through upsampling, downsampling, and concatenation operations, ultimately outputting a set of fused feature maps. The fused feature maps are then fed into a decoupled detection head. This detection head contains two independent branches: one branch outputs the probability distribution of each preset anchor box belonging to each preliminary category through convolutional layers. The other branch outputs the position offset of each anchor box. Predict the final coordinates of the bounding box. Calculated using the following formula: .in, This represents the sigmoid activation function; and It is the coordinate of the top-left corner of the current grid cell on the feature map; and This refers to the preset width and height of the anchor frame; This represents an exponential function. Each prediction box is also accompanied by an object confidence score. .

[0110] For trash targets partially submerged in water, the system performs additional bounding box expansion logic. First, the Canny edge detection algorithm is applied to the target region to identify the boundary between the trash and the water surface. Based on the outline of the portion of the target above water, the edge detection results, and the inferred relationship to the waterline, the system infers its overall outline below the water surface. Based on this inference, the height of the initial detection bounding box is expanded vertically downwards to cover the inferred complete trash area. The expanded bounding box coordinates, along with the initial class probability and confidence level, constitute the output of the target detection trajectory.

[0111] In the feature analysis phase, the second convolutional neural network employs a multi-branch structure to perform fine-grained feature encoding on the input target region image patches. The first branch is a deep semantic branch, using ResNet-101 as the backbone network to extract the deep semantic features of the target region. After the input image passes through the forward propagation of ResNet-101, the feature maps of the final convolutional layers are obtained, and then converted into feature vectors through global average pooling. .

[0112] The second branch is the material recognition branch, which uses a multi-scale Gabor filter bank to extract the surface texture and reflective properties of the target area. The Gabor filter is defined in the spatial domain as: .in, λ represents the wavelength of the sine wave; θ represents the direction of the filter, and in this embodiment, filters are set in four directions: 0°, 45°, 90°, and 135°; ψ is the phase offset, set to 0; σ is the standard deviation of the Gaussian envelope; and γ is the spatial aspect ratio. A multi-scale filter bank is constructed by setting different λ and σ values. The grayscale image of the input image is convolved with each filter, and the mean and standard deviation of each response image are calculated. All statistics are then concatenated to form a texture feature vector. .

[0113] The third branch is the shape feature branch, which extracts the Hu invariant moments and Fourier descriptors of the target contour. For the binary contour map B(x,y) of the target, its (p+q)th order central moment is defined as... ,in Let be the coordinates of the centroid. The normalized central moments are... ,in Seven Hu invariant moments were calculated based on second- and third-order normalized central moments. Simultaneously, the complex representation of the contour point sequence is extracted. Perform a discrete Fourier transform: The Fourier descriptor takes the first K low-frequency components after normalization. The Hu moment vector is concatenated with the Fourier descriptor vector to form the shape feature vector. .

[0114] The fourth branch is the color feature branch, which operates in the HSV color space. After converting the input image to the HSV space, the quantized color histograms are calculated for the hue (H), saturation (S), and lightness (V) channels respectively, and L1 normalization is performed. The histograms are then concatenated to form the color histogram feature vector. Simultaneously identify the primary color tone.

[0115] The feature vectors output from the above four branches The features are concatenated to form a high-dimensional concatenated feature vector. This vector is then adaptively weighted using channel attention and spatial attention mechanisms. The channel attention module generates channel weight vectors through global average pooling, a fully connected layer, and a sigmoid function, weighting the input features channel by channel. The spatial attention module generates a spatial weight map by aggregating spatial information, weighting the features spatially. Finally, the weighted features are dimensionality-reduced through a fully connected layer, outputting the final multi-dimensional fine-grained feature vector. This vector comprehensively represents the semantic, texture, shape, and color attributes of the target.

[0116] Step S30: Based on the spatiotemporal correlation and coordinate mapping mechanism, the bounding box coordinates, the preliminary category probability and the multidimensional fine-grained feature vector are fused and processed, and combined with the water surface velocity vector field, the geographical coordinates, classification attributes and drift trajectory of river garbage are output.

[0117] In this embodiment, the geographic coordinates of the output river waste include:

[0118] Step S31: For fixed monitoring images, the image pixel coordinates are back-projected to three-dimensional space using pre-calibrated camera imaging model parameters, and the plane coordinates of the target in the world coordinate system are calculated by introducing the water surface elevation plane constraint, and then converted into standard geographic coordinates.

[0119] Step S32: For UAV inspection images, combine the sensor pose parameters acquired in real time, back-project the image pixel coordinates to the object space according to the collinearity equation, and use the water surface elevation constraint to solve the three-dimensional geographic coordinates of the target in the standard geodetic coordinate system.

[0120] For images captured by fixed surveillance cameras, the system establishes an imaging geometric model using pre-calibrated camera intrinsic parameter matrix K and extrinsic parameter matrix [R|t] (where R is the rotation matrix and t is the translation vector). When a garbage target with pixel coordinates (u,v) is detected in the image, its ray direction in the camera coordinate system is first calculated through back projection. Specifically, the normalized camera coordinates are calculated. This coordinate corresponds to a ray originating from the camera's optical center. Since the river debris floats on the surface, the height value Z of the plane it occupies is... wThis can be obtained from external water level elevation data. In the camera coordinate system, a point on this ray can be represented as... ,in The depth value is unknown. This is determined through the transformation relationship between the world coordinate system and the camera coordinate system. And using the known Z w Solving the third line of the transformation equation yields the depth λ. Then, the planar coordinates (X, Y, λ) of the target in the world coordinate system can be calculated. w ,Y w Finally, based on the transformation relationship between the world coordinate system and the standard geographic coordinate system (such as WGS84), (X...) w ,Y w Convert to latitude and longitude coordinates.

[0121] For UAV inspection images, the sensor spatial position and attitude parameters (latitude, longitude, geodetic height, roll angle, pitch angle, and yaw angle) at the moment of imaging are acquired in real time by the POS system on the UAV. The coordinates (u,v) of the identified garbage pixels are transformed to the image space coordinate system using camera intrinsic parameters. Then, based on the sensor rotation matrix R and translation vector T calculated from the POS data, the image points are back-projected to the object space using collinearity equations. By introducing the known river surface elevation as a constraint, the intersection point of the projected ray and the 3D water surface model is solved, thereby directly calculating the 3D geographic coordinates of the garbage target in the CGCS2000 geodetic coordinate system (or a corresponding standard coordinate system), i.e., longitude, latitude, and elevation.

[0122] In this embodiment, the drift trajectory of the discharged river debris includes:

[0123] Step S33: Between consecutive frames, a multi-target tracking algorithm is used to integrate motion information and appearance features, and associate the detected bounding box of the current frame with the existing tracking trajectory;

[0124] Step S34: Use a filtering algorithm to predict and update the status of the successfully associated trajectories to form a continuous motion trajectory;

[0125] Step S35: For targets across different monitoring sources, spatiotemporal constraints are constructed based on the water surface velocity vector field and time difference, and the similarity of the multidimensional fine-grained feature vectors between targets is used to determine the identity of the targets, thereby realizing cross-domain trajectory association and continuation.

[0126] In continuous video frames from a single camera or drone, an improved DeepSORT multi-target tracking algorithm is used for inter-frame correlation to form the motion trajectory of each debris target. For each established trajectory, a Kalman filter is used to predict its position in the next frame. The state vector of the Kalman filter is set as follows: This includes the coordinates of the bounding box center point, width, height, and their respective rates of change. The estimated state of the current frame is obtained through prediction using the state transition matrix F and the process noise covariance matrix Q. and estimate covariance Then, all detection boxes in the current frame are matched with the predicted states of each trajectory.

[0127] The matching metric integrates motion information and appearance features. First, the first... The detection box and the first Mahalanobis distance between the predicted states of the trajectory Where H is the observation matrix, To innovate the covariance, R is the observation noise covariance. Simultaneously, the appearance feature vector of the detection box is calculated. (From the feature analysis track) and the historical mean of the appearance feature set saved by the trajectory Cosine distance between The two distances are combined through weighting or setting cascading thresholds to form a comprehensive matching cost matrix. Finally, the Hungarian algorithm is used to solve this cost matrix to achieve the optimal matching between the detection box and the existing trajectory. Unmatched detection boxes are initialized as new trajectories, and unmatched trajectories are terminated after a certain number of consecutive lost frames.

[0128] For the association (i.e., re-identification) of the same target across different fixed cameras or across drone inspection segments, the system integrates spatiotemporal constraints and feature similarity for judgment. When a target disappears from the field of view of camera A and reappears in the field of view of camera B at a later time, the system first uses the time difference Δt between the appearance of the two targets and the water flow velocity of the river segment estimated in the preprocessing stage. Establish spatiotemporal constraints: The maximum distance the target can drift along the water flow direction within Δt is... ,in An empirical margin is introduced to account for errors in water flow velocity estimation and lateral movement. If the geographical coordinate distance between two targets is less than... If the similarity is higher than a preset threshold, it is considered a candidate matching pair. Then, the cosine similarity between the depth feature vectors of the two targets is calculated. If the similarity is higher than a preset threshold, they are confirmed as the same target, and the trajectory IDs are merged and continued.

[0129] After obtaining the spatiotemporally correlated trajectory point sequence, the trajectory is smoothed and analyzed. A more stable trajectory is obtained by applying a Kalman smoother or moving average filter to historical trajectory points. The instantaneous drift velocity can be obtained by dividing the geographic coordinate difference between adjacent frames by the time interval. Combined with the water flow velocity field, the consistency between the debris movement and the main water flow movement can be analyzed. For trajectory prediction, the current motion state can be used to estimate the location in the near future through the prediction step of the Kalman filter, thereby predicting the drift direction and possible accumulation areas of the debris.

[0130] In this embodiment, the classification attributes of the output river waste include:

[0131] Step S36: Concatenate the preliminary category probability vector with the multidimensional fine-grained feature vector to form a fused feature vector;

[0132] Step S37: Input the fused feature vector into the classifier network for processing, and output a refined garbage category as the classification attribute.

[0133] The initial category probability vector output by the target detection trajectory (C represents the initial number of categories) and the multidimensional fine-grained feature vector output by the feature analysis trajectory. The features are concatenated to form a fused feature vector. The fused feature vector is then input into a lightweight, fully connected neural network classifier. This classifier typically consists of one or two hidden layers and ultimately outputs the probability distribution of each refined waste category (such as "plastic bottles," "styrofoam boards," "dried branches," "plastic bags," etc.) through a softmax layer. The category with the highest probability is output as the final refined classification attribute of that waste target. The output of this classifier, together with the aforementioned geographic coordinates and drift trajectory information, constitutes the final output of the system, providing a decision-making basis for the precise management and scientific cleanup of river waste.

[0134] The second embodiment of the present invention provides a river debris identification and localization system based on deep learning, which is based on a river debris identification and localization method based on deep learning, such as... Figure 2 As shown, the system includes:

[0135] The data processing module is configured to collect multi-source image data of the river channel and preprocess the multi-source image data to generate preprocessed image data and environmental feature data containing water surface velocity and disturbance area information.

[0136] The target detection module is configured to input the preprocessed image data in parallel into the target detection track and feature analysis track of the deep learning visual processing model;

[0137] In the target detection trajectory, the preprocessed image data is processed by a first convolutional neural network to output the bounding box coordinates and preliminary category probabilities of the floating objects in the river.

[0138] In the feature analysis track, the target region in the preprocessed image data is processed by a second convolutional neural network to output a multidimensional fine-grained feature vector of the target region;

[0139] The fusion output module is configured to fuse the bounding box coordinates, the preliminary category probabilities, and the multidimensional fine-grained feature vectors based on a spatiotemporal correlation and coordinate mapping mechanism, and combine them with the water surface velocity vector field to output the geographic coordinates, classification attributes, and drift trajectory of river debris. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related explanations of the method described above can be found in the corresponding processes of the aforementioned system embodiments, and will not be repeated here.

[0140] It should be noted that the deep learning-based river debris identification and location system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0141] A device according to a third embodiment of the present invention includes:

[0142] At least one processor;

[0143] and a memory communicatively connected to at least one of the processors;

[0144] The memory stores instructions that can be executed by the processor to implement the aforementioned deep learning-based method for river debris identification and location.

[0145] A fourth embodiment of the present invention provides a computer-readable storage medium storing computer instructions, which are executed by the computer to implement the above-described deep learning-based method for river debris identification and location.

[0146] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the storage device and processing device described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0147] The following is for reference. Figure 3 It shows a schematic diagram of the structure of a computer system for implementing the methods, systems, and devices of this application. Figure 3 The server shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0148] like Figure 3 As shown, the computer system includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in Read Only Memory (ROM) 302 or programs loaded from storage section 308 into Random Access Memory (RAM) 303. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0149] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card and a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0150] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0151] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0153] The terms “first”, “second”, etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.

[0154] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.

[0155] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for river debris identification and localization based on deep learning, characterized in that, Includes the following steps: Multi-source image data of the river channel is collected and preprocessed to generate preprocessed image data and environmental feature data containing water surface velocity and disturbance area information. The generated water surface flow velocity includes: For a continuous video stream acquired by a fixed camera, take two consecutive grayscale images, and use a Gaussian kernel to perform convolution smoothing to obtain a smoothed frame. A quadratic polynomial model is used to model the gray-level distribution of a point's neighborhood in the image. By assuming constant brightness and consistency with local motion, the displacement vector of a pixel from the previous frame to the next frame is solved. After traversing all pixels, the initial dense optical flow field is obtained. ; Estimating a global affine transformation model using feature point matching and the RANSAC algorithm Calculate the relative motion field ; Set an amplitude threshold Filter out those that meet the requirements The stationary vector; The median of the remaining effective vectors is calculated, and the estimated value representing the main water flow motion is extracted as the water surface velocity vector field. The preprocessed image data is input in parallel into the target detection track and feature analysis track of the deep learning visual processing model; In the target detection trajectory, the preprocessed image data is processed by a first convolutional neural network to output the bounding box coordinates and preliminary category probabilities of the floating objects in the river. In the feature analysis track, the target region in the preprocessed image data is processed by a second convolutional neural network to output a multidimensional fine-grained feature vector of the target region; Based on the spatiotemporal correlation and coordinate mapping mechanism, the bounding box coordinates, the preliminary category probability and the multidimensional fine-grained feature vector are fused and processed, and combined with the water surface velocity vector field, the geographic coordinates, classification attributes and drift trajectory of river garbage are output. The geographical coordinates of the river waste output include: For fixed surveillance images, the image pixel coordinates are back-projected to three-dimensional space using pre-calibrated camera imaging model parameters, and the plane coordinates of the target in the world coordinate system are calculated by introducing the water surface elevation plane constraint, and then converted into standard geographic coordinates. For UAV inspection images, the image pixel coordinates are back-projected to the object space based on the collinearity equation, and the three-dimensional geographic coordinates of the target in the standard geodetic coordinate system are solved by combining the sensor pose parameters acquired in real time. The drift trajectories of the garbage discharged into the river include: Between consecutive frames, a multi-target tracking algorithm is used to integrate motion information and appearance features, and associate the detected bounding box of the current frame with the existing tracking trajectory; The filtering algorithm is used to predict and update the status of the successfully associated trajectories to form a continuous motion trajectory; For the association of the same target across different fixed cameras or across drone inspection segments, the spatiotemporal constraints and feature similarity are combined for judgment; When the target disappears from the field of view of camera A and then reappears in the field of view of camera B, the time difference Δ between the appearance of the two targets is used to determine the target's position. t and estimated river flow velocity Construct spatiotemporal constraints, the spatiotemporal constraints including: the target in Δ t The maximum distance that the inner part of the water flow direction drifts is ,in An empirical margin is introduced to account for errors in water flow velocity estimation and lateral motion; If the geographical coordinate distance between two targets is less than If the match is 0, it is considered a candidate match. Calculate the cosine similarity between the depth feature vectors of two targets. If the similarity is higher than a preset threshold, they are identified as the same target, and the trajectory IDs are merged and continued.

2. The method according to claim 1, characterized in that, The preprocessing steps also include: If the scene is determined to be a strong light scene, an image enhancement algorithm based on multi-scale decomposition is executed to separate and reconstruct the reflection component of the image in the logarithmic domain. If the scene is determined to be in low light, an adaptive histogram equalization algorithm with limited contrast is executed to perform block processing and interpolation on the image brightness channel. If the weather is determined to be rainy or foggy, a defogging algorithm based on dark channel priors is executed to reconstruct the image by constructing an atmospheric scattering model and estimating transmittance. The image processed by the adaptively selected algorithm is used as the preprocessed image data.

3. The method according to claim 1, characterized in that, The processing steps of the first convolutional neural network include: Multi-scale features of the preprocessed image data are extracted using the backbone network; The multi-scale features are fused using a feature pyramid structure; The decoupled detection heads predict the target's category and bounding box location based on the fused features. For partially submerged targets, the bounding box is expanded to cover the inferred complete area by combining edge detection results and water surface boundary information, and the bounding box coordinates are obtained.

4. The method according to claim 1, characterized in that, The second convolutional neural network has a multi-branch structure, and its processing includes: Deep semantic features of the target region are extracted using the first branch network; Surface texture features of the target region are extracted using a multi-scale filter bank of the second branch network; The shape invariant moments and Fourier descriptors of the target region contour are extracted using the third branch network. The color histogram and dominant hue of the target region are extracted in the HSV color space using the fourth branch network. The deep semantic features, surface texture features, shape features, and color features are concatenated and then weighted and dimensionality reduced using an attention mechanism to generate the multidimensional fine-grained feature vector.

5. The method according to claim 1, characterized in that, The classification attributes of river waste output include: The preliminary category probability vector is concatenated with the multidimensional fine-grained feature vector to form a fused feature vector; The fused feature vector is input into a classifier network for processing, and the refined waste category is output as the classification attribute.

6. The method according to claim 1, characterized in that, The environmental feature data generated in the preprocessing step includes: The optical flow field between consecutive frames is calculated using a dense optical flow algorithm. After motion compensation and statistical filtering, the water surface velocity vector field is estimated. A single frame image is converted to the HSV color space, and a mask for highly reflective areas is generated based on threshold segmentation and morphological operations of the luminance and saturation channels. A two-dimensional frequency domain transformation is performed on the water surface region in a single frame image. Periodic texture features are extracted by a ring bandpass filter. After inverse transformation and threshold segmentation, a ripple interference region mask is generated. The highly reflective area mask and the ripple interference area mask are combined to form an optical interference area mask.

7. A deep learning-based river debris identification and localization system, based on the deep learning-based river debris identification and localization method according to any one of claims 1-6, characterized in that, The system includes: The data processing module is configured to collect multi-source image data of the river channel and preprocess the multi-source image data to generate preprocessed image data and environmental feature data containing water surface velocity and disturbance area information. The target detection module is configured to input the preprocessed image data in parallel into the target detection track and feature analysis track of the deep learning visual processing model; In the target detection trajectory, the preprocessed image data is processed by a first convolutional neural network to output the bounding box coordinates and preliminary category probabilities of the floating objects in the river. In the feature analysis track, the target region in the preprocessed image data is processed by a second convolutional neural network to output a multidimensional fine-grained feature vector of the target region; The fusion output module is configured to fuse the bounding box coordinates, the preliminary category probability, and the multidimensional fine-grained feature vector based on a spatiotemporal correlation and coordinate mapping mechanism, and combine them with the water surface velocity vector field to output the geographic coordinates, classification attributes, and drift trajectory of river garbage.

8. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to at least one of the processors; The memory stores instructions that can be executed by the processor to implement a deep learning-based method for identifying and locating river debris, as described in any one of 1-6.

Images