An unmanned ship floating object identification method and device

Abstract

This invention proposes a method and device for identifying floating objects on an unmanned surface vessel (USV), belonging to the field of USV technology. The method includes the following steps: A rotating linear polarizer is placed in front of the imaging optical path of a dual-camera module; visible light and near-infrared images at different rotation angles of the linear polarizer are acquired in a time-division manner and then weighted and fused to obtain an image without specular reflection; water ripple texture is further suppressed to obtain a pre-processed water surface image; reflection restoration is performed on the pre-processed water surface image to obtain a water surface image that retains floating objects and water surface texture; motion salient candidate regions are obtained from the water surface image that retains floating objects and water surface texture; a garbage-non-garbage discrimination model trained by contrastive learning is used to perform binary classification judgment on the motion salient candidate regions to identify whether the motion salient candidate regions are garbage; for the garbage collection set, a garbage collection path is planned by combining the energy consumption of the USV and the revenue from garbage recycling.

Description

Technical Field

[0001] This invention relates to the field of unmanned vessel technology, and in particular to a method and device for identifying floating objects on unmanned vessels. Background Technology

[0002] Various floating pollutants, including oil spills and solid waste, often float on the surface of rivers, lakes, or reservoirs. These pollutants are not only difficult to degrade naturally, but also cause continuous harm to the ecological environment. It is necessary to arrange personnel for temporary or regular cleanup. During cleanup, professional personnel drive specialized vessels to designated locations and manually operate machinery to collect the pollutants. However, manual cleanup is very inefficient and can only be carried out during the day.

[0003] With the continuous development of technology, new energy vessels for collecting pollutants have emerged, replacing traditional fuel-powered waste collection vessels. These vessels do not produce pollutants such as exhaust fumes or oil leaks and can operate continuously when there is sufficient power. However, they still require manual visual identification of floating debris and operation of mechanical equipment for cleaning, resulting in relatively high costs. If machine vision technology can be integrated, direct human intervention can be eliminated, improving adaptability to the environment and the automatic identification of floating debris, while also enabling continuous cleaning of floating debris in water bodies regardless of weather conditions.

[0004] Therefore, this paper provides a method and device for identifying floating objects on unmanned vessels. By identifying floating objects in water surface images, it is of great significance for improving the efficiency of target identification and water surface cleaning, and for better planning the trajectory of unmanned vessels. Summary of the Invention

[0005] In view of this, the present invention proposes a floating object identification method and device based on an unmanned vessel that can reliably eliminate the interference of water surface mirror reflection and water temperature texture, accurately separate the sky, shoreline and water surface areas and eliminate the misjudgment effect of reflections, effectively distinguish floating objects and background water flow movement through optical flow residual vector and spatial clustering, and better identify potential garbage.

[0006] On the one hand, the present invention provides a method for identifying floating objects based on unmanned vessels, comprising the following steps: S1: A visible light-near infrared dual-camera module is used. A rotating linear polarizer is set in front of the imaging optical path of the dual-camera module. Visible light and near infrared images at different rotation angles of the linear polarizer are acquired in a time-division manner and then weighted and fused to obtain an image without specular reflection. Then, water ripple texture is removed from the acquired image without specular reflection through inter-frame alignment, local displacement field modeling and blind deconvolution processing to obtain a pre-processed water surface image. S2: Perform reflection restoration on the preprocessed water surface image, separating the water surface area from the sky or shoreline area to obtain a water surface image that retains floating objects and water surface texture; S3: Based on the sequence of water surface images that preserve floating objects and water surface texture, and water flow motion vectors, extract and calculate the optical flow residual vectors of corner points, perform spatial clustering on the optical flow residual vectors, and determine whether the clustered cluster belongs to the floating object cluster based on the number of corner points in the current corner point's neighborhood, the amplitude and directional divergence of the optical flow residuals of all corner points in the neighborhood; fit the minimum bounding rectangle to the floating object cluster and perform morphological dilation processing to obtain motion saliency candidate regions; S4: Using the garbage-non-garbage discrimination model trained by contrastive learning, binary classification is performed on the moving salient candidate region to identify whether the moving salient candidate region is garbage; S5: For the set of salient candidate regions identified as waste, combine the energy consumption of the unmanned vessel with the benefits of waste recycling to plan waste cleaning paths.

[0007] Based on the above technical solutions, preferably, the visible-near-infrared dual-camera module described in step S1, with a rotating linear polarizer placed in front of the imaging optical path of the dual-camera module, and visible light and near-infrared images at different rotation angles of the linear polarizer acquired in a time-division manner and then weighted and fused to obtain an image without specular reflection, includes the following: the rotating linear polarizer is mounted on the output shaft of a stepper motor, and a polarization angle encoder disk is also mounted on the output shaft of the stepper motor. The stepper motor drives the rotating linear polarizer to rotate, and the dual-camera module acquires RGB images and near-infrared channel images in four polarization directions (0°, 45°, 90°, and 135°) in real time. I 0、 I 45 , I 90 、I 135 Calculate the RGB-NIR spectral intensity map of each pixel based on the RGB images and near-infrared channel images in each polarization direction; calculate the degree of polarization map for each channel separately. DoP and polarization angle diagram AoP Based on near-infrared channel intensity map I 135 Average brightness map of RGB images I gary Obtain near-infrared difference map D NIR Polarization diagram DoP Difference between near-infrared and near-infrared images D NIR Normalize all values ​​to the range [0, 1], then convert the polarization diagram. DoP Difference map with near-infrared D NIR Weighted fusion constructs fusion weights W Using fusion weights WObtain approximate values ​​of specular reflection components based on polarization degree maps and near-infrared difference maps. I reflection The image without specular reflection is obtained by subtracting the approximate value of the specular reflection component from the RGB image in the 0° polarization direction. I clean .

[0008] Preferably, the water ripple texture of the acquired anti-specular reflection image is removed through inter-frame alignment, local displacement field modeling, and blind deconvolution processing to obtain a preprocessed water surface image. This includes the following steps: Inter-frame alignment of several consecutive anti-specular reflection images; compensation for global rigid body motion; calculation of the local residual displacement field relative to the reference frame for each anti-specular reflection image to obtain a local displacement field characterizing the water ripple motion; modeling point spread functions for local image patches of the anti-specular reflection image based on the local displacement field; performing blind deconvolution processing on each local image patch using an alternating minimization method to obtain a background image with suppressed water ripple texture; and constructing a residual feature map based on the background image with suppressed water ripple texture and the anti-specular reflection image. R The residual feature map R The weighted fusion is applied to the destigmatized image to obtain the preprocessed water surface image. .

[0009] Preferably, step S2 includes the following: using a lightweight semantic segmentation network model as the backbone feature extraction network, taking the preprocessed water surface image as the model input, and segmenting it through the lightweight semantic segmentation network model to obtain the water surface area and the sky or shoreline area; setting reflection candidate areas in the water surface area, calculating the gradient distribution direction of pixels in the reflection candidate areas, and identifying reflection pixels by statistically analyzing the concentration of gradient direction distribution; using the optical flow alignment results between preprocessed water surface images of multiple consecutive frames, searching for similar texture blocks in the non-reflecting water surface area and the temporal neighborhood, filling and boundary repairing the areas where the identified reflection pixels are located, and obtaining a water surface image that retains floating objects and water surface texture.

[0010] Further optimized, the lightweight semantic segmentation network model uses MobileNetV3. The training samples are images from unmanned surface vessel (USV) usage scenarios, including different weather conditions. The images contain clear sky and water boundaries. The training samples are expanded through random horizontal flipping, rotation, brightness and contrast adjustments, and random cropping. Boundary-aware loss is used. The lightweight semantic segmentation network model outputs the predicted probabilities of pixels belonging to the sky, shoreline, and water surface. The MobileNetV3 lightweight semantic segmentation network model's training hyperparameters are set as follows: training sample size 512×512, batch size 4-8, 50-100 training iterations, Adam optimizer, and learning rate 10. -3 Or 5×10 -4 Using a freezing strategy, the backbone feature extraction network is frozen for the first 10 rounds of training, and only the classification head with 1×1 convolutions is trained.

[0011] Further preferably, step S3 includes: extracting corner points from the sequence of water surface images that retain floating objects and water surface texture; calculating the optical flow vector corresponding to each corner point; and calculating the optical flow residual vector by combining it with the water flow motion vector; calculating the amplitude of the optical flow residual vector; and filtering out optical flow residual vectors that are less than a threshold. T mag The corner points are used to remove the water surface background caused by the translational flow of water. The image coordinates and residual direction vectors of the remaining corner points are used as the feature space. A density-based spatial clustering algorithm, DBSCAN, is used for clustering. Taking any corner point as the center, if the number of corner points in the neighborhood is not less than the minimum number of corner points, the corner points in the neighborhood are determined to belong to the same core point cluster. The average residual amplitude and directional divergence are calculated for each core point in the core point cluster. The average residual amplitude is greater than the threshold 1.2. T mag If the directional divergence is below a set upper limit, the core point cluster is determined to be a floating object cluster. Morphological dilation is then applied to the minimum bounding rectangle of the corner points within the floating object cluster to obtain motion-significant candidate regions. If the amplitude of the corner point optical flow residual vector is less than a threshold... T mag The number of corner points within the neighborhood is less than the minimum number of corner points, and the average residual amplitude of the core point cluster does not exceed the threshold of 1.2. T mag If the directional divergence of the core point is not less than the set upper limit, it is judged as noise.

[0012] Furthermore, the DBSCAN algorithm is a parameter-free, unsupervised clustering algorithm with a neighborhood radius of [missing information]. The value ranges from 5 to 15 pixels, and the minimum number of corner points is [not specified]. The value range is 5-10, and the threshold is... T mag The value range is 0.5-1.5px / frame; the upper limit for directional divergence is ≤18° for angular standard deviation.

[0013] Further preferred, step S4 involves: collecting floating debris samples and non-debris samples, performing random geometric and photometric enhancements on each sample, and generating two data-enhanced views with equivalence of positive and negative values ​​for the same sample; the debris-non-debris discrimination model includes an encoder, a projection head, and a binary classification head; the encoder uses the convolutional backbone of the lightweight convolutional network MobileNetV3-Small, which is used to expand the number of channels according to the expansion factor, extract features through convolution in the expanded high-dimensional space, and then compress the number of channels back to the target number of channels; the projection head is used to perform L2 normalization on the feature vector and output the projection vector; the binary classification head is set on the output side of the encoder, which is used to map the feature vector to represent the probability that the sample is debris in 1-dimensional form; in the online inference stage after training, the trained debris-non-debris discrimination model is used to output the probability that the motion salient candidate region is debris.

[0014] In a further optimized version, the garbage-non-garbage discrimination model first optimizes the encoder and projection head with contrastive learning loss during training to separate garbage samples from non-garbage samples in the embedding space. Then, the encoder is frozen and the binary classification head is adjusted to optimize the garbage-non-garbage binary classification performance. During the inference process after training, only the encoder and binary classification head are used to output the probability that the motion salient candidate region is garbage, and the projection head is discarded.

[0015] On the other hand, the present invention also provides a floating object identification device based on an unmanned vessel for implementing the above-mentioned method, comprising: The image preprocessing unit is used to set a rotating linear polarizer in front of the imaging optical path of the visible light-near infrared dual camera module, and to acquire visible light and near infrared images at different rotation angles of the linear polarizer in a time-division manner and perform weighted fusion to obtain an image with anti-specular reflection; then, the water ripple texture of the acquired anti-specular reflection image is further suppressed to obtain a preprocessed water surface image. The reflection restoration unit is used to restore the reflection of the preprocessed water surface image, separate the water surface area from the sky or shoreline area, fill and restore the boundary of the area where the identified reflection pixels are located, and obtain a water surface image that retains floating objects and water surface texture. The floating object recognition unit acquires a sequence of water surface images that preserves floating objects and water surface texture. It combines the water flow motion vector to calculate the optical flow residual vector, performs spatial clustering on the optical flow residual vector, and determines whether the clustered clusters belong to the floating object clusters. It then fits the minimum bounding rectangle to the floating object clusters and performs morphological dilation processing to obtain motion saliency candidate regions. The waste classification and identification unit uses a trained waste-non-waste discrimination model to perform binary classification on moving salient candidate regions and identify whether the moving salient candidate regions are waste. The waste path planning unit is used to plan waste disposal paths for multiple identified wastes, combining the energy consumption of unmanned vessels with the revenue from waste recycling.

[0016] The present invention provides a method and device for identifying floating objects on unmanned vessels, which has the following advantages compared with the prior art: 1. By combining a visible-near-infrared dual-camera module with a rotating linear polarizer, and by weighted fusion of the polarization degree map and the near-infrared difference map, the specular reflection of the water surface is effectively eliminated; further, through inter-frame alignment, local displacement field modeling and blind deconvolution processing, the interference of water ripple texture on the identification of floating objects is suppressed.

[0017] 2. A lightweight semantic segmentation network is used to accurately separate the water surface, sky, and shoreline areas. Gradient direction statistics and texture filling are used to repair the reflection areas, preserving floating objects and real water surface textures, thus avoiding misjudgment of subsequent garbage caused by reflections.

[0018] 3. By calculating optical flow residual vectors and using DBSCAN spatial clustering, floating objects are effectively distinguished from background water flow, and significant candidate regions are extracted from dynamic water surfaces, reducing interference from non-target objects. The encoder is trained using contrastive learning, which separates garbage samples from non-garbage samples in the embedding space. Then, a binary classification head is used for precise judgment, which effectively improves the efficiency of garbage identification in complex backgrounds.

[0019] 4. After identifying multiple types of waste, the system comprehensively considers the energy consumption of the unmanned vessel and the benefits of waste recycling to conduct efficient path planning, thereby optimizing resource utilization and improving operational efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the steps of a floating object identification method and device based on an unmanned vessel according to the present invention. Detailed Implementation

[0022] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] New energy vessels have emerged to replace traditional fuel-powered waste cleaning vessels. They do not produce pollutants such as exhaust fumes or oil leaks and can operate continuously when there is sufficient electricity. However, they still require manual visual identification of floating objects and operation of mechanical equipment for cleaning, which remains costly. If machine vision technology can be combined, direct human intervention can be eliminated, improving adaptability to the environment and the ability to automatically identify floating objects. It can also achieve continuous cleaning of floating objects in water bodies regardless of weather conditions.

[0024] In view of this, such as Figure 1 As shown, in one aspect, the present invention provides a method for identifying floating objects based on an unmanned vessel, comprising the following steps: S1: A visible-near-infrared dual-camera module is used. A rotating linear polarizer is set in front of the imaging optical path of the dual-camera module. Visible light and near-infrared images at different rotation angles of the linear polarizer are acquired in a time-division manner and then weighted and fused to obtain an image without specular reflection. Then, water ripple texture is removed from the acquired image without specular reflection through inter-frame alignment, local displacement field modeling and blind deconvolution processing to obtain a pre-processed water surface image.

[0025] In one embodiment, acquiring an image with specular reflection eliminated includes the following: the rotating linear polarizer is mounted on the output shaft of a stepper motor, and a polarization angle encoder disk is also mounted on the output shaft of the stepper motor. The rotating linear polarizer is driven to rotate by the stepper motor, and RGB images and near-infrared channel images in four polarization directions (0°, 45°, 90°, and 135°) are acquired in real time by a dual-camera module. I 0、 I 45 , I 90 、I 135 Calculate the RGB-NIR spectral intensity map of each pixel based on the RGB images and near-infrared channel images in each polarization direction; calculate the degree of polarization map for each channel separately. DoP , making the total light intensity S 0=( I 0+ I 45 + I 90 + I 135 ) / 2, Horizontal-vertical linear polarization difference S 1= I 0- I 90 The polarization difference S2 = ±45 degrees I 45 - I 135 Polarization diagram , ,constant Polarization angle diagram Based on the near-infrared channel intensity map I 135 Average brightness map of RGB images I gary Obtain near-infrared difference map D NIR Average brightness diagram I gary It is obtained by converting an RGB image with a 0° polarization direction into a grayscale image. I gary =0.299 R +0.587 G +0.114 B , R , G , B Near-infrared difference map, representing channel pixel values. D NIR =| I 135 - α • I gary |, α The normalization coefficient is used to represent the polarization degree diagram. DoP Difference between near-infrared and near-infrared images D NIR Normalize all values ​​to the range [0, 1], then convert the polarization diagram. DoP Difference map with near-infrared D NIR Weighted fusion constructs fusion weights W , W = β • DoP+ ( 1-β ) • D NIR , W ∈[0,1], β For empirical coefficients, use fusion weights W Obtain approximate values ​​of specular reflection components based on polarization degree maps and near-infrared difference maps. I reflection = W • I 0. Subtract the approximate value of the specular reflection component from the RGB image in the 0° polarization direction to obtain the image without specular reflection. I clean , I clean = I 0- I reflection .

[0026] Then, the water ripple texture of the acquired anti-specular reflection image is removed through inter-frame alignment, local displacement field modeling, and blind deconvolution processing to obtain a preprocessed water surface image. This process includes: inter-frame alignment of several consecutive anti-specular reflection images; compensation for global rigid body motion; calculation of the local residual displacement field relative to the reference frame for each anti-specular reflection image to obtain a local displacement field characterizing the water ripple motion; modeling point spread functions for local image patches of the anti-specular reflection image based on the local displacement field; and performing blind deconvolution processing on each local image patch using an alternating minimization method to obtain a background image with suppressed water ripple texture; and constructing a residual feature map based on the background image with suppressed water ripple texture and the anti-specular reflection image. R The residual feature map R The weighted fusion is applied to the destigmatized image to obtain the preprocessed water surface image. .

[0027] Specifically, several consecutive frames of images without specular reflection are aligned between frames. First, a global affine transformation is estimated and compensated to eliminate the overall translation and rotation caused by the unmanned vessel's swaying. After compensation, the local residual displacement field relative to the reference frame is calculated for each frame. By applying a divergence constraint ▽•d≈0 to conform to the incompressible flow characteristics of water surface, a local displacement field characterizing the water ripple motion is obtained. Recorded as Based on the local displacement field, local image patches of the image to be destigmatized are modeled as Gaussian point spread functions (PSFs), and the expectation of each local image patch is shifted by the local water ripple displacement. σ is determined based on the wave frequency and frame interval, completing the Gaussian point spread function (PSF) modeling for each local image patch; alternating minimization is performed for each local image patch, fixing the local water ripple displacement offset. We use Wiener filtering to estimate potential sharp background patches, then fix these patches and update the local water ripple displacement using gradient descent. The process converges after 3-5 iterations, and the update process is constrained by hydrodynamics. A blind deconvolution is performed on each local block, and the images are then stitched together according to the coordinates of the image after specular reflection removal to obtain the water surface background image. Subtract the water background image from the image after removing specular reflections. Obtain the water ripple residual term The preprocessed water surface image is obtained. , , p The enhancement coefficient is used to strengthen the floating object signal, and its value ranges from [0, 1].

[0028] S2: Perform reflection restoration on the preprocessed water surface image, separating the water surface area from the sky or shoreline area to obtain a water surface image that retains floating objects and water surface texture.

[0029] Specifically, this includes the following: Using the preprocessed water surface image as model input, the lightweight semantic segmentation network MobileNetV3 is used as the backbone feature extraction network. After outputting the feature map of the last layer of the network model, a 1×1 convolutional classification head is used to map the number of channels to three categories: sky, shoreline, and water surface. The resolution of the preprocessed water surface image is restored through bilinear sampling, and softmax processing is performed on each pixel to obtain a three-component probability map. , representing the probabilities that pixels in the preprocessed water surface image belong to the sky, shoreline, and water surface, respectively; given a threshold for water surface category determination. The probability that the current pixel belongs to the water surface. When the current pixel is in the water surface mask, assign a water surface mask to the current pixel; otherwise, assign a sky or shoreline mask to distinguish the water surface area from the sky or shoreline area in the preprocessed water surface image. Let the row number of the top boundary of the sky or shoreline area in the preprocessed water surface image be... The row number of the center of the reflection candidate region in the preprocessed water surface image is Satisfying the constraints , The offset is used to calibrate the correction amount caused by the pitch of the dual-camera module, installation height, and slight curvature of the water surface. The vertical pixel length of the reflection in the water surface area is defined, and constraints are used to limit the candidate reflection region. A Sobel convolution kernel is applied to the preprocessed water surface image to calculate the pixel length. Brightness gradient component at the location Thus, pixels are obtained. gradient direction at , For each pixel in the reflection candidate region, a histogram of the gradient direction is statistically analyzed. Calculate histogram With uniform distribution The chi-square distance between them , k This indicates the gradient space Divided into K Sequence number of a fixed interval In this embodiment ; Indicates the first k A fixed interval observation frequency, This represents the expected frequency when the distribution is uniform. The number of pixels in the reflection candidate area is used. When the chi-square distance is greater than a set threshold, the pixel is determined to be a reflection pixel. The system acquires three consecutive preprocessed water surface images, including the current frame and the preceding and following frames. It calculates the optical flow in the water surface region and predicts the corresponding position in the adjacent frame for the current frame position, performing optical flow alignment. The reflection pixels are used as the target to be filled. PatchMatch is used to search for texture blocks in the water surface region neighborhood of the non-reflection pixels of the target to be filled. The similarity is measured by L2 distance and gradient direction consistency. The most matching similar texture block is selected and filled with the target. Poisson fusion is performed on the filling boundary to obtain a water surface image that retains floating objects and water surface texture.

[0030] The training process of the lightweight semantic segmentation network model MobileNetV3 used in the above scheme is as follows: The images of the training samples are from scenes of unmanned boat use, including different weather conditions. The images contain clear sky and water boundaries. The training samples are expanded by random horizontal flipping, rotation, brightness and contrast adjustment, and random cropping. Boundary-aware loss is selected. , ,in N This represents the total number of pixels in the image. It is the boundary pixel weight. , It is a pixel i To the nearest semantic boundary E distance, The boundary width range shall not exceed 3 pixels. It is a pixel i In category c The three-class truth values ​​on, Representing the sky, shoreline, and water respectively. These are the pixels output by the lightweight semantic segmentation network model. i The predicted probabilities of belonging to the sky, shoreline, and water surface; the training hyperparameters of the lightweight semantic segmentation network model MobileNetV3 are set as follows: training sample size is 512×512, batch size is 4-8, training iterations are 50-100 epochs, the optimizer is Adam, and the learning rate is 10. -3 Or 5×10 -4 Using a freezing strategy, the backbone feature extraction network is frozen for the first 10 rounds of training, and only the classification head with 1×1 convolutions is trained.

[0031] S3: Based on the sequence of water surface images that preserve floating objects and water surface texture, and water flow motion vectors, extract and calculate the optical flow residual vectors of corner points, perform spatial clustering on the optical flow residual vectors, and determine whether the clustered cluster belongs to the floating object cluster based on the number of corner points in the neighborhood of the current corner point, the amplitude and directional divergence of the optical flow residuals of all corner points in the neighborhood; fit the minimum bounding rectangle to the floating object cluster and perform morphological dilation processing to obtain motion saliency candidate regions.

[0032] Specifically, the process involves: extracting corner points from a sequence of water surface images that preserve floating objects and water surface texture; calculating the optical flow vector corresponding to each corner point; combining this with the water flow motion vector to calculate the optical flow residual vector; calculating the amplitude of the optical flow residual vector; and filtering out optical flow residual vectors smaller than a threshold. T mag The corner points are used to remove the water surface background caused by the translational flow. The image coordinates and residual direction vectors of the remaining corner points are used as the feature space. A density-based spatial clustering algorithm, DBSCAN, is used for clustering. With any corner point as the center, if the number of corner points in the neighborhood is not less than the minimum number of corner points, the corner points in the neighborhood are determined to belong to the same core point cluster. The average optical flow residual amplitude and directional divergence are calculated for each core point of the core point cluster. The average residual amplitude is greater than the threshold 1.2. T mag If the directional divergence is below a set upper limit, the core point cluster is determined to be a floating object cluster. Morphological dilation is then applied to the minimum bounding rectangle of the corner points within the floating object cluster to obtain motion-significant candidate regions. If the amplitude of the corner point optical flow residual vector is less than a threshold... T mag The number of corner points within the neighborhood is less than the minimum number of corner points, and the average residual amplitude of the core point cluster does not exceed the threshold of 1.2. T mag If the directional divergence of the core point is not less than the set upper limit, it is judged as noise.

[0033] In one specific embodiment, the details of step S3 are as follows: In the sequence of water surface images that preserve floating objects and water surface texture, the instantaneous motion vector obtained from sparse optical flow is... , p These are the corner points in the current frame that have been filtered by the corner detector ShiTomasi. The image plane optical flow vector corresponding to the corner point. These represent the pixel displacements of the optical flow vector in the horizontal and vertical directions, respectively, and the optical flow vector residuals at the corner points. for , For water flow motion vector, and for optical flow residual vector Calculate amplitude Filter out values ​​with amplitude less than the threshold T mag The corner points are used to remove the water surface background caused by the translation of water flow; the residual vector of the remaining corner points is... The density-based spatial clustering algorithm DBSCAN is used to perform clustering in the joint space of image coordinates and residual directions. The feature space of the density-based spatial clustering algorithm DBSCAN is: , For the first i Image coordinates of the corner points For the first i The residual direction unit vector at each corner point, with With the center of the circle, The number of corner points within the neighborhood of the radius is greater than or equal to the minimum number of corner points. If the corner points within the neighborhood are determined to be core points, and these core points form a core point cluster, then the average residual amplitude and directional divergence of the core point cluster are further evaluated. If the average residual amplitude is greater than the threshold 1.2... T mag If the divergence of the core point direction is below the set upper limit, it is determined to be a floating object cluster. If the corner point is not a core point, and the average residual amplitude of the core point cluster does not exceed the threshold of 1.2... T mag If the directional divergence of the core point is not less than the set upper limit, it is judged as noise; take the circumscribed rectangle for the corner point judged as a floating object cluster, and perform morphological expansion along the outline of the circumscribed rectangle to obtain the motion significant candidate region.

[0034] The DBSCAN algorithm is a parameter-free, unsupervised clustering algorithm that does not require pre-training. Its parameters are as follows: neighborhood radius. The value ranges from 5 to 15 pixels, and the minimum number of corner points is [not specified]. The value range is 5-10, and the threshold is... T mag The value range is 0.5-1.5px / frame; the upper limit for directional divergence is ≤18° for angular standard deviation.

[0035] S4: Using the garbage-non-garbage discrimination model trained by contrastive learning, binary classification is performed on the moving salient candidate region to identify whether the moving salient candidate region is garbage.

[0036] Step S4 involves collecting floating debris and non-debris samples from the water surface, performing random geometric and photometric enhancements on each sample, and generating two data-enhanced views with equivalence of positive and negative values ​​for the same sample. The debris-non-debris discrimination model includes an encoder, a projection head, and a binary classification head. The encoder uses the convolutional backbone of the lightweight convolutional network MobileNetV3-Small, including a first 3×3 convolutional layer, multiple inverse residual bottleneck blocks, and a global average pooling layer arranged sequentially. The first 3×3 convolutional layer has 3 input channels, 16 output channels, and a kernel stride of 2. The inverse residual bottleneck block includes a first 1×... The system consists of a first convolutional layer, a second 3×3 convolutional layer, and a second 1×1 convolutional layer. The first 1×1 convolutional layer expands the number of channels according to the expansion factor. The second 3×3 convolutional layer extracts features through convolution in the high-dimensional space expanded by the first 1×1 convolutional layer. The second 1×1 convolutional layer compresses the number of channels back to the target number of channels. When the target number of channels is the same as the number of input channels of the first 1×1 convolutional layer, residual connections are used; when the target number of channels is different from the number of input channels of the first 1×1 convolutional layer, residual connections are not used. A global average pooling layer is used to generate the feature map from the output of the last inverse residual bottleneck block, which is then used to output the feature vector. As a high-dimensional semantic representation, x This is a sample; the projection head includes a two-layer fully connected network. The first fully connected layer is followed by batch normalization and a ReLU activation function, expanding the dimension of the input feature vector to 256 dimensions. The second fully connected layer has no activation function, compressing the input dimension from 256 dimensions to 128 dimensions. The projection head is used to process the feature vector. Perform L2 normalization and output the projection vector. The binary classification head is set on the output side of the encoder and is used to classify the feature vectors. The mapping is represented as the probability that a 1-dimensional sample is garbage; in the online inference stage after training, the trained garbage-non-garbage discrimination model is used to output the probability that a motion salient candidate region is garbage.

[0037] During training, the garbage-non-garbage discrimination model first optimizes the encoder and projector head using contrastive learning loss to separate garbage samples from non-garbage samples in the embedding space. Then, the encoder is frozen and the binary classification head is adjusted to optimize the garbage-non-garbage binary classification performance. During inference after training, only the encoder and binary classification head are used to output the probability that the motion salient candidate region is garbage, and the projector head is discarded.

[0038] The specific sample preparation, training, and application process is as follows: The training process for the litter-non-litter discrimination model involves using floating objects such as PET bottles, foam, plastic bags, and dead branches as positive samples, and using water surface images containing only water surface textures, preprocessed water surface images with reflection restoration and only background content, and images of birds and insects as negative samples. Each sample undergoes data augmentation processing, including random cropping, horizontal flipping, rotation, brightness and contrast adjustments, or the addition of Gaussian noise, to generate SimCLR positive pairs. The sample size is adjusted to 64×64 or 96×96 to be compatible with the input of the lightweight convolutional network MobileNetV3-Small. In the first stage of training, the binary classification head is not involved; only the encoder and projection head are used. The loss function at this time is... , B This represents the number of samples corresponding to the batch size. τ This is the temperature coefficient, with a value ranging from 0.5 to 1.0. For cosine similarity calculation, and These are positive examples of projection heads based on SimCLR. The output; batch size is 128 or 256, training epochs are 100-200, the optimizer is Adam, the learning rate is cosine annealing, and the minimum learning rate is 10. -3 ; In the second phase of training, the encoder is frozen, and a binary classification head is connected after the encoder. The loss function at this point is Category tags , Let be the probability that a sample is classified as garbage, with a value in the range [0, 1]. This is the output of the binary classification head. The training uses the sigmoid function, with 20-50 training epochs, a batch size of 32-64, and a learning rate of 1e. 4 ~5e -5 The optimizer chosen is Adam; During the online inference phase, the probability that a salient candidate region is classified as garbage is: , As a significant candidate region for motion, repeat F Sub-forward online inference to calculate average probability Based on the average probability value, output the judgment result. , The positive threshold is set to 0.7. The negative threshold is 0.3.

[0039] S5: For the set of salient candidate regions identified as waste, combine the energy consumption of the unmanned vessel with the benefits of waste recycling to plan waste cleaning paths.

[0040] Make each piece of waste have a corresponding recycling value The expression for the recovery value is: The recovery value is a dimensionless operation and takes the value of a positive real number, where For the equivalent volume of garbage, As a garbage priority level, For easy recycling of waste, As a weighting term; introduce the water flow vector. Construct a single-segment path energy consumption model for unmanned vessels. ,in From the current position a To the target location b The estimated travel time, where the denominator represents the velocity vector of the unmanned vessel relative to the water. Subtract local flow vector The obtained effective velocity relative to the ground is given by the molecule as the target position. b With current location a The distance between them To improve the energy efficiency ratio, Represents drag penalty, with a value range of [0, 2], and velocity vector. With local water flow vector The smaller the included angle, the smaller the value. This is the drag coefficient; Current location a To the target location b The change in heading, The turning energy consumption coefficient ranges from 0.5 to 5 J / rad; the path energy consumption model is a dimensionless calculation. Based on the above modeling, evaluate the unit energy consumption benefit ratio of unvisited garbage. A greedy algorithm with backtracking is employed to prioritize the trash with the highest unit energy consumption benefit ratio as the target location corresponding to the current position. Based on the remaining energy of the unmanned vessel, the corresponding path energy consumption is deducted. When the accumulated energy consumption of several single-segment paths exceeds a dynamic energy consumption threshold, backtracking is triggered, returning to the most recently visited, safe return location for recharging. Unvisited targets await subsequent cleanup. The dynamic energy consumption threshold is the sum of the energy required for the current single-segment path and the energy required for a direct return from the target location. The energy required for a direct return from the target location is calculated using the unmanned vessel's single-segment path energy consumption model.

[0041] To ensure the efficiency of floating debris collection and navigation safety, the task of collecting and clearing floating debris will only be carried out when the wind speed on the water surface is ≤5m / s and the wave height is ≤0.3m. At this time, the floating debris will not change position quickly, reducing the difficulty of collection.

[0042] On the other hand, the present invention also provides a floating object identification device based on an unmanned vessel for implementing the above-mentioned method, comprising: The image preprocessing unit is used to set a rotating linear polarizer in front of the imaging optical path of the visible light-near infrared dual camera module, and to acquire visible light and near infrared images at different rotation angles of the linear polarizer in a time-division manner and perform weighted fusion to obtain an image with anti-specular reflection; then, the water ripple texture of the acquired anti-specular reflection image is further suppressed to obtain a preprocessed water surface image. The reflection restoration unit is used to restore the reflection of the preprocessed water surface image, separate the water surface area from the sky or shoreline area, fill and restore the boundary of the area where the identified reflection pixels are located, and obtain a water surface image that retains floating objects and water surface texture. The floating object recognition unit acquires a sequence of water surface images that preserves floating objects and water surface texture. It combines the water flow motion vector to calculate the optical flow residual vector, performs spatial clustering on the optical flow residual vector, and determines whether the clustered clusters belong to the floating object clusters. It then fits the minimum bounding rectangle to the floating object clusters and performs morphological dilation processing to obtain motion saliency candidate regions. The waste classification and identification unit uses a trained waste-non-waste discrimination model to perform binary classification on moving salient candidate regions and identify whether the moving salient candidate regions are waste. The waste path planning unit is used to plan waste disposal paths for multiple identified wastes, combining the energy consumption of unmanned vessels with the revenue from waste recycling.

[0043] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying floating objects based on unmanned vessels, characterized in that, Includes the following steps: S1: A visible light-near infrared dual-camera module is used. A rotating linear polarizer is set in front of the imaging optical path of the dual-camera module. Visible light and near infrared images at different rotation angles of the linear polarizer are acquired in a time-division manner and then weighted and fused to obtain an image without specular reflection. Then, water ripple texture is removed from the acquired image without specular reflection through inter-frame alignment, local displacement field modeling and blind deconvolution processing to obtain a pre-processed water surface image. S2: Perform reflection restoration on the preprocessed water surface image, separating the water surface area from the sky or shoreline area to obtain a water surface image that retains floating objects and water surface texture; S3: Based on the sequence of water surface images that preserve floating objects and water surface texture, and water flow motion vectors, extract and calculate the optical flow residual vectors of corner points, perform spatial clustering on the optical flow residual vectors, and determine whether the clustered cluster belongs to the floating object cluster based on the number of corner points in the current corner point's neighborhood, the amplitude and directional divergence of the optical flow residuals of all corner points in the neighborhood; fit the minimum bounding rectangle to the floating object cluster and perform morphological dilation processing to obtain motion saliency candidate regions; S4: Using the garbage-non-garbage discrimination model trained by contrastive learning, binary classification is performed on the moving salient candidate region to identify whether the moving salient candidate region is garbage; S5: For the set of salient candidate regions identified as waste, combine the energy consumption of the unmanned vessel with the benefits of waste recycling to plan waste cleaning paths.

2. The method for identifying floating objects based on an unmanned vessel according to claim 1, characterized in that, Step S1 describes the use of a visible-near-infrared dual-camera module. A rotating linear polarizer is placed in front of the imaging optical path of the dual-camera module. Visible and near-infrared images at different rotation angles of the linear polarizer are acquired in a time-division manner and then weighted and fused to obtain an image free of specular reflection. This includes the following: the rotating linear polarizer is mounted on the output shaft of a stepper motor, which also has a polarization angle encoder disk. The stepper motor drives the rotating linear polarizer to rotate, and the dual-camera module acquires RGB images and near-infrared channel images in four polarization directions (0°, 45°, 90°, and 135°) in real time. I 0、 I 45 , I 90 、I 135 Calculate the RGB-NIR spectral intensity map of each pixel based on the RGB images and near-infrared channel images in each polarization direction; calculate the degree of polarization map for each channel separately. DoP and polarization angle diagram AoP ; Based on near-infrared channel intensity map I 135 Average brightness map of RGB images I gary Obtain near-infrared difference map D NIR Polarization diagram DoP Difference between near-infrared and near-infrared images D NIR Normalize all values ​​to the range [0, 1], then convert the polarization diagram. DoP Difference map with near-infrared D NIR Weighted fusion constructs fusion weights W Using fusion weights W Obtain approximate values ​​of specular reflection components based on polarization degree maps and near-infrared difference maps. I reflection The image without specular reflection is obtained by subtracting the approximate value of the specular reflection component from the RGB image in the 0° polarization direction. I clean .

3. The method for identifying floating objects based on an unmanned vessel according to claim 2, characterized in that, The water ripple texture of the acquired anti-specular reflection image is removed through inter-frame alignment, local displacement field modeling, and blind deconvolution processing, resulting in a preprocessed water surface image. The process includes: inter-frame alignment of several consecutive anti-specular reflection images; compensation for global rigid body motion; calculation of the local residual displacement field relative to the reference frame for each anti-specular reflection image to obtain a local displacement field characterizing the water ripple motion; modeling point spread functions for local image patches of the anti-specular reflection image based on the local displacement field; and performing blind deconvolution processing on each local image patch using an alternating minimization method to obtain a background image with suppressed water ripple texture; and constructing a residual feature map based on the background image with suppressed water ripple texture and the anti-specular reflection image. R The residual feature map R The weighted fusion is applied to the destigmatized image to obtain the preprocessed water surface image. .

4. The method for identifying floating objects based on an unmanned vessel according to claim 3, characterized in that, Step S2 includes the following: using a lightweight semantic segmentation network model as the backbone feature extraction network, taking the preprocessed water surface image as the model input, and segmenting it through the lightweight semantic segmentation network model to obtain the water surface area and the sky or shoreline area; setting reflection candidate areas in the water surface area, calculating the gradient distribution direction of the pixels in the reflection candidate areas, and judging the reflection pixels by statistically analyzing the concentration of the gradient direction distribution. By utilizing the optical flow alignment results between preprocessed water surface images of multiple consecutive frames, similar texture blocks are searched in the non-reflecting water surface region and the temporal neighborhood. The regions where the identified reflection pixels are located are filled and the boundaries are repaired to obtain water surface images that retain floating objects and water surface texture.

5. The method for identifying floating objects based on an unmanned vessel according to claim 4, characterized in that, The lightweight semantic segmentation network model used is MobileNetV3. The training samples are images from unmanned surface vessels (USVs) in various weather conditions, and the images contain clear sky and water boundaries. The training samples are expanded by random horizontal flipping, rotation, brightness and contrast adjustment, and random cropping. Boundary-aware loss is used. The lightweight semantic segmentation network model outputs the predicted probabilities of pixels belonging to the sky, shoreline, and water surface. The MobileNetV3 lightweight semantic segmentation network model's training hyperparameters are set as follows: training sample size 512×512, batch size 4-8, 50-100 training iterations, Adam optimizer, and learning rate 10. -3 Or 5×10 -4 Using a freezing strategy, the backbone feature extraction network is frozen for the first 10 rounds of training, and only the classification head with 1×1 convolutions is trained.

6. The method for identifying floating objects based on an unmanned vessel according to claim 4, characterized in that, Step S3 involves: extracting corner points from the sequence of water surface images that preserve floating objects and water surface texture; calculating the optical flow vector corresponding to each corner point; combining this with the water flow motion vector to calculate the optical flow residual vector; calculating the amplitude of the optical flow residual vector; and filtering out optical flow residual vectors smaller than a threshold. T mag The corner points are used to remove the water surface background caused by the translational flow of water. The image coordinates and residual direction vectors of the remaining corner points are used as the feature space. A density-based spatial clustering algorithm, DBSCAN, is used for clustering. Taking any corner point as the center, if the number of corner points in the neighborhood is not less than the minimum number of corner points, the corner points in the neighborhood are determined to belong to the same core point cluster. The average residual amplitude and directional divergence are calculated for each core point in the core point cluster. The average residual amplitude is greater than the threshold 1.

2. T mag If the directional divergence is lower than the set upper limit, then the core point cluster is determined to be a floating object cluster. Morphological expansion is performed on the minimum bounding rectangle of the inner corner points of the floating object cluster to obtain the motion significant candidate region. If the magnitude of the corner optical flow residual vector is less than the threshold T mag The number of corner points within the neighborhood is less than the minimum number of corner points, and the average residual amplitude of the core point cluster does not exceed the threshold of 1.

2. T mag If the directional divergence of the core point is not less than the set upper limit, it is judged as noise.

7. The method for identifying floating objects based on an unmanned vessel according to claim 6, characterized in that, The DBSCAN algorithm is a parameter-free, unsupervised clustering algorithm, with a neighborhood radius... The value ranges from 5 to 15 pixels, and the minimum number of corner points is [not specified]. The value range is 5-10, and the threshold is... T mag The value range is 0.5-1.5px / frame; the upper limit for directional divergence is ≤18° for angular standard deviation.

8. The method for identifying floating objects based on an unmanned vessel according to claim 6, characterized in that, Step S4 involves collecting floating debris and non-debris samples, performing random geometric and photometric enhancements on each sample, and generating two data-enhanced views with equivalence of positive and negative values ​​for the same sample. The debris-non-debris discrimination model includes an encoder, a projection head, and a binary classification head. The encoder uses the convolutional backbone of the lightweight convolutional network MobileNetV3-Small to expand the number of channels according to the expansion factor, extract features through convolution in the expanded high-dimensional space, and then compress the number of channels back to the target number. The projection head is used to perform L2 normalization on the feature vector and output the projection vector. The binary classification head is set on the output side of the encoder to map the feature vector to a 1-dimensional probability that the sample is debris. In the online inference stage after training, the trained debris-non-debris discrimination model is used to output the probability that the motion salient candidate region is debris.

9. The method for identifying floating objects based on an unmanned vessel according to claim 8, characterized in that, During training, the garbage-non-garbage discrimination model first optimizes the encoder and projector head using contrastive learning loss to separate garbage samples from non-garbage samples in the embedding space. Then, the encoder is frozen and the binary classification head is adjusted to optimize the garbage-non-garbage binary classification performance. During inference after training, only the encoder and binary classification head are used to output the probability that the motion salient candidate region is garbage, and the projector head is discarded.

10. A floating object identification device based on an unmanned vessel, used to implement the method according to any one of claims 1-9, characterized in that, include: The image preprocessing unit is used to set a rotating linear polarizer in front of the imaging optical path of the visible light-near infrared dual camera module, and to acquire visible light and near infrared images at different rotation angles of the linear polarizer in a time-division manner and perform weighted fusion to obtain an image with anti-specular reflection; then, the water ripple texture of the acquired anti-specular reflection image is further suppressed to obtain a preprocessed water surface image. The reflection restoration unit is used to restore the reflection of the preprocessed water surface image, separate the water surface area from the sky or shoreline area, fill and restore the boundary of the area where the identified reflection pixels are located, and obtain a water surface image that retains floating objects and water surface texture. The floating object recognition unit acquires a sequence of water surface images that preserves floating objects and water surface texture. It combines the water flow motion vector to calculate the optical flow residual vector, performs spatial clustering on the optical flow residual vector, and determines whether the clustered clusters belong to the floating object clusters. It then fits the minimum bounding rectangle to the floating object clusters and performs morphological dilation processing to obtain motion saliency candidate regions. The waste classification and identification unit uses a trained waste-non-waste discrimination model to perform binary classification on moving salient candidate regions and identify whether the moving salient candidate regions are waste. The waste path planning unit is used to plan waste disposal paths for multiple identified wastes, combining the energy consumption of unmanned vessels with the revenue from waste recycling.

Images