A material level perception, underwater degradation self-adaptation and long-tail category optimization method and system for marine garbage detection

Abstract

This invention discloses a material hierarchy perception, underwater degradation adaptation, and long-tail category optimization method and system for marine debris detection, belonging to the fields of computer vision and intelligent marine environmental monitoring. The method constructs a material category system, calculates image color shift, contrast reduction, blurring degree, and scattering intensity to obtain an underwater degradation score, and adaptively fuses branch features from the original and enhanced images. The detection network outputs the location, category, and material probability of debris targets, applying consistency constraints based on the hierarchical mapping between fine-grained and material categories. To address the long-tail distribution problem, category weights and resampling factors are constructed, and a long-tail loss function is introduced to enhance tail category recognition capabilities. Finally, multi-frame video results are temporally fused to output stable material classification and pollution statistics. This invention effectively improves the accuracy and statistical reliability of debris detection in complex underwater environments and is applicable to scenarios such as marine monitoring, environmental assessment, and intelligent cleanup assistance.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, intelligent monitoring of marine environment and target detection technology. Specifically, it relates to an intelligent detection method and system for garbage targets in marine, near-shore and deep-sea scenarios. In particular, it relates to a marine garbage detection method, device, electronic device and storage medium that integrates material-level semantic constraints, underwater degradation adaptive feature modulation and long-tail category optimization mechanism. Background Technology

[0002] Among existing publicly available technologies, several solutions have been studied from different perspectives to address the problem of marine debris or underwater target identification. For example, Chinese invention patent (CN117994499A), "A Marine Debris Detection Method and Electronic Device and Storage Medium Based on Deep Learning," discloses a technical solution for marine debris detection. This solution enhances marine scene images and expands sample data using virtual rendering, then utilizes a deep learning detection model to identify marine debris targets, thus improving the accuracy of marine debris detection to some extent. Another example is Chinese invention patent (CN118570625A), "A Method for Detecting Underwater Prominent Objects Based on Secondary Degradation," which discloses a detection scheme for underwater salient targets. This scheme improves the robustness of target detection in complex underwater environments through underwater image enhancement, unsupervised learning based on secondary degradation, and multi-stream feature fusion.

[0003] However, the aforementioned existing technologies still primarily focus on single aspects of marine debris detection or underwater target detection: the former emphasizes marine debris target detection and sample expansion, without further constructing material-level semantic representation, long-tail category optimization, and video-level stable statistical mechanisms to meet the needs of marine debris management; the latter focuses on salient target detection under underwater image degradation conditions, without establishing material differentiation, category imbalance optimization, and continuous video statistical analysis schemes specifically for marine debris. Therefore, existing technologies still lack a unified marine debris detection technology solution that integrates underwater degradation modeling, material-level semantics, long-tail category optimization, and video-level statistical analysis. Summary of the Invention

[0004] This invention provides a marine debris detection method based on deep representation learning and manifold distribution alignment, mainly including: acquiring multi-source heterogeneous image and video stream data, constructing a data manifold and distribution alignment layer, implementing label ontology mapping and semantic aggregation to obtain a unified material classification topology; constructing a degradation perception module for the input data, quantifying spatial visual degradation representation, and obtaining underwater visual degradation scores; constructing nonlinear gating coefficients using the underwater visual degradation scores, and adaptively converging and fusing the original image branch features and enhanced branch features to obtain fused features; inputting the fused features into a fully convolutional single-stage perception network for deep representation decoupling and prediction, and outputting target spatial coordinates, category confidence, and material probability distribution; constructing a fine-grained mapping matrix to the core material category for sample manifolds with fine-grained historical labels and introducing hierarchical consistency loss for hierarchical semantic constraints and consistency alignment; and implementing cost-sensitive intervention and dynamic penalty for long-tail distribution based on the sample frequency of each material category in the manifold space, and superimposing long-tail modulation terms in the classification loss space. The network is trained by weighted orthogonal joint optimization of various losses to form a high-dimensional multi-objective joint optimization mechanism; posterior knowledge fusion in the temporal dimension is implemented for the trajectory of the same physical target in high spatial resolution video streaming media, and high-throughput streaming perception and posterior filtering fusion are performed to output regional material proportion and decision evaluation results.

[0005] Furthermore, multi-source heterogeneous image and video stream data are collected, a data manifold and distribution alignment layer is constructed, and label ontology mapping and semantic aggregation are implemented to obtain a unified material classification topology system. This includes: collecting multi-source heterogeneous image and video stream data with long-tailed and chaotic distribution characteristics from real marine environments, and constructing a training manifold space. For the original high-variance discrete categories, a label ontology dimensionality reduction mapping mechanism is introduced to aggregate multi-source fine-grained categories into a unified material classification topology system in the semantic space. Preferably, the material topology system is dimensionality-reduced to nine core semantic categories: Plastic, Glass, Metal, Paper, Fiber, Foam, Rubber, Electronics, and Other, to significantly reduce the inter-class feature manifold variance.

[0006] Furthermore, a degradation perception module is constructed for the input data to quantify spatial visual degradation characteristics and obtain an underwater visual degradation score. This includes calculating the color shift, contrast attenuation, blur divergence, and scattering intensity of the input image matrix, and constructing an underwater visual degradation scoring function. The degradation scoring function is defined as: D(χ)=α1,Ccast(χ)+α2·(1-Contr(χ))+α3·BLur(χ)+α4·Scatter(χ), where α1, α2, α3, α4 are weight hyperparameters and satisfy ∑αi=1. Preferably, in one embodiment of the present invention, α1=0.30, α2=0.25, α3=0.20, and α4=0.25. Furthermore, a nonlinear gating coefficient is constructed using underwater visual degradation scoring to adaptively converge and fuse the branch features of the original image and the enhanced branch features, resulting in fused features. This includes: constructing a nonlinear gating coefficient ɡ(χ)=σ(ḳ(D(χ)-τ)) using the degradation score D(X), where σ is the Sigmoid activation function and τ is the threshold, and then fusing the branch features of the original image... Fr With domain adaptive enhancement of branch features Fe Perform dynamic weighted fusion: F= (1-ɡ)Fr+ɡFe,,This mechanism ensures that the response weights of the enhanced features are adaptively increased when the degradation level is high, while more original high-frequency spatial textures are preserved in low degradation scenarios.

[0007] Furthermore, the fused features are input into a fully convolutional single-stage perceptual network for deep representation decoupling and prediction, outputting target spatial coordinates, category confidence, and material probability distribution, including: fusion features F The target is deconstructed using a fully convolutional single-stage perceptron. Preferably, the backbone network adopts the YOLO26x architecture, balancing high receptive field feature extraction and real-time inference efficiency against complex marine backgrounds. The network's prediction task is mapped to a decoupled multi-objective optimization space, outputting the target space coordinates (Box), class confidence (Cls), and material probability distribution. Receptive field enhancement units, shallow-deep composite connections, and Distribution Focal Loss (DFL) units for refining bounding box ambiguity are introduced into the feature fusion layer and the detection head. Furthermore, for sample manifolds with fine-grained historical labels, a fine-grained mapping matrix to the core material category is constructed, and a hierarchical consistency loss is introduced for hierarchical semantic constraints and consistency alignment, including: constructing a fine-grained mapping matrix to the core material category for sample manifolds with fine-grained historical labels. AThe high-dimensional fine-grained probability distribution is projected onto the material subspace, and a hierarchical consistency loss Lhier=Lmat+λh·|pm-pm| is introduced to constrain the output of the deep representation to maintain rigid consistency with the prior topological structure of the historical categories while satisfying the current spatial classification distribution.

[0008] Furthermore, based on the sample frequency of each material category in the manifold space, cost-sensitive intervention and dynamic penalty are implemented for the long-tail distribution, and a long-tail modulation term is superimposed on the classification loss space, including: based on the sample frequency Nc of each material category in the manifold space, A dynamic category-weighted penalty boundary ωc = min(ωmax, (Navg / Nc)Y) is introduced, and a heuristic resampling factor rc = min(rmax, Nref) / Nc is applied. A long-tail modulation term, defined as Llt = ∑ωc·CE(ycyc) + βLseesaw, is superimposed on the classification loss space. By setting loss truncation and high-rate resampling for minority classes, the model is forced to cross local optima, alleviating the gradient suppression of the minority class by the majority class. Furthermore, the various losses are jointly optimized using weighted orthogonal methods to form a high-dimensional multi-objective joint optimization mechanism for training the network. This includes: jointly optimizing the bounding box localization loss, hierarchical consistency semantic loss, long-tail cost-sensitive loss, and degradation-gated modulation loss using weighted orthogonal methods, defining the overall empirical risk function of the system as: L = λ1Ldet + λ2Lhier + λ3Llt + λ4Ldeg. At the end of training, the distribution focus loss adaptively converges to an extremely low threshold approaching 0.007, achieving a high-precision continuous distribution representation.

[0009] Furthermore, temporal posterior knowledge fusion is implemented for the trajectory of the same physical target in high spatial resolution video streaming media. High-throughput streaming perception and posterior filtering are fused to output regional material proportion and decision evaluation results. This includes: For the trajectory K of the same physical target in high spatial resolution video streaming media, temporal posterior knowledge fusion is implemented. The expected score of its material probability response at multiple frame timest t is calculated as pt,c: Sk,c = 1 / Tk∑t(pt,c); and the final material semantics of the target is determined by ck = argmaxcSk,c. In the posterior filtering stage, heuristic confidence truncation constraints and non-maximum suppression (NMS) spatial intersection-union ratio constraints are jointly applied to effectively filter out false positive interference caused by ocean highlights or waves, thereby outputting regional material proportion, pollution spatial distribution, and cleanup priority decision evaluation results.

[0010] The technical solutions provided by the embodiments of this invention can include the following beneficial effects: This invention discloses a method for hierarchical perception of marine debris materials based on manifold distribution alignment and deep representation learning. Addressing the technical bottlenecks commonly found in existing technologies, such as severe underwater image degradation, insufficient utilization of fine-grained semantic representations, weak long-tail category perception capabilities, and poor posterior stability of high-resolution streaming media predictions, this method proposes a full-link visual perception architecture encompassing data manifold reconstruction, deep representation decoupling optimization, and high-dimensional parameter space scheduling. First, by employing a quantized degradation scoring function and a dynamic feature gating mechanism, adaptive alignment and enhancement of the data manifold at the feature level are achieved for complex physical degradation scenarios such as color distortion, high scattering, and low contrast. Then, innovatively applying label ontology mapping and hierarchical consistency loss, the discrete, fine-grained historical structure prior is dimensionality-reduced and projected onto the core material space, significantly reducing the feature confusion variance between heterogeneous and junk materials. Simultaneously, cost-sensitive learning and a heuristic high-rate resampling mechanism are introduced, coupled with a long-tailed modulation loss with dynamic penalty boundaries, effectively curbing gradient suppression caused by extreme class imbalances and significantly improving the sensitivity to rare classes. Finally, based on a decoupled multi-objective optimization spatial and temporal posterior fusion algorithm, combined with heuristic truncation and NMS constraints, high-order temporal stability of video-level material identification is achieved while maintaining near real-time throughput for extremely high-resolution streaming. This invention significantly overcomes the static limitations of traditional underwater image enhancement algorithms, achieving an exponential leap in generalization ability on massive image manifolds, ultimately approximating the joint distribution of real marine physical data. Attached Figure Description

[0011] Figure 1 This is an overall flowchart of the method of the present invention.

[0012] Figure 2 This is a block diagram of the system structure of the present invention.

[0013] Figure 3 This is a schematic diagram of a degenerate adaptive dual-branch feature fusion structure.

[0014] Figure 4 This is a schematic diagram of the semantic constraint structure of the material hierarchy.

[0015] Figure 5 A schematic diagram of the optimization process for long-tail categories.

[0016] Figure 6 This is a schematic diagram of the video-level material statistics and pollution assessment process. Detailed Implementation

[0017] The technical solutions in the embodiments of the present invention are described clearly and in detail below. The described embodiments are merely a part of the embodiments of the present invention. Figures 1-6This embodiment of a method and system for marine debris detection and video-level material pollution statistics may specifically include the following steps and module collaboration process: First, as Figure 2 As shown, the system structure diagram of this invention mainly consists of a data acquisition module, a degradation assessment module, a feature modulation module, a target detection module, a hierarchical semantic constraint module, a long-tail optimization module, a video statistics module, and an application output module cascaded together to complete end-to-end processing from data input to pollution assessment. Step S101: Marine debris image / video input. Marine debris images and video data are acquired through the data acquisition module, serving as the input basis for subsequent model training and detection. Step S102: Sample annotation and mapping of 9 material types. A marine debris training sample set is constructed, and bounding box annotations and material annotations are performed on the debris targets in the samples.

[0018] In one implementation, fine-grained categories from different data sources are integrated into a 9-category material system through mapping rules for subsequent unified training and statistical analysis. For example, in an engineering context, 75,156 images and 307,530 instances can be used as the training sample base. Steps S103-S105 involve underwater degradation score calculation and degradation adaptive dual-branch gating feature fusion.

[0019] like Figure 3 The diagram shows a degradation adaptive dual-branch feature fusion structure. For the input image, the degradation evaluation module calculates its color shift, blurriness, contrast, and scattering intensity to obtain the degradation score D(x).

[0020] Specifically, the feature modulation module generates a gating coefficient g(x) based on the degradation score, and uses this coefficient to dynamically weight the original image branch feature Fr and the enhanced image branch feature Fe. When the score D(x) exceeds a threshold τ, the enhanced branch feature Fe takes on a larger weight, performing color restoration, local contrast enhancement, or scattering suppression on the image; when the score is below the threshold τ, the original image branch feature Fr is used more. The final output fusion feature equation is: F= (1-ɡ)Fr+ɡFe, and input this fused feature into the detection / material joint prediction branch. Preferably, the degradation score related parameter K can be 5, and τ can be 0.45. Step S106, target detection and material probability output.

[0021] A YOLO26x-based object detection network is used as the basic backbone network, with a material classification head added in addition to the detection head. After the feature layer passes through the receptive field enhancement module and the prediction refinement module, it simultaneously outputs the detection box, object confidence, and material probability. Step S107: Layer consistency and long-tail optimization.

[0022] On the one hand, such as Figure 4 The diagram shows the hierarchical semantic constraint structure for materials. The hierarchical semantic constraint module receives the predicted material probability pm output by the detection backbone network and the material classification head. Simultaneously, fine-grained historical labels are converted into prior material probabilities through a mapping matrix A. These two parameters are jointly used to calculate the hierarchical consistency loss Lhier, which, along with the cross-entropy loss, is used for joint optimization training of the material branches.

[0023] On the other hand, such as Figure 5 The diagram illustrates the long-tail category optimization process. The long-tail optimization module first performs category sample statistics (Nc), calculates the category weight (wc) and resampling factor (rc) based on the number of samples in each category, and then performs tail category sample enhancement. Preferably, the upper limit of the category weight is 5.0, and the maximum resampling factor for the rare category "Electronics" is 20. A Seesaw modulation term (containing a long-tail modulation term based on category frequency and online prediction bias) is introduced on top of the classification loss (CE) to jointly constitute the long-tail optimization loss (Llt) to alleviate gradient suppression of the head class. For example, the value of Y in this modulation term ranges from 0.5 to 1.0, and the value of β ranges from 0.2 to 1.0. It should be noted that the model training parameters of the above network are configured as follows: input image size is 640; optimizer is AdamW; initial learning rate is 0.001, final learning rate is 0.0001057; total batch size is 64; FP16 mixed precision training is used; early stopping parameter patience is 50. Furthermore, marine scene data augmentation and progressive mosaic closing strategies can be combined during training to improve the model's generalization ability. Step S108: Video trajectory fusion and pollution statistics output.

[0024] like Figure 6 The diagram illustrating the video-level material statistics and pollution assessment workflow shows that the video statistics module and the application output module work together. After inputting the monitoring video stream, marine debris detection is performed frame-by-frame, and target trajectories are generated using a target association algorithm. The material probabilities of the same target trajectory are temporally fused to obtain Sk,c, thus outputting stable material category results. Subsequently, regional material proportion statistics are performed (e.g., calculating the proportions of categories such as Plastic, Metal, Glass, Foam, and Electronics), ultimately generating a pollution composition and cleanup priority report.

[0025] For example, in an engineering example, for a 2304×1440 resolution video, a confidence threshold of 0.3 and an IoU threshold of 0.45 can achieve an inference speed of approximately 20.07 fps under single-GPU conditions. It will be apparent to those skilled in the art that this application is not limited to the details of the above exemplary embodiments, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Therefore, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A material hierarchy perception, underwater degradation adaptation, and long-tail category optimization method for marine debris detection, characterized in that, The process includes the following steps: acquiring marine debris image or video data; labeling marine debris targets in the images or videos; constructing a marine debris training sample set; and integrating multi-source fine-grained categories into a material category system through a preset mapping relationship. The material category system includes at least one or more of Plastic, Glass, Metal, Paper, Fiber, Foam, Rubber, Electronics, and Other. Underwater degradation features are calculated on the input images according to D(x) = α1·Ccast(x). The degradation score, representing color shift, contrast reduction, blur degree, and scattering intensity, is obtained by α1, α2, α3, and α4 as weight coefficients with a sum of 1. Based on the degradation score, feature modulation coefficients are generated through the gating function g(x)=σ(k(D(x)-τ)). Adaptive fusion is performed on the original image branch feature Fr and the enhanced image branch feature Fe to obtain the degradation adaptive fusion feature F=(1-g)Fr+gFe. The target detection network should be integrated with the feature inputs to output the location, category confidence, and material category probability of marine debris targets. Based on the fine-grained category-to-material category mapping matrix A, the predicted probabilities of fine-grained categories are mapped to the material category space, and a hierarchical consistency loss Lhier = Lmat + λh·|p̂m - pm| is constructed to apply hierarchical consistency constraints to the material category results output by the target detection network, where Lmat is the material classification loss, pm is the material classification header output result, and p̂m is the material prior obtained from the fine-grained category mapping. The results are then processed according to the sample distribution of each material category. The class weights are constructed according to wc=min(wmax,(Navg / Nc)^γ), and the resampling factor is constructed according to rc=min(rmax,ceil(Nref / Nc)). The tail class is optimized by combining the long-tail loss function, where wmax is the upper limit of the class weight and not greater than 5.0, and rmax is the upper limit of the resampling and not greater than 20. The detection results of the same marine debris target in the video at multiple time points are fused temporally. The stable target material category result is obtained according to Sk,c=(1 / Tk)Σpt,c, and ck*=argmaxc Sk,c is used as the final material category. The statistical results of the marine debris material composition are output. The joint training objective is L=λ1Ldet+λ2Lhier+λ3Llt+λ4Ldeg, where Ldet is the detection loss, Llt is the long-tail optimization loss, and Ldeg is the degradation adaptive loss.

2. The method according to claim 1, characterized in that, The target detection network is a single-stage or multi-stage target detection network; in one embodiment, the target detection network adopts a YOLO26x-based detection network, and introduces receptive field enhancement units, shallow-deep composite connection units and / or prediction refinement units in the feature extraction layer, feature fusion layer or detection head.

3. The method according to claim 1, characterized in that, The sum of the weighting coefficients α1, α2, α3, and α4 in the degradation score is 1; in the gating function, k is the steepness parameter, τ is the threshold parameter, and σ is the sigmoid function.

4. The method according to claim 1, characterized in that, The hierarchical consistency constraint includes: constructing a mapping matrix A from fine-grained categories to material categories, mapping the predicted probabilities of fine-grained categories to the material category space, constructing a hierarchical consistency loss based on the mapping results and the output results of the material classification head, and supervising network training with the hierarchical consistency loss.

5. The method according to claim 1, characterized in that, The upper limit of the category weight wmax is no greater than 5.0; the upper limit of resampling rmax is no greater than 20; in one embodiment, the Electronics category is resampled by a maximum of 20 times.

6. The method according to claim 1, characterized in that, The long-tail loss function includes a mitigation term based on class frequency relationships and a compensation term based on online prediction probability relationships, in order to reduce the classification suppression of tail classes by head classes.

7. The method according to claim 1, characterized in that, In the temporal fusion step, Tk represents the frame number corresponding to the k-th target trajectory, and pt,c represents the probability that the k-th target in the t-th frame belongs to the c-th material type. ck*=argmaxc Sk,c is used as the final material type of the k-th target trajectory, and the statistical results of the proportion of contaminated materials in the output area are based on the area, confidence level, and / or frequency of occurrence of each target trajectory.

8. A material hierarchy perception, underwater degradation adaptation, and long-tail category optimization system for marine debris detection, characterized in that, include: The data acquisition module is used to acquire marine debris image or video data and build a training sample set; the degradation assessment module is used to calculate the degradation score of the input image. The feature modulation module is used to adaptively fuse the original branch features and the enhanced branch features based on the degradation score; The target detection module outputs the location, category, and material probability of marine debris targets; the hierarchical semantic constraint module executes consistency constraints based on the mapping relationship between fine-grained categories and material categories; the long-tail optimization module performs category weight calculation, resampling, and long-tail loss optimization; and the video statistics module performs temporal fusion of target trajectories and outputs statistical results of material composition.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1 to 7.

Images