Water surface floating garbage monitoring method and system based on unmanned aerial vehicle

Abstract

The invention discloses a water surface floating garbage monitoring method and system based on an unmanned aerial vehicle. The method comprises the steps of data acquisition, environment feature extraction, modal weight dynamic adjustment, feature weighted fusion, cross-modal consistency constraint and confidence correction and detection result output. The unmanned aerial vehicle carries a multi-mode sensor to collect RGB, polarization, spectrum, depth and environment data at the same time, and a multi-mode data set is generated. An environmental feature-driven modal weight control network generates each modal fusion weight and calculates an environmental dependency factor to dynamically adjust consistency constraints. Performing weighted fusion and multi-layer convolution processing on each modal feature graph to generate a fused feature graph and reserving a modal response graph; candidate targets are generated based on the fusion feature map, and the confidence coefficient is corrected in combination with the environmental dependence factor, so that high-precision and high-robustness water surface garbage detection is realized. According to the invention, through an environment-driven dynamic closed-loop mechanism, organic cooperation of a multi-modal algorithm is realized, and the accuracy and adaptability of water surface garbage identification are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of environmental protection, and particularly relates to a water surface floating garbage monitoring method and system based on a UAV. BACKGROUND

[0002] With the acceleration of industrialization and urbanization, the problem of floating garbage in rivers, lakes, ports and offshore waters is becoming increasingly serious. These floating garbage not only affects the water landscape, but also threatens navigation safety, aquatic ecosystems and drinking water sources. Therefore, how to efficiently and accurately monitor and identify water surface floating garbage has become an important research direction in environmental monitoring and management. The existing monitoring methods of water surface floating garbage mainly fall into two categories: relying on manual boat patrol, taking photos, and manually marking the position of garbage. This method has the problems of low efficiency, limited coverage, and being greatly affected by weather and water conditions, and high labor cost. Using a UAV or a shore-based fixed camera, RGB images are collected, and traditional image processing or deep learning detection algorithms are used to identify floating garbage. However, the water surface environment has characteristics such as high reflectivity, wave disturbance, and water color change, and a single modality is easily disturbed by factors such as light, reflection, and turbidity, resulting in high false detection and missed detection rates.

[0003] In order to solve the problem of single modality, in recent years, multi-modal fusion monitoring technology has appeared, such as using multiple modal sensors to obtain multiple feature information at the same time, and fusing the results of each modality at the algorithm backend. However, the existing multi-modal fusion technology generally has the following problems:

[0004] 1. Fixed weight fusion: Most methods use fixed weights or simple splicing when fusing different modal features, lacking the ability to adaptively adjust to different light conditions, water color changes, and reflection intensities, resulting in a decrease in fusion effect in some scenes.

[0005] 2. Lack of mutual constraint between modalities: In conventional multi-modal detection, each modality is independently inferred, and only a simple combination is performed at the result layer, for example, a high-reflectivity area shows a strong response in the polarization modality, but if there is no corresponding texture feature in the RGB / spectral modality, the target is likely to be a wave or a bird, but it is not filtered. SUMMARY

[0006] (I) Technical problems solved

[0007] In view of the deficiencies of the prior art, the present application aims to provide a water surface floating garbage monitoring method and system based on a UAV, which solves the problem that the existing multi-modal fusion method mostly adopts a fixed weight or a simple splicing method, the contribution degree of each modal feature remains unchanged in different environments, and the detection performance is reduced in complex scenes such as strong light, reflection, turbid water, etc. The present application introduces a modal weight control subnetwork (MCN), dynamically calculates the fusion weight of different modes according to the real-time collected environmental features (light intensity, water turbidity, reflection intensity, weather conditions, etc.), so that each mode automatically plays the optimal role in different scenes. For example, the weight of the polarization mode is increased in strong light conditions, the weight of the spectral mode is increased in turbid water, and the weight of the RGB mode is increased in overcast low color temperature conditions, so that the system has self-adaptive capability.

[0008] (II) Technical solutions

[0009] To achieve the above-mentioned purpose, the present application provides the following technical solutions: a water surface floating garbage monitoring method based on a UAV, comprising the following steps:

[0010] S1 data acquisition: synchronously collecting the RGB image, polarization image, spectral data, depth data and environmental data of the water surface area by the multi-modal sensor carried by the UAV to generate a multi-modal data set D;

[0011] S2 environmental feature vector generation: extracting the light feature, water quality feature, reflection feature and weather feature from the environmental data and performing normalization processing to obtain an environmental feature vector E;

[0012] S3 modal weight dynamic adjustment: inputting the environmental feature vector E into the modal weight control subnetwork MCN to output the fusion weight W of each modal, and mapping the environmental feature vector E through the MCN to generate g(E) for dynamically adjusting the subsequent consistency constraint;

[0013] S4 feature extraction and weighted fusion: inputting the preprocessed modal images into a feature extraction network to obtain corresponding modal feature maps Fk; performing weighted fusion on the modal feature maps according to the fusion weight W to generate a fusion feature map F fusion , and simultaneously retaining the response map R k of each modal;

[0014] S5 cross-modal consistency constraint and confidence correction: generating a candidate target based on the fusion feature map and attaching an initial confidence; calculating the consistency score C i of the candidate target in the modal response map, and dynamically scaling the consistency score based on the environmental dependence factor g(E) to obtain a corrected score C i '; weighting and combining the corrected score and the initial detection confidence to obtain a final confidence;

[0015] S6 result output: output the candidate target with final confidence greater than the threshold as the water surface floating garbage detection result.

[0016] Preferably, the network structure of the MCN in S3 is composed of a modal weight branch and an environment-dependent factor branch.

[0017] Preferably, the modal weight branch comprises:

[0018] Input layer: used for receiving the environment feature vector E;

[0019] Hidden layer: two fully connected layers, each followed by a ReLU activation function, used for capturing the nonlinear relationship between the environment features and the modal weights;

[0020] Output layer: a fully connected layer, outputting the modal fusion weights W, with the expression being:

[0021] W = Softmax(W2·σ(W1·E+b1)+b2)

[0022] wherein W1 and W2 are trainable weight matrices; b1 and b2 are bias vectors; and σ(·) is a nonlinear activation function.

[0023] Preferably, the environment-dependent factor branch comprises:

[0024] Input layer: receiving the environment feature vector E;

[0025] Hidden layer: a fully connected layer followed by a Sigmoid activation function, used for generating the environment-dependent factor g(E):

[0026] g(E) = Sigmoid(FC2(ReLU(FC1(E))))

[0027] wherein FC1 and FC2 are fully connected layers, and Sigmoid limits the output to the range of [0, 1].

[0028] Preferably, the pre-processing of the images of each modality in S4 comprises distortion correction, spatial registration, and time synchronization.

[0029] Preferably, the generation of the fused feature map F fusion in S4 comprises:

[0030] a. inputting each modality into a feature extraction network for processing to obtain a feature map set Fk;

[0031] b. inputting the modal weights W = {w RGB , w Pol , w Spec , w Depth} and each modal feature map is element-wise weighted to obtain a single-modal feature map F' k ;

[0032] c. The weighted modal feature map F' k is fused to obtain a fusion feature map F fusion .

[0033] Preferably, the fusion processing includes:

[0034] Channel splicing: splicing each modal feature map in the channel dimension to obtain a preliminary fusion feature map;

[0035] Channel compression and convolution fusion: compressing the channels by one-dimensional convolution, and extracting deep fusion features by using continuous convolution, batch normalization and activation function;

[0036] Multi-layer convolution fusion: through cross-layer connection, different scale modal features are complementary superimposed to obtain the final fusion feature map F fusion .

[0037] Preferably, the expression of the cross-modal consistency score in S5 is:

[0038]

[0039] Wherein, M is the number of modalities; cos(·) represents the cosine similarity; C i reflects the feature consistency of the candidate target in different modalities.

[0040] Preferably, the expression of the final confidence in S5 is:

[0041]

[0042] Wherein: α, β are weight coefficients, and α+β=1; the size of β is dynamically adjusted by the environmental factor g(E).

[0043] An unmanned aerial vehicle-based water surface floating garbage monitoring system, comprising:

[0044] An unmanned aerial vehicle platform: the unmanned aerial vehicle platform is used to carry multiple modal sensors and complete the covering flight of the target area of the water area under the control of a preset flight route;

[0045] An RGB camera: the RGB camera is installed on the unmanned aerial vehicle platform and is used to acquire visible light image information of the water surface area;

[0046] A polarization imager: the polarization imager is installed on the unmanned aerial vehicle platform and is used to collect water surface reflection and polarization feature information. By analyzing the optical response of different polarization angles, high-reflectivity garbage targets can be significantly distinguished from background water or spray, thereby reducing false detection under a single RGB modality.

[0047] a spectral imager mounted on the UAV platform for acquiring spectral information of water quality features and color distribution;

[0048] a depth camera mounted on the UAV platform for acquiring three-dimensional structure information and depth data of the water surface;

[0049] an environmental sensor mounted on the UAV platform for collecting environmental parameters such as illumination intensity, color temperature, turbidity, reflectivity, and weather type;

[0050] a cloud processor connected with the UAV platform through wireless communication for integrated processing and storage of the multi-modal data uploaded by the UAV.

[0051] (III) Beneficial effects

[0052] The present application provides a water surface floating garbage monitoring method and system based on a UAV, which has the following beneficial effects compared with the existing monitoring technology relying on a single mode or a fixed fusion strategy:

[0053] Firstly, by introducing a modal weight control subnetwork (MCN), the system can dynamically adjust the fusion weights of RGB, polarization, spectral, and depth modal according to environmental features such as illumination intensity, water turbidity, reflectivity, and weather type, realizing joint optimization among modes. This mechanism avoids the drawbacks of artificial preset fixed weights, significantly improving detection adaptability and robustness in different scenarios.

[0054] Secondly, by designing a cross-modal consistency constraint mechanism, the present application not only complements modes in the feature fusion stage, but also introduces cross-modal spatial consistency verification in the candidate target determination stage. If a mode makes a mistake, other modes can constrain and correct it, thereby reducing the interference of non-garbage targets such as spray, birds, and reflective objects.

[0055] Finally, the fused feature map retains the multi-modal complementarity while combining dynamic weight adjustment and cross-modal consistency correction, so that the detection model still maintains high confidence output in complex environments, reduces missed detection and false detection, and improves overall detection accuracy and system stability. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described below in a clear and complete manner. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0057] Embodiment 1:

[0058] An unmanned aerial vehicle-based water surface floating garbage monitoring system, comprising:

[0059] An unmanned aerial vehicle platform: the unmanned aerial vehicle platform is used to carry multi-modal sensors and a computing module, and provides flight control, power supply and data transmission support required for system operation. The unmanned aerial vehicle platform can complete coverage flight of a target area of a water area under a preset route or manual control, and ensure the spatiotemporal consistency of multi-modal data acquisition.

[0060] An RGB camera: the RGB camera is installed on the unmanned aerial vehicle platform, and is used to acquire visible light image information of a water surface area, and provides basic features such as color and texture as one of important inputs for multi-modal data fusion.

[0061] A polarization imager: the polarization imager is installed on the unmanned aerial vehicle platform, and is used to acquire water surface reflection and polarization feature information. By analyzing optical responses at different polarization angles, high-reflectivity garbage targets can be significantly distinguished from background water or spray, thereby reducing false detection under a single RGB mode.

[0062] A spectral imager: the spectral imager is installed on the unmanned aerial vehicle platform, and is used to acquire spectral information of water quality features and color distribution. By acquiring image data at multiple narrow-band wavebands, the spectral imager can be used to identify water color differences, pollutant distribution and spectral features of floating garbage, thereby providing rich representation capabilities for subsequent modal fusion.

[0063] A depth camera: the depth camera is installed on the unmanned aerial vehicle platform, and is used to acquire three-dimensional structure information and depth data of a water surface. By using structure light or binocular imaging technology, the depth camera can provide spatial geometric relationships between garbage targets and background water surfaces, and is used to assist positioning and reduce interference of false targets.

[0064] An environmental sensor: the environmental sensor is installed on the unmanned aerial vehicle platform, and is used to acquire environmental parameters such as illumination intensity, color temperature, turbidity, reflectivity and weather category. The parameters are input as an environmental feature vector into a modal weight control network (MCN), and are used to guide dynamic weight distribution of different modes in a feature fusion process.

[0065] A cloud processor: the cloud processor establishes a data connection with the unmanned aerial vehicle platform through wireless communication, and is used to integrally process and store multi-modal data uploaded by the unmanned aerial vehicle. The cloud processor can run a deep learning inference framework, perform candidate target detection, cross-modal consistency constraint and final result output on uploaded fusion features, and can also perform historical data archiving and model iterative updating.

[0066] An unmanned aerial vehicle-based water surface floating garbage monitoring method, comprising the following steps:

[0067] S1 Data collection

[0068] In this step, the target water area is monitored by a multi-modal sensor carried by a UAV (Unmanned Aerial Vehicle), and raw observation data is obtained. Specifically, it includes:

[0069] S1-1 Multi-modal sensor configuration

[0070] The UAV-borne sensor includes:

[0071] RGB camera: Obtain three-channel visible light images of the water surface area, with a resolution of HxW, to provide texture and color information;

[0072] Polarization camera: Capture polarization intensity in horizontal, vertical, and 45° directions, output polarization image I p , to enhance reflection suppression and floating object edge features;

[0073] Spectral sensor: Collect spectral reflectance data I s (λ) in multiple wavebands, where λ is the wavelength, to distinguish different materials of floating objects;

[0074] Depth camera: Obtain depth information I d of the water surface and floating objects, supporting three-dimensional position estimation and volume perception;

[0075] Environmental sensor: Including illumination sensor, water quality turbidity sensor, temperature and humidity and barometric pressure sensor, to obtain environmental state I e .

[0076] S1-2 Data synchronization and timestamp alignment

[0077] Due to different sampling frequencies, each modal data needs to be timestamped based on a unified clock:

[0078]

[0079] Where t is the timestamp, and T is the sampling time sequence set.

[0080] Through the time synchronization mechanism, the RGB, polarization, spectral, depth, and environmental data collected at the same time point have spatial and temporal consistency.

[0081] S1-3 Data set construction: Spatial registration of the above synchronized data to generate a spatiotemporally consistent multi-modal data set D, providing input for subsequent processing.

[0082] S2 Environment feature vector generation

[0083] In the multi-modal data set obtained in step S1, in addition to the perception data such as images, spectra and depths, the application further utilizes the external environment data collected by the environment sensor to extract an environment feature vector E.

[0084] Specifically, the environment data includes:

[0085] Illumination feature: the ambient brightness value is collected by the illumination intensity sensor, and the brightness distribution information of the RGB image histogram is combined to generate a feature index describing the current illumination intensity and contrast;

[0086] Water quality feature: the turbidity, pH value, conductivity and other parameters collected by the water quality sensor form an index representing the transparency and chemical properties of the water body, which is used to assist in judging the visibility of the garbage target in the water surface background;

[0087] Reflective feature: the polarization degree and polarization angle distribution of the water surface are analyzed through the polarization image to extract a water surface mirror reflection degree index to describe the interference level;

[0088] Weather feature: the wind speed, humidity, temperature and weather state (such as sunny, cloudy, rainy) obtained by the weather sensor are used to represent the influence of external climate conditions on imaging and garbage distribution.

[0089] For the above multi-source environment features, the following processing methods are adopted:

[0090] First, different physical quantities are converted to a unified dimension (such as interval [0, 1]) by using normalization method to eliminate the dimension difference;

[0091] Second, principal component analysis (PCA) or feature screening algorithm is used to remove redundant features and improve the compactness and representation efficiency of the vector;

[0092] Finally, the processed multi-dimensional features are spliced to form the environment feature vector E, and the structure of the vector is:

[0093] E = [e light , e water , e reflection , e weather ]

[0094] Wherein, e light represents the illumination feature sub-vector, e water represents the water quality feature sub-vector, e reflection represents the reflective feature sub-vector, and e weather represents the weather feature sub-vector.

[0095] The environment feature vector E is used as input, which will drive the modal weight control sub-network (MCN) to calculate the fusion weight of each mode, and also be used for dynamic adjustment of cross-modal consistency constraint to realize global adaptive control.

[0096] S3 Modal Weight Generation and Dynamic Adjustment

[0097] After obtaining the environmental feature vector E in step S2, this invention inputs the vector into the Modality Control Network (MCN) to achieve dynamic weighted adjustment of different modal features.

[0098] S3-1 Network Structure

[0099] The MCN receives the environmental feature vector E and outputs the fusion weights W for each modality and the environmental dependency factor g(E). The network consists of two parallel branches:

[0100] (1) Modal weight branch (WeightBranch)

[0101] The input layer receives the environmental feature vector E.

[0102] The hidden layer consists of two fully connected layers, each followed by a ReLU activation function to capture the nonlinear relationship between environmental features and modal weights.

[0103] The output layer is a fully connected layer, outputting the fusion weights W for each modality:

[0104] W = {w RGB w Pol w Spec w Depth},in:

[0105] w RGB For RGB modal weights; w Pol For polarization mode weights; w Spec For spectral mode weights; w Depth For depth modal weights.

[0106] The weighting result calculated by MCN based on the environmental feature vector E is expressed as follows:

[0107] W=Softmax(W2·σ(W1·E+b1)+b2)

[0108] Where: W1 and W2 are trainable weight matrices; b1 and b2 are bias vectors; σ(·) is a nonlinear activation function;

[0109] Softmax normalization processing:

[0110] Where m∈{r, p, s, d} corresponds to four modes, namely RGB, polarization, spectral and depth modes.

[0111] (2) Environment-dependent factor branch

[0112] The input layer also receives the environmental feature vector E.

[0113] The hidden layer is a fully connected layer followed by a Sigmoid activation function, which is used to generate the environmental dependence factor g(E):

[0114] g(E) = Sigmoid(FC2(ReLU(FC1(E))))

[0115] where FC1, FC2 are fully connected layers, and Sigmoid limits the output to the range [0, 1]. g(E) can dynamically amplify or suppress the contribution of cross-modal consistency scores, achieving environment-driven closed-loop regulation.

[0116] The environmental dependence factor is used to dynamically scale the confidence of the cross-modal consistency constraint, adjusting the scoring threshold of the candidate target according to environmental conditions such as light, water surface reflection, and weather, thereby enhancing the robustness of the system in complex environments.

[0117] S3-2 MCN weight distribution capability training

[0118] The weight distribution capability of the MCN is obtained through supervised training, which specifically includes:

[0119] S3-2-1 Training data preparation: Collect scene sample data sets covering different water environment conditions

[0120] where E i is the environmental feature vector of scene i, is the optimal modal weight label under this scene;

[0121] The modal weight label is calculated based on the normalized performance indicators by manually evaluating the accuracy and confidence of each modal detection result in multiple typical scenes, for example:

[0122] where P m is the detection accuracy of modal m in this scene.

[0123] S3-2-2 Loss function design

[0124] During training, the difference between the MCN output weight and the labeled weight is measured by a weighted cross-entropy loss:

[0125]

[0126] S3-2-3 Optimization and iteration

[0127] The parameter is updated using the Adam optimizer, the initial learning rate is set as η0, and the learning rate is automatically decayed when the performance of the verification set does not improve; after each round of training, whether the weight distribution output by the MCN can improve the overall detection accuracy is evaluated on the verification set, and the model parameters are continuously updated until convergence.

[0128] S3-2-4 reasoning phase running mechanism

[0129] In actual monitoring tasks, the MCN receives the real-time collected environmental feature vector E t , and generates the optimal modal weight through forward reasoning.

[0130] S3-3 weight distribution logic:

[0131] Real-time acquisition of environmental features E; input MCN, output weight vector W; adaptive adjustment of weight distribution under different environments:

[0132] Strong light reflection scene: the MCN increases the polarization modal weight w Pol , and reduces the RGB weight w RGB , to reduce the interference caused by the high reflection of the water surface;

[0133] Low light / night scene: the MCN increases the weights w Spec ,w Depth of the spectral modal and the depth modal, to compensate for the insufficient brightness of RGB;

[0134] Turbid water scene: the MCN reduces the depth modal weight w Depth , and increases the proportion of RGB and polarization modal.

[0135] S3-4 output and bidirectional interaction

[0136] The output results of the MCN include:

[0137] Each modal fusion weight W: used for feature weighted fusion in step S4.

[0138] Environment-dependent factor g(E): used for candidate target confidence correction in step S5, to realize environment-driven global closed-loop adjustment.

[0139] Further, the present application introduces a bidirectional interaction mechanism: the cross-modal consistency score result calculated in S5 will be input back to the MCN as a feedback signal; the MCN updates or fine-tunes its parameters according to the feedback, so that it is more consistent with the actual detection result in the next reasoning;

[0140] For example, when a modality repeatedly shows low reliability in the consistency score, the MCN will gradually reduce the weight of that modality, and vice versa.

[0141] S4 Feature Extraction and Weighted Fusion

[0142] In step S3, the dynamic weights W = {w} of each mode are obtained. RGB ,w Pol ,w Spec ,w Depth After that, this step performs feature extraction and weighted fusion on the data of each modality to generate a fused feature map, and at the same time retains the response map of each modality for use in the consistency constraint of the subsequent step S5.

[0143] S4-1 Modal Image Preprocessing and Registration

[0144] To ensure spatial correspondence between different modal features, each modal image first undergoes geometric correction and registration. The specific steps are as follows:

[0145] 1. Distortion correction: using the camera's intrinsic K... i Distortion correction is performed on each modal image using distortion coefficients:

[0146]

[0147] 2. Spatial registration: Select a reference mode (usually RGB), and register it using the extrinsic parameter R. i , t i Project other modes onto the reference plane: p ref =Π ref (R i→ref P i +t i→ref )

[0148] Among them Π ref For the reference modal projection function, P i Let i be the pixel 3D coordinates or depth point cloud of mode i.

[0149] 3. Time Synchronization: Time interpolation or optical flow registration is performed on frames of different modalities to align each modal frame to the reference frame t0 in time. After preprocessing and registration, an aligned modal image with a uniform resolution of H×W is obtained.

[0150] S4-2 Modal Feature Extraction

[0151] Align each modal image (RGB, polarization, spectrum, depth) are respectively input into the respective feature extraction network for processing. The feature extraction network can use a conventional convolutional neural network (CNN) or a special module designed according to the specific modality. In the specification, the feature extraction network is a conventional technical means and will not be described in detail. The corresponding modality feature map Fk is obtained:

[0152] The RGB modality generates a feature map F RGB ; the polarization modality generates a feature map F Pol ; the spectrum modality generates a feature map F Spec , and the depth modality generates a feature map F Depth .

[0153] S4-3 modality weighting

[0154] The modality weights W = {w RGB , w Pol , w Spec , w Depth} output by step S3 are element-wise weighted with the respective modality feature maps:

[0155] F′ k = w k ·F k , k∈ {RGB, Pol, Spec, Depth}

[0156] wherein F′ k represents the weighted single-modality feature map. The weighted feature map is used for subsequent fusion, embodying the environment self-adaptive characteristic of the MCN.

[0157] S4-4 feature fusion

[0158] The weighted modality feature maps are subjected to multi-level fusion, specifically including:

[0159] Channel splicing: F′ k is spliced in the channel dimension to obtain a preliminary fusion feature map;

[0160] Channel compression and convolution fusion: channel compression is performed through 1×1 convolution, and then deep fusion features are extracted through a convolution block of continuous convolution + BN (batch normalization) + ReLU activation;

[0161] Multi-layer convolution fusion: cross-layer connection is introduced to ensure that modality features of different scales can be complementary and superimposed to obtain the final fusion feature map F fusion .

[0162] This process not only ensures the full use of complementary information between modalities, but also retains the dynamic adjustment characteristic driven by the environment.

[0163] S4-5 reserved modality response map

[0164] At the same time of generating the fusion feature map, the system reserves the response map of each modality:

[0165] The method is: intercept the activation map of the middle layer before the last layer of the feature extraction network, and keep consistent with the spatial size of the fusion feature map;

[0166] Noted as R RGB , R Pol , R Spee , R Depth ; the response map is used for cross-modal consistency scoring and confidence correction in step S5.

[0167] S4-6 output result

[0168] This step outputs:

[0169] The fusion feature map F fusion , for candidate target generation and detection;

[0170] The response map R k of each modality, for calling the consistency constraint mechanism.

[0171] S5: Cross-modal consistency constraint and confidence correction

[0172] After obtaining the fusion feature map F fusion and the response map R k of each modality in step S4, this step is used to generate candidate targets and correct the detection confidence through the cross-modal consistency constraint mechanism, thereby enhancing the robustness and accuracy of multi-modal detection.

[0173] S5-1 candidate target generation

[0174] Using the fusion feature F fusion , a set of candidate targets is obtained through a detection head (convolutional detection network):

[0175] B={b1, b2,..., b n}

[0176] Where each candidate b i represents a candidate target region, and is accompanied by an initial confidence

[0177] S5-2 cross-modal consistency score calculation

[0178] For each candidate target b i , the corresponding candidate region is intercepted in the response map R k of each modality:

[0179] R k (b i), k ∈ {RGB, Pol, Spec, Depth}

[0180] Calculate the similarity between each modality to get the cross-modal consistency score C i :

[0181]

[0182] Where M is the number of modalities (4 in this application); cos(·) represents the cosine similarity; C i reflects the feature consistency of the candidate target in different modalities.

[0183] S5-3 Environment-driven dynamic consistency adjustment

[0184] Unlike the prior art, the present application not only uses the MCN output modality weight, but also uses the environmental feature vector E to dynamically adjust the threshold and weight of the consistency constraint:

[0185] When the light is too strong and the water surface reflection is significant, the system increases the weight of the polarization modality in the consistency constraint;

[0186] When the water is turbid or at night, the system reduces the consistency requirement of RGB and improves the contribution of the spectral modality;

[0187] The introduction of the environmental feature E makes the threshold and fusion method of the consistency score C i be able to adaptively adjust.

[0188] The specific implementation is to combine the environmental dependence factor g(E) generated by S3 with the cross-modal consistency score:

[0189] C′ i = g(E)·C i

[0190] S5-4 Confidence correction

[0191] The final confidence is determined by the initial detection confidence and the dynamic consistency score:

[0192]

[0193] Where α, β are weight coefficients, and α+β=1; The size of β is dynamically adjusted by the environmental factor g(E).

[0194] In this way, under certain environmental conditions, the system can automatically amplify or suppress the consistency contribution of certain modalities, thereby improving the reliability of detection.

[0195] The cross-modal consistency score distribution calculated in this step will also be fed back to the modal weight control network (MCN) in step S3: when a certain modality consistently contributes low in multiple candidate targets, MCN will automatically reduce its weight in the next round of reasoning; when a certain modality consistently provides high consistency support, MCN will increase its weight.

[0196] S6 Result output

[0197] After completing the cross-modal consistency constraint and confidence correction in S5, the system performs final screening and result output on all candidate targets, including the following steps:

[0198] S6-1 Final detection result generation

[0199] According to the candidate target set B = {b1, b2,... b n} obtained in step S5 and its modified confidence The system screens targets that meet the set threshold:

[0200]

[0201] Where θ is the detection threshold, and the output target set O is the result of the water surface floating garbage target detected in this round.

[0202] S6-2 Global closed-loop feedback mechanism

[0203] At the same time as outputting the target set, the consistency score distribution {C′ i} and detection performance (such as false positive rate, false negative rate) of each candidate target are fed back to MCN:

[0204] When a certain modality has low consistency score in multiple targets, MCN reduces its fusion weight in the next round;

[0205] When a certain modality consistently provides stable support under environmental conditions, MCN increases its fusion weight in the next round.

[0206] The feedback path forms the following closed loop: environmental feature driving → modal weight adjustment → fusion feature generation → consistency constraint → confidence correction → result output → global dynamic optimization of environment-weight update.

[0207] S6-3 Dynamic adaptive update

[0208] In continuous monitoring tasks, MCN will periodically update internal parameters, enabling the system to: adapt to changes in environmental conditions such as light, water quality, and weather over the long term; automatically correct the dependency relationship between modalities to avoid detection instability caused by failure of a single modality.

[0209] Through dynamic closed-loop linkage of output results and environmental characteristics, the system has self-learning and self-adaptive characteristics;

[0210] Compared with the one-time weighting fusion in the prior art, the application can continuously optimize the modal weight and consistency constraint during task execution, and significantly improves the robustness, real-time performance and accuracy of water surface floating garbage detection.

[0211] Although embodiments of the application have been shown and described, it is to be understood that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the application, the scope of which is defined by the appended claims and their equivalents.

Claims

1.A method for monitoring water surface floating garbage based on a UAV, characterized in that, The method comprises the following steps: S1 data acquisition: synchronously collecting RGB images, polarization images, spectral images, depth images and environment data of the water surface area by a multi-modal sensor carried by a UAV to generate a multi-modal data set D; S2 environment feature vector generation: extracting illumination features, water quality features, reflection features and weather features from the environment data and performing normalization processing to obtain an environment feature vector E; S3 modal weight dynamic adjustment: inputting the environment feature vector E into a modal weight control subnetwork MCN to output fusion weights W of each modality, and mapping the environment feature vector E through the MCN to generate g(E) for dynamically adjusting subsequent consistency constraints; S4 feature extraction and weighted fusion: input the preprocessed RGB image, polarization image, spectral image and depth image obtained in the S1 step into a feature extraction network to obtain corresponding modal feature maps Fk; weight and fuse the modal feature maps according to the fusion weight W to generate a fusion feature map F fusion , and meanwhile reserve the response maps R of each modality k ​ S5 cross-modal consistency constraint and confidence correction: generating a candidate target based on the fusion feature map and attaching an initial confidence; calculating a consistency score C of the candidate target in the modal response map i and dynamically scaling the consistency score based on the environment-dependent factor g(E) to obtain a modified score Ci'; weighting and combining the modified score with the initial detection confidence to obtain a final confidence; S6 result output: outputting the candidate target with a final confidence greater than a threshold as a water surface floating garbage detection result. 2.The unmanned aerial vehicle based water surface floating garbage monitoring method according to claim 1, characterized in that, The network structure of the MCN in S3 is composed of a modal weight branch and an environment dependent factor branch. 3.The unmanned aerial vehicle based water surface floating garbage monitoring method according to claim 2, characterized in that, The modal weight branch comprises: an input layer for receiving the environment feature vector E; a hidden layer which is a two-layer fully connected layer, each layer being connected with a ReLU activation function at the rear, for capturing the nonlinear relationship between the environment features and the modal weights; an output layer which is a fully connected layer, outputting the fusion weights W of each modality, and the expression is: W=Softmax(W2σ(W1·E+b1)+b2) wherein W1 and W2 are trainable weight matrices; b1 and b2 are bias vectors; and σ(·) is a nonlinear activation function. 4.The unmanned aerial vehicle based water surface floating garbage monitoring method according to claim 2, characterized in that, The environment dependent factor branch comprises: an input layer for receiving the environment feature vector E; a hidden layer which is a fully connected layer, connected with a Sigmoid activation function at the rear, for generating the environment dependent factor g(E): g(E)=Sigmoid(FC2(ReLU(FC1(E)))) wherein FC1 and FC2 are fully connected layers, and Sigmoid limits the output to the range of [0, 1]. 5.The unmanned aerial vehicle based water surface floating garbage monitoring method according to claim 1, wherein, The pre-processing of the images of each modality in S4 comprises distortion correction, spatial registration and time synchronization. 6.The unmanned aerial vehicle based water surface floating garbage monitoring method according to claim 1, wherein, The fusion feature map F in the S4 fusion The generating comprises: a. The images of each modality are input into a feature extraction network for processing to obtain a feature map set Fk; b. The modal weight W = {w RGB , w Pol , w Spec , w Depth} is element-wise weighted with each modal feature map to obtain a single-modal feature map F′ k ; c. The weighted feature maps F' of each modality are fused to obtain a fused feature map F k fusion .​ 7.The unmanned aerial vehicle based water surface floating garbage monitoring method according to claim 1, wherein, The fusion processing comprises: channel splicing: splicing the feature maps of each modality in the channel dimension to obtain a preliminary fusion feature map; channel compression and convolution fusion: compressing the channels through one-dimensional convolution, and extracting deep fusion features by using continuous convolution, batch normalization and an activation function; Multi-layer convolution fusion: through cross-layer connection, the modal features of different scales are complementary superimposed to obtain the final fusion feature map F fusion . 8.The unmanned aerial vehicle based water surface floating garbage monitoring method of claim 1, wherein, The expression of the cross-modal consistency score in S5 is: where M is the number of modalities; cos(·) denotes cosine similarity; C i reflects the consistency of the features of the candidate target in different modalities. 9.The unmanned aerial vehicle based water surface floating garbage monitoring method of claim 1, wherein, The expression of the final confidence in the step S5 is: wherein α and β are weight coefficients, and α+β=1; the size of β is dynamically adjusted by the environment factor g(E). 10.A system for performing the unmanned aerial vehicle-based water surface floating garbage monitoring method of any one of claims 1-9, characterized in that, It comprises: a UAV platform for carrying multi-modal sensors and completing coverage flight on a target area of a water area under the control of a preset flight route; an RGB camera installed on the UAV platform for acquiring visible light image information of the water surface area; Polarization imager: The polarization imager is installed on the UAV platform for collecting water surface reflection and polarization characteristic information. By analyzing the optical response of different polarization angles, the high-reflectivity garbage target can be significantly distinguished from the background water or spray, thereby reducing the false detection under a single RGB mode. Spectral imager: The spectral imager is installed on the UAV platform for obtaining spectral information of water quality characteristics and color distribution; Depth camera: The depth camera is installed on the UAV platform for obtaining three-dimensional structure information and depth data of the water surface; Environmental sensor: The environmental sensor is installed on the UAV platform for collecting environmental parameters such as illumination intensity, color temperature, turbidity, reflectivity, and weather category; Cloud processor: The cloud processor establishes a data connection with the UAV platform through wireless communication, and is used for integrated processing and storage of the multi-modal data uploaded by the UAV.

Images