Water surface floating object detection method based on RGB-D fusion and state space network

Abstract

The application discloses a water surface floating object detection method based on RGB-D fusion and state space network, and relates to the technical field of target detection. The method comprises the following steps: acquiring an original color image and an original depth image of a water surface monitoring area; performing numerical dynamic range correction and environmental noise confidence weighting preprocessing on the original depth image to obtain an effective depth image; splicing the original color image and the effective depth image to construct a multi-channel input tensor; inputting the tensor into a pre-trained water surface floating object detection model, utilizing a depth gradient driven selective state space mechanism to perform cross-modal context feature modeling and extract multi-scale features; and outputting a category and a positioning result based on the multi-scale features. The application can effectively suppress the interference of water surface reflection and dynamic ripples, improve the perception accuracy of complex environment targets under a moving perspective, and has low overall computational complexity, so that real-time deployment on edge devices is easy to realize.

Description

Technical Field

[0001] This invention relates to the field of water surface target detection technology, and mainly to a method for detecting floating objects on the water surface. Background Technology

[0002] Dynamic monitoring of floating objects on water surfaces is a crucial technical means to ensure the safety of waterway facilities and navigation. With the increasing prevalence of unmanned surface vessels (USVs), deploying visual perception systems on mobile platforms for autonomous inspection has become a current development trend. Compared to fixed-viewpoint monitoring, this mobile sensing method can expand the physical coverage of monitoring and provide real-time dynamic visual feedback in complex and ever-changing water environments. It has key technical value for improving the continuous tracking and spatial positioning capabilities of surface targets.

[0003] Current visual detection solutions largely rely on extracting single visible light image features and performing target recognition within a conventional convolutional neural network framework. In practical deployments, these perception models, which depend solely on two-dimensional textures, are easily affected by ambient light. On one hand, the inherent specular reflection and high-frequency ripples of water surfaces can distort local image features, causing confusion between the physical edges of floating objects and the dynamic water background. On the other hand, from the perspective of a continuously moving unmanned vessel, the image exhibits drastic scale changes and background blurring. Existing networks, when processing image streams accompanied by large-scale dynamic background changes, are limited by a single local receptive field or quadratic global computational complexity, making it difficult to achieve stable output that balances real-time performance and environmental resilience on resource-constrained edge computing devices.

[0004] In summary, existing methods for water surface target detection still lack robustness and environmental adaptability under complex physical conditions. Especially under highly reflective and dynamic water surface background interference, the model's ability to distinguish the geometric boundaries of targets with varying scales from a moving perspective needs improvement, easily leading to feature confusion and false positives / false negatives. Therefore, it is necessary to investigate a method to improve the robustness of mobile sensing platforms in data processing and the stability of feature extraction in complex water surface scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a method for detecting floating objects on the water surface based on RGB-D fusion and state space networks, in order to solve the aforementioned problems in the prior art.

[0006] Technical solution: A method for detecting floating objects on the water surface based on RGB-D fusion and state-space networks, comprising: Acquire raw color and raw depth images of the water surface monitoring area; Numerical dynamic range correction and environmental noise confidence weighted preprocessing are performed on the original depth image to obtain an effective depth map; The original color image and the effective depth map are concatenated by channel to construct a multi-channel input tensor; The multi-channel input tensor is input into a pre-trained floating object detection model, and cross-modal context feature modeling is performed using a depth gradient-driven selective state space mechanism to extract multi-scale features of the water surface monitoring area. Based on the multi-scale features, the category and bounding box location results of floating objects in the water surface monitoring area are output.

[0007] Beneficial effects: To address the issues of depth measurement failure caused by high water surface reflectivity and high-frequency noise interference introduced by dynamic ripples, a pixel-by-pixel depth confidence evaluation mechanism based on physical characteristics is constructed and weighted fusion is performed with the original image. This effectively suppresses specular reflection blind spots and geometric artifacts, and improves the true three-dimensional spatial measurement of the water surface.

[0008] To address the challenge of overall depth distribution drift caused by continuous hull movement under a moving perception perspective, a frame-level adaptive temporal smoothing correction mechanism is introduced to dynamically iterate the normalized baseline. This enables the system to smoothly track the macroscopic shift in the field of view depth and reduces the loss of target absolute distance information.

[0009] Furthermore, to address the bottleneck of visual feature confusion between variable water backgrounds and floating objects at spatial edges, a continuous depth gradient at the physical geometry level is extracted as a control driver. A spatially variable decay gating factor is generated to rigidly intervene in the temporal transmission process of the state space network. This effectively blocks the cross-domain penetration of erroneous background contexts at depth abrupt boundary, reducing the false detection rate of targets. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the method for detecting floating objects on the water surface provided in the embodiments of this application.

[0011] Figure 2 This application provides a schematic diagram of the numerical dynamic range correction and environmental noise confidence weighted preprocessing process.

[0012] Figure 3 This is a schematic diagram illustrating the process of pre-expanding the number of first-layer convolutional channels of the water surface floating object detection model provided in this application embodiment to a multi-channel dimension.

[0013] Figure 4 This is a schematic diagram of the process for extracting multi-scale features of the water surface monitoring area provided in an embodiment of this application. Detailed Implementation

[0014] Combination Figure 1Description of Example 1: This example provides a method for detecting floating objects on the water surface based on RGB-D fusion and state-space networks, mainly including the following steps: Step 101: Obtain the original color image and original depth image of the water surface monitoring area; Specifically, the original color image and original depth image of the water surface monitoring area are acquired in real time and synchronously by environmental sensing equipment mounted on the unmanned surface vessel. Based on this, the environmental sensing equipment transmits the acquired dual-modal images in real time to the edge computing unit inside the unmanned surface vessel for subsequent processing.

[0015] In this embodiment, the edge computing unit can be implemented using a pre-built high-performance microcomputing platform, and an accelerated inference computing engine, such as TensorRT or ONNX Runtime, a conventional model inference acceleration tool in the art, can be deployed on it to ensure the low latency and high frame rate requirements of water surface dynamic monitoring.

[0016] In order to obtain color and depth information that is absolutely aligned in physical space, the environmental perception device uses an integrated binocular structured light depth camera to directly output modal data.

[0017] In some alternative implementations, as an alternative to a binocular structured light depth camera, the environmental perception device may also consist of a separate color camera coupled with an external laser time-of-flight ranging radar array.

[0018] Furthermore, the system dynamically projects the spatial three-dimensional point cloud obtained by radar scanning onto the two-dimensional imaging coordinate system of the color camera through an offline pre-calibrated extrinsic parameter matrix. After densification processing, it also obtains the original depth image with spatiotemporal consistency.

[0019] Step 102: Perform numerical dynamic range correction and environmental noise confidence weighted preprocessing on the original depth image to obtain an effective depth map; Specifically, due to the continuous large-scale regional light variations, sunlight specular reflection, and dynamic ripples in the real water surface environment, the raw depth images directly output by the sensor are mixed with a large number of numerical missing blind spots and high-frequency fluctuating geometric noise.

[0020] Meanwhile, the camera pose of the unmanned surface vessel changes constantly with the undulations of the waves during its surface cruise, causing a drastic overall shift in the absolute depth distribution of the entire image. Based on the aforementioned physical interference, the edge computing unit uses statistical algorithms to dynamically correct the global extreme value distribution pattern in the original depth image, so that the drifting depth values ​​can be stably mapped to a fixed standard dynamic range.

[0021] By extracting the spatial continuity and reflectivity physical features of depth values, a pixel-by-pixel soft weight mask operation is applied to the corrected depth map, i.e., environmental noise confidence-weighted preprocessing is performed. After this two-stage preprocessing filtering, an effective depth map unaffected by specular reflections and surface ripples is finally separated, thus providing reliable spatial geometric measurement information for subsequent networks.

[0022] Step 103: Perform channel stitching between the original color image and the effective depth map to construct a multi-channel input tensor; Furthermore, in order to achieve early fusion of visual two-dimensional appearance information and three-dimensional depth geometric information in the initial stage of network forward propagation, the edge computing unit directly superimposes and merges the pre-extracted original color image containing three color information channels with the effective depth map of a single channel dimension in the channel dimension.

[0023] The specific stitching logic involves combining channels in a specific order to construct a four-channel 3D data block tensor. This multi-channel input tensor structure not only perfectly preserves the rich representation of the texture attributes and color details of the floating objects on the water surface in the original color image, but also simultaneously embeds the target's relative physical distance and 3D geometric contour boundary information provided by the effective depth map.

[0024] In some optional implementations, the multi-channel input tensor is formatted as a matrix structure with three mutually orthogonal dimensions: height, width, and number of channels, using C++. in The tensor layout format is *H*W. In this data layout, C... in H represents the total number of input channels, while H and W represent the absolute spatial pixel resolution of the image in the vertical and horizontal directions of the water surface monitoring area, respectively.

[0025] Step 104: Input the multi-channel input tensor into the pre-trained water surface floating object detection model, and use the depth gradient-driven selective state space mechanism to perform cross-modal context feature modeling to extract multi-scale features of the water surface monitoring area. In this step, the pre-trained floating object detection model serves as the central hub for feature extraction and logical reasoning, and is deployed and runs in the high-speed memory of the aforementioned edge computing unit. The backbone network of this floating object detection model abandons the traditional method of expanding the perception range by relying on pure local convolutional kernel stacking, and introduces a state-space scanning topology structure with global linear time complexity.

[0026] Specifically, cross-modal context feature modeling using a depth gradient-driven selective state-space mechanism means that when extracting features from the current level, the model does not indiscriminately diffuse contextual information to the surrounding neighborhood. Instead, it calculates the depth gradient physical quantity between adjacent pixels in real time from the depth physical dimension carried by the multi-channel input tensor. Then, based on the relative intensity of this numerical gradient, it dynamically blocks or allows the temporal transmission process of the feature state at that moment.

[0027] By utilizing a cross-modal hard intervention mechanism that guides the network's perception attention through physical depth boundaries, it effectively isolates the reflection of water surface background features from penetrating and contaminating the target area of ​​floating objects. As the fused features are downsampled and converged layer by layer in the feedforward network path, the network ultimately extracts corresponding multi-scale features in parallel on multiple feature map nodes of different scales, thus meeting the dual perception needs of large obstacles and small floating debris in the water.

[0028] Step 105: Based on the multi-scale features, output the category and bounding box location results of floating objects in the water surface monitoring area.

[0029] Specifically, the detection head network at the back end of the floating object detection model is responsible for receiving and parsing the aforementioned multi-scale features, and performing corresponding numerical decoding operations through parallel classification prediction branches and bounding box regression prediction branches.

[0030] The decoded category parameters are used to indicate that the floating object on the water surface is identified as a specific preset object type. At the same time, the bounding box localization result is output as a pure numerical combination of two-dimensional spatial center coordinates, absolute width, and height to provide the target's precise physical spatial location in the two-dimensional monitoring screen.

[0031] Finally, the edge computing unit packages the above category information and bounding box coordinate parameters into a standard communication data stream and outputs it outward, providing high-precision dynamic environmental perception basis for the subsequent autonomous driving obstacle avoidance strategy of the unmanned vessel equipped with the system or the intelligent robotic arm salvage trajectory planning.

[0032] Example 2: In order to solve the problem of overall drift and dynamic range loss of water surface depth distribution caused by the continuous movement of the unmanned vessel and the dynamic change of the viewpoint in Example 1, this example provides an adaptive lossless depth normalization pipeline.

[0033] In one implementation, numerical dynamic range correction includes the following steps: like Figure 2 As shown, in step 201, an adaptive normalization operation is performed on the original depth image to obtain a normalized depth map; An adaptive normalization operation is performed on the original depth image to map the absolute physical depth values ​​to a standardized numerical range that is easily processed by the neural network. Considering the differences in boundary conditions between offline construction and online runtime of deep learning models, this operation executes corresponding processing logic based on the different stages of the model's operation.

[0034] In the first case, during the model training phase, static truncation normalization is performed based on the global upper limit quantile and global lower limit quantile of the pre-stored global depth sample statistics to obtain the normalized depth map.

[0035] Specifically, the system reads all valid depth pixel values ​​greater than zero from the entire multimodal water surface floating object training set, and uses these as extreme value reference points for the statistical overall calculation distribution.

[0036] To eliminate outlier noise, a pre-stored global lower bound quantile is used as the lower bound reference for the effective depth, and a pre-stored global upper bound quantile is used as the upper bound reference. Based on these statistical quantiles, a truncation-normalization mapping operation is performed. The calculation is performed using the following formula: D norm =clip((D raw -P min ) / (P max -P min ),0,1)*255; Among them, D norm To normalize the pixel values ​​of each pixel in the depth map, `clip` is a truncation function that restricts the values ​​to upper and lower bounds. raw P represents the physical depth value of each pixel in the original depth image. min Let P be the global lower bound quantile. max This represents the global upper limit quantile.

[0037] In the second scenario, during the model inference stage, a frame-level adaptive temporal smoothing correction is introduced to update the current quantile benchmark for the original depth image in the currently acquired video frame, and normalization is performed based on the updated current quantile benchmark to obtain the normalized depth map.

[0038] Specifically, when an unmanned surface vessel carrying a detection model is performing a patrol mission on the water surface, the movement of the vessel from near the shore to deep water will cause an overall shift in the depth distribution within the field of view.

[0039] At this point, the system extracts the current statistical quantile of the current original depth image. Then, using a pre-configured exponential moving average algorithm, the smoothed quantile of the historical video frame output from the previous video frame through this step is weighted and fused with the current statistical quantile to obtain an updated current quantile benchmark, so that the current quantile benchmark adaptively tracks the overall distribution shift of the water surface depth.

[0040] The specific calculation formula for the exponential moving average algorithm is as follows: P * t = μP * t-1 +(1-μ) P t ; Among them, P * t The updated current quantile benchmark, μ is the pre-configured smoothing coefficient, and P * t-1 P is the smooth quantile of historical video frames. t This is the current statistical quantile extracted from the current original depth image.

[0041] When applying the exponential moving average algorithm described above, independent weighted fusion calculations are performed for the lower and upper reference quantiles. The pre-configured smoothing coefficient is set to a range of 0.90 to 0.99, and the selected smoothing coefficient is set to 0.95. This coefficient value can maintain the stability of the depth reference values ​​between consecutive frames while avoiding a sluggish response of the system to depth changes.

[0042] In one embodiment, it is assumed that the global lower bound quantile of the normalized baseline obtained through offline statistics is 0.1, and the global upper bound quantile is 0.9. During the model training phase, if the obtained physical depth value is 0.5, substituting it into the truncation normalization formula, the normalized median value is calculated to be (0.5-0.1) / (0.9-0.1)=0.5.

[0043] In the model inference stage, assuming the smoothed lower quantile of historical video frames is 0.12, and the current video frame's pose change causes the extracted current lower quantile to jump to 0.22, with the pre-configured smoothing coefficient set to 0.9, the updated current lower quantile baseline is 0.9*0.12 + 0.1*0.22 = 0.13.

[0044] The calculation process shows that the exponential moving average algorithm buffers the normalization baseline jump caused by single-frame jitter and maintains the numerical stability of the depth input tensor.

[0045] Example 3: Considering that the specular reflection of sunlight on the water surface will cause the sensor's output depth value to be 0, and that high-frequency ripples on the water surface will introduce geometric undulation noise, in some scenarios, this example is executed in parallel with Example 2 to refine the depth preprocessing process and provide a pixel-by-pixel depth confidence weighting mechanism.

[0046] In one implementation, the following steps are included: Step 301: Calculate the pixel-by-pixel depth confidence map based on the spatial geometric metric information features of the original depth image; In this step, the reliability of the data is inferred using the physical geometric properties inherent in the original depth image. For noise interference regions with different physical causes, the system divides the calculation process into parallel physical feature extraction branches, and then performs the following operation: combining the validity factor and the smoothness factor to generate the depth confidence map.

[0047] In the first physical feature extraction branch, for each pixel in the original depth image, an effectiveness factor is calculated to filter out invalid areas of water surface specular reflection.

[0048] Specifically, the validity factor is calculated by exponential mapping based on the ratio of the original depth value of the pixel to the pre-configured effective distance threshold, so that the confidence of the region with a depth value of zero approaches the minimum.

[0049] Mirror reflections from the water surface cause depth measurements to fail and return a value of 0. A nonlinear suppression function is constructed using exponential mapping.

[0050] In the formula, the effectiveness factor c valid =1-exp(-D raw / d th ); Where exp is the natural exponential function, and D raw The original depth value of the pixel, in mm, d th The pre-configured effective distance threshold.

[0051] Those skilled in the art will know that other methods can also be used, such as D raw / D ref D ref For reference only.

[0052] The pre-configured effective distance threshold is set to a range of 30 to 80, with the physical unit being mm, and the selected value is set to 50. Based on the above calculation logic, when the original depth value of a pixel is 0, the natural exponent term equals 1, and the resulting validity factor equals 0; when the original depth value of a pixel is greater than the pre-configured effective distance threshold, the natural exponent term approaches 0, and the resulting validity factor approaches 1.

[0053] In the second physical feature extraction branch, depth fluctuation features within the neighborhood of each pixel in the original depth image are extracted, and a smoothness factor is calculated to suppress high-frequency noise in water surface ripples.

[0054] Specifically, the local depth variance within a preset neighborhood window centered on the pixel is calculated. Then, the ratio of this local depth variance to the square of a pre-configured depth smoothing scale parameter is used.

[0055] The smoothness factor is calculated using exponential decay mapping to suppress the confidence level of regions with drastic depth fluctuations. High-frequency fluctuations caused by water surface ripples manifest as a discrete distribution of depth values ​​within a local spatial range. The specific dimensions of the preset neighborhood window are set to a 5×5 square spatial array. The fluctuation measure is obtained by calculating the statistical variance of the pixel depth values ​​within this spatial array.

[0056] Specifically, the smoothness factor c smooth =exp(-V ω / (η 2 )); In the formula, exp is the natural exponential function, and V ω The local depth variance is the average of the sum of squares of the differences between the depth values ​​and the mean, expressed in mm. 2 η is a pre-configured depth smoothing scale parameter, in mm.

[0057] The pre-configured depth smoothing scale parameter is set to a range of 300 to 600, with the physical unit set to millimeters and the value set to 500.

[0058] Based on the above calculation logic, when the local depth variance is small, indicating that the physical surface is flat, the smoothness factor approaches 1; when the local depth variance is greater than the square of the pre-configured depth smoothing scale parameter, the smoothness factor decays non-linearly and approaches 0.

[0059] After completing the calculations of the above two branches, the obtained validity factor and smoothness factor are multiplied pixel by pixel to obtain the comprehensive confidence scalar of each pixel. The value of this scalar is between 0 and 1. The comprehensive confidence scalar of all pixels together constitutes the depth confidence map.

[0060] Step 302: Perform pixel-wise weighting on the depth confidence map and the normalized depth map to obtain the effective depth map.

[0061] After obtaining the depth confidence map characterizing the spatial reliability of the data and the normalized depth map after numerical interval standardization, the two are combined numerically with spatial coordinate alignment.

[0062] Specifically, D eff =c*D norm ; In the formula, D eff Let c be the depth value of the corresponding pixel in the effective depth map, and D be the overall confidence scalar in the depth confidence map.norm These are the original normalized pixel values ​​in the normalized depth map.

[0063] Using the above numerical combination method, the original spatial geometric measurement of depth is retained in areas where the sensor's ranging performance is stable, while its value is attenuated to near 0 in unreliable areas where reflection is missing or fluctuates drastically.

[0064] In some alternative implementations, as an alternative to the above-described mechanism of calculating depth confidence maps by extracting physical features, pixel interpolation repair of reflective regions containing 0 values ​​can be performed by introducing a pre-trained depth data reconstruction network.

[0065] Specifically, the original depth image features and the color image features corresponding to the spatial location are extracted, and the two sets of features are jointly input into the depth data reconstruction network. The multi-layer convolution operator is used to infer the fitted depth value of the invalid region, and the fitted depth value is used to replace the original data, thereby outputting an effective depth map.

[0066] Example 4: After obtaining a noise-free effective depth map using Examples 2 and 3, in order to organically integrate depth geometric spatial information with color visual information and solve the problem of easy missed detection of small targets under moving perspective, this example refines and limits the channel stitching and network input topology structure in Example 1.

[0067] In one possible implementation, the following steps are included: Step 401: In order to adapt to the multi-channel input tensor, the number of first-layer convolutional input channels of the water surface floating object detection model is pre-expanded to a dimension that matches the number of channels of the multi-channel input tensor. The weight parameters in the first convolutional layer corresponding to the effective depth map dimension are initialized by averaging the pre-stored color pre-trained weight parameters corresponding to the original color image dimension, so as to provide an initial numerical response for the depth channel based on the inherited image pre-training knowledge.

[0068] Specifically, for a multi-channel input tensor containing three color channels (red, green, and blue) and a single-dimensional depth channel, the receiving dimension at the entry point of the feature extraction network needs to be adjusted.

[0069] During model construction, the input channel dimension of the first convolutional kernel, originally designed to receive three-channel images, was modified to 4. When loading the initial network parameters, for the first three channels, the weight matrix corresponding to the color dimension in the pre-stored image pre-trained model was directly inherited and loaded to reuse the optimized filter parameters of the visual dataset to extract basic texture features.

[0070] For a newly added depth channel, the system extracts the weight parameters of the first three channels and performs an arithmetic mean operation on the channel dimension, using the result as the initial weight of that depth channel.

[0071] Specifically, the following calculation formula is used: W depth =(1 / 3)*(W R +W G +W B ); In the formula, W depth W represents the weight parameters in the first convolutional layer that correspond to the effective depth map dimension. R For the pre-stored color pre-trained weight parameters, W represents the weight parameters corresponding to the red channel dimension. G For the pre-stored color pre-trained weight parameters, W represents the weight parameters corresponding to the green channel dimension. B The weight parameters corresponding to the blue channel dimension in the pre-stored color pre-trained weight parameters.

[0072] By using this arithmetic average initialization operation, the depth channel can obtain a numerical response benchmark that matches the scale of the color features in the early stages of training, which solves the problem of inconsistent multimodal gradient propagation amplitude and network convergence hindrance caused by random distribution initialization.

[0073] Step 402: At the output end of the shallow downsampling stage of the backbone network of the water surface floating object detection model, a high-resolution feature extraction branch is added to extract shallow high-resolution features for small floating objects on the water surface. Specifically, for tiny floating objects such as broken plastic and small twigs that are widely distributed in the water environment and occupy a small pixel area, multiple downsampling operations will cause irreversible loss of their fine-grained spatial information.

[0074] In this embodiment, network topology splitting is implemented at the computing nodes after the first downsampling operation in the backbone network. At this stage, the system pulls out an independent signal bypass branch to output a feature matrix with a feature map size equivalent to 1 / 4 of the original resolution of the multi-channel input tensor.

[0075] This high-resolution feature extraction branch is configured with 32 feature channels to preserve sufficient geometric edges and underlying texture details. Adding this branch enables the network to capture the physical features of small-scale targets under the system constraint of introducing 0.15M additional network parameters and increasing computation time by 15%.

[0076] Step 403: Extract deep multi-scale features using the deep stages of the backbone network; In this step, the input data stream continues to be passed layer by layer and downsampled and compressed within the backbone network. The deeper stages of the backbone network output corresponding hierarchical feature maps at three computation nodes with resolutions equivalent to 1 / 8, 1 / 16, and 1 / 32 of the original input tensor, respectively.

[0077] The three feature maps described above, with decreasing spatial dimensions but increasing receptive fields, together constitute deep multi-scale features. This process relies on high-level abstract semantic features generated after multi-layer nonlinear activation and dimensionality reduction compression calculations to characterize the main structural information of obstacles with large physical dimensions in the water surface monitoring area.

[0078] Step 404: Perform path aggregation and fusion processing on the shallow high-resolution features and the deep multi-scale features to obtain the final extracted multi-scale features.

[0079] Furthermore, in order to enable the final feature set used for detection to simultaneously possess high-resolution detailed localization features and semantic recognition features of deep networks, the system activates path aggregation network nodes at multiple feature scales.

[0080] The fusion process includes upsampling feature interpolation and lateral concatenation operations from deep to shallow, as well as downsampling path enhancement calculations from shallow to deep. The shallow high-resolution features are connected to the bottom-level fusion node of the path aggregation network through the feature concatenation operator, so that the abstract semantic features contained in the deep multi-scale features are numerically coupled with the fine-grained positional features retained by the shallow branches.

[0081] After path aggregation and fusion processing, the multi-level feature set outputs multi-scale features, which are then fed into the subsequent detection branch structure to perform bounding box regression and category inference operations.

[0082] In some alternative implementations, as an alternative to the above-described method of initializing depth channel weights using an arithmetic mean, a physical variance-weighted average algorithm can be used.

[0083] Specifically, the local variance data of the pre-trained weights for each of the three channels (red, green, and blue) are extracted separately. The summation weight factor for the corresponding channel is set based on the reciprocal of the local variance value.

[0084] The weighted sum is used as the weight parameter corresponding to the effective depth map dimension in the first convolution, so that the color channel features with richer texture details have a higher mapping weight in the depth initialization calculation.

[0085] Example 5: In order to build upon the topology constructed in Example 4 and to solve the problem of numerical confusion between floating object features and water surface background features at the geometric spatial boundary during feature extraction, this example refines the algorithm mechanism inside the feature extraction backbone network in Example 1.

[0086] In one implementation, the following steps are included: like Figure 3 As shown, in step 501, the current scale feature map inside the network layer is obtained, and the effective depth map is spatially downsampled to the same resolution as the current scale feature map to obtain a scaled depth map. Specifically, during the multi-level extraction process in the backbone network, the spatial resolution of the input tensor decreases layer by layer.

[0087] To ensure that physical depth information and high-dimensional semantic features remain pixel-aligned in the spatial dimension, the system extracts the effective depth map output by the previous implementation and performs spatial downsampling processing on it using a pre-configured average pooling operator.

[0088] The downsampling scaling parameter strictly corresponds to the downsampling factor of the current scale feature map, thereby generating a scaled depth map with consistent spatial dimensions, which serves as a preliminary physical reference for controlling feature flow.

[0089] Step 502: Extract the depth gradient between adjacent pixels on the scaled depth map along the preset scanning direction; In order to capture edge abrupt changes in physical structures in the water surface environment, the system uses differential operations to process the scaled depth map.

[0090] The preset scanning direction is defined as the extension path of feature serialization. The system sequentially extracts the depth measure values ​​of spatially adjacent pixels along a specific geometric direction and calculates the numerical difference between the two as the depth gradient. This depth gradient characterizes the continuity or breakage of the current physical surface in three-dimensional space.

[0091] Step 503: Based on the relative change magnitude of the depth gradient, generate a spatially variable decay gating factor to characterize the physical geometric boundary; In this step, the extracted depth gradient values ​​are converted into continuous scalars that control the network's perceptual weights. To achieve numerical suppression of abrupt depth changes, an exponential mapping operation is used to construct a nonlinear function.

[0092] The specific calculation is performed using the following formula: α ij =exp(-(▽ (D_ij) 2 ) / (2*τ s 2 )); Where, α ij Let ∠exp be a spatially variable decay gate factor, and ∠exp be a natural exponential function. (D_ij) Let τ be the depth gradient at the current pixel. s The initial values ​​of the pre-configured scale temperature parameters can be determined by those skilled in the art through validation set experiments based on the numerical resolution of the depth map.

[0093] By setting this pre-configured scale temperature parameter, the system can control its sensitivity to depth fluctuations. When the unmanned vessel cruises to the boundary between the floating object and the water surface, the depth gradient relative to τ... s The parameters increase significantly, causing the gating factor to approach zero, thereby blocking the propagation of background features.

[0094] Step 504: Spatial processing is performed on the current scale feature map using a pair of asymmetric depthwise separable convolutions of preset size to extract local visual features; To acquire sufficient local neighborhood information before performing global cross-modal perception, the system constructs parallel local feature extraction branches. Independent one-dimensional spatial convolutional kernels are used for sliding filtering in the horizontal and vertical dimensions.

[0095] Local visual features z local =DWConv 1x7 (u)+DWConv 7x1 (u); In the formula, z local For local visual features, DWConv 1x7 DWConv is a depthwise separable convolution operator with a kernel size spatial dimension of 1 row and 7 columns. 7x1 is a depthwise separable convolution operator with a kernel size spatial dimension of 7 rows and 1 column, and u is the input feature tensor after mapping the feature map at the current scale.

[0096] By limiting the dimensionality of asymmetric depthwise separable convolutions, the system controls the computational overhead of floating-point operations while preserving the local receptive field coverage within the range of 14 to 28 original pixels.

[0097] Step 505: Apply the spatially variable decay gating factor to the state recursion process of the state space network, and perform dynamic weighted feature extraction on the current scale feature map to truncate the propagation of the feature context state at the deep abrupt change boundary to obtain cross-modal context features.

[0098] In this step, the system breaks away from the traditional state-space model's mechanism of relying solely on the input features for state transitions, and introduces cross-modal physical depth features into the temporal control process.

[0099] The specific update rules for the state recursion process are calculated using the following formula: hij =α ij *h (ij_minus_1) +X ij ; Among them, h ij The hidden state after the current pixel is updated, α ij h is a spatially variable decay gate factor. (ij_minus_1) For the preceding hidden state, X ij This represents the input feature corresponding to the current pixel in the current scale feature map.

[0100] To maintain numerical stability during the recursive process, the system synchronously calculates normalized tracking parameters.

[0101] The specific calculation is performed using the following formula: N ij =α ij *N (ij_minus_1) +1; Where, N ij α is the normalization coefficient after the current pixel is updated. ij N is a spatially variable decay gate factor. (ij_minus_1) The preceding normalization coefficient is denoted as .

[0102] Finally, a one-way normalized output feature is obtained by performing a division operation.

[0103] The specific calculation is performed using the following formula: h (hat_ij) =h ij / N ij ; Among them, h (hat_ij) h is the unidirectional normalized feature output for the current pixel. ij N represents the hidden state after the current pixel is updated. ij This is the normalization coefficient after the current pixel is updated.

[0104] To cope with the complex physical characteristics of the water surface environment, the system sets up independent execution branches for different environmental conditions within this step.

[0105] In the first operating condition, in each of the preset scanning directions, when the depth gradient at the current pixel exceeds the smooth scale represented by the pre-configured scale-temperature parameter, it is determined that there is an object boundary. The spatial variable decay gating factor corresponding to the current pixel is decayed to near zero through exponential decay mapping, so as to block the cumulative propagation of the previous hidden state to the current pixel.

[0106] When the depth gradient at the current pixel is characterized as a continuous surface, the spatially variable decay gating factor is kept close to its maximum value to allow lossless transfer of the preceding hidden state and ensure that the flow of feature context within the continuous physical surface is not interfered with.

[0107] For example, in a certain scenario, suppose the depth gradient at the current pixel is 1.5, and the pre-configured scale temperature parameter is 0.5. Substituting into the exponential mapping formula for the gate factor, the numerator of the exponential term is the calculated value -2.25, and the denominator is the calculated value 0.5, resulting in a spatially variable decay gate factor of exp(-4.5) equal to 0.011. This factor's value approaches 0, forcing the preceding hidden state to be assigned this extremely small weight, thereby preventing water surface background features from crossing the physical boundary and contaminating the floating object's body features. When the magnitude of the depth gradient at the current pixel is relatively large compared to the scale temperature parameter, the spatially variable decay gate factor approaches zero after exponential decay mapping, thus blocking the cumulative propagation of the preceding hidden state to the current pixel.

[0108] In the second scenario, when the depth gradient at the current pixel represents a continuous surface, the spatially variable decay gate factor is kept close to its maximum value to allow lossless propagation of the preceding hidden state. In this scenario, the algorithm exhibits rigorous degradation consistency. When the scaled depth map is uniform, i.e., the depth gradient is always 0, the decay gate factor is always equal to 1. At this point, the aforementioned linear formula for state updates degenerates into an indiscriminate numerical accumulation; that is, the updated hidden state at the current pixel is equivalent to the arithmetic sum of the preceding hidden state and the current input feature, maintaining mathematical equivalence to the mechanism of conventional state-space networks.

[0109] Step 506: The features obtained through the state recursion process in the four preset scanning directions are fused to output the final cross-modal context features.

[0110] Because floating objects are randomly distributed on the water surface, in order to obtain omnidirectional features without spatial bias, the preset scanning direction is configured to include both horizontal and vertical forward and reverse directions. The system performs an arithmetic mean operation to integrate the unidirectional normalized feature outputs generated in the four independent directions.

[0111] The specific calculation is performed using the following formula: S = 0.25 * (h) (hat_lr) +h (hat_rl) +h (hat_tb) +h (hat_bt) ); Where S represents the cross-modal context feature, h (hat_lr) For the feature output from left to right, h (hat_rl) For the feature output from right to left, h (hat_tb) For top-down feature output, h (hat_bt) This is the feature output from the bottom up direction.

[0112] Step 507: After obtaining the cross-modal context features, the local visual features and the cross-modal context features are activated and fused to obtain the output features of the current network layer, so as to simultaneously obtain the local texture details and global scene dependencies of the floating objects on the water surface.

[0113] The system uses a pre-configured nonlinear activation function to perform numerical threshold truncation on the fused features after numerical superposition, and performs Hadamard product operation on the activated features and the modulation signal controlled by the gating mechanism.

[0114] The specific calculation is performed using the following formula: Y = Conv (1x1) (σ(z local +S)*g); Where Y is the output feature tensor after fusion processing, and Conv 1x1 This is a linear convolution operation that uses a 1x1 kernel size for channel mapping, where σ is a pre-configured non-linear activation function, and z... local S represents local visual features, S represents cross-modal context features, and g represents a modulation gating signal generated within the network layer. This signal is generated by performing linear projection and nonlinear activation operations on the input of the current scale feature map and is used to control the selective passage of output features.

[0115] Finally, the system sums the fused output feature tensor with the original input data of the current layer of the network through the residual connection path to ensure the flow of backpropagation gradients and complete the comprehensive calculation of semantic and geometric features at specific scales.

[0116] Example 6: The foregoing examples all pertain to the floating object detection process during online operation of unmanned vessels. In order to enable the floating object detection model to have the above-mentioned feature extraction and cross-modal modeling capabilities, this example provides a two-stage decoupled localization enhancement training method in an offline state.

[0117] The pre-trained floating object detection model is pre-built using a two-stage decoupled localization enhancement training method. In one embodiment, the pre-building process includes the following steps: like Figure 4 As shown, in step 601, a multimodal training set of floating objects containing annotation information is obtained; Specifically, the system reads pre-acquired pairs of color and depth images on an offline computer device and combines them into multimodal sample data. Each set of multimodal sample data is associated with corresponding annotation information, which records the category identifiers of known floating objects on the water surface in the image and their corresponding two-dimensional planar bounding box coordinates in a preset text format.

[0118] The multimodal floating debris training set can be obtained through on-site data acquisition in real water environments using a data acquisition platform equipped with RGB-D sensors, or by acquiring and expanding depth modal information from publicly available water surface debris target detection datasets. The specific scale and category distribution of the training data can be adjusted according to the environmental characteristics of the actual deployment water area. This multimodal floating debris training set serves as the data foundation for the subsequent two training stages and is loaded into the physical memory of the computer device for iterative access.

[0119] Step 602: In the first training phase, based on the multimodal floating object training set, global feature learning is performed on the initial detection network to obtain the first phase network weights. In the initial stage of model training, the optimization objective focuses on enabling the model to learn global, universal semantic features of multiple types of floating objects on the water surface. The spatial resolution of the input image is set as the base extraction size, with a constraint range of 512 x 512 to 720 x 720, and 640 x 640 in a typical case.

[0120] Simultaneously, a complete random perturbation algorithm, including color space data augmentation and spatial geometric data augmentation, is activated at this stage to expand the distribution diversity of the data samples. When calculating the gradient during backpropagation, a combination of standard loss for object detection is used. Specifically, the calculation is performed using the following formula: L total =γ box *L box +γ cls *L cls +γ dfl *L dfl ; Among them, L total Let γ be the total loss function. box For the bounding box regression loss weights, L box For bounding box regression loss, γ cls For the classification loss weights, L cls For classification loss, γ dfl For the distribution focus loss weights, L dfl This represents the distribution focus loss.

[0121] During training, those skilled in the art can use conventional optimization algorithms (such as stochastic gradient descent or adaptive moment estimation algorithms) to update parameters. Hyperparameters during training (such as learning rate, batch size, and number of iterations) can be determined by conventional hyperparameter tuning methods.

[0122] After a preset number of iterations and optimizations, the parameters generated by the network on the validation set are recorded, and the output is the first-stage network weights.

[0123] In some embodiments, during the first training phase, the system adopts a basic loss weight distribution that takes into account both classification and bounding box regression. After a preset number of iterations, the system records the parameters generated by the network on the validation set and outputs the first-stage network weights.

[0124] Step 603: In the second training phase, the network weights of the first phase are loaded to improve the spatial resolution of the input images of the multimodal floating object training set. Under the conditions of constrained color space perturbation and increased bounding box localization penalty bias, localization enhancement fine-tuning is performed to obtain the pre-trained floating object detection model.

[0125] To address the distortion of floating objects caused by changes in perspective in real-world mobile scenarios, the network needs to have anti-interference positioning capabilities for blurred edges.

[0126] The system reads and loads the network weights from the first stage, using them as the initial parameter benchmark for the second training stage. Subsequently, the system forcibly increases the data channel dimension of the read data, raising the spatial resolution of the input images of the multimodal floating object training set to the range of 800x800 to 1024x1024, typically set to 960x960, to provide richer physical pixel coordinate nodes for fine-tuning calculations.

[0127] After the input resolution is increased, localization enhancement fine-tuning is performed to update the model parameters until the loss function reaches the convergence condition.

[0128] Step 604: In the second training phase, all color space data augmentation operations for the multimodal water surface floating object training set are turned off to maintain the numerical consistency of depth image spatial geometric measure information. Specifically, conventional color space perturbation algorithms apply random offsets to the hue, saturation, and brightness of input data. In the case of multi-channel feature coupling, if the aforementioned perturbation affects the depth channel, it will disrupt the fixed mapping ratio between physical distance and pixel value.

[0129] In order to enable the optimizer to return to the true visual spatial distribution during the fine-tuning stage and to maintain the spatial geometric measurement information of depth features unchanged, the system disables the corresponding perturbation subroutine call interface by configuring control words, and only retains the scaling, translation and flipping operations of the image, thereby avoiding the unpredictable interference of the enhancement algorithm on the physical coordinate values.

[0130] Step 605: In the second training phase, relative to the loss weight distribution configured in the first training phase, the bounding box regression loss weight is increased and the classification loss weight is decreased simultaneously, so that the network optimization process focuses on learning spatial localization information.

[0131] After disabling color space data augmentation, the system resets the optimizer's hyperparameter configuration. The bounding box regression loss weight in the linear formula above is increased from 7.5 in the first training phase to 10.0, while the classification loss weight is decreased from 0.5 to 0.3. By increasing the influence coefficient of spatial distance error in the total gradient direction and simultaneously suppressing the proportion of classification error, the network's computational resources are forced to tilt towards calculating the Gaussian intersection-union ratio of the bounding boxes.

[0132] After this fine-tuning operation, the bounding box regression parameters output by the network under the set intersection-union ratio overlap threshold are closer to the real physical contour annotation. As a result, the focus loss weights of the pre-trained floating object detection model remain unchanged in the second training stage.

[0133] In some alternative implementations, a dynamic weight scheduling strategy can be used instead of rigid phased training as an alternative to the two-stage decoupled localization enhancement training. Specifically, a linear decay operator that increases with each iteration batch is configured. During model training, this linear decay operator is used to calculate and reduce the trigger probability of color space data enhancement operations in real time, round by round. Simultaneously, a logarithmic growth operator that increases with each iteration batch is configured. This operator drives the continuous up-adjustment of the bounding box regression loss weights, thereby achieving a continuous transition from global feature learning to localization enhancement fine-tuning.

[0134] Example 7: Further explanation of the unmanned vessel system action coordination process after outputting the detection results.

[0135] In one implementation, the following steps are included: Step 701: The category and bounding box positioning results are used as dynamic environment perception parameters and sent in real time to the main control system of the unmanned vessel equipped with a perception device for acquiring the original color image and the original depth image.

[0136] Specifically, after acquiring and decoding the multi-scale features of floating objects on the water surface, the edge computing unit generates structured data for the corresponding target.

[0137] The categories are quantized using pre-assigned integer identifiers, and the bounding box localization results are quantized using the target center point coordinates and relative aspect ratio in the image pixel coordinate system. Specifically, the two-dimensional spatial coordinates and dimensions are represented in the format (x, y, w, h). To enable data interaction across the underlying hardware of devices, the edge computing unit packages and reassembles the above discrete quantized data to construct a standard data frame conforming to the Controller Area Network Bus protocol or the Universal Asynchronous Receiver / Transmitter serial communication protocol.

[0138] Subsequently, the constructed standard data frames are continuously transmitted to the main control system as dynamic environment perception parameters via physical cables inside the unmanned vessel. The main control system can be implemented using a microcontroller unit or a programmable logic controller, and its hardware circuitry is used to receive and analyze the dynamic environment perception parameters and maintain the kinematic state of the unmanned vessel's chassis.

[0139] Step 702: The category and bounding box positioning results are used as dynamic environment perception parameters and sent in real time to the main control system of the unmanned vessel equipped with an environment perception device for acquiring the original color image and original depth image. This triggers the main control system to perform autonomous path planning and obstacle avoidance actions for large floating objects based on the category and bounding box positioning results, or to guide automated cleaning equipment to salvage the target floating object.

[0140] Specifically, the determination is as follows: when the target area corresponding to the bounding box positioning result is greater than the preset safe passage area threshold, an autonomous path planning and obstacle avoidance action is executed for large floating objects; when the target area is not greater than the safe passage area threshold, automated cleaning equipment is guided to salvage the target floating object.

[0141] In this step, the main control system executes nonlinear conditional branch control commands based on the parsed category and bounding box positioning results, thereby converting the software output from the vision algorithm side into control actions from the ship's hardware side.

[0142] In the first execution path, if the category indicates that the current target is an obstacle affecting navigation safety, or if the physically calculated area corresponding to the bounding box positioning result is greater than the pre-configured safe passage area threshold, the main control system is triggered to perform autonomous path planning and obstacle avoidance actions for large floating objects. Specifically, the main control system extracts the center point coordinates (x, y) of the bounding box and converts them into polar coordinates relative to the heading of the unmanned vessel based on the camera imaging model intrinsic parameters. Simultaneously, combined with the effective depth map obtained in Example 3, the depth value corresponding to the pixel position of the center point is extracted to calculate the physical distance. Based on the extracted relative angle and physical distance, the main control system updates the local obstacle avoidance trajectory point sequence and outputs a control voltage to adjust the speed difference between the left and right thrusters of the unmanned vessel, controlling the hull to generate a yaw torque to bypass the obstacle, thereby maintaining the physical integrity of the equipment hardware.

[0143] In the second execution path, if the category indicates that the current target is surface debris to be cleaned and the calculated physical area is less than the pre-configured safe passage area threshold, the main control system is triggered to guide the automated cleaning equipment to salvage the target floating object. Specifically, the main control system calculates the vector approximation route between the current position of the unmanned vessel and the surface debris, and outputs a control signal to reduce the absolute speed of the dual thrusters to perform a deceleration approach operation. When the relative distance measured by combining the effective depth map is reduced to within the pre-configured salvage action trigger threshold range, the main control system sends a high-level enable signal to the automated cleaning equipment mounted on the front of the vessel. The automated cleaning equipment can be implemented using a multi-degree-of-freedom salvage robotic arm or a conveyor belt-type collection net. After receiving the enable signal, the automated cleaning equipment starts its internal closed-loop servo control program, fine-tunes the spatial pose of the robotic arm end effector or the collection net based on the dynamically updated bounding box positioning results, and completes the physical capture and recovery of the surface debris.

[0144] In this application, the addition of high-resolution feature branches at the lower level in the early stage, and the decoupling fine-tuning under the conditions of constrained color perturbation and increased positioning bias in the later stage, force the model to focus on the absolute correction of spatial position in the iteration, and achieve high recall and strict bounding box fit for multi-scale and even small debris targets under complex dynamic water surface conditions.

Claims

1. A method for detecting floating objects on the water surface based on RGB-D fusion and state-space networks, characterized in that, include: Acquire raw color and raw depth images of the water surface monitoring area; Numerical dynamic range correction and environmental noise confidence-weighted preprocessing are performed on the original depth image to obtain an effective depth map; The original color image and the effective depth map are concatenated by channel to construct a multi-channel input tensor; The multi-channel input tensor is input into a pre-trained floating object detection model, and cross-modal context feature modeling is performed using a depth gradient-driven selective state space mechanism to extract multi-scale features of the water surface monitoring area. Based on the multi-scale features, the category and bounding box location results of floating objects in the water surface monitoring area are output.

2. The method according to claim 1, characterized in that, Numerical dynamic range correction and environmental noise confidence-weighted preprocessing are performed on the original depth image to obtain an effective depth map, including: An adaptive normalization operation is performed on the original depth image to obtain a normalized depth map; Based on the spatial geometric metric information features of the original depth image, a pixel-by-pixel depth confidence map is calculated; The effective depth map is obtained by weighting the depth confidence map and the normalized depth map pixel by pixel.

3. The method according to claim 2, characterized in that, An adaptive normalization operation is performed on the original depth image to obtain a normalized depth map, including: During the model training phase, static truncation normalization is performed based on the pre-stored global upper and lower bound quantiles of global depth samples. During the model inference phase, for the original depth image in the currently acquired video frame, a frame-level adaptive temporal smoothing correction is introduced to update the current quantile benchmark, and normalization is performed based on the updated current quantile benchmark to obtain the normalized depth map.

4. The method according to claim 2, characterized in that, Based on the spatial geometric metric information features of the original depth image, a pixel-by-pixel depth confidence map is calculated, including: For each pixel in the original depth image, calculate the effectiveness factor used to filter out invalid areas of specular reflection from the water surface; Extract the depth fluctuation features in the neighborhood of each pixel in the original depth image, and calculate the smoothness factor used to suppress high-frequency noise of water surface ripples; The depth confidence map is generated by combining the validity factor and the smoothness factor.

5. The method according to claim 1, characterized in that, The extraction of multi-scale features of the water surface monitoring area includes: In the shallow downsampling stage output of the backbone network of the water surface floating object detection model, a high-resolution feature extraction branch is added to extract shallow high-resolution features for small floating objects on the water surface. Deep multi-scale features are extracted using the deep stages of the backbone network; The shallow high-resolution features and the deep multi-scale features are subjected to path aggregation and fusion processing to obtain the final extracted multi-scale features.

6. The method according to claim 1, characterized in that, The method of using a depth gradient-driven selective state-space mechanism for cross-modal context feature modeling includes: Obtain the current scale feature map within the network layer, and the scaled depth map corresponding to the current scale feature map; Along a preset scanning direction, extract the depth gradient between adjacent pixels on the scaled depth map; Based on the relative change magnitude of the depth gradient, a spatially variable decay gating factor is generated to characterize the physical geometric boundary. The spatially variable decay gating factor is applied to the state recursion process of the state space network to dynamically weight the feature map at the current scale, thereby truncating the propagation of the feature context state at the boundary of deep abrupt change and obtaining cross-modal context features.

7. The method according to claim 6, characterized in that, The preset scanning directions include four directions: horizontal forward and reverse, and vertical forward and reverse. Accordingly, applying the spatially variable decay gating factor to the state recursion process of the state-space network includes: In each of the preset scanning directions, when the depth gradient at the current pixel exceeds the pre-configured scale temperature parameter, it is determined that there is an object boundary, and the spatial variable decay gating factor corresponding to the current pixel is decayed to close to zero, so as to block the cumulative propagation of the previous hidden state to the current pixel. When the depth gradient at the current pixel is characterized as a continuous surface, the spatially variable decay gating factor is kept close to its maximum value to allow lossless transfer of the preceding hidden state; The features obtained through the state recursion process in the four preset scanning directions are fused to output the final cross-modal context features.

8. The method according to claim 1, characterized in that, The pre-trained floating object detection model is pre-built using a two-stage decoupled localization enhancement training method. The pre-building process includes: Obtain a multimodal training set of floating objects on the water surface that includes labeled information; In the first training phase, based on the multimodal floating object training set, global feature learning is performed on the initial detection network to obtain the first-stage network weights; In the second training phase, the network weights from the first phase are loaded to improve the spatial resolution of the input images for the multimodal floating object training set. Under the conditions of constrained color space perturbation and increased bounding box localization penalty bias, localization enhancement fine-tuning is performed to obtain the pre-trained floating object detection model.

9. The method according to claim 8, characterized in that, Positioning enhancement fine-tuning is performed under constraints of color space perturbation and increased bounding box positioning penalty offset, including: In the second training phase, all color space data augmentation operations for the multimodal floating object training set are turned off to maintain the numerical consistency of depth image spatial geometric measure information. Compared to the loss weight distribution in the first training phase, the bounding box regression loss weight is increased in the second training phase, while the classification loss weight is decreased simultaneously, so that the network optimization process focuses on learning spatial localization information.

10. The method according to claim 7, characterized in that, While applying the spatially variable decay gating factor to the state recursion process of the state-space network, the method also includes: By using a pair of asymmetric depthwise separable convolutions of preset sizes, spatial processing is performed on the current scale feature map to extract local visual features; After obtaining the cross-modal context features, the local visual features and the cross-modal context features are activated and fused to simultaneously obtain the local texture details and global scene dependencies of floating objects on the water surface.

Images