River and lake aquatic plant salvage system based on image recognition and intelligent robot

Abstract

The invention discloses a river and lake aquatic plant salvage system based on image recognition and an intelligent robot, and the system comprises an image preprocessing module which outputs an enhanced multi-view aquatic plant image; the improved YOLOv8n network module outputs a multi-view aquatic plant image feature tensor; the aquatic plant feature corresponding module obtains a multi-view aquatic plant feature corresponding relation; the distortion correction module is used for obtaining distortionless aquatic plant characteristic coordinates; the target feature module outputs a unified aquatic plant target feature map; the middle positioning module outputs a middle positioning result; and the control module is used for performing error threshold filtering and positioning confidence evaluation on the intermediate positioning result to generate a final positioning result, and performing aquatic plant fishing by adopting an intelligent machine according to the final positioning result. According to the invention, high-precision and high-robustness three-dimensional position information support can be provided for intelligent aquatic plant fishing equipment.

Description

Technical Field

[0001] This invention relates to the field of aquatic plant harvesting technology, and in particular to a river and lake aquatic plant harvesting system and intelligent robot based on image recognition. Background Technology

[0002] With the increasing demand for ecological protection and smart water management of rivers and lakes, the monitoring and spatial positioning of aquatic plant distribution is of great significance in applications such as ecological restoration of rivers and lakes, water treatment and automated dredging. Due to the optical characteristics of river and lake water and the complexity of the operation scenario, multi-view underwater image fusion and computer vision methods are currently used to perform three-dimensional positioning of aquatic plants in rivers and lakes.

[0003] In aquatic environments, light undergoes strong refraction and scattering at the water surface and medium interface, resulting in image color distortion, blurred details, and severe noise interference. Existing image preprocessing methods are mostly based on simple color equalization and filtering algorithms, which fail to differentiate and enhance the weak contrast and complex texture features between the slender leaves of aquatic plants and the background water, making it difficult to provide a robust foundation for subsequent feature extraction and matching.

[0004] For feature extraction and fusion of multi-view images of aquatic plants in rivers and lakes, existing methods often directly adopt a general deep neural network architecture. However, they fail to take into account the spatial self-similarity, flexible deformation, and longitudinal texture distribution characteristics of aquatic plant structures, resulting in limited feature representation capabilities, low cross-view feature matching accuracy, and affecting positioning accuracy.

[0005] Regarding underwater imaging refraction distortion correction, most traditional methods are based on approximate single-viewpoint imaging models or ignore multi-medium refraction effects. They fail to fully utilize the correspondence of multi-viewpoint features to construct physically consistent refraction geometric constraints, resulting in large refraction correction errors and affecting the accuracy of 3D positioning. Summary of the Invention

[0006] One objective of this invention is to propose a river and lake aquatic weed harvesting system and intelligent robot based on image recognition. This invention can provide high-precision and highly robust three-dimensional position information support for intelligent aquatic weed harvesting equipment.

[0007] A river and lake aquatic plant harvesting system based on image recognition according to an embodiment of the present invention includes:

[0008] The image preprocessing module acquires and preprocesses the original multi-view images of the target aquatic plants in the river and lake water, and outputs enhanced multi-view images of the aquatic plants.

[0009] An improved YOLOv8n network module was developed. The enhanced multi-view aquatic plant image was input into the feature extraction network, which is based on the improved YOLOv8n network containing the Ghost convolutional backbone network and the GSConv neck network. The feature extraction model was constructed and the multi-view aquatic plant image feature tensor was output.

[0010] The aquatic plant feature correspondence module uses the SF module to group and fuse the local aquatic plant structural features and global spatial distribution features of the feature tensor of multi-view aquatic plant images, and uses the FAU module to perform feature alignment and matching based on the differences in aquatic plant feature distribution under different views to obtain the correspondence of aquatic plant features from multiple perspectives.

[0011] The distortion correction module uses the correspondence of aquatic plant features from multiple perspectives to construct refractive geometric constraints, performs refractive distortion correction on the correspondence of aquatic plant features from multiple perspectives, and obtains the coordinates of aquatic plant features without distortion.

[0012] The target feature module inputs the distortion-free aquatic plant feature coordinates and matching confidence into the TFE module to extract the mid-layer structural features and high-layer semantic features of the aquatic plants. It then uses a dynamic weighted fusion strategy to assign weights to the aquatic plant features from each perspective based on the matching confidence and image quality evaluation indicators, and outputs a unified aquatic plant target feature map.

[0013] The intermediate localization module constructs a three-dimensional localization model of aquatic plants based on a unified aquatic plant target feature map, refraction correction parameters, and refraction geometric constraints. It optimizes the prediction of the aquatic plant target bounding box using an improved WIoUv3 loss function and iteratively solves the three-dimensional spatial coordinates of the aquatic plant target using the sparse Levberg–Marquardt algorithm, outputting the intermediate localization results.

[0014] The control module performs error threshold filtering and location reliability assessment on the intermediate positioning results to generate the final positioning result. Based on the final positioning result, an intelligent machine is used to remove aquatic plants.

[0015] Optionally, the image preprocessing module includes:

[0016] Obtain original multi-view images of target aquatic plants in rivers and lakes;

[0017] Calculate the brightness component based on the original multi-view aquatic plant images;

[0018] Multi-scale Retinex processing is performed on the luminance component to construct a scale set. Each scale parameter corresponds to a Gaussian filter scale, and a weight value is assigned to each Gaussian filter scale to calculate the multi-scale enhanced luminance component.

[0019] Based on the multi-scale enhanced brightness component, the red, green and blue channels of the original multi-view aquatic plant image are color corrected respectively. Channel color correction coefficients are introduced to obtain the image enhanced by the improved ProMSRCR.

[0020] For the image enhanced by the improved ProMSRCR, luminance equalization mapping is performed on the pixel intensity of each channel to obtain the luminance equalized image.

[0021] Based on the multi-scale enhancement of brightness components, a water plant texture preservation mask is constructed. The brightness-equalized image is then filtered according to the water plant texture preservation mask to obtain the filtered output image. The filtered output image is then used as the enhanced multi-view water plant image corresponding to the vth viewpoint.

[0022] Optionally, the improved YOLOv8n network module includes:

[0023] The enhanced multi-view aquatic plant images are input into the feature extraction model constructed by the improved YOLOv8n network. The resolution of the enhanced multi-view aquatic plant images is scale-aligned and normalized to obtain the input feature tensor.

[0024] In the backbone network of the improved YOLOv8n network for constructing a feature extraction model, Ghost convolution is used to extract features from the input feature tensor layer by layer. In each backbone layer, basic aquatic plant morphology features are generated through main convolution. The basic aquatic plant morphology features are expanded using the cheap linear mapping operator. The basic aquatic plant morphology features and the expanded feature results are concatenated along the channel dimension to form the output feature tensor of the backbone layer.

[0025] Based on the feature tensors output from different levels of the backbone network, a multi-scale feature set is constructed according to spatial resolution from high to low.

[0026] In the neck network of the improved YOLOv8n network for constructing a feature extraction model, the GSConv module is used to perform channel-wise convolution, channel concatenation, and channel rearrangement on the multi-scale feature set, and output the fused feature tensor of each scale.

[0027] The feature tensors of each scale are fused together in a preset order to form a multi-view aquatic plant image feature tensor.

[0028] Optionally, the aquatic plant feature corresponding module includes:

[0029] Each scale-fused feature tensor in the feature tensor of each multi-view aquatic plant image is mapped to a unified scale index set according to a preset scale alignment rule to construct an aquatic plant feature tensor at a unified scale.

[0030] The aquatic plant feature tensor under a uniform scale is divided into K groups of sub-feature tensors according to the channel dimension. The SF module performs a global self-attention fusion operation on each group of sub-feature tensors to obtain the SF fused aquatic plant feature tensor under each view.

[0031] Based on the SF-fused aquatic plant feature tensor, the FAU module is used to align and match the aquatic plant features between different perspectives. For any two perspectives, the feature response values ​​of the SF-fused aquatic plant feature tensor on the preset spatial grid are normalized to obtain the feature distribution representation of perspective v and the feature distribution representation of perspective q.

[0032] Based on the feature distribution representations of arbitrary viewpoint v and viewpoint q, the distribution difference between the feature distribution representations is quantified by the Earthhover distance, and the feature distribution difference between viewpoint v and viewpoint q is defined as the distribution distance.

[0033] A matching confidence score is generated based on the distribution distance. Based on the matching confidence score, a set of multi-view aquatic plant feature correspondences is established between each viewpoint to obtain multi-view aquatic plant feature correspondences with matching confidence scores.

[0034] Optionally, the distortion correction module includes:

[0035] Based on a simplified refraction camera model, refraction geometric constraints are constructed, and a refraction correction parameter vector is introduced. The pixel coordinates of any viewpoint v are extended into a three-dimensional homogeneous pixel vector. The inverse transformation and normalization are performed using the camera intrinsic parameter matrix to obtain the normalized imaging direction vector. The normalized imaging direction vector is then subjected to refraction correction to obtain the refraction-corrected unit exit direction. The refraction-corrected unit exit direction at viewpoint q is obtained in the same way.

[0036] By utilizing refractive geometric constraints, a convex optimization solution model is constructed to solve for the refractive correction parameter vector;

[0037] Based on the refraction correction parameter vector obtained by convex optimization, the pixel coordinates in the multi-view aquatic plant feature correspondence set are corrected for refraction distortion to generate distortion-free aquatic plant feature coordinates.

[0038] Optionally, the target feature module includes:

[0039] In the feature tensor of multi-view aquatic plant images, a mid-level feature tensor is selected to represent the mid-level structural features of aquatic plants, and a high-level feature tensor is selected to represent the high-level semantic features of aquatic plants.

[0040] The distortion-free aquatic plant feature coordinates are mapped to the feature coordinate systems of the middle layer feature tensor and the upper layer feature tensor. The same sampling rule is used to sample the features of the middle layer feature tensor and the upper layer feature tensor at the positions corresponding to the distortion-free aquatic plant feature coordinates, so as to obtain the middle layer structure sampling feature vector and the upper layer semantic sampling feature vector respectively.

[0041] The mid-level structural sampling feature vector and the high-level semantic sampling feature vector are input into the TFE module. Through feature enhancement and semantic consistency integration, the enhanced feature vector of the m-th group of aquatic plant targets under the v-th view is obtained.

[0042] For the v-th viewpoint, the matching confidence scores related to all other viewpoints are counted and averaged to obtain the viewpoint confidence convergence value corresponding to the v-th viewpoint, and the image quality evaluation index corresponding to the v-th viewpoint is obtained at the same time.

[0043] Based on the view confidence convergence of the vth viewpoint and the image quality evaluation index, the dynamic weighted fusion weight corresponding to the vth viewpoint is generated. The enhanced feature vectors of aquatic plants from all viewpoints are weighted and fused according to their respective dynamic weighted fusion weights to obtain the mth group of unified aquatic plant target feature vectors. All groups of unified aquatic plant target feature vectors are backfilled into the unified feature plane according to the spatial index relationship of the distortion-free aquatic plant feature coordinates to generate a unified aquatic plant target feature map.

[0044] Optionally, the intermediate positioning module includes:

[0045] A prediction model for the bounding box of aquatic plants is constructed based on a unified aquatic plant target feature map;

[0046] The improved WIoUv3 loss function was used to optimize the aquatic plant target bounding box prediction model, and the optimized aquatic plant target bounding box prediction results were obtained.

[0047] Based on the optimized prediction results of the aquatic plant target bounding box, the pixel coordinates of the aquatic plant feature coordinates that fall within the coverage area of ​​the prediction results of the aquatic plant target bounding box are selected from the distortion-free aquatic plant feature coordinates and combined into a set of distortion-free aquatic plant feature coordinates.

[0048] Based on the refraction correction parameter vector, refraction geometric constraints, and the set of distortion-free aquatic plant feature coordinates, a three-dimensional spatial coordinate model of aquatic plants is constructed to obtain the three-dimensional spatial coordinates of aquatic plants.

[0049] S65. Using the weighted sum of the point-line consistency residuals under all views as the reprojection consistency cost function, the sparse Levenberg-Marquardt algorithm is used to iteratively solve the three-dimensional spatial coordinates of aquatic plants. In each iteration, the reprojection consistency cost function is incrementally updated based on the sparse Jacobian matrix, and the intermediate positioning result of the current iteration step is output.

[0050] Optionally, the control module includes:

[0051] Error assessment is performed on the intermediate positioning results. The point-line consistency residuals corresponding to each intermediate positioning result under each view are summarized, and the weighted average of the point-line consistency residuals is used as the comprehensive positioning error index of the intermediate positioning results.

[0052] The overall positioning error index is compared with a preset positioning error threshold, and intermediate positioning results are filtered based on the comparison results.

[0053] When the overall positioning error index is not greater than the positioning error threshold, the corresponding intermediate positioning result is determined to be a valid positioning result;

[0054] When the overall positioning error index exceeds the positioning error threshold, the corresponding intermediate positioning results will be judged as invalid positioning results and discarded.

[0055] The three-dimensional spatial coordinates of aquatic plants that are determined to be valid positioning results are evaluated for positioning reliability, and corresponding positioning reliability parameters are generated.

[0056] Based on the range of values ​​for the location reliability parameter, the valid location results are classified and labeled to form the final location result set;

[0057] The high-confidence positioning results and the medium-confidence positioning results are combined to form the three-dimensional spatial coordinates of the aquatic plants and their corresponding positioning confidence parameters to form the final positioning result, and the final positioning result is output.

[0058] Based on the final positioning result, the three-dimensional spatial coordinates of the aquatic plants are input into the intelligent machine control system as target location information. According to the three-dimensional spatial coordinates of the aquatic plants and the positioning confidence parameters, the corresponding aquatic plant retrieval operation command is generated, and the intelligent machine is controlled to perform the aquatic plant retrieval operation under a unified water body spatial coordinate system.

[0059] Optionally, the final location result set includes:

[0060] High-confidence positioning results are achieved when the positioning confidence parameter is greater than or equal to the preset high-confidence threshold, and the comprehensive positioning error index is at the lower limit of the preset error range.

[0061] The mid-confidence positioning result is where the positioning confidence parameter is less than the preset high confidence threshold and greater than or equal to the preset low confidence threshold.

[0062] Low-confidence localization results are obtained when the localization confidence parameter is less than the preset low-confidence threshold.

[0063] An intelligent robot includes: an intelligent robot body, a power module, a camera, a lidar, a memory, and a processor disposed on the intelligent robot body;

[0064] The memory is used to store programs;

[0065] The processor is used to execute the program to implement the various steps of a river and lake aquatic plant harvesting system based on image recognition.

[0066] The beneficial effects of this invention are:

[0067] This invention employs an improved ProMSRCR algorithm in the multi-view image preprocessing stage, combined with multi-scale Retinex enhancement and adaptive texture mask-guided filtering strategies, to achieve differentiated processing of the slender structure of aquatic plant leaves and water noise. It can adaptively identify and enhance the edge and longitudinal texture features of aquatic plants, effectively suppress the noise of suspended particles in the water, and make the aquatic plant structure and morphology information obtained by the feature extraction network more complete and realistic.

[0068] This invention proposes to simulate the local rotation and deformation perturbation of aquatic plants using Ghost convolution, enhance the channel-level transfer of longitudinal texture and pyramid feature fusion using GSConv structure, and combine it with SF-FAU module to achieve adaptive alignment of feature distribution and confidence modeling under multi-view conditions.

[0069] This invention employs refractive geometric constraints and convex optimization methods to achieve joint refractive correction of aquatic plant features from multiple perspectives in multi-media environments. It also minimizes reprojection errors using the sparse Levenberg-Marquardt algorithm, ultimately outputting the 3D spatial coordinates of the aquatic plants, including confidence assessments. Compared to existing methods that directly utilize the PINHOLE model or single-viewpoint approximation for positioning, this significantly reduces spatial errors caused by refractive distortion, viewpoint variations, and local occlusion. The 3D positioning accuracy of aquatic plant targets is improved to the millimeter level. Low-confidence error filtering and dynamic weighting strategies make the final output more adaptable to engineering applications and more reliable for real-time operations, providing high-precision and robust 3D positional information support for intelligent aquatic plant harvesting equipment. Attached Figure Description

[0070] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0071] Figure 1 This is a flowchart of a river and lake aquatic weed harvesting system and intelligent robot based on image recognition proposed in this invention. Detailed Implementation

[0072] Example 1: Reference Figure 1 A river and lake aquatic plant harvesting system based on image recognition includes:

[0073] The image preprocessing module acquires and preprocesses the original multi-view images of the target aquatic plants in the river and lake water, and outputs enhanced multi-view images of the aquatic plants.

[0074] An improved YOLOv8n network module was developed. The enhanced multi-view aquatic plant image was input into the feature extraction network, which is based on the improved YOLOv8n network containing the Ghost convolutional backbone network and the GSConv neck network. The feature extraction model was constructed and the multi-view aquatic plant image feature tensor was output.

[0075] The aquatic plant feature correspondence module uses the SF module to group and fuse the local aquatic plant structural features and global spatial distribution features of the feature tensor of multi-view aquatic plant images, and uses the FAU module to perform feature alignment and matching based on the differences in aquatic plant feature distribution under different views to obtain the correspondence of aquatic plant features from multiple perspectives.

[0076] The distortion correction module uses the correspondence of aquatic plant features from multiple perspectives to construct refractive geometric constraints, performs refractive distortion correction on the correspondence of aquatic plant features from multiple perspectives, and obtains the coordinates of aquatic plant features without distortion.

[0077] The target feature module inputs the distortion-free aquatic plant feature coordinates and matching confidence into the TFE module to extract the mid-layer structural features and high-layer semantic features of the aquatic plants. It then uses a dynamic weighted fusion strategy to assign weights to the aquatic plant features from each perspective based on the matching confidence and image quality evaluation index, and outputs a unified aquatic plant target feature map.

[0078] The intermediate localization module constructs a three-dimensional localization model of aquatic plants based on a unified aquatic plant target feature map, refraction correction parameters, and refraction geometric constraints. It optimizes the prediction of the aquatic plant target bounding box using an improved WIoUv3 loss function and iteratively solves the three-dimensional spatial coordinates of the aquatic plant target using the sparse Levberg–Marquardt algorithm, outputting the intermediate localization results.

[0079] The control module performs error threshold filtering and location reliability assessment on the intermediate positioning results to generate the final positioning result. Based on the final positioning result, an intelligent machine is used to remove aquatic plants.

[0080] In this embodiment, the image preprocessing module includes:

[0081] Obtain original multi-view images of target aquatic plants in rivers and lakes;

[0082] The original multi-view aquatic plant image obtained from the vth viewpoint is defined as a three-channel image tensor indexed by pixel coordinates. The three-channel image tensor is composed of the pixel intensity of the red channel, the pixel intensity of the green channel, and the pixel intensity of the blue channel. Each pixel coordinate represents the pixel intensity value of the corresponding pixel in the red, green, and blue channels of the aquatic plant image.

[0083] Calculate the brightness component based on the original multi-view aquatic plant images;

[0084] The brightness component is the arithmetic mean of the pixel intensity of the red channel, the pixel intensity of the green channel, and the pixel intensity of the blue channel, used to represent the overall brightness intensity of aquatic plant images at each pixel coordinate in river and lake water environments.

[0085] Multi-scale Retinex processing is applied to the luminance component to construct a scale set consisting of several scale parameters. Each scale parameter corresponds to a Gaussian filter scale, and a weight value is assigned to each Gaussian filter scale to calculate the multi-scale enhanced luminance component.

[0086] The multi-scale enhanced luminance component is obtained by weighted summation of the logarithmic differences at each scale. The logarithmic difference is the difference between the luminance component and the logarithm of the Gaussian filter result. To avoid singularity in logarithmic operations, a positive constant is introduced to correct the denominator, and the sum of the weight values ​​of all scales is 1.

[0087] Based on the multi-scale enhanced brightness component, the red, green and blue channels of the original multi-view aquatic plant image are color corrected respectively. Channel color correction coefficients are introduced to obtain the image enhanced by the improved ProMSRCR.

[0088] In Example 1, based on the multi-scale enhanced luminance component, for each pixel coordinate (x, y), corresponding channel color correction coefficients are introduced for the pixel intensity of the red, green, and blue channels of the original multi-view aquatic plant image. The pixel intensity of each channel is divided by the luminance component and multiplied by the channel color correction coefficient and the exponential mapping result of the multi-scale enhanced luminance component to obtain the color-corrected pixel value of each channel. The three channel pixel values ​​are combined to form the image enhanced by the improved ProMSRCR.

[0089] For the image enhanced by the improved ProMSRCR, luminance equalization mapping is performed on the pixel intensity of each channel to obtain the luminance equalized image.

[0090] Based on the multi-scale enhanced brightness component, a water plant texture preservation mask is constructed. The brightness-equalized image is filtered according to the water plant texture preservation mask to obtain the filtered output image. The filtered output image is used as the enhanced multi-view water plant image corresponding to the v-th view.

[0091] In Example 1, the gradient magnitude of the multi-scale enhanced brightness component at each pixel coordinate (x, y) is calculated. The gradient magnitude is obtained by synthesizing the brightness change rate of the pixel in the horizontal and vertical directions, and is used to characterize the intensity of the seaweed edge and texture changes at the corresponding pixel. The gradient magnitude is normalized with a positive constant to prevent the denominator from being zero, thereby generating a seaweed texture preservation mask with a limited value range at each pixel coordinate (x, y). The value of the seaweed texture preservation mask is used to represent the salience of the seaweed texture structure at the corresponding pixel. A larger value indicates a more obvious change in the edge and texture of the aquatic plants at the corresponding pixel. A smaller value for the aquatic plant texture preservation mask indicates a flatter water area at the corresponding pixel. Based on the aquatic plant texture preservation mask, the standard deviation in the spatial domain and the standard deviation in the grayscale domain are adaptively modulated. In the aquatic plant edge area (where the aquatic plant texture preservation mask value is large), the standard deviation is reduced to weaken the filtering intensity. In the flat water area (where the aquatic plant texture preservation mask value is small), the standard deviation is increased to enhance the noise reduction intensity. The adaptive parameters are used to perform texture-aware bilateral filtering. The image after brightness equalization is filtered to obtain the filtered output image.

[0092] ;

[0093] in, Indicates the first From each perspective, after texture-aware bilateral filtering, at pixel coordinates Color channel That is, the filtered output image pixel values ​​of the red, green, or blue channels. Represents the normalization coefficient, indicating the coefficient at the th... A perspective, a channel Pixel coordinates At that location, all neighboring pixels The sum of weighted coefficients, For The set of pixels centered on the center This indicates that for all neighboring pixels Summation is performed, where exp is an exponential function. Indicates the current pixel With neighboring pixels The square of the distance in spatial location, The benchmark value representing the standard deviation of the spatial domain. Indicates the first Pixel coordinates from each perspective The aquatic plant texture remains masked. Indicates the first From each viewpoint, the image after brightness equalization is in pixel coordinates. Color channel pixel values, Indicates the first From each viewpoint, the image after brightness equalization is in pixel coordinates. Color channel pixel values, The reference value representing the standard deviation of the grayscale range.

[0094] In this embodiment, the YOLOv8n network module is improved, including:

[0095] The enhanced multi-view aquatic plant images are input into the feature extraction model constructed by the improved YOLOv8n network. The resolution of the enhanced multi-view aquatic plant images is scale-aligned and normalized to obtain the input feature tensor.

[0096] The enhanced multi-view aquatic plant image corresponding to the v-th viewpoint is defined as a three-channel image tensor indexed by pixel coordinates (x, y). The pixel intensity of the three channels under each pixel coordinate represents the pixel value of the red channel, green channel, and blue channel in the river and lake aquatic plant scene, respectively. The numbering range of the v-th viewpoint is from 1 to V, and V is not less than 2.

[0097] In the backbone network of the improved YOLOv8n network for constructing a feature extraction model, Ghost convolution is used to extract features from the input feature tensor layer by layer. In each backbone layer, basic aquatic plant morphology features are generated through main convolution. The basic aquatic plant morphology features are expanded using the cheap linear mapping operator. The basic aquatic plant morphology features and the expanded feature results are concatenated along the channel dimension to form the output feature tensor of the backbone layer.

[0098] In Example 1, in the backbone network of the improved YOLOv8n network for constructing the feature extraction model, Ghost convolution is used to extract features layer by layer from the input feature tensor. In each backbone layer, the input feature tensor is convolved by the main convolution to obtain basic aquatic plant morphological features, which are used to represent the main structural information of aquatic plants at the current spatial resolution. Using the basic aquatic plant morphological features as input, the cheap linear mapping operator is used to perform linear transformation on the basic aquatic plant morphological features. The cheap linear mapping operator generates extended features that correspond one-to-one with the basic aquatic plant morphological features in spatial position by performing a channel-by-channel linear mapping on the basic aquatic plant morphological features. The extended features are used to simulate the morphological perturbations caused by the rotation, bending and scale changes of aquatic plant leaves under different perspectives. The basic aquatic plant morphological features and the extended features are concatenated in the channel dimension to form the output feature tensor of the current backbone layer. The output feature tensor contains both the main structural features and the morphological perturbation features.

[0099] Based on the feature tensors output from different levels of the backbone network, a multi-scale feature set is constructed according to spatial resolution from high to low.

[0100] The feature tensors at different scales represent the edge detail features of aquatic plant leaves, the overall morphological contour features, and the semantic distinction features between aquatic plants and the background. The multi-scale feature set dynamically adjusts the number of channels during scale changes to match the morphological complexity of aquatic plants at the corresponding spatial resolution.

[0101] In the neck network of the improved YOLOv8n network for constructing a feature extraction model, the GSConv module is used to perform channel-wise convolution, channel concatenation, and channel rearrangement on the multi-scale feature set, and output the fused feature tensor of each scale.

[0102] In Example 1, the GSConv module is used to construct a feature pyramid and perform cross-scale fusion on a multi-scale feature set. For the morphological features of aquatic plants in rivers and lakes with an aspect ratio greater than a threshold and textures exhibiting longitudinal correlation along the growth direction, the GSConv module performs channel-wise convolution processing on the aquatic plant feature tensor at each scale to achieve channel compression and extract local response features while maintaining spatial resolution. After completing the channel-wise convolution processing, the GSConv module performs channel-level concatenation of the channel-wise convolution output features with the main feature channels at the corresponding scale, and performs channel rearrangement operation on the concatenated feature tensor, so that the aquatic plant longitudinal texture responses originally distributed in different channels are re-interwoven and arranged in the channel dimension, restoring the longitudinal texture correlation weakened during the channel-wise convolution process. Through the joint processing of channel-wise convolution, channel concatenation, and channel rearrangement, a fused feature tensor containing continuous texture information of slender aquatic plant structures is obtained at each scale. The fused feature tensors obtained at each scale are then arranged in order from high resolution to low resolution to form the fused feature tensors of each layer in the multi-scale feature pyramid.

[0103] The feature tensors of each scale are fused together in a preset order to form a multi-view aquatic plant image feature tensor.

[0104] The feature tensor of the multi-view aquatic plant image at the vth viewpoint is defined as a set composed of fused feature tensors at each scale. The numbering range of the vth viewpoint is 1 to V, where V is not less than 2, and L is the scale layer number and L is not less than 2. The spatial resolution of the fused feature tensor at each scale decreases sequentially as the scale layer number increases.

[0105] In this embodiment, the aquatic plant feature corresponding module includes:

[0106] Each scale-fused feature tensor in the feature tensor of each multi-view aquatic plant image is mapped to a unified scale index set according to a preset scale alignment rule to construct an aquatic plant feature tensor at a unified scale.

[0107] The aquatic plant feature tensor at a unified scale is obtained by upsampling the fused feature tensors at different scales and then splicing them in the channel dimension. It simultaneously represents the edge detail features of aquatic plant leaves, the overall morphological outline features, and the semantic distinction features between aquatic plants and the background.

[0108] The aquatic plant feature tensor under a uniform scale is divided into K groups of sub-feature tensors according to the channel dimension. The SF module performs a global self-attention fusion operation on each group of sub-feature tensors to obtain the SF fused aquatic plant feature tensor under each view.

[0109] In Example 1, the aquatic plant feature tensor at a uniform scale is divided into K groups of sub-feature tensors according to the channel dimension, where K is not less than 2. Each group of sub-feature tensors corresponds to a portion of the channel set in the aquatic plant feature tensor at a uniform scale, used to represent the local structural response and global spatial distribution response of aquatic plants in different feature subspaces. The SF module uses each group of sub-feature tensors as an independent input to perform global self-attention calculation on each group of sub-feature tensors. By calculating the correlation weight between each spatial location within the spatial dimension of the corresponding group of sub-feature tensors, and weighting and recombining the feature responses of the corresponding group of sub-feature tensors according to the correlation weight, the self-attention fusion output corresponding to the corresponding group of sub-feature tensors is obtained. The K self-attention fusion outputs obtained by the global self-attention calculation of the K groups of sub-feature tensors are spliced ​​in the channel dimension to form the SF fused aquatic plant feature tensor corresponding to each viewpoint. The SF fused aquatic plant feature tensor simultaneously contains the adaptive fusion result of the local structural features and global spatial distribution features of aquatic plants.

[0110] Based on the SF-fused aquatic plant feature tensor, the FAU module is used to align and match the aquatic plant features between different perspectives. For any two perspectives, the feature response values ​​of the SF-fused aquatic plant feature tensor on the preset spatial grid are normalized to obtain the feature distribution representation of perspective v and the feature distribution representation of perspective q.

[0111] In Example 1, based on the SF-fused aquatic plant feature tensor, the FAU module is used to align and match the aquatic plant features between different perspectives. The FAU module takes the SF-fused aquatic plant feature tensor of any two perspectives v and q as input, and performs spatial expansion, feature normalization and distribution modeling on the SF-fused aquatic plant feature tensor to obtain the feature distribution representation of the corresponding perspective.

[0112] Specifically, the FAU module first divides the SF-fused aquatic plant feature tensor of viewpoint v according to a preset spatial grid, and aggregates the feature response values ​​within each spatial grid cell to form a feature response set of viewpoint v at each spatial grid position. The feature response set is then normalized so that the sum of the feature response values ​​at each spatial grid position is 1, thus obtaining the feature distribution representation of viewpoint v. In the same way, the SF-fused aquatic plant feature tensor of viewpoint q is divided according to the same preset spatial grid, and the feature responses are aggregated and normalized to obtain the feature distribution representation of viewpoint q. The feature distribution representations of viewpoint v and viewpoint q are used to characterize the distribution state of aquatic plant features in a unified spatial grid under the corresponding viewpoints.

[0113] Based on the feature distribution representations of arbitrary viewpoint v and viewpoint q, the distribution difference between the feature distribution representations is quantified by the Earthhover distance, and the feature distribution difference between viewpoint v and viewpoint q is defined as the distribution distance.

[0114] In Example 1, based on the feature distribution representations of arbitrary viewpoint v and viewpoint q, the distribution difference between the two feature distribution representations is calculated using the Earthhover distance. The Earthhover distance is calculated by treating the feature distribution representations of viewpoint v and viewpoint q as probability mass distributions on the same spatial grid, with the goal of minimizing the total cost required to transfer probability mass from the feature distribution representation of viewpoint v to the feature distribution representation of viewpoint q. The total cost is obtained by weighted summation of the probability mass transfer amount between each spatial grid position and the distance between the corresponding spatial positions. Thus, the feature distribution difference between viewpoint v and viewpoint q is defined as the distribution distance. The distribution distance is used to measure the degree of feature distribution shift caused by the difference between the slender leaf structure of aquatic plants and the semantics of the background under different viewpoints.

[0115] A matching confidence score is generated based on the distribution distance. Based on the matching confidence score, a set of multi-view aquatic plant feature correspondences is established between each viewpoint to obtain multi-view aquatic plant feature correspondences with matching confidence scores.

[0116] The matching confidence score is used to perform consistent alignment of aquatic plant feature responses from different perspectives. The matching confidence score is defined as the negative exponential mapping result of the distribution distance. The matching confidence score ranges from 0 to 1. The matching confidence score is used to represent the reliability of the correspondence between aquatic plant features between perspective v and perspective q. The larger the matching confidence score, the more reliable the feature correspondence.

[0117] Each pair of aquatic plant feature correspondences between viewpoints v and q is represented by a triplet. The triplet contains the pixel coordinates of the m-th aquatic plant feature in viewpoint v, the pixel coordinates of the m-th aquatic plant feature in viewpoint q, and the matching confidence of the m-th aquatic plant feature correspondence. The matching confidence is used to represent the reliability of the aquatic plant feature correspondence.

[0118] In this embodiment, the distortion correction module includes:

[0119] Based on a simplified refraction camera model, refraction geometric constraints are constructed, and a refraction correction parameter vector is introduced. The pixel coordinates of any viewpoint v are extended into a three-dimensional homogeneous pixel vector. The inverse transformation and normalization are performed using the camera intrinsic parameter matrix to obtain the normalized imaging direction vector. The normalized imaging direction vector is then subjected to refraction correction to obtain the refraction-corrected unit exit direction. The refraction-corrected unit exit direction at viewpoint q is obtained in the same way.

[0120] In Example 1, the camera intrinsic matrix, camera extrinsic rotation matrix, and camera extrinsic translation vector corresponding to viewpoints v and q are obtained. The camera intrinsic matrix, extrinsic rotation matrix, and extrinsic translation vector of viewpoint v and viewpoint q are used to map the above pixel coordinates into an imaging ray expression in the camera coordinate system. The camera intrinsic matrix is ​​a third-order matrix, the extrinsic rotation matrix is ​​a third-order matrix, and the extrinsic translation vector is a three-dimensional vector.

[0121] For any pixel coordinates at viewpoint v, they are extended into a 3D homogeneous pixel vector. The homogeneous pixel vector is then inversely mapped and normalized using the camera intrinsic parameter matrix of the corresponding viewpoint to obtain a normalized imaging direction vector representing the initial propagation direction of the pixel in the camera coordinate system. A refraction correction parameter vector is introduced on the normalized imaging direction vector, and a direction correction operation based on refraction geometry is applied to the normalized imaging direction vector, causing the imaging direction vector to be redirected after passing through the multi-medium interface. This yields the refraction-corrected unit exit direction corresponding to the propagation path inside the water body. The refraction-corrected unit exit direction represents the effective propagation direction from the camera imaging center, through refraction at the multi-medium interface, and pointing towards the water body space under the current refraction correction parameter constraints. Using the same processing flow as above, the refraction-corrected unit exit direction is obtained for the corresponding pixel coordinates at viewpoint q, so that the refraction-corrected unit exit direction at different viewpoints is constructed under the constraints of a unified water body space propagation model.

[0122] The simplified refraction camera model is a geometric imaging model that equates the multi-medium refraction imaging process to a single-direction correction. Its core is to reduce the complexity of underwater refraction modeling and optimization while maintaining the geometric consistency of the rays.

[0123] By utilizing refractive geometric constraints, a convex optimization solution model is constructed to solve for the refractive correction parameter vector;

[0124] In Example 1, based on refraction geometric constraints, for each set of aquatic plant feature correspondences, corresponding 3D aquatic plant feature point variables are introduced, and these 3D aquatic plant feature point variables are mapped to the camera coordinate systems of viewing angles v and q, respectively. At each viewing angle, based on the geometric relationship between the refraction-corrected unit outgoing direction and the 3D aquatic plant feature point variables, a point-line consistency constraint is constructed between the refraction-corrected imaging ray and the 3D aquatic plant feature point variables. This point-line consistency constraint is used to characterize the orthogonal distance from the 3D aquatic plant feature point variables to the corresponding imaging ray. The point-line consistency constraints formed by all aquatic plant feature correspondences at each viewing angle are then... The algorithm summarizes the results and constructs a global optimization objective function with matching confidence as the weight. In the global optimization objective function, the refraction correction parameter vector is taken as one of the variables to be optimized, and the three-dimensional aquatic plant feature point variables are taken as joint variables to be optimized. By minimizing the weighted residual of all point and line consistency constraints, the value of the refraction correction parameter vector satisfies the optimal requirement of aquatic plant feature imaging consistency under multi-view conditions. A preset convex feasible region constraint is applied to the refraction correction parameter vector to limit its value range to meet the physical rationality under the conditions of air-waterproof porthole-water multi-media interface. The refraction correction parameter vector is obtained through the convex optimization solution process.

[0125] Based on the refraction correction parameter vector obtained by convex optimization, the pixel coordinates in the multi-view aquatic plant feature correspondence set are corrected for refraction distortion to generate distortion-free aquatic plant feature coordinates.

[0126] The distortion-free aquatic plant feature coordinates for each group under viewpoint v are defined as follows: the unit outgoing direction after refraction correction is projected back to the pixel coordinate system using the camera intrinsic parameter matrix of viewpoint v through the projection operator to obtain the distortion-free aquatic plant feature coordinates; the distortion-free aquatic plant feature coordinates for each group under viewpoint q are obtained in the same way; the distortion-free aquatic plant feature coordinates and the corresponding matching confidence are combined to form the corrected multi-view aquatic plant feature correspondence set.

[0127] The distortion-free aquatic plant feature coordinates of the m-th group under the v-th viewpoint are defined as the distortion-free aquatic plant feature coordinates of the m-th group under the v-th viewpoint. The numbering range of the v-th viewpoint is 1 to V, where V is not less than 2. The numbering range of the m-th group is 1 to M, where M is not less than 1. The matching confidence of the distortion-free aquatic plant feature coordinates of the m-th group between viewpoint v and viewpoint q is defined as the matching confidence of the distortion-free aquatic plant feature coordinates of the m-th group between viewpoint v and viewpoint q.

[0128] In this embodiment, the target feature module includes:

[0129] In the feature tensor of multi-view aquatic plant images, a mid-level feature tensor is selected to represent the mid-level structural features of aquatic plants, and a high-level feature tensor is selected to represent the high-level semantic features of aquatic plants.

[0130] The intermediate-level feature tensor and the high-level feature tensor correspond to different scale indices, and the two scale indices are not the same.

[0131] The distortion-free aquatic plant feature coordinates are mapped to the feature coordinate systems of the middle layer feature tensor and the upper layer feature tensor. The same sampling rule is used to sample the features of the middle layer feature tensor and the upper layer feature tensor at the positions corresponding to the distortion-free aquatic plant feature coordinates, so as to obtain the middle layer structure sampling feature vector and the upper layer semantic sampling feature vector respectively.

[0132] The mid-level structural sampling feature vector and the high-level semantic sampling feature vector are input into the TFE module. Through feature enhancement and semantic consistency integration, the enhanced feature vector of the m-th group of aquatic plant targets under the v-th view is obtained.

[0133] In Example 1, the mid-level structural sampling feature vector and the high-level semantic sampling feature vector are input to the TFE module. The TFE module performs feature standardization on the mid-level structural sampling feature vector and the high-level semantic sampling feature vector respectively. Then, the TFE module applies channel attention enhancement operation to the standardized mid-level structural sampling feature vector and the high-level semantic sampling feature vector respectively. The channel attention enhancement operation assigns adaptive attention weights to each feature component, highlighting the response of feature components that contribute more to the edge and spatial structure of aquatic plants, and weakening the response to redundant information. The attention weights are automatically adjusted according to the significance of different feature components. The TFE module weights and superimposes the enhanced mid-level structural sampling feature vector and the enhanced high-level semantic sampling feature vector according to the preset fusion weights, and adds them element by element in the vector dimension to output the enhanced feature vector of the m-th group of aquatic plant targets under the v-th view. The enhanced feature vector of aquatic plant targets includes both the edge structure response of aquatic plant leaves and the semantic consistency response of aquatic plants in the vector dimension, and under the action of the channel attention mechanism, it highlights the feature components with the most discriminative power for target localization among the multi-level aquatic plant features.

[0134] For the v-th viewpoint, the matching confidence scores related to all other viewpoints are counted and averaged to obtain the viewpoint confidence convergence value corresponding to the v-th viewpoint, and the image quality evaluation index corresponding to the v-th viewpoint is obtained at the same time.

[0135] In Example 1, the matching confidence scores between the v-th viewpoint and all other viewpoints are obtained in all distortion-free aquatic plant feature correspondences. These matching confidence scores are used as statistical objects. While keeping the viewpoint number and feature number unchanged, the matching confidence scores between the v-th viewpoint and all other viewpoints are summarized, and the arithmetic mean of the summarized matching confidence scores is taken to obtain the viewpoint confidence convergence value corresponding to the v-th viewpoint. Based on the enhanced multi-view aquatic plant image corresponding to the v-th viewpoint, the image quality evaluation index corresponding to the v-th viewpoint is calculated according to a preset image quality evaluation rule. The image quality evaluation index is used to represent the overall quality performance of the enhanced multi-view aquatic plant image from the v-th viewpoint in terms of sharpness, contrast, and noise level, and the image quality evaluation index is a non-negative scalar.

[0136] The preset image quality evaluation rules refer to a set of rules for comprehensively evaluating image quality based on the statistical characteristics of enhanced multi-view aquatic plant images, and include at least one or more of the following evaluation criteria:

[0137] Brightness stability evaluation based on the statistical distribution of brightness components is used to measure whether there are overexposed or underexposed areas in the overall brightness range of the enhanced aquatic plant image; local contrast evaluation based on the gray-scale variation amplitude of local areas is used to measure the degree of distinguishability between the edges of aquatic plant leaves and the background water; noise suppression effect evaluation based on high-frequency components or local gradient response is used to measure the degree of residual noise of suspended particles in the water after enhancement processing.

[0138] Based on the view confidence convergence of the vth viewpoint and the image quality evaluation index, the dynamic weighted fusion weight corresponding to the vth viewpoint is generated. The enhanced feature vectors of aquatic plants from all viewpoints are weighted and fused according to their respective dynamic weighted fusion weights to obtain the mth group of unified aquatic plant target feature vectors. All groups of unified aquatic plant target feature vectors are backfilled into the unified feature plane according to the spatial index relationship of the distortion-free aquatic plant feature coordinates to generate a unified aquatic plant target feature map.

[0139] In this embodiment, the intermediate positioning module includes:

[0140] A prediction model for the bounding box of aquatic plants is constructed based on a unified aquatic plant target feature map;

[0141] The feature vector at each spatial location in the unified aquatic plant target feature map is used as the input feature. The neural network regression module predicts the probability score of the location being the center of the aquatic plant target, the x and y coordinates of the center, and the width and height of the bounding box for each spatial location. The prediction results of all spatial locations form a candidate set of aquatic plant target bounding boxes. The prediction results with scores higher than a preset threshold in the candidate set are selected as the output of the aquatic plant target bounding box prediction model. The prediction result of the aquatic plant target bounding box is defined as a four-dimensional vector, which contains the x and y coordinates of the center of the aquatic plant target bounding box and the width and height of the aquatic plant target bounding box.

[0142] The improved WIoUv3 loss function was used to optimize the aquatic plant target bounding box prediction model, and the optimized aquatic plant target bounding box prediction results were obtained.

[0143] The improved WIoUv3 loss function is calculated as follows: The WIoU loss term between the predicted bounding box and the labeled bounding box of the aquatic plant target is calculated. The WIoU loss term is calculated based on the ratio of the overlapping area between the predicted bounding box and the labeled bounding box to the area of ​​the enclosed region. An IoU ratio factor is calculated; this IoU ratio factor is a target-scale adaptive ratio factor, and its value is equal to the preset target IoU loss reference value divided by the IoU loss value of the WIoU loss term. The IoU ratio factor, the non-monotonic focusing factor parameter, and the focusing modulation index are substituted into the focusing modulation formula to calculate the focusing modulation coefficient. The WIoU loss term is multiplied by the focusing modulation coefficient to obtain the improved WIoUv3 loss function value, which is used to guide the parameter optimization of the aquatic plant target bounding box prediction model.

[0144] Based on the optimized prediction results of the aquatic plant target bounding box, the pixel coordinates of the aquatic plant feature coordinates that fall within the coverage area of ​​the prediction results of the aquatic plant target bounding box are selected from the distortion-free aquatic plant feature coordinates and combined into a set of distortion-free aquatic plant feature coordinates.

[0145] Based on the refraction correction parameter vector, refraction geometric constraints, and the set of distortion-free aquatic plant feature coordinates, a three-dimensional spatial coordinate model of aquatic plants is constructed to obtain the three-dimensional spatial coordinates of aquatic plants.

[0146] In Example 1, the feature coordinates of each distortion-free aquatic plant are associated with its corresponding refraction-corrected unit emission direction from its viewpoint. Under a unified water body spatial coordinate system, the three-dimensional spatial coordinates of the aquatic plant are introduced as the three-dimensional variable to be solved. These coordinates are then mapped to the camera coordinate system using the camera extrinsic rotation matrix and translation vector from the corresponding viewpoint. This ensures that the three-dimensional spatial coordinates of the aquatic plant and its refraction-corrected unit emission direction are within the same geometric reference frame in the camera coordinate system. Based on refraction geometric constraints, a point-line consistency constraint is constructed between the three-dimensional spatial coordinates of the aquatic plant and its refraction-corrected unit emission direction at each viewpoint. This point-line consistency constraint is determined by calculating the three-dimensional spatial coordinates of the aquatic plant. The orthogonal distance to the corresponding refraction-corrected unit exit direction is quantified. The orthogonal distance is used to characterize whether the assumed three-dimensional spatial coordinates of the aquatic plants satisfy the imaging geometric relationship under the current refraction correction parameter vector constraint conditions. The point-line consistency constraints from multiple viewpoints are combined to form a three-dimensional spatial coordinate model of the aquatic plants, so that the three-dimensional spatial coordinates of the aquatic plants simultaneously satisfy the minimum point-line deviation condition under multiple refraction-corrected unit exit direction constraints. By optimizing the three-dimensional spatial coordinate model of the aquatic plants, the three-dimensional spatial coordinates of the aquatic plants that simultaneously achieve the optimal point-line consistency constraints under all viewpoints are determined. The three-dimensional spatial coordinates of the aquatic plants are the desired three-dimensional spatial coordinates of the aquatic plants.

[0147] The weighted sum of the point-line consistency residuals from all perspectives is used as the reprojection consistency cost function. The sparse Levenberg-Marquardt algorithm is used to iteratively solve the three-dimensional spatial coordinates of aquatic plants. In each iteration, the reprojection consistency cost function is incrementally updated based on the sparse Jacobian matrix, and the intermediate positioning result of the current iteration step is output.

[0148] In Example 1, during each iteration, based on the estimated 3D spatial coordinates of the aquatic plants in the current step, combined with the refraction-corrected unit exit direction, camera extrinsic parameters, and distortion-free aquatic plant feature coordinates from all perspectives, the point-line consistency residuals for each pair of corresponding relationships under each perspective are calculated. Using a pre-set matching confidence level as a weight, the weighted sum of all residuals is used as the current reprojection consistency cost function. Based on the currently estimated 3D spatial coordinates of the aquatic plants, the partial derivative of the reprojection consistency cost function with respect to the 3D spatial coordinates of the aquatic plants is calculated, and a sparse Jacobian matrix is ​​constructed. Following the incremental update strategy of the Levenberg-Marquardt algorithm, the update increment of the 3D spatial coordinates of the aquatic plants is solved based on the sparse Jacobian matrix and the cost function. The update increment is used to correct the current estimated 3D spatial coordinates of the aquatic plants, and the updated 3D spatial coordinates of the aquatic plants are used as the new estimated values ​​for the next iteration. This process is repeated until the decrease in the reprojection consistency cost function is less than the preset convergence threshold or the maximum number of iterations is reached. The 3D spatial coordinates of the aquatic plants output in the last iteration are used as the intermediate positioning result.

[0149] In this embodiment, the control module includes:

[0150] Error assessment is performed on the intermediate positioning results. The point-line consistency residuals corresponding to each intermediate positioning result under each view are summarized, and the weighted average of the point-line consistency residuals is used as the comprehensive positioning error index of the intermediate positioning results.

[0151] The overall positioning error index is compared with a preset positioning error threshold, and intermediate positioning results are filtered based on the comparison results.

[0152] When the overall positioning error index is not greater than the positioning error threshold, the corresponding intermediate positioning result is determined to be a valid positioning result;

[0153] When the overall positioning error index exceeds the positioning error threshold, the corresponding intermediate positioning results will be judged as invalid positioning results and discarded.

[0154] The three-dimensional spatial coordinates of aquatic plants that are determined to be valid positioning results are evaluated for positioning reliability, and corresponding positioning reliability parameters are generated.

[0155] Based on the distribution of matching confidence scores of effective positioning results under multi-view conditions and the magnitude of the comprehensive positioning error index, corresponding location confidence parameters are generated. These location confidence parameters are used to represent the reliability of the three-dimensional spatial coordinates of aquatic plants under multi-view observation conditions.

[0156] Based on the range of values ​​for the location reliability parameter, the valid location results are classified and labeled to form the final location result set;

[0157] The high-confidence positioning results and the medium-confidence positioning results are combined to form the three-dimensional spatial coordinates of the aquatic plants and their corresponding positioning confidence parameters to form the final positioning result, and the final positioning result is output.

[0158] Based on the final positioning results, the three-dimensional spatial coordinates of the aquatic plants are input into the intelligent machine control system as target location information. According to the three-dimensional spatial coordinates of the aquatic plants and the positioning confidence parameters, the corresponding aquatic plant retrieval operation instructions are generated, and the intelligent machine is controlled to perform aquatic plant retrieval operations in a unified water body spatial coordinate system.

[0159] In this embodiment, the final location result set includes:

[0160] High-confidence positioning results are achieved when the positioning confidence parameter is greater than or equal to the preset high-confidence threshold, and the comprehensive positioning error index is at the lower limit of the preset error range.

[0161] The mid-confidence positioning result is where the positioning confidence parameter is less than the preset high confidence threshold and greater than or equal to the preset low confidence threshold.

[0162] Low-confidence localization results are obtained when the localization confidence parameter is less than the preset low-confidence threshold.

[0163] An intelligent robot includes: an intelligent robot body, a power module, a camera, a lidar, a memory, and a processor disposed on the intelligent robot body;

[0164] Memory, used to store programs;

[0165] The processor is used to execute programs to implement the various steps of a river and lake aquatic plant harvesting system based on image recognition.

[0166] Example 2: During a river and lake water body maintenance task, the intelligent operation and maintenance platform detected a large area of ​​dense aquatic plants growing in the central area of ​​the water body, affecting navigation and the water's self-purification capacity. The management decided to use the multi-view image fusion and high-precision positioning method based on this invention, utilizing underwater intelligent equipment to achieve automated aquatic plant removal. The entire operation process and data comparison are as follows:

[0167] The maintenance robot deployed four multi-view underwater high-definition cameras in the work area. All cameras acquired multi-view raw images of aquatic plants in rivers and lakes through a unified synchronization signal. In one task, the system acquired a total of 16,000 raw images within 10 minutes, covering 35,000 aquatic plant targets in different spatial locations.

[0168] Statistical analysis revealed that the original images exhibited significant color distortion, localized overexposure, and noise from suspended particles in the water. In Example 2, data from a set of original image samples showed that the mean color distortion was 17.4 (normalized RGB difference), the mean noise intensity was 24.8 (normalized noise index), and the average contrast between the aquatic plant leaves and the background was only 0.32.

[0169] This invention applies multi-scale Retinex enhancement and adaptive texture-preserving filtering to all original images. Comparing the pixel gradient distribution of aquatic plant edges in the same region before and after filtering reveals that the average edge gradient increased from 6.1 to 16.7, and the edge detail preservation rate increased from 57.2% to 91.8%. The average noise intensity decreased from 24.8 to 3.7, and the average color distortion decreased to 2.9. The control group, using traditional global equalization + Gaussian filtering, only reduced the noise intensity to 11.1 and the color distortion to 9.7.

[0170] The enhanced multi-view images are input into a feature extraction network based on Ghost convolution and GSConv. Taking a batch of images as an example, the number of channels in the backbone feature map is dynamically adjusted from the traditional 256 to 128-320 to match the complexity of the aquatic plant structure, and the number of feature pyramid layers L=4. The network automatically identifies the slender structure and longitudinal texture of the main vein in the aquatic plant region. During feature alignment, the SF module divides all feature tensors into 6 groups according to the channel dimension and performs self-attention fusion on each group of features. The FAU module constructs 2312 high-confidence feature correspondences across all views.

[0171] Statistical data shows that, after using the feature extraction network of this invention, the average feature matching confidence score reached 0.88 (maximum 0.97, minimum 0.79), and the matching accuracy was 89.6%. In contrast, the traditional YOLOv5+ mean pooling scheme achieved an average matching confidence score of only 0.61 and a matching accuracy of 63.5% on the same batch of data. On the aquatic plant leaf group with sample number A1012, this method identified 68 valid feature points, while the traditional method only identified 39.

[0172] Utilizing multi-view feature correspondences, the system automatically constructs refraction geometric constraints based on camera calibration parameters. Through a convex optimization algorithm, it automatically solves for the optimal refraction correction parameter vector and corrects the coordinates of each set of feature points to distortion-free aquatic plant feature coordinates. Taking feature correspondence #273 as an example, the point-line consistency error was 2.18 pixels before correction and decreased to 0.41 pixels after correction. Overall statistics: the average error for all feature points before correction was 1.87 pixels, and after correction, it was 0.46 pixels.

[0173] In the comparative experiment, the average error of the traditional PINHOLE model plus approximate correction was 1.32 pixels, and the maximum error of some feature points was as high as 4.1 pixels.

[0174] The coordinates and confidence scores of distortion-free aquatic plant features are mapped to a unified coordinate system with multi-scale feature maps. A TFE module is then used to fuse mid-level structural features and high-level semantic features. Taking a specific aquatic plant region in frame 17 as an example, this invention dynamically assigns viewpoint-weighted fusion weights, with a maximum weight of 0.37 (high confidence + high image quality viewpoint) and a minimum of 0.08. After fusion, the spatial consistency score of the feature map is improved to 0.93 (compared to 0.66 in traditional methods), and the continuity of aquatic plant main vein recognition is increased from 71.4% to 94.2%.

[0175] After all features were fused, the feature intensity value of the aquatic plant target at spatial location (183, 246) was improved by 28% compared with the traditional method, and the recognition rate of the main stem structure location was improved by 22%.

[0176] A unified feature map of aquatic plants was used as input to a bounding box regression model. An improved WIoUv3 loss function was employed for bounding box optimization, and the system automatically selected 93 high-confidence target bounding boxes (score ≥ 0.75). 97 target bounding boxes were labeled within the actual area. The recall rate of this method was 95.9%, while the recall rate of the traditional method was 79.1%.

[0177] The sparse Levenberg-Marquardt algorithm was employed, dynamically adjusting the Jacobian matrix in each iteration. The final mean positioning error for all 3D spatial coordinate points was 0.85 mm, with a maximum error of 2.4 mm and a standard deviation of 0.61 mm. In contrast, the traditional method for the same batch of data had a mean error of 2.03 mm, a maximum error of 8.5 mm, and a standard deviation of 1.97 mm. For a typical aquatic plant target (#17), the spatial positioning error was 0.62 mm using this method, compared to 3.3 mm using the traditional method.

[0178] The positioning results were filtered with an error threshold of 1.5mm, resulting in 91 sets of high-confidence 3D coordinates for aquatic plants. The mean confidence parameter for all high-confidence results was 0.91. Seven sets of low-confidence results were filtered out due to strong image noise or extreme refraction errors. The system automatically generated a retrieval path for the intelligent robotic arm based on the spatial coordinates. The robot operated point-by-point, performing 100 consecutive retrievals with an average retrieval time of 12.2 seconds per attempt and a success rate of 98.8%. Traditional methods yielded positioning results with an average confidence level of 0.68, an average retrieval time of 19.3 seconds per attempt, and a success rate of only 73.5%.

[0179] Within a specific task cycle (24 hours), this invention detected and processed 536 high-density aquatic plant targets with a positioning accuracy rate of 97.4% and a retrieval error rate (accidental retrieval / missed retrieval) of 1.3%. Traditional methods, processing the same amount of data, only achieved a positioning accuracy rate of 74.6% and a retrieval error rate as high as 15.1%.

[0180] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A river and lake aquatic plant harvesting system based on image recognition, characterized in that, include: The image preprocessing module acquires and preprocesses the original multi-view images of the target aquatic plants in the river and lake water, and outputs enhanced multi-view images of the aquatic plants. An improved YOLOv8n network module was developed. The enhanced multi-view aquatic plant image was input into the feature extraction network, which is based on the improved YOLOv8n network containing the Ghost convolutional backbone network and the GSConv neck network. The feature extraction model was constructed and the multi-view aquatic plant image feature tensor was output. The aquatic plant feature correspondence module uses the SF module to group and fuse the local aquatic plant structural features and global spatial distribution features of the feature tensor of multi-view aquatic plant images, and uses the FAU module to perform feature alignment and matching based on the differences in aquatic plant feature distribution under different views to obtain the correspondence of aquatic plant features from multiple perspectives. The distortion correction module uses the correspondence of aquatic plant features from multiple perspectives to construct refractive geometric constraints, performs refractive distortion correction on the correspondence of aquatic plant features from multiple perspectives, and obtains the coordinates of aquatic plant features without distortion. The target feature module inputs the distortion-free aquatic plant feature coordinates and matching confidence into the TFE module to extract the mid-layer structural features and high-layer semantic features of the aquatic plants. It then uses a dynamic weighted fusion strategy to assign weights to the aquatic plant features from each perspective based on the matching confidence and image quality evaluation index, and outputs a unified aquatic plant target feature map. The intermediate localization module constructs a three-dimensional localization model of aquatic plants based on a unified aquatic plant target feature map, refraction correction parameters, and refraction geometric constraints. It optimizes the prediction of the aquatic plant target bounding box using an improved WIoUv3 loss function and iteratively solves the three-dimensional spatial coordinates of the aquatic plant target using the sparse Levberg–Marquardt algorithm, outputting the intermediate localization results. The control module performs error threshold filtering and location reliability assessment on the intermediate positioning results to generate the final positioning result. Based on the final positioning result, an intelligent machine is used to remove aquatic plants.

2. The river and lake aquatic plant harvesting system based on image recognition according to claim 1, characterized in that, The image preprocessing module includes: Obtain original multi-view images of target aquatic plants in rivers and lakes; Calculate the brightness component based on the original multi-view aquatic plant images; Multi-scale Retinex processing is performed on the luminance component to construct a scale set, where each scale parameter corresponds to a Gaussian filter scale, and a weight value is assigned to each Gaussian filter scale to calculate the multi-scale enhanced luminance component. Based on the multi-scale enhanced brightness component, the red, green and blue channels of the original multi-view aquatic plant image are color corrected respectively. Channel color correction coefficients are introduced to obtain the image enhanced by the improved ProMSRCR. For the image enhanced by the improved ProMSRCR, luminance equalization mapping is performed on the pixel intensity of each channel to obtain the luminance equalized image. Based on the multi-scale enhancement of brightness components, a water plant texture preservation mask is constructed. The brightness-equalized image is then filtered according to the water plant texture preservation mask to obtain the filtered output image. The filtered output image is then used as the enhanced multi-view water plant image corresponding to the v-th viewpoint.

3. The river and lake aquatic plant harvesting system based on image recognition according to claim 1, characterized in that, The improved YOLOv8n network module includes: The enhanced multi-view aquatic plant images are input into the feature extraction model constructed by the improved YOLOv8n network. The resolution of the enhanced multi-view aquatic plant images is scale-aligned and normalized to obtain the input feature tensor. In the backbone network of the improved YOLOv8n network for constructing a feature extraction model, Ghost convolution is used to extract features from the input feature tensor layer by layer. In each backbone layer, basic aquatic plant morphology features are generated through main convolution. The basic aquatic plant morphology features are expanded using the cheap linear mapping operator. The basic aquatic plant morphology features and the expanded feature results are concatenated along the channel dimension to form the output feature tensor of the backbone layer. Based on the feature tensors output from different levels of the backbone network, a multi-scale feature set is constructed according to spatial resolution from high to low. In the neck network of the improved YOLOv8n network for constructing a feature extraction model, the GSConv module is used to perform channel-wise convolution, channel concatenation, and channel rearrangement on the multi-scale feature set, and output the fused feature tensor of each scale. The feature tensors of each scale are fused together in a preset order to form a multi-view aquatic plant image feature tensor.

4. The river and lake aquatic plant harvesting system based on image recognition according to claim 1, characterized in that, The aquatic plant feature corresponding module includes: Each scale-fused feature tensor in the feature tensor of each multi-view aquatic plant image is mapped to a unified scale index set according to a preset scale alignment rule to construct an aquatic plant feature tensor at a unified scale. The aquatic plant feature tensor under a uniform scale is divided into K groups of sub-feature tensors according to the channel dimension. The SF module performs a global self-attention fusion operation on each group of sub-feature tensors to obtain the SF fused aquatic plant feature tensor under each view. Based on the SF-fused aquatic plant feature tensor, the FAU module is used to align and match the aquatic plant features between different perspectives. For any two perspectives, the feature response values ​​of the SF-fused aquatic plant feature tensor on the preset spatial grid are normalized to obtain the feature distribution representation of perspective v and the feature distribution representation of perspective q. Based on the feature distribution representations of arbitrary viewpoint v and viewpoint q, the distribution difference between the feature distribution representations is quantified by the Earthhover distance, and the feature distribution difference between viewpoint v and viewpoint q is defined as the distribution distance. A matching confidence score is generated based on the distribution distance. Based on the matching confidence score, a set of multi-view aquatic plant feature correspondences is established between each viewpoint to obtain multi-view aquatic plant feature correspondences with matching confidence scores.

5. A river and lake aquatic plant harvesting system based on image recognition according to claim 1, characterized in that, The distortion correction module includes: Based on a simplified refraction camera model, refraction geometric constraints are constructed, and a refraction correction parameter vector is introduced. The pixel coordinates of any viewpoint v are extended into a three-dimensional homogeneous pixel vector. The inverse transformation and normalization are performed using the camera intrinsic parameter matrix to obtain the normalized imaging direction vector. The normalized imaging direction vector is then subjected to refraction correction to obtain the refraction-corrected unit exit direction. The refraction-corrected unit exit direction at viewpoint q is obtained in the same way. By utilizing refractive geometric constraints, a convex optimization solution model is constructed to solve for the refractive correction parameter vector; Based on the refraction correction parameter vector obtained by convex optimization, the pixel coordinates in the multi-view aquatic plant feature correspondence set are corrected for refraction distortion to generate distortion-free aquatic plant feature coordinates.

6. The river and lake aquatic plant harvesting system based on image recognition according to claim 1, characterized in that, The target feature module includes: In the feature tensor of multi-view aquatic plant images, a mid-level feature tensor is selected to represent the mid-level structural features of aquatic plants, and a high-level feature tensor is selected to represent the high-level semantic features of aquatic plants. The distortion-free aquatic plant feature coordinates are mapped to the feature coordinate systems of the middle layer feature tensor and the upper layer feature tensor. The same sampling rule is used to sample the features of the middle layer feature tensor and the upper layer feature tensor at the positions corresponding to the distortion-free aquatic plant feature coordinates, so as to obtain the middle layer structure sampling feature vector and the upper layer semantic sampling feature vector respectively. The mid-level structural sampling feature vector and the high-level semantic sampling feature vector are input into the TFE module. Through feature enhancement and semantic consistency integration, the enhanced feature vector of the m-th group of aquatic plant targets under the v-th view is obtained. For the v-th viewpoint, the matching confidence scores related to all other viewpoints are counted and averaged to obtain the viewpoint confidence convergence value corresponding to the v-th viewpoint, and the image quality evaluation index corresponding to the v-th viewpoint is obtained at the same time. Based on the view confidence convergence of the vth viewpoint and the image quality evaluation index, the dynamic weighted fusion weight corresponding to the vth viewpoint is generated. The enhanced feature vectors of aquatic plants from all viewpoints are weighted and fused according to their respective dynamic weighted fusion weights to obtain the mth group of unified aquatic plant target feature vectors. All groups of unified aquatic plant target feature vectors are backfilled into the unified feature plane according to the spatial index relationship of the distortion-free aquatic plant feature coordinates to generate a unified aquatic plant target feature map.

7. A river and lake aquatic plant harvesting system based on image recognition according to claim 1, characterized in that, The intermediate positioning module includes: A prediction model for the bounding box of aquatic plants is constructed based on a unified aquatic plant target feature map; The improved WIoUv3 loss function was used to optimize the aquatic plant target bounding box prediction model, and the optimized aquatic plant target bounding box prediction results were obtained. Based on the optimized prediction results of the aquatic plant target bounding box, the pixel coordinates of the aquatic plant feature coordinates that fall within the coverage area of ​​the prediction results of the aquatic plant target bounding box are selected from the distortion-free aquatic plant feature coordinates and combined into a set of distortion-free aquatic plant feature coordinates. Based on the refraction correction parameter vector, refraction geometric constraints, and the set of distortion-free aquatic plant feature coordinates, a three-dimensional spatial coordinate model of aquatic plants is constructed to obtain the three-dimensional spatial coordinates of aquatic plants. S65. Using the weighted sum of the point-line consistency residuals under all views as the reprojection consistency cost function, the sparse Levenberg-Marquardt algorithm is used to iteratively solve the three-dimensional spatial coordinates of aquatic plants. In each iteration, the reprojection consistency cost function is incrementally updated based on the sparse Jacobian matrix, and the intermediate positioning result of the current iteration step is output.

8. A river and lake aquatic plant harvesting system based on image recognition according to claim 1, characterized in that, The control module includes: Error assessment is performed on the intermediate positioning results. The point-line consistency residuals corresponding to each intermediate positioning result under each view are summarized, and the weighted average of the point-line consistency residuals is used as the comprehensive positioning error index of the intermediate positioning results. The overall positioning error index is compared with a preset positioning error threshold, and intermediate positioning results are filtered based on the comparison results. When the overall positioning error index is not greater than the positioning error threshold, the corresponding intermediate positioning result is determined to be a valid positioning result; When the overall positioning error index exceeds the positioning error threshold, the corresponding intermediate positioning results will be judged as invalid positioning results and discarded. The three-dimensional spatial coordinates of aquatic plants that are determined to be valid positioning results are evaluated for positioning reliability, and corresponding positioning reliability parameters are generated. Based on the range of values ​​for the location reliability parameter, the valid location results are classified and labeled to form the final location result set; The high-confidence positioning results and the medium-confidence positioning results are combined to form the three-dimensional spatial coordinates of the aquatic plants and their corresponding positioning confidence parameters to form the final positioning result, and the final positioning result is output. Based on the final positioning result, the three-dimensional spatial coordinates of the aquatic plants are input into the intelligent machine control system as target location information. According to the three-dimensional spatial coordinates of the aquatic plants and the positioning confidence parameters, the corresponding aquatic plant retrieval operation command is generated, and the intelligent machine is controlled to perform the aquatic plant retrieval operation under a unified water body spatial coordinate system.

9. A river and lake aquatic plant harvesting system based on image recognition according to claim 8, characterized in that, The final location result set includes: High-confidence positioning results are achieved when the positioning confidence parameter is greater than or equal to the preset high-confidence threshold, and the comprehensive positioning error index is at the lower limit of the preset error range. The mid-confidence positioning result is where the positioning confidence parameter is less than the preset high confidence threshold and greater than or equal to the preset low confidence threshold. Low-confidence localization results are obtained when the localization confidence parameter is less than the preset low-confidence threshold.

10. An intelligent robot, characterized in that, include: The intelligent robot body, including a power module, camera, lidar, memory, and processor. The memory is used to store programs; The processor is used to execute the program to implement the various steps of the image recognition-based river and lake aquatic plant harvesting system as described in any one of claims 1-9.

Images