Underwater vegetation classification method based on unsupervised learning and feature fusion
By extracting visual and semantic features of underwater vegetation using deep convolutional neural networks and multimodal large models, and combining L2 normalization and principal component analysis, the problems of noise interference and domain shift in underwater vegetation classification were solved, achieving higher classification accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DALI UNIV
- Filing Date
- 2026-01-28
- Publication Date
- 2026-05-19
AI Technical Summary
Existing underwater vegetation classification methods are severely affected by noise interference, rely on manual annotation which is costly and inaccurate, and the domain offset problem between different lakes makes it difficult for the model to generalize. Existing unsupervised methods suffer from noise submerging semantic features and the curse of dimensionality when fusing visual and semantic features.
A deep convolutional neural network is used to extract high-dimensional visual features, which are then transformed into high-dimensional semantic features by combining a multimodal large model. An orthogonal subspace is constructed through L2 normalization and principal component analysis, and low-dimensional visual and semantic features are fused together. An unsupervised clustering algorithm is then used for classification.
It improves the accuracy of underwater vegetation classification, overcomes the problem of visual feature noise covering semantic features, and achieves higher classification accuracy.
Smart Images
Figure SMS_1 
Figure SMS_2
Abstract
Description
Technical Field
[0001] This invention relates to the field of aquatic ecological monitoring and image processing, and in particular to an underwater vegetation classification method based on unsupervised learning and feature fusion. Background Technology
[0002] Health assessment and eutrophication control of lake ecosystems are crucial for environmental protection. Underwater vegetation, as a core component of lake ecosystems, has its species composition, distribution range, and growth status as key indicators reflecting water quality. With the widespread adoption of underwater robots (ROVs) and automated guided vehicles (AGVs), numerous cameras are deployed in ecological monitoring sites, continuously collecting massive amounts of underwater video and image data. This data is typically affected by environmental factors such as water turbidity, light refraction, equipment shadows, and air bubbles, accompanied by high-intensity noise interference. Corresponding these unstructured image data, rich in "visual noise," to "vegetation species" with strict biological definitions, and performing unified characterization and accurate classification, is inherently difficult. These factors pose significant challenges to the feature extraction and cross-lake application of underwater vegetation.
[0003] Traditional monitoring methods rely on divers sampling or experts manually annotating video frame by frame. This approach is not only time-consuming, labor-intensive, and costly, but also highly susceptible to biases due to subjective judgment. Furthermore, natural lakes are vast, and different regions (such as deep lakes in plateaus and shallow lakes in cities) exhibit significant differences in hydrological conditions, lighting environments, and dominant species, leading to severe "domain shift" in data distribution. This limits the cost and feasibility of supervised learning schemes that rely on large amounts of labeled data: a model trained on one lake often suffers a significant drop in accuracy when transferred to another due to "acclimatization," making it difficult to establish universal standards. Based on research, unsupervised annotation systems are currently divided into two main categories: image-based and text-based. In the image domain, for example, the invention patent CN112507937B discloses an unsupervised classification method for satellite imagery that integrates multi-source data. This method incorporates k-means clustering loss into each encoder and decoder, achieving constraints between different data sources. In the text domain, for example, the invention patent with publication number CN113806527B discloses a cross-lingual unsupervised classification method with multi-view transfer learning. This method transforms text data into latent representations as input to a classification algorithm, achieving the output of text document tags. Since images correspond to visual features, existing unsupervised classification methods using visual multi-source data fusion have higher accuracy than methods using text to extract image features. However, due to the significant differences between visual features and text semantic features, when fusing visual and semantic features, visual noise can easily overwhelm semantic features, leading to conflicts between them. Directly concatenating visual and semantic features can result in the "curse of dimensionality" or the dilution of semantic information by high-dimensional visual noise, reducing feature purity. Therefore, the accuracy of combining visual and semantic features for tag classification is still far inferior to tag classification methods using single visual multi-source data fusion. To obtain higher accuracy classification results, visual feature multi-source data fusion classification methods have become the primary classification approach. The classification method of this invention aims to overcome the feature fusion barrier between visual features and semantic features in label classification, thereby obtaining an underwater vegetation classification method based on unsupervised learning and feature fusion with higher classification accuracy. Summary of the Invention
[0004] In view of this, the present invention provides an underwater vegetation classification method based on unsupervised learning and feature fusion with high classification accuracy.
[0005] The objective of this invention is achieved through the following technical solutions.
[0006] An underwater vegetation classification method based on unsupervised learning and feature fusion includes the following steps: S1. Collect underwater video data and extract the video data into discrete image sequences according to a fixed frequency to form the original image; S2. Input the original image into a deep convolutional neural network to extract the macroscopic texture and morphological features of the original image, output a high-dimensional visual feature vector, and then remove redundant information from the high-dimensional visual feature vector to generate a low-dimensional visual feature vector. S3. The original image is quantized by a multimodal large model. Based on biological prompts, the unstructured image is converted into a high-dimensional scoring vector. The high-dimensional scoring vector is then ID-ized and input into the EAPCR-AE model to extract low-dimensional, high-density semantic feature vectors. S4. After L2 normalization of the low-dimensional visual feature vector and the low-dimensional high-density semantic feature vector, they are concatenated to obtain the concatenated vector. Then, principal component analysis is used to construct an orthogonal subspace, extract the principal components whose cumulative variance reaches a preset threshold, map the concatenated vector to the orthogonal subspace, and output the fused feature vector. S5. Cluster and vote on the fused feature vectors to form a classification result.
[0007] Preferably, the deep convolutional neural network is the ConvNeXt pre-trained model.
[0008] Preferably, in step S1, the high-dimensional visual features are input into the UMAP manifold learning algorithm for nonlinear dimensionality reduction to remove redundant information.
[0009] The preferred multimodal large model is the Tongyi Thousand Questions.
[0010] Preferably, the biological cue words in step S3 are: object boundary clarity, plant density, branching level, main stem orientation, root attachment method, leaf color depth, leaf vein visibility, leaf curvature, leaf transparency, stem / petiole color, leaf color distribution, stem node swelling, petiole length, stipe / leaf sheath visibility, stem color stripes, leaf margin curling direction, leaf angle, internode length, leaf dorsal and ventral color difference, floating leaf attachment method, floating leaf shape, liquid surface wax / water droplets, floating leaf upper and lower surface color difference, flower color main hue, number of petals, petiole / pedicel flexibility, and fruit / seed sac; the low-dimensional high-density semantic feature vector is 8-dimensional.
[0011] Preferably, the high-dimensional visual feature vector is 2048-dimensional and the low-dimensional visual feature vector is 100-dimensional.
[0012] Preferably, in step S3, the EAPCR-AE model inputs the ID-processed high-dimensional scoring vector into the Embedding module to align it to a unified semantic space, forming matrix E; then, it calculates the inner product similarity between any two feature vectors in matrix E to obtain the relation matrix G; then, it rearranges the relation matrix G to shuffle the correspondence between features in the relation matrix, obtaining the permutation matrix G'; using a CNN convolutional neural network, it performs one convolutional branch on each of the relation matrix G and the permutation matrix, concatenates the convolutional features of the two convolutional branches in the channels, and then unfolds them to obtain the encoding vector C of the fully connected output relation branch; it flattens matrix E along the feature dimension and feeds it into a multilayer perceptron (MLP) to extract global features to obtain the output R; it then fuses the output R and the encoding vector C with learnable weights; finally, it performs mirror decoding to obtain a low-dimensional, high-density semantic feature vector.
[0013] Preferably, in step S5, the fused feature vector is simultaneously input into three unsupervised clustering algorithms: K-means, Agg, and Birch. The number of clusters K adopts an over-clustering strategy, and the intersection is calculated using a bagging voting strategy. Results with the same judgment result from the three unsupervised clustering algorithms are retained for classification.
[0014] Preferably, the preset threshold in step S4 is 95%.
[0015] The beneficial effects of this invention are: By concatenating the positional visual feature vector and positional high-density semantic features after eliminating dimensional differences through L2 normalization, and then mapping the concatenated vector to a low-dimensional subspace through an orthogonal subspace constructed by principal component analysis, the common features that coexist with visual and semantic features are automatically preserved, while random noise that only appears in a single modality of visual or semantic features is eliminated. This overcomes the problem of visual feature noise covering semantic features, and achieves feature-level filtering of environmental noise, thereby improving classification accuracy. Detailed Implementation
[0016] The technical solution of the present invention will be further described below.
[0017] In this specific embodiment, "dimension" refers to existing technology; for example, "100-dimensional visual features" is defined as a feature vector Vvis=[v1,v2,...,v100] containing 100 floating-point values. These 100 dimensions are not simple pixel compression or mapping of a single physical attribute, but rather the topological projection coordinates of the image on a non-linear visual manifold. The same applies to 2048-dimensional, 27-dimensional, and 8-dimensional features, which will not be elaborated upon here. Example 1
[0018] This embodiment provides an underwater vegetation classification method based on unsupervised learning and feature fusion, including the following steps: S1. Collect underwater video data of the lake using camera equipment carried by unmanned boats or underwater robots, and extract the video data at one frame per second. After adaptive median filtering and cropping to remove equipment shadows and bubble interference areas, the data is uniformly adjusted into a discrete image sequence of 1280×720 pixels to form the original image. S2. Input the original image into the ConvNeXt pre-trained model as the backbone network, use its large kernel convolution to obtain the macroscopic receptive field, overcome the underwater overlapping occlusion problem, extract the macroscopic texture and morphological features of the original image, and output a 2048-dimensional high-dimensional visual feature vector. Then, input the high-dimensional feature vector into the UMAP manifold learning algorithm for nonlinear dimensionality reduction, remove redundant information while maintaining the global topological structure of the data, compress and generate a 100-dimensional low-dimensional visual feature vector, thereby capturing the visual features in the original image. S3. The original image is quantized using the Tongyi Qianwen multimodal large model. Based on the biological prompt words, the clarity of the object boundary, plant density, branching level, main stem direction, root attachment method, leaf color depth, leaf vein visibility, leaf curvature, leaf transparency, stem / petiole color, leaf color distribution, stem node swelling, petiole length, stalk / leaf sheath visibility, stem color stripes, leaf margin curling direction, leaf inclination angle, internode length, leaf dorsal and ventral color difference, floating leaf attachment method, floating leaf shape, liquid surface wax / water droplets, floating leaf upper and lower surface color difference, flower color main color, number of petals, petiole / pedicel flexibility, and fruit / seed sac, the unstructured image is transformed into a 27-dimensional high-dimensional scoring vector covering biological semantic features through PromptEngineering. The high-dimensional scoring vector is then ID-ized and input into the EAPCR-AE model to extract an 8-dimensional low-dimensional high-density semantic feature vector. Features corresponding to relevant biological prompt words in the original image are then extracted. S4. The 100-dimensional low-dimensional visual feature vector and the low-dimensional high-density semantic feature vector are L2 normalized respectively, and the two sets of heterogeneous features are forced to be mapped onto the same unit hypersphere to eliminate the difference in magnitude. Then, they are concatenated in the channel dimension to obtain a 108-dimensional concatenated vector. Then, principal component analysis is used to construct an orthogonal subspace, and the eight principal components with a cumulative variance contribution rate of 95% are extracted as orthogonal bases. The 108-dimensional concatenated vector is mapped to the orthogonal subspace to output the fused feature vector. S5. Cluster and vote on the fused feature vectors to form a classification result.
[0019] In step S3, the EAPCR-AE model inputs the ID-processed high-dimensional scoring vector into the Embedding module to align it to a unified semantic space, forming matrix E. Then, it calculates the inner product similarity between any two feature vectors in matrix E to obtain the relation matrix G. Next, it rearranges the relation matrix G, shuffling the correspondence between features to obtain the permutation matrix G'. Using a CNN convolutional neural network, it performs convolutional branches on both the relation matrix G and the permutation matrix, concatenates the convolutional features of the two branches in the channels, and then unfolds them to obtain the fully connected output relation branch encoding vector C. Matrix E is flattened along the feature dimension and fed into a multilayer perceptron (MLP) to extract global features, obtaining the output R. The output R and the encoding vector C are then fused with learnable weights. Finally, it is mirror-decoded to obtain a low-dimensional, highly dense semantic feature vector.
[0020] In step S5, the low-dimensional, highly dense semantic feature vector is simultaneously input into three unsupervised clustering algorithms: K-means, Agg, and Birch. The number of clusters K is set to 2-5 times the expected number of species using an over-clustering strategy. A bagging voting strategy is used to calculate the intersection, retaining the results from the three unsupervised clustering algorithms that have the same classification outcome, thus performing classification. The judgment logic is to retain only samples with consistent classification structures from the three unsupervised clustering algorithms as "high-confidence samples," discarding marginal samples with discrepancies. Sample validation: Experts identify and assign species labels to the selected high-purity clusters, integrate similar clusters, and validate the final classification structure.
[0021] Comparative Example 1 The underwater vegetation classification method in this comparative example is similar to that in Example 1, except that it does not include step S3 in the classification method of Example 1.
[0022] Comparative Example 2 The underwater vegetation classification method in this comparative example is similar to that in Example 1, except that it does not include step S2 in the classification method of Example 1.
[0023] In this specific implementation, the classification methods of Example 1, Comparative Example 1, and Comparative Example 2 were used to classify the public datasets Deepseagrass, Erhai Lake dataset, and Wuhan East Lake dataset. The accuracy is shown in Table 1.
[0024] The Erhai Lake dataset and the Wuhan East Lake dataset both use unmanned vessels equipped with high-definition underwater cameras to cruise in the corresponding waters, maintaining an observation distance of 0.5-1.5 meters from the top of the vegetation to obtain continuous video streams.
[0025]
[0026] As shown in Table 1, the underwater vegetation classification method in Example 1 has a higher classification accuracy than Comparative Example 1 (single visual feature) and Comparative Example 2 (word semantic feature).
[0027] Comparative Example 3 The underwater vegetation classification method in this comparative example is similar to that in Example 1, except that in step S4 of this comparative example, the normalized spliced 108-dimensional spliced vector is input into step S5 for classification.
[0028] Comparative Example 4 The underwater vegetation classification method in this comparative example is similar to that in Example 1, except that in step S4, principal component analysis is used to compress the 100-dimensional low-dimensional visual feature vector to 8 dimensions, and then it is fused with the 8-dimensional low-dimensional high-density semantic feature vector according to the linear weighting formula Z=0.5×Vpca+0.5×Ssem to output an 8-dimensional fused feature vector, which is then classified according to step S5. Here, Z represents the fused feature vector, Vpca represents the 8-dimensional low-dimensional visual feature vector after dimensionality reduction by principal component analysis, and Ssem represents the 8-dimensional low-dimensional high-density semantic feature vector.
[0029] Comparative Example 5 This comparative example is similar to the underwater vegetation classification method in Example 1. The difference is that step S4 in this comparative example uses a traditional correlation analysis algorithm to find the maximum correlation projection direction between the low-dimensional visual feature vector and the low-dimensional high-density semantic feature vector, thereby outputting an 8-dimensional fused feature vector.
[0030] Table 2 shows the classification accuracy of the underwater vegetation classification methods of Example 1 and Comparative Examples 3-5 on the Erhai Lake dataset.
[0031]
[0032] As shown in Table 2, the underwater vegetation classification method in Example 1 has a higher classification accuracy. The classification accuracy of the three different fusion strategies of visual features and semantic features in Comparative Examples 3-5 is lower than the classification accuracy of 92.43% of the single visual feature in Comparative Example 1.
[0033] The above are specific implementations of the present invention, described in detail, but should not be construed as limiting the scope of the invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the invention, and these obvious substitutions all fall within the protection scope of the present invention.
Claims
1. A method for underwater vegetation classification based on unsupervised learning and feature fusion, characterized in that, Includes the following steps: S1. Collect underwater video data and extract the video data into discrete image sequences according to a fixed frequency to form the original image; S2. Input the original image into a deep convolutional neural network, extract the macroscopic texture and morphological features of the original image, output a high-dimensional visual feature vector, and then remove redundant information from the high-dimensional visual feature vector to generate a low-dimensional visual feature vector. S3. The original image is quantized using a multimodal large model. Based on biological prompts, the unstructured image is converted into a high-dimensional scoring vector. The high-dimensional scoring vector is then ID-ized and input into the EAPCR-AE model to extract low-dimensional, high-density semantic feature vectors. S4. The low-dimensional visual feature vector and the low-dimensional high-density semantic feature vector are L2 normalized and then concatenated to obtain a concatenated vector; then, principal component analysis is used to construct an orthogonal subspace, extract principal components whose cumulative variance contribution rate reaches a preset threshold, map the concatenated vector to the orthogonal subspace, and output the fused feature vector. S5. Cluster and vote the fused feature vectors to form a classification result.
2. The underwater vegetation classification method based on unsupervised learning and feature fusion according to claim 1, characterized in that, The deep convolutional neural network is a ConvNeXt pre-trained model.
3. The underwater vegetation classification method based on unsupervised learning and feature fusion according to claim 1, characterized in that, Step S1 involves inputting the high-dimensional visual features into the UMAP manifold learning algorithm for nonlinear dimensionality reduction to remove redundant information.
4. The underwater vegetation classification method based on unsupervised learning and feature fusion according to claim 1, characterized in that, The multimodal large model is Tongyi Qianwen.
5. The underwater vegetation classification method based on unsupervised learning and feature fusion according to claim 1, characterized in that, The biological cue words mentioned in step S3 are: object boundary clarity, plant density, branching level, main stem orientation, root attachment method, leaf color depth, leaf vein visibility, leaf curvature, leaf transparency, stem / petiole color, leaf color distribution, stem node swelling, petiole length, stipule / leaf sheath visibility, stem color stripes, leaf margin curling direction, leaf angle, internode length, leaf dorsal and ventral color difference, floating leaf attachment method, floating leaf shape, liquid surface wax / water droplets, floating leaf upper and lower surface color difference, flower color main tone, number of petals, petiole / pedicel flexibility, and fruit / seed sac; the low-dimensional high-density semantic feature vector is 8-dimensional.
6. The underwater vegetation classification method based on unsupervised learning and feature fusion according to claim 5, characterized in that, The high-dimensional visual feature vector has 2048 dimensions, and the low-dimensional visual feature vector has 100 dimensions.
7. The underwater vegetation classification method based on unsupervised learning and feature fusion according to claim 6, characterized in that, In step S3, the EAPCR-AE model inputs the ID-processed high-dimensional scoring vector into the Embedding module to align it to a unified semantic space, forming matrix E. Then, it calculates the inner product similarity between any two feature vectors in matrix E to obtain the relation matrix G. Next, it rearranges the relation matrix G, shuffling the correspondence between features to obtain the permutation matrix G'. Using a CNN convolutional neural network, it performs a convolutional branch on both the relation matrix G and the permutation matrix, concatenates the convolutional features of the two branches in the channels, and then unfolds them to obtain the encoding vector C of the fully connected output relation branch. The matrix E is flattened along the feature dimension and fed into a multilayer perceptron (MLP) to extract global features, obtaining the output R. The output R and the encoding vector C are then fused with learnable weights. Finally, it is mirror-decoded to obtain the low-dimensional, high-density semantic feature vector.
8. The underwater vegetation classification method based on unsupervised learning and feature fusion according to claim 7, characterized in that, In step S5, the fused feature vector is simultaneously input into three unsupervised clustering algorithms: K-means, Agg, and Birch. The number of clusters K adopts an over-clustering strategy, and the intersection is calculated using a bagging voting strategy. Results with the same judgment result from the three unsupervised clustering algorithms are retained for classification.
9. The underwater vegetation classification method based on unsupervised learning and feature fusion according to any one of claims 1-8, characterized in that, The preset threshold in step S4 is 95%.