Blue-green algae image recognition method and system based on hierarchical self-adaptation and domain driving

Abstract

The application provides a cyanobacteria image recognition method and system based on hierarchical adaptation and field driving. In the recognition method, an improved dark channel algorithm is used to enhance the image; a cyanobacteria biological attribute text database is constructed and synonym replacement and sentence restructuring are performed; multi-scale visual features are extracted through a hierarchical adaptive Swin Transformer model, and key region representation is enhanced in combination with dynamic spectrum attention; text is input into a Bio-ALBERT model, and field-optimized semantic embedding is generated through term mask prediction and attribute relationship pre-training; a two-layer heterogeneous graph is constructed using a graph attention interaction network (GAIN), cross-modal correlation weights are calculated through a bidirectional graph attention mechanism, and cross-modal graph features are output; multi-scale cross-modal correlations are modeled through a hierarchical graph attention fusion mechanism, the model is optimized in combination with a contrast alignment loss, and high-precision cyanobacteria recognition is achieved. Through multi-scale perception, field semantic adaptation and graph structure fusion, the application improves the accuracy of cyanobacteria detection in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and particularly relates to the field of cyanobacteria detection technology, specifically a cyanobacteria image recognition method and system based on hierarchical adaptation and domain-driven approaches. Background Technology

[0002] The rapid development of deep neural networks has improved the performance of image classification models.

[0003] In related technologies, most existing neural network systems are based on the assumption of independent and identically distributed data, that is, assuming that the training data (source domain data) and the test data (target domain data) have the same statistical distribution. However, in real-world applications, this assumption is often difficult to hold due to various factors, leading to a significant performance drop in well-trained models on test data with different distributions.

[0004] Currently, image classification domain generalization methods can effectively solve the above problems. However, existing image classification domain generalization methods can be mainly divided into two categories: one is based on data manipulation, and the other is based on alignment. The former aims to enhance source domain data through different image transformations or generate new source domain data to simulate target domain data; the latter is committed to reducing the representation differences between multiple source domains in a specific space and learning domain-invariant latent representations for the target domain.

[0005] In the field of cyanobacteria image classification, relevant research methods mainly include cyanobacteria image classification methods based on convolutional neural networks (CNNs), cyanobacteria image classification methods based on traditional machine learning algorithms, and cyanobacteria image classification methods based on deep learning (especially Transformer models). Among them, the CNN-based methods classify and detect cyanobacteria images by designing CNN models; the traditional machine learning algorithm-based methods rely on manual feature extraction and use traditional machine learning algorithms such as support vector machines (SVMs) and k-nearest neighbors (KNNs) to classify cyanobacteria images; and the deep learning-based methods use deep learning models, especially the global self-attention mechanism of the Transformer model, to classify cyanobacteria images, often employing a joint network architecture of deep convolutional and Transformer networks to improve classification performance.

[0006] Currently, cyanobacteria image classification methods based on Convolutional Neural Networks (CNNs) have certain limitations, namely the lack of local receptive fields and global information. Specifically, traditional CNN models extract features layer by layer through local convolutional kernels, and their limited receptive fields make it difficult to capture global dependencies across regions in an image. In cyanobacteria recognition scenarios, due to water disturbance, lighting changes, and the irregularity of cyanobacteria community distribution (such as filamentous structures spanning the entire image), CNN models cannot effectively model long-distance spatial relationships, leading to a higher false negative rate for fragmented cyanobacteria clusters or targets with blurred edges. Furthermore, while pooling operations can improve translation invariance, they lose crucial detail information (such as cell wall texture), weakening the ability to distinguish between microscopic cyanobacteria subclasses.

[0007] The technical bottleneck of cyanobacteria identification methods based on traditional machine learning algorithms lies in the coupling defects between manual feature engineering and complex background interference. Specifically, traditional methods (such as SVM and random forest) rely on manually designed features (such as color moments and LBP textures), whose feature representation capabilities are limited by expert prior knowledge and are difficult to adapt to the variability of cyanobacteria images. For example, cyanobacteria and suspended particles and algal debris in water bodies highly overlap in color space, and manual features cannot extract their microscopic morphological differences (such as pseudo-vacuolar structures). At the same time, complex background noise can contaminate the feature statistical distribution, leading to blurred decision boundaries for classifiers and misclassifying highly similar non-target objects (such as aquatic plant textures) as cyanobacteria, seriously affecting the robustness of real-world scenarios.

[0008] Finally, general deep learning image recognition methods are inefficient. While existing deep learning frameworks (such as ResNet and DenseNet) improve accuracy through deep networks, their massive parameter count requires training with huge amounts of labeled data and GPU clusters. However, data acquisition in the cyanobacteria field is costly (requiring microscopic imaging and biomarkers), and overfitting is easily caused under small sample conditions. In addition, such models are sensitive to input scale, and multi-scale cyanobacteria imaging requires repeated data augmentation and model fine-tuning, significantly increasing deployment complexity. Although techniques such as knowledge distillation can compress models, the accuracy loss is still significant in resource-constrained edge devices (such as water quality monitoring buoys). Summary of the Invention

[0009] The purpose of this invention is to provide a cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approach, so as to at least solve the technical problems of existing technologies that are difficult to detect cyanobacteria in complex aquatic environments and have low detection accuracy.

[0010] To achieve the above objectives, the present invention provides the following technical solution.

[0011] According to one embodiment of this application, a method for cyanobacteria image recognition based on hierarchical adaptation and domain-driven approach is provided;

[0012] The identification method includes the following steps:

[0013] Multispectral images of target water samples were acquired, and the images were enhanced using an improved dark channel algorithm. The dark channel algorithm includes fusing RGB-NIR four-dimensional spectral features and introducing a dynamic attenuation coefficient to adjust the near-infrared channel gradient. The enhanced images were processed using a hierarchical adaptive Swin Transformer model to output multi-scale visual feature vectors.

[0014] A text database of cyanobacterial biological attributes was constructed. The biological attribute text was enhanced by synonym replacement, sentence restructuring and attribute relation triple expansion. The enhanced biological attribute text was used as the input of the domain-driven Bio-ALBERT model to generate a semantic embedding matrix. Each row of the matrix corresponds to a biological attribute feature description, and the text feature vector is obtained.

[0015] The visual feature vector and text feature vector are aligned using the graph attention interaction network GAIN to construct a heterogeneous graph containing visual nodes and text attribute nodes. Cross-modal association weights are calculated using a bidirectional graph attention mechanism, and the cross-modal graph feature matrix is ​​generated by aggregation to obtain cross-modal fused features. The visual features and cross-modal fused features are concatenated, and a hierarchical graph attention fusion mechanism is adopted to achieve cross-modal feature aggregation through a multi-layer graph structure and hierarchical attention, and finally the fused feature vector is output.

[0016] The extracted fusion feature vectors are used to make decisions to identify cyanobacteria in the image.

[0017] As a further limitation of the embodiment of the present invention, in the step of constructing the text database of cyanobacterial biological attributes, the constructed text database of cyanobacterial biological attributes includes annotation information and a text feature dictionary; wherein, the annotation information includes cyanobacterial presence tags, cyanobacterial species tags and density grading tags; the text feature dictionary includes biological attribute descriptions and environmental parameters.

[0018] As a further limitation of the embodiment of the present invention, in the improved dark channel algorithm, the pixel values ​​of the four channels of red, green, blue and near-infrared are minimized in the neighborhood Ω(x) centered on the pixel x of the multispectral image to calculate the dark channel of the image, and the near-infrared channel is introduced to construct a four-dimensional spectral feature space RGB-NIR.

[0019] Background suppression is achieved by adjusting the gradient of the near-infrared channel in real time using a dynamic attenuation coefficient.

[0020] Building upon the previous methods of highlighting dark areas and dynamically suppressing the background, the original image is enhanced using a hyperbolic tangent image restoration strategy.

[0021] As a further limitation of the embodiment of the present invention, the step of processing the enhanced image using the hierarchical adaptive Swin Transformer model includes: dividing the image into blocks, extracting multi-scale visual features through four stages of hierarchical window attention calculation and window shifting operation, and enhancing the representation of key spectral channels using a dynamic spectral attention mechanism to obtain a multi-scale visual feature vector; wherein:

[0022] For the enhanced input image with dimensions of 244×244×3 First, it is divided into multi-scale blocks. Visual features are extracted through four stages of hierarchical window attention calculation and window shifting operations, and a dynamic spectral attention mechanism is adopted.

[0023]

[0024]

[0025] In the formula, Indicates the input spectral features, For the c-th spectral channel feature, GAP represents global average pooling. This is a learnable parameter matrix used to calculate the attention weights for the spectral channels; Represents multi-scale visual feature vectors;

[0026] By enhancing the representation of key spectral channels through dynamic spectral attention, a multi-scale visual feature vector is ultimately output. .

[0027] As a further limitation of the embodiment of the present invention, in the domain-driven Bio-ALBERT model, semantic understanding is optimized through term mask prediction and attribute relationship pre-training tasks, and a hybrid embedding layer is used to fuse word vectors and biological attribute labels.

[0028] As a further limitation of the embodiment of the present invention, after aggregating and generating a cross-modal graph feature matrix, a hierarchical graph fusion network is used to perform cross-modal feature fusion, including:

[0029] A bottom-level heterogeneous graph is constructed using 14×14 visual nodes and 30 text attribute nodes; a high-level heterogeneous graph is constructed using 49 visual supernodes and 30 text attributes generated through clustering.

[0030] Cross-modal association weights are calculated using a bidirectional graph attention mechanism. The underlying association weights are represented as follows:

[0031]

[0032] The weight of high-level associations is expressed as follows:

[0033]

[0034] Aggregate node features to generate cross-modal graph feature matrix The aggregated features of the bottom-level graph are represented as follows:

[0035]

[0036] The feature matrix of the bottom layer graph is represented as follows:

[0037]

[0038] High-level graph feature aggregation is represented as:

[0039]

[0040] The high-level graph feature matrix is ​​represented as follows:

[0041]

[0042] Cross-layer integration is represented as:

[0043]

[0044] The hierarchical gating coefficients are respectively expressed as:

[0045]

[0046]

[0047] In the formula, Let be the attention weight between the i-th visual node and the j-th text node in the bottom layer graph; This is a learnable parameter matrix used to calculate the attention score after feature concatenation; Let i be the feature of the i-th visual node, with dimension 1. ; Let j be the feature of the j-th text node, with dimension 1. ; Let be the set of neighbors of visual node i; || represents the splicing operation; Indicates the activation function;

[0048] Let m be the attention weight between the m-th visual supernode and the n-th text node in the high-level graph;

[0049] It represents a learnable parameter matrix that adapts to the concatenation dimension of supernodes and text features;

[0050] Let m be the feature vector of the m-th visual supernode, with dimension 1. ; The feature of the nth text node has a dimension of ; Let m be the set of neighbors of the visual supernode m. This is the intermediate feature vector of the i-th visual node in the bottom layer graph; The value projection matrix maps text features to the aggregation dimension. ; Let be the alignment feature vector of the m-th visual supernode in the high-level graph; The value projection matrix maps text features to higher-level aggregation dimensions. GAP indicates global average pooling. For visual feature dimensions; For text feature dimensions; For visual supernode feature dimensions; These are the underlying aggregation dimension and the high-level aggregation dimension, respectively.

[0051] As a further limitation of the embodiments of the present invention, in In the cross-modal feature fusion of hierarchical graph fusion networks, hierarchical gating coefficients are calculated based on the global context;

[0052] The hierarchical gating coefficients are respectively expressed as:

[0053]

[0054]

[0055] in, It is a learnable parameter matrix, where D is the total dimension of the concatenated features, used to dynamically adjust the contribution of low-level and high-level features.

[0056] As a further limitation of the embodiments of the present invention, in In hierarchical graph fusion networks for cross-modal feature fusion, a hierarchical multi-head attention mechanism based on global context is computed, as follows:

[0057]

[0058]

[0059] In the formula, variables This indicates a hierarchical index, representing the first and second levels constructed. This represents the head index in a multi-head attention mechanism, where the number of attention heads is . ; Let i be the set of neighboring nodes of node i; This is the attention vector, used to calculate the attention score for the concatenated features; The purpose of this is to reduce the dimensionality of node features by using a learnable weight matrix. Let be the feature vectors of the i-th and j-th nodes in layer l.

[0060] As a further limitation of the embodiment of the present invention, in the step of outputting the final feature vector after adaptive fusion, the concatenated two-stream features are fused based on the hierarchical gating coefficient, as follows:

[0061]

[0062] In the formula, Indicates the hierarchical gating coefficient; Indicates the final fused features; || represents the splicing operation; Represents the feature matrix of the cross-modal graph; This represents a visual feature vector.

[0063] As a further limitation of the embodiment of the present invention, it also includes: constructing a multi-task loss function that includes a contrast alignment loss term, dynamically balancing the weights of classification and regression tasks, and in model training, first freezing the underlying parameters of the Swin Transformer model and the domain-driven Bio-ALBERT model, and training only the fusion module and the classification head; then unfreezing all parameters and using a linear warmup strategy for training.

[0064] According to another embodiment of this application, the present invention provides a recognition system for implementing the cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approach provided in the above embodiments, comprising the following modules:

[0065] The visual feature extraction module is used to acquire multispectral images of target water samples and enhance the images using an improved dark channel algorithm. The dark channel algorithm includes fusing RGB-NIR four-dimensional spectral features and introducing a dynamic attenuation coefficient to adjust the near-infrared channel gradient. The enhanced image is processed using a hierarchical adaptive Swin Transformer model to output multi-scale visual feature vectors.

[0066] The text feature extraction module is used to construct a cyanobacterial biological attribute text database. It enhances the biological attribute text by using synonym replacement, sentence recombination and attribute relation triple expansion. The enhanced biological attribute text is used as the input of the domain-driven Bio-ALBERT model to generate a semantic embedding matrix. Each row of the matrix corresponds to a biological attribute feature description, resulting in a text feature vector.

[0067] The feature fusion module aligns visual feature vectors and text feature vectors through a graph attention interaction network (GAIN) to construct a heterogeneous graph containing visual nodes and text attribute nodes. It then uses a bidirectional graph attention mechanism to calculate cross-modal association weights, aggregates them to generate a cross-modal graph feature matrix, and obtains cross-modal fused features. Finally, it concatenates the visual features and cross-modal fused features and employs a hierarchical graph attention fusion mechanism. Through a multi-layer graph structure and hierarchical attention, it achieves cross-modal feature aggregation and outputs the fused feature vector.

[0068] The cyanobacteria identification module is used to make decisions based on the extracted fusion feature vectors to identify cyanobacteria in images.

[0069] Compared with existing technologies, the beneficial effects of the cyanobacteria image recognition method and system based on hierarchical adaptation and domain-driven approaches of this invention are:

[0070] This invention enhances the capture of background information, including suspended particles, by introducing a near-infrared channel to construct a four-dimensional spectral feature space; it adjusts background suppression in real time based on the near-infrared channel gradient by designing a dynamic attenuation coefficient; it reduces interference by adopting a hyperbolic tangent recovery strategy, and improves image quality by combining denoising and adaptive enhancement, thus providing accurate input for the detection model.

[0071] This invention constructs a cross-modal graph attention interaction architecture, extracts multi-scale visual features through hierarchical adaptive SwinTransformer branches, optimizes the semantic understanding of biological attributes through domain-driven Bio-ALBERT branches, and realizes complex visual-text association modeling through the GAIN module, breaking down semantic barriers. Furthermore, it adopts hierarchical graph attention to fuse multi-modal features, preserves the long-range topological structure features of cyanobacteria, and dynamically adjusts weights by combining multi-task learning, significantly improving recognition accuracy and inference efficiency of edge devices. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.

[0073] Figure 1 This is a flowchart illustrating the implementation of the cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approach of the present invention.

[0074] Figure 2 This is a structural block diagram of the cyanobacteria image recognition system based on hierarchical adaptation and domain-driven architecture of the present invention;

[0075] Figure 3 A structural block diagram of a computer device provided by the present invention. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0077] Unlike existing cyanobacteria image recognition methods, this invention provides a cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approaches. In this method, multispectral images of target water samples are acquired, and the images are enhanced using an improved dark channel algorithm. A text database of cyanobacteria biological attributes is constructed and synonym replacement and sentence recombination are performed. Multi-scale visual features are extracted through a hierarchical adaptive SwinTransformer model, and key region representation is enhanced by combining dynamic spectral attention.

[0078] Text is input into the Bio-ALBERT model, which generates domain-optimized semantic embeddings through term mask prediction and attribute relationship pre-training; the graph attention interaction network GAIN is used to model complex visual-textual relationships and output cross-modal graph features; cross-modal feature adaptive aggregation is achieved through a hierarchical graph fusion network, and the model is optimized by combining contrastive alignment loss; thus achieving high-precision cyanobacteria identification.

[0079] This invention significantly improves the accuracy of detecting cyanobacteria in complex environments by using multi-scale perception, domain semantic adaptation, and graph structure fusion.

[0080] The following describes in detail the specific implementation of the cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approach of the present invention with reference to specific embodiments.

[0081] Please refer to Figure 1 In one embodiment of the present invention, a method for cyanobacteria image recognition based on hierarchical adaptation and domain-driven approach is provided, comprising the following steps:

[0082] S101. Acquire multispectral images of the target water sample and enhance the images using an improved dark channel algorithm. The dark channel algorithm includes fusing RGB-NIR four-dimensional spectral features and introducing a dynamic attenuation coefficient to adjust the near-infrared channel gradient. The enhanced image is processed using a hierarchical adaptive Swin Transformer model to output a multi-scale visual feature vector.

[0083] Specifically, for the acquisition of multispectral images, this invention utilizes an underwater drone equipped with a multispectral camera to acquire water images from all directions, simultaneously deploys various sensors to collect environmental parameters such as light intensity and water temperature, collects water samples to obtain cyanobacterial biological attribute data, and constructs a text database of cyanobacterial biological attributes.

[0084] S102. Construct a text database of cyanobacterial biological attributes. Enhance the biological attribute text by using synonym replacement, sentence restructuring and attribute relation triple expansion. Use the enhanced biological attribute text as input to the domain-driven Bio-ALBERT model to generate a semantic embedding matrix. Each row of the matrix corresponds to a biological attribute feature description, and the text feature vector is obtained.

[0085] In the domain-driven Bio-ALBERT model of this invention, semantic understanding is optimized through term mask prediction and attribute relationship pre-training tasks, and a hybrid embedding layer is used to fuse word vectors and biological attribute labels.

[0086] Furthermore, in the data augmentation process, the improved dark channel algorithm is used to suppress the background, random flipping and rotation are used to enhance the image, and synonym replacement and sentence restructuring are performed on the text to improve the usability and richness of multimodal data.

[0087] S103. Align visual feature vectors and text feature vectors through the graph attention interaction network GAIN to construct a heterogeneous graph containing visual nodes and text attribute nodes; calculate cross-modal association weights using a bidirectional graph attention mechanism, aggregate to generate a cross-modal graph feature matrix and obtain cross-modal fusion features; concatenate the visual features and cross-modal fusion features, and use a hierarchical graph attention fusion mechanism to achieve cross-modal feature aggregation through a multi-layer graph structure and hierarchical attention.

[0088] S104. Construct a multi-task loss function that includes a contrast alignment loss term, dynamically balance the weights of classification and regression tasks. During model training, first freeze the underlying parameters of the Swin Transformer model and the domain-driven Bio-ALBERT model, and train only the fusion module and the classification head. Then, unfreeze all parameters and use a linear warmup strategy for training. Finally, use the fusion feature vector extracted by the model to make decisions to identify cyanobacteria in the image.

[0089] Specifically, in this embodiment, the acquired input image is x, and the constructed cyanobacterial biological attribute text database includes annotation information and a text feature dictionary; wherein, the annotation information includes:

[0090] Cyanobacteria Presence Tag: y exist ∈{0,1};

[0091] Blue-green algae species tag: y species ∈{1, ..., C};

[0092] Density grading label: y density ∈{Ⅰ,Ⅱ,Ⅲ};

[0093] In addition, the text feature dictionary includes: biological attribute descriptions (such as 30 features such as pseudocavitation reflectivity and filament length ratio) and environmental parameters (light intensity, water temperature, etc.).

[0094] Furthermore, in this embodiment of the invention, in order to improve image quality and suppress the influence of interference factors such as suspended matter in water on cyanobacteria detection, a series of image enhancement techniques are adopted;

[0095] Specifically, this invention improves upon the traditional dark channel prior algorithm. In the improved dark channel algorithm, the pixel values ​​of the four channels—red (R), green (G), blue (B), and near-infrared (NIR)—are minimized within a neighborhood Ω(x) centered on pixel x in the multispectral image to calculate the dark channel I of the image. dark (x) By introducing a near-infrared (NIR) channel to construct a four-dimensional spectral feature space RGB-NIR, image information can be captured more comprehensively, highlighting dark areas containing background information such as suspended matter in water, laying the foundation for subsequent background suppression and image enhancement.

[0096] Among them, the dark channel I of the present invention dark The expression for calculating (x) is:

[0097] ;

[0098] In the formula, R, G, B, and NIR represent the four channels: red, green, blue, and near-infrared, respectively; Ω(x) represents the central neighborhood; and x represents a pixel. y represents the pixel value of the image in channel c; y is the pixel in the neighborhood Ω(x) centered at pixel x.

[0099] This invention enhances the capture of background information, including suspended matter, by introducing a near-infrared channel to construct a four-dimensional spectral feature space.

[0100] To further optimize the background suppression effect, this invention introduces a dynamic attenuation coefficient λ(x). Specifically, the dynamic attenuation coefficient λ(x) replaces the fixed parameter, and the background suppression effect is dynamically adjusted according to the gradient intensity of the near-infrared channel. This dynamic adjustment mechanism can flexibly suppress the background according to the gradient changes of the near-infrared channel in different regions of the image, avoiding the problem of over-suppression or under-suppression that may occur in different scenarios with fixed parameters, making the background suppression more accurate.

[0101] Therefore, this invention uses a dynamic attenuation coefficient to adjust the gradient of the near-infrared channel in real time to suppress background.

[0102] In one implementation of this disclosure, the dynamic attenuation coefficient λ(x) is expressed as:

[0103]

[0104] In the formula, This represents the gradient strength of the near-infrared channel; k represents the adjustment factor, with a value of 3; ||·|| represents the norm operation of the vector, used to calculate the magnitude of the gradient;

[0105] Furthermore, building upon the previous methods of highlighting dark areas and dynamically suppressing the background, this invention also utilizes a hyperbolic tangent image restoration strategy to enhance the original image, resulting in an enhanced image. , is represented as:

[0106] ;

[0107] In the formula, ε is a constant with a value of 0.01, and t represents the original image; It is a Gaussian kernel with a standard deviation of σ. This means that in this formula, element-wise multiplication is performed, multiplying the elements at corresponding positions to obtain a new matrix of the same dimension;

[0108] Therefore, this invention employs a hyperbolic tangent recovery strategy to reduce interference, and combines denoising and adaptive enhancement to improve image quality and provide accurate input for the detection model.

[0109] Furthermore, in terms of image enhancement, this invention also increases the diversity of images and improves the model's adaptability to cyanobacteria images in different scenarios by randomly combining operations such as flipping (probability of 0.5), rotating (angle range of ±30°) and color jittering (brightness change range of ±0.2, contrast change range of ±0.3).

[0110] For text enhancement, this invention performs synonym replacement and sentence restructuring on biological attribute text, making the text expression richer and more diverse, thereby improving the generalization of text semantics and better integrating it with image information.

[0111] This invention enhances the capture of background information, including suspended particles, by introducing a near-infrared channel to construct a four-dimensional spectral feature space; it also improves the image quality by designing a dynamic attenuation coefficient and adjusting background suppression in real time based on the near-infrared channel gradient; and it employs a hyperbolic tangent recovery strategy to reduce interference, combined with denoising and adaptive enhancement, thus providing accurate input for the detection model.

[0112] Furthermore, in step S103, the process of processing the enhanced image using the hierarchical adaptive Swin Transformer model specifically includes:

[0113] The enhanced image is segmented into blocks. The Swin Transformer model extracts multi-scale visual features through four stages of hierarchical window attention calculation and window shifting operation, and uses a dynamic spectral attention mechanism to enhance the representation of key spectral channels, thereby obtaining multi-scale visual feature vectors.

[0114] Specifically, for the enhanced input image with dimensions of 244×244×3... First, it is divided into multi-scale blocks. Visual features are extracted through four stages of hierarchical window attention calculation and window shifting operations. Then, a dynamic spectral attention mechanism is used to enhance the representation of key spectral channels, as shown below:

[0115]

[0116]

[0117] In the formula, Indicates the input spectral features, For the c-th spectral channel feature, GAP represents global average pooling. This is a learnable parameter matrix used to calculate the attention weights for the spectral channels, i.e. This represents the attention weight of the c-th spectral channel in the dynamic spectral attention mechanism; Represents multi-scale visual feature vectors;

[0118] Then, this invention enhances the representation of key spectral channels through dynamic spectral attention, and finally outputs a multi-scale visual feature vector. .

[0119] Furthermore, in step S105, the process of aggregating and generating the cross-modal graph feature matrix specifically includes:

[0120] A heterogeneous graph is constructed from 14×14 visual nodes and 30 text attribute nodes.

[0121] A high-level heterogeneous graph was constructed by generating 49 visual supernodes and 30 text attributes through clustering;

[0122] Among them, cross-modal association weights are calculated using a bidirectional graph attention mechanism. The underlying association weights are represented as follows:

[0123]

[0124] The weight of high-level associations is expressed as follows:

[0125]

[0126] Aggregate node features to generate cross-modal graph feature matrix The aggregated features of the bottom-level graph are represented as follows:

[0127]

[0128] The feature matrix of the bottom layer graph is represented as follows:

[0129]

[0130] High-level graph feature aggregation is represented as:

[0131]

[0132] The high-level graph feature matrix is ​​represented as follows:

[0133]

[0134] Cross-layer fusion is represented as:

[0135]

[0136] In the formula, Let be the attention weight between the i-th visual node and the j-th text node in the bottom layer graph;

[0137] This is a learnable parameter matrix used to calculate the attention score after feature concatenation;

[0138] Let i be the feature of the i-th visual node, with dimension 1. ;

[0139] Let j be the feature of the j-th text node, with dimension 1. ;

[0140] Let be the set of neighbors of visual node i; || represents the splicing operation; This represents the activation function, used to map the input to the [0,1] interval;

[0141] Let m be the attention weight between the m-th visual supernode and the n-th text node in the high-level graph;

[0142] This represents the learnable parameter matrix, used to adapt the concatenation dimension of supernodes and text features;

[0143] Let m be the feature vector of the m-th visual supernode, with dimension 1. ;

[0144] The feature of the nth text node has a dimension of ;

[0145] Let m be the set of neighbors of the visual supernode m.

[0146] This is the intermediate feature vector of the i-th visual node in the bottom layer graph; The value projection matrix maps text features to the aggregation dimension. ; Let be the alignment feature vector of the m-th visual supernode in the high-level graph; The value projection matrix maps text features to higher-level aggregation dimensions. GAP indicates global average pooling. For visual feature dimensions; For text feature dimensions; For visual supernode feature dimensions; These are the underlying aggregation dimension and the high-level aggregation dimension, respectively.

[0147] This invention constructs a cross-modal graph attention interaction architecture, extracts multi-scale visual features through hierarchical adaptive SwinTransformer branches, optimizes the semantic understanding of biological attributes through domain-driven Bio-ALBERT branches, and realizes complex visual-text association modeling through the GAIN module, thus breaking down semantic barriers.

[0148] Furthermore, in step S106 of this embodiment, in the step of generating dynamic weights using a hierarchical graph fusion network, hierarchical gating coefficients are calculated based on the global context. The calculation of the hierarchical gating coefficients is expressed as follows:

[0149]

[0150]

[0151] in, It is a learnable parameter matrix, where D is the total dimension of the concatenated features, used to dynamically adjust the contribution of low-level and high-level features.

[0152] Furthermore, in this embodiment of the application, the cross-modal graph feature matrix is... With visual feature vectors In the step of inputting the hierarchical graph fusion network, a two-layer hierarchical graph structure is constructed. The first layer consists of 196 visual nodes and 30 text nodes; the second layer consists of 49 visual supernodes and 30 text nodes. A hierarchical multi-head attention mechanism is applied, which involves two types of computation: inter-layer propagation weight calculation and node feature update, as shown below:

[0153]

[0154]

[0155] In the formula, variables This indicates a hierarchical index, representing the first and second levels constructed. This represents the head index in a multi-head attention mechanism, where the number of attention heads is . ; Let i be the set of neighboring nodes of node i; This is the attention vector, used to calculate the attention score for the concatenated features; This is a learnable weight matrix used to reduce the dimensionality of node features; Let be the feature vectors of the i-th and j-th nodes in layer l.

[0156] Furthermore, in the process of outputting the final feature vector after adaptive fusion in step S106, the features are fused based on the hierarchical gating coefficient, as shown below:

[0157]

[0158] In the formula, Indicates the hierarchical gating coefficient; Indicates the final fused features; || represents the splicing operation; Represents the feature matrix of the cross-modal graph; This represents a visual feature vector.

[0159] As can be seen, the present invention adopts a hierarchical graph fusion network to adaptively aggregate multimodal features, retains the long-range topological features of cyanobacteria through a two-layer graph structure, and dynamically adjusts the multi-scale contribution by combining a gating mechanism, which significantly improves the recognition accuracy and the inference efficiency of edge devices.

[0160] Furthermore, in step S107 of this embodiment, the multi-task loss function provided in this embodiment is expressed as:

[0161]

[0162]

[0163]

[0164]

[0165]

[0166] In the formula, weight α = 1.0; weight β = 0.8; weight γ = 0.6; δ = 0.4; τ represents the temperature coefficient, with a value of 0.07; s represents the similarity function, used to calculate the similarity between the visual feature vector and the text feature vector; This represents the cross-modal graph feature vector of the j-th negative sample; Represents visual feature vectors; Represents the feature matrix of the cross-modal graph; This is the total loss function; The existence loss function; For the overall class loss function; For density grading loss function, To compare the alignment loss function; A label indicating actual existence; The probability of existence predicted by the model; Labels for actual categories; The probability distribution of the types predicted by the model; This represents the true density level; The density level probability predicted by the model; For focus loss function; The label smoothing loss function; This is the ordinal regression loss function.

[0167] Furthermore, this invention divides the model training into two stages:

[0168] Phase 1 is the feature adaptation phase, where the underlying parameters of the Bio-ALBERT and Swin Transformer models are frozen, and only the fusion module and classification head are trained; the Lion optimizer (β1=0.95, β2=0.98) is used with a learning rate of 3×10⁻⁶. -5 ;

[0169] Phase two is the global fine-tuning phase, where all parameters are unfrozen, and a linear warmup strategy is adopted (peak learning rate of 5×10). -6 The weight decay coefficient is 0.01 to prevent overfitting.

[0170] Finally, during the deployment and verification of the detection model of this invention, the Taihu dataset was used to conduct a comprehensive performance verification of the trained model.

[0171] The comparative experiment used ResNet-101 and Swin-T as baseline models and compared them with the model of this invention.

[0172] The ablation experiment compared pure Swing Transformer with Bio-ALBERT+Swin Transformer, and compared spatial channel gating fusion with splicing fusion.

[0173] Based on the verification and experimental results, the qualified models will be deployed to real-world application scenarios for monitoring and analyzing cyanobacteria in water bodies.

[0174] Please refer to Figure 2In another embodiment of this application, a cyanobacteria image recognition system based on hierarchical adaptation and domain-driven approach is provided. This recognition system includes the following modules:

[0175] The visual feature extraction module 201 is used to acquire multispectral images of the target water sample and enhance the images using an improved dark channel algorithm. The dark channel algorithm includes fusing RGB-NIR four-dimensional spectral features and introducing a dynamic attenuation coefficient to adjust the near-infrared channel gradient. The enhanced image is processed using a hierarchical adaptive Swin Transformer model to output a multi-scale visual feature vector.

[0176] The text feature extraction module 202 is used to construct a cyanobacterial biological attribute text database. It enhances the biological attribute text by using synonym replacement, sentence recombination and attribute relationship triple expansion. The enhanced biological attribute text is used as the input of the domain-driven Bio-ALBERT model to generate a semantic embedding matrix. Each row of the matrix corresponds to a biological attribute feature description, thus obtaining a text feature vector.

[0177] The feature fusion module 203 is used to align visual feature vectors and text feature vectors through the graph attention interaction network GAIN to construct a heterogeneous graph containing visual nodes and text attribute nodes; calculate cross-modal association weights using a bidirectional graph attention mechanism and aggregate them to generate a cross-modal graph feature matrix; then, a hierarchical graph fusion network is used to perform cross-modal feature fusion: first, a two-layer hierarchical graph structure is constructed, and the bottom and high-level features are aggregated respectively through a hierarchical graph attention mechanism; then, cross-layer features are fused based on hierarchical gating coefficients, and the original visual features and cross-modal graph features are weighted and concatenated to output the final feature vector;

[0178] The cyanobacteria identification module 204 is used to construct a multi-task loss function that includes a contrast alignment loss term and dynamically balances the weights of classification and regression tasks. During model training, the underlying parameters of the Swin Transformer model and the domain-driven Bio-ALBERT model are first frozen, and only the fusion module and the classification head are trained. Then, all parameters are unfrozen, and a linear warmup strategy is used for training. Finally, the fusion feature vector extracted by the model is used to make decisions to identify cyanobacteria in the image.

[0179] In yet another embodiment of the present invention, a computer device is also provided.

[0180] Please refer to the details. Figure 3 , Figure 3 This is a basic structural block diagram of the computer device in this embodiment;

[0181] The computer device 300 includes a memory, a processor, and a network interface that are interconnected via a system bus.

[0182] It should be noted that the figure only shows a computer device 300 with component-connected memory, processor, and network interface. However, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, embedded devices, etc.

[0183] The network interface may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 300 and other electronic devices.

[0184] The present invention also provides another embodiment, namely, a computer-readable storage medium storing an interface display program, which can be executed by at least one processor to cause the at least one processor to perform the steps of the above-described hierarchical adaptive and domain-driven cyanobacteria image recognition method.

[0185] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0186] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0187] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, whether directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for cyanobacteria image recognition based on hierarchical adaptation and domain-driven approach, characterized in that, Includes the following steps: Multispectral images of target water samples were acquired, and the images were enhanced using an improved dark channel algorithm. The dark channel algorithm includes fusing RGB-NIR four-dimensional spectral features and introducing a dynamic attenuation coefficient to adjust the gradient of the near-infrared channel. The enhanced image was processed using a hierarchical adaptive Swin Transformer model to output a multi-scale visual feature vector. In the improved dark channel algorithm, the pixel values ​​of the four channels (red, green, blue, and near-infrared) are minimized within a neighborhood Ω(x) centered on the pixel x of the multispectral image to calculate the dark channel of the image, and the near-infrared channel is introduced to construct a four-dimensional spectral feature space RGB-NIR. Background suppression is achieved by adjusting the gradient of the near-infrared channel in real time through a dynamic attenuation coefficient. Based on highlighting dark areas and dynamically suppressing the background, the original image is enhanced using a hyperbolic tangent image restoration strategy. A text database of cyanobacterial biological attributes was constructed. The biological attribute text was enhanced by synonym replacement, sentence restructuring and attribute relation triple expansion. The enhanced biological attribute text was used as the input of the domain-driven Bio-ALBERT model to generate a semantic embedding matrix. Each row of the matrix corresponds to a biological attribute feature description, and the text feature vector is obtained. By aligning visual feature vectors and text feature vectors through the graph attention interaction network GAIN, a cross-modal multi-level graph structure containing a low-level feature association layer and a high-level semantic aggregation layer is constructed. A heterogeneous graph is constructed using visual nodes and text attribute nodes. Cross-modal association weights are calculated using a bidirectional graph attention mechanism and aggregated to generate a cross-modal graph feature matrix, resulting in cross-modal fused features. The visual features and cross-modal fused feature vectors are input into a hierarchical graph fusion network. By constructing a two-layer hierarchical graph structure and applying a hierarchical attention mechanism, the final feature vector is output after adaptive fusion. The extracted fusion feature vectors are used to make decisions to identify cyanobacteria in the image.

2. The cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approach according to claim 1, wherein the constructed cyanobacteria biological attribute text database includes annotation information and a text feature dictionary in the step of constructing the cyanobacteria biological attribute text database; The labeling information includes cyanobacteria presence tags, cyanobacteria species tags, and density grading tags; The text feature dictionary includes descriptions of biological attributes and environmental parameters.

3. The cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approach according to claim 2, characterized in that, The steps of processing the enhanced image using the hierarchical adaptive Swin Transformer model include: The image is segmented into blocks, and multi-scale visual features are extracted through a four-stage hierarchical window attention calculation and window shifting operation. A dynamic spectral attention mechanism is then employed to enhance the representation of key spectral channels, resulting in a multi-scale visual feature vector. Wherein: For the enhanced input image with dimensions of 244×244×3 First, the image The system is segmented into multi-scale blocks, and visual features are extracted through a four-stage hierarchical window attention calculation and window shifting operation. A dynamic spectral attention mechanism is employed, as shown below: ； ； In the formula, Indicates the input spectral features, For the c-th spectral channel feature, GAP represents global average pooling. This represents the attention weight of the c-th spectral channel in the dynamic spectral attention mechanism. This is a learnable parameter matrix used to calculate the attention weights for the spectral channels; Represents multi-scale visual feature vectors; By enhancing the representation of key spectral channels through dynamic spectral attention, a multi-scale visual feature vector is ultimately output. .

4. The cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approach according to claim 3, characterized in that, In the domain-driven Bio-ALBERT model, semantic understanding is optimized through term mask prediction and attribute relationship pre-training tasks, and a hybrid embedding layer is used to fuse word vectors and biological attribute labels.

5. The cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approach according to claim 4, characterized in that, The steps involved in aggregating and generating a cross-modal graph feature matrix include: A bottom-level heterogeneous graph is constructed using 14×14 visual nodes and 30 text attribute nodes; a high-level heterogeneous graph is constructed using 49 visual supernodes and 30 text attributes generated through clustering. Cross-modal association weights are calculated using a bidirectional graph attention mechanism. The underlying association weights are represented as follows: ； The weight of high-level associations is expressed as follows: ； Aggregate node features to generate cross-modal graph feature matrix The aggregated features of the bottom-level graph are represented as follows: ； The feature matrix of the bottom layer graph is represented as follows: ； High-level graph feature aggregation is represented as: ； The high-level graph feature matrix is ​​represented as follows: ； Cross-layer fusion is represented as: ； The hierarchical gating coefficients are expressed as follows: ； ； In the formula, Let be the attention weight between the i-th visual node and the j-th text node in the bottom layer graph; This is a learnable parameter matrix used to calculate the attention score after feature concatenation; Let i be the feature of the i-th visual node, with dimension 1. ; Let j be the feature of the j-th text node, with dimension 1. ; Let be the set of neighbors of visual node i; || represents the splicing operation; For activation functions; Let m be the attention weight between the m-th visual supernode and the n-th text node in the high-level graph; This represents the learnable parameter matrix, used to adapt the concatenation dimension of supernodes and text features; Let m be the feature vector of the m-th visual supernode, with dimension 1. ; The feature of the nth text node has a dimension of ; Let m be the set of neighbors of the visual supernode m. This is the intermediate feature vector of the i-th visual node in the bottom layer graph; The value projection matrix maps text features to the aggregation dimension. ; Let be the alignment feature vector of the m-th visual supernode in the high-level graph; The value projection matrix maps text features to higher-level aggregation dimensions. ; This represents the learnable parameter matrix, where D represents the total dimension of the concatenated features, used to dynamically adjust the contribution of low-level and high-level features. For visual feature dimensions; For text feature dimensions; For visual supernode feature dimensions, Indicates the underlying gating coefficient. represents the high-level gating coefficient; GAP represents global average pooling.

6. The cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approach according to claim 5, characterized in that, The cross-modal graph feature matrix With visual feature vectors Input hierarchical graph fusion network, including: Construct a two-layer hierarchical graph structure: the first layer consists of 196 visual nodes and 30 text nodes; the second layer consists of 49 visual supernodes and 30 text nodes. Applying a hierarchical multi-head attention mechanism, inter-layer propagation weight calculation and node feature update are performed separately, as follows: ； ； In the formula, variables This indicates a hierarchical index, representing the first and second levels constructed. This represents the head index in a multi-head attention mechanism, where the number of attention heads is . ; Let i be the set of neighboring nodes of node i; This is the attention vector, used to calculate the attention score for the concatenated features; This is a learnable weight matrix used to reduce the dimensionality of node features; Let be the feature vectors of the i-th and j-th nodes in layer l.

7. The cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approach according to claim 6, characterized in that, In the step of outputting the final feature vector after adaptive fusion, the visual features and cross-modal graph features are weighted and fused based on hierarchical gating coefficients, as follows: In the formula, , Indicates the hierarchical gating coefficient; Indicates the final fusion characteristics; || is for splicing operations; Represents the feature matrix of the cross-modal graph; This represents a visual feature vector.

8. The cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approach according to claim 7, characterized in that, Also includes: A multi-task loss function with a contrast alignment loss term is constructed to dynamically balance the weights of classification and regression tasks. During model training, the underlying parameters of the Swin Transformer model and the domain-driven Bio-ALBERT model are first frozen, and only the fusion module and the classification head are trained. Then, all parameters are unfrozen, and a linear warmup strategy is used for training.

9. A recognition system for implementing the cyanobacteria image recognition method based on hierarchical adaptation and domain-driven approach as described in any one of claims 1 to 8, characterized in that, Includes the following modules: The visual feature extraction module is used to acquire multispectral images of target water samples and enhance the images using an improved dark channel algorithm. The dark channel algorithm includes fusing RGB-NIR four-dimensional spectral features and introducing a dynamic attenuation coefficient to adjust the near-infrared channel gradient. The enhanced image is processed using a hierarchical adaptive Swin Transformer model to output a multi-scale visual feature vector. The text feature extraction module is used to construct a cyanobacterial biological attribute text database. It enhances the biological attribute text by using synonym replacement, sentence recombination and attribute relation triple expansion. The enhanced biological attribute text is used as the input of the domain-driven Bio-ALBERT model to generate a semantic embedding matrix. Each row of the matrix corresponds to a biological attribute feature description, resulting in a text feature vector. The feature fusion module aligns visual feature vectors and text feature vectors through the graph attention interaction network GAIN, constructing a heterogeneous graph containing visual nodes and text attribute nodes; it calculates cross-modal association weights using a bidirectional graph attention mechanism, aggregates and generates a cross-modal graph feature matrix, and obtains cross-modal fused features; it concatenates the visual features and cross-modal fused features, and uses a hierarchical graph attention fusion mechanism to achieve cross-modal feature aggregation through a two-layer hierarchical graph structure and hierarchical multi-head attention. The cyanobacteria identification module is used to make decisions based on the fused feature vectors extracted by the model in order to identify cyanobacteria in the image.

Images