A multi-modal hypergraph neural network-based accurate identification method for fungal species

Abstract

This application discloses a method for accurate fungal species identification based on a multimodal hypergraph neural network, comprising: acquiring multimodal data of a target fungal sample, including microscopic visual images, segmentation mask images, satellite remote sensing images, and text metadata; extracting features from the multimodal data respectively, and projecting the extracted multimodal features onto a semantic space of a preset dimension to obtain a fused feature vector; constructing a hypergraph structure containing multiple hyperedges using each fungal sample as a node and the fused feature vector as the initial feature of the node; inputting the hypergraph structure into a pre-constructed hypergraph neural network for information propagation and node feature updating; inputting the updated node features into a classifier to output the species identification result of the target fungal sample. This invention effectively improves the accuracy and robustness of fungal species identification in complex field environments, achieving efficient and accurate identification of fungal species.

Description

Technical Field

[0001] This application relates to the fields of computer vision and bioinformatics processing technology, and more specifically, to a method for accurate identification of fungal species based on a multimodal hypergraph neural network. Background Technology

[0002] Fungi are an important component of ecosystems, and their accurate species identification has crucial applications in biodiversity conservation, ecological environment monitoring, and agricultural disease control. However, existing automatic fungal identification technologies face numerous technical challenges due to limitations in technical principles and data utilization methods. The identification accuracy and practical application effects are insufficient to meet requirements. Specific problems include: (1) Traditional identification methods have significant limitations and neglect the role of the environment in the selection of fungal species. Traditional fungal identification mainly relies on experts to observe fungi under a microscope and describe macroscopic morphological characteristics. At the same time, molecular biology methods use DNA sequencing as the "gold standard" for fungal identification. However, manual identification is time-consuming and labor-intensive, and the identification results are highly dependent on expert experience, which can easily lead to human error. Although molecular sequencing provides accurate identification results, the required experimental equipment is expensive and the detection cost is high, making it impossible to achieve large-scale real-time application in the field. In addition, some fungal species have significant intraspecific variations, which leads to blurred classification boundaries at the molecular level. Furthermore, fungal growth is strictly constrained by macroscopic environmental factors such as latitude, vegetation type, and humidity. However, existing technologies mostly rely on microscopic images of fungi for identification, ignoring the important biological prior knowledge of "environmental selection of species," resulting in a lack of identification basis.

[0003] (2) Single visual recognition models based on deep learning lack fine-grained discrimination ability and are significantly affected by data distribution. Existing single visual models based on convolutional neural networks, visual Transformers, etc. can perform end-to-end feature extraction and classification of fungal photos. However, the fungal kingdom generally has the characteristics of "high similarity between classes" and "drastic variation within classes". Such models are difficult to capture the decisive subtle features used for species differentiation and cannot achieve accurate identification of similar fungi. At the same time, such models are significantly affected by the long tail distribution of data. For rare fungal species with few samples, they are prone to serious overfitting or direct recognition failure due to insufficient training data.

[0004] (3) Basic multimodal fusion schemes suffer from shallow fusion levels and insufficient utilization of geometric features. Some schemes attempt to use metadata such as geographic coordinates and collection time as auxiliary variables, and simply concatenate them with fungal image features before inputting them into the classifier. However, this approach only treats environmental factors as independent discrete variables and fails to establish a deep coupling logic between "habitat-morphology". The fusion level is shallow and cannot fully leverage the complementary role of multimodal data. Furthermore, in existing technologies, the masking of fungal segmentation masks is only used as a noise reduction tool, ignoring the physical geometric features such as cap roundness and edge jaggedness behind the mask. These geometric features are important bases for fungal taxonomy, resulting in the loss of key features.

[0005] (4) Traditional graph neural network schemes cannot model high-order relationships and are prone to introducing noise when dealing with the heterogeneity of multimodal data. Traditional graph neural networks transmit information through pairwise relationships and can only handle one-to-one associations between samples. They cannot express complex high-order relationships such as "niche overlap" and "group similarity" between multiple samples, and it is difficult to fully characterize the association features between fungal samples. At the same time, when dealing with the heterogeneity of multimodal data, the simple graph connection method is very easy to introduce noise, which leads to the failure of information propagation and affects the final recognition effect. Summary of the Invention

[0006] To address the aforementioned issues, this application provides a method for accurate fungal species identification based on a multimodal hypergraph neural network. This method aims to overcome the problems of low identification accuracy and poor generalization performance in complex field conditions caused by the lack of environmental background, insufficient utilization of geometric features, and lack of high-order topological relationships between samples when dealing with visually highly similar species.

[0007] The first aspect of this invention provides a method for accurate identification of fungal species based on a multimodal hypergraph neural network, comprising: Acquire multiple modal data of the target fungal sample, including microscopic visual images, segmentation mask images, satellite remote sensing images, and text metadata; Feature extraction is performed on the various modal data respectively, and the extracted multimodal features are uniformly projected onto a semantic space of a preset dimension to obtain a fused feature vector; Using each fungal sample as a node and the fused feature vector as the initial feature of the node, a hypergraph structure containing multiple hyperedges is constructed. The hypergraph structure is input into a pre-constructed hypergraph neural network, through which information propagation and node feature updates are performed; The updated node features are input into the classifier, which outputs the species identification results of the target fungal sample.

[0008] In one optional implementation, feature extraction is performed on the multiple modal data, including: The microscopic visual image is processed by an integrated Transformer framework consisting of DINOv2, BEIT and ViT to extract topological features, texture details and global morphological features in parallel. The extracted features are then stitched together and compressed to obtain the visual features. The segmented mask image is processed using a dual-path parallel processing method to extract mask features. The first path extracts deep spatial layout features of the segmented mask image using a convolutional neural network. The second path performs connected component analysis on the segmented mask image to extract geometric descriptor features, and then fuses the deep spatial layout features and the geometric descriptor features to obtain a mask fusion feature. The geometric descriptor features are at least one of area, perimeter, circularity, and aspect ratio. Environmental features were extracted from the satellite remote sensing images using a pre-trained ResNet50 network. Discrete features in the text metadata are embedded and encoded, while continuous features are standardized to obtain metadata features.

[0009] In one alternative implementation, the construction involves a hypergraph structure with multiple hyperedges, employing various construction strategies in parallel processing: Cluster the fusion features of all nodes and construct a hyperedge for multiple nodes belonging to the same cluster; In at least one dimension of the fused feature vector, the interval is divided according to the quantile of the feature value in that dimension, and multiple nodes whose feature values ​​are located in the same quantile interval are constructed as a hyperedge; Multiple nodes are selected by random sampling and constructed into a superedge for the propagation of features of rare species.

[0010] In one optional implementation, the information propagation and node feature update via the hypergraph neural network includes a first-level attention aggregation: For each hyperedge, calculate the similarity between the features of each node within the hyperedge and the mean of the features of all its nodes, and determine the contribution weight of each node to the hyperedge based on the similarity. The features of each node within the hyperedge are weighted and summed based on the contribution weights to obtain the feature representation of the hyperedge.

[0011] In one optional implementation, the information propagation and node feature update via the hypergraph neural network further includes a second-level attention fusion: For the current node, obtain the feature representations of each hyperedge connected to the node; Calculate the attention coefficient between the feature representation of each hyperedge and the feature of the current node, and determine the contribution weight of each hyperedge to the update of the node based on the attention coefficient; The feature representations of each hyperedge are weighted and fused based on the contribution weights to obtain the aggregated features of the current node.

[0012] In one optional implementation, the information propagation and node feature update via the hypergraph neural network further includes residual connections: The aggregated features of the current node are superimposed with its input features in the current layer to obtain the updated node features; and the first-level attention aggregation and the second-level attention fusion are repeatedly executed during the multi-layer propagation process.

[0013] In an alternative implementation, adaptive compensation for missing modes is also included: When a missing modality exists in the acquired multimodal data, the feature dimension corresponding to the missing modality is filled with a zero vector; and the information propagation mechanism of the hypergraph neural network is used to implicitly complete the fused feature vector of the current sample node based on the known modal features of the neighboring nodes that are connected to the current missing node by a hyperedge.

[0014] A second aspect of this invention provides a device for accurate fungal species identification based on a multimodal hypergraph neural network, the device comprising: The data acquisition module is used to acquire multiple modal data of the target fungal sample, including microscopic visual images, segmentation mask images, satellite remote sensing images, and text metadata. The feature extraction and projection module is used to extract features from the multiple modal data respectively, and project the extracted multimodal features into a semantic space of a preset dimension to obtain a fused feature vector; The hypergraph construction module is used to construct a hypergraph structure containing multiple hyperedges, using each fungal sample as a node and the fused feature vector as the initial feature of the node. A hypergraph neural network propagation module is used to input the hypergraph structure into a pre-constructed hypergraph neural network, and to perform information propagation and node feature updating through the hypergraph neural network; and The classification output module is used to input the updated node features into the classifier and output the species identification results of the target fungal sample.

[0015] A third aspect of the present invention provides an electronic device, characterized in that it includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a method for accurate identification of fungal species based on a multimodal hypergraph neural network.

[0016] A fourth aspect of this invention provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, a method for accurate identification of fungal species based on a multimodal hypergraph neural network is provided.

[0017] This application has at least the following advantages or beneficial effects: Experiments on the FungiTastic-Mini dataset have verified that the recognition accuracy of the method in this invention reaches 98.74%. In particular, for "twin" species that are extremely similar in appearance, this invention achieves accurate differentiation by introducing environmental background and geometric parameters. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a method for accurate fungal species identification based on a multimodal hypergraph neural network proposed in an embodiment of this application; Figure 2 This is a structural diagram of a fungal species accurate identification device based on a multimodal hypergraph neural network proposed in an embodiment of this application; Figure 3 This is an overall framework diagram of a fungal species accurate identification method based on a multimodal hypergraph neural network proposed in an embodiment of this application; Figure 4 This is a training curve diagram of a fungal species accurate identification method based on a multimodal hypergraph neural network proposed in an embodiment of this application; Figure 5 This is a visualization result of t-random distribution neighborhood embedding proposed in an embodiment of this application; Figure 6 This is a category performance heatmap proposed in one embodiment of this application; Figure 7 This is a schematic diagram of an electronic device according to this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Please refer to Figure 1 and Figure 3 , Figure 1 This is a flowchart of a method for accurate fungal species identification based on a multimodal hypergraph neural network, as proposed in an embodiment of this application. Figure 3 This is an overall framework diagram of a method for accurate fungal species identification based on a multimodal hypergraph neural network proposed in one embodiment of this application. Figure 1 and Figure 3 As shown, a method for accurate fungal species identification based on a multimodal hypergraph neural network includes: S100: Acquire multiple modal data of the target fungal sample, including microscopic visual images, segmentation mask images, satellite remote sensing images, and text metadata; In this embodiment, the acquired target fungal sample's multimodal data comprises four types of heterogeneous data, specifically including microscopic visual images, segmentation mask images, satellite remote sensing images, and textual metadata. The microscopic visual images are close-up images of the fungal body, used to capture core visual features such as the fungal's appearance and texture details; the segmentation mask images are binarized labeled images of the fungal body's outline, used to extract the fungal's geometric morphological features; the satellite remote sensing images are pixel patches corresponding to the fungal collection site, used to analyze macroscopic habitat information such as vegetation index, water distribution, and terrain features of the collection site; the textual metadata includes information such as the time (season), geographical location (country / region), and habitat type (leaf layer / grassland) of fungal collection, providing environmental auxiliary features for fungal species identification.

[0022] After data acquisition, various modal data are preprocessed: microscopic visual images are adjusted to a preset size, and data enhancement operations such as rotation and mirroring are performed, followed by normalization to a specified range; segmented mask images are binarized to clarify the boundary between the fungal subject and the background; satellite remote sensing images are cropped to obtain pixel patches matching the acquisition location; and text metadata is preliminarily cleaned to remove invalid information, preparing for subsequent feature extraction. Simultaneously, sample UUIDs are used as primary keys to achieve accurate alignment of all modal data, ensuring data correlation and consistency. S200: Extract features from the various modal data respectively, and project the extracted multimodal features into a semantic space of a preset dimension to obtain a fused feature vector; In this embodiment, a dedicated feature extraction strategy is designed for the characteristics of the four types of modal data to achieve accurate extraction of heterogeneous features. Then, all modal features are uniformly projected onto a 256-dimensional semantic subspace to obtain a fused feature vector. The specific process is as follows: S210: For microscopic visual images, an integrated Transformer framework consisting of DINOv2, BEIT, and standard ViT is used to extract three types of visual features in parallel: DINOv2 extracts the topological structure information of fungi through contrastive learning, enhancing the ability to perceive the spatial position of the stipe and cap; BEIT extracts high-frequency texture details such as gills and spots on the cap surface based on mask reconstruction; and ViT extracts the global morphological layout features of fungi through supervised learning. These three types of features are concatenated and then compressed using a combination module of linear layer + BatchNorm normalization + ReLU activation function to obtain 256-dimensional visual features.

[0023] S220: A dual-path parallel processing strategy is used to extract mask fusion features from the segmented mask image. The first path is the depth path, which uses a 3-layer lightweight convolutional neural network (CNN) to extract features from the mask image and capture its deep spatial layout features. The second path is the geometric path, which performs connected component analysis on the mask image and explicitly extracts at least 8 key geometric descriptor features, including area (total number of pixels), perimeter (number of edge pixels), roundness (the closer the value is to 1, the rounder the fungal outline), aspect ratio (length and width ratio of the circumscribed rectangle), as well as centroid coordinates, fill rate, and intensity standard deviation. The 8 geometric descriptor features are integrated into a geometric feature vector, mapped to 256-dimensional features by MLP, and the features from the depth path and geometric path are added element-wise and fused to obtain the mask fusion features.

[0024] The formula for calculating roundness is:

[0025] Where Area is the area (total number of pixels) and Perimeter is the perimeter (number of edge pixels). The closer the circularity value is to 1, the closer the fungal outline is to a circle.

[0026] S230: A pre-trained ResNet50 network is used to extract features from satellite remote sensing images. The fully connected layers of the network are removed, and 2048-dimensional global environmental features are extracted. After feature dimensionality reduction, 256-dimensional environmental features are obtained. These features can effectively capture macroscopic background information such as "coniferous forest", "broadleaf forest" and "grassland" of the collection site.

[0027] S240: Differentiate the text metadata by using Embedding for discrete features (such as habitat type and collection area) and StandardScaler for continuous features (such as temperature and humidity at the time of collection). Integrate the processed features to obtain 256-dimensional metadata features.

[0028] After extracting the features of the four modalities, in order to solve the problems of uneven distribution of heterogeneous features and inconsistent semantic spaces between modalities, independent nonlinear mapping layers (MLPs) are configured for visual features, mask fusion features, environmental features, and metadata features, respectively. All modal features are uniformly projected onto a preset 256-dimensional semantic subspace. Finally, the four types of features after projection are fused through dimension concatenation operation to obtain the fused feature vector of the target fungal sample.

[0029] S300: Using each fungal sample as a node and the fused feature vector as the initial feature of the node, construct a hypergraph structure containing multiple hyperedges; In this embodiment, each fungal sample is treated as a node, and the fused feature vector obtained in S200 is used as the initial feature of the node. Multiple construction strategies are used in parallel to construct a multi-scale hypergraph structure containing multiple types of hyperedges. V represents the set of nodes, and E represents the set of hyperedges. The specific construction strategies are as follows: S310: The Mini-Batch KMeans algorithm is used to dynamically cluster the fused feature vectors of all nodes. A reasonable number of clusters is preset, and multiple fungal sample nodes belonging to the same cluster are aggregated into a hyperedge. This hyperedge can mine the implicit classification consistency in multimodal data and capture the global semantic association between samples.

[0030] S320: In at least one semantic dimension of the fused feature vector, calculate the quantiles (10%, 20%...90%) of the feature values ​​in that dimension and divide the feature intervals. Construct a hyperedge by connecting multiple nodes whose feature values ​​are within the same quantile interval. This construction strategy does not rely on Euclidean distance but captures the local continuity of the feature distribution, which can effectively enhance the model's ability to distinguish subtle feature differences in fungal samples.

[0031] S330: Select multiple nodes from all nodes through random sampling and construct them into a sparse hyperedge. This method increases the connectivity density of the hypergraph structure, offsets the risk of information propagation interruption caused by the sparse data distribution in the high-dimensional feature space, and ensures the stability of information flow within the hypergraph.

[0032] The hyperedges constructed in parallel using the three strategies described above together form a multi-scale hypergraph structure. This structure can comprehensively characterize the high-order associations between fungal samples from three dimensions: global semantics, local features, and topological robustness.

[0033] S400: Input the hypergraph structure into the pre-constructed hypergraph neural network, and perform information propagation and node feature update through the hypergraph neural network; In this embodiment, the pre-constructed hypergraph neural network introduces a two-level attention mechanism and a residual connection structure to achieve high-order nonlinear propagation of information within the hypergraph and efficient updating of node features. Simultaneously, it effectively suppresses the oversmoothing problem in deep networks. The specific information propagation and node feature updating process includes three stages: first-level attention aggregation, second-level attention fusion, and residual connection. The first two stages are repeated throughout the multi-layer propagation process, as detailed below: S410: For each hyperedge in the hypergraph, first calculate the similarity between the fused feature vector of each node within the hyperedge and the mean feature value of all nodes within the hyperedge. Based on this similarity measure, quantify the representativeness of each node in the hyperedge, and then determine the contribution weight of each node to the hyperedge. This ensures that representative fungal sample nodes dominate the population feature representation of the hyperedge, rather than using a simple arithmetic mean. Based on the calculated contribution weights, perform a weighted summation of the initial features of each node within the hyperedge to obtain the overall feature representation of the hyperedge, completing the information aggregation within the hyperedge. The update formula for node features is:

[0034] in, It is a non-linear activation function used to enhance the feature extraction capability of the model. This is a two-layer aggregation process: first, the node information within the hyperedge is aggregated, and then the information is propagated back from the hyperedge to the nodes. The weights, calculated by the attention mechanism, represent the nodes. For the current update node The importance of this is that it allows the model to automatically ignore noisy samples (such as mislabeled fungi) and focus on high-quality samples.

[0035] S420: For each current node in the hypergraph, firstly, obtain the feature representations of all hyperedges (clustering hyperedges, quantile hyperedges, and topology compensation hyperedges) connected to the node; then, calculate the attention coefficient between the feature representation of each hyperedge and the initial features of the current node through an attention mechanism; based on the attention coefficient, automatically evaluate the information value of each type of hyperedge and determine the contribution weight of each hyperedge to the feature update of the current node; finally, perform weighted fusion of the feature representations of each hyperedge based on the contribution weight to obtain the aggregated features of the current node, realizing the dynamic transmission of hyperedge information at different scales to the node.

[0036] S430: To prevent a decrease in the discriminative power of fungal sample features due to excessive feature fusion during multi-layer hypergraph convolutional propagation, the aggregated features of the current node are element-wise superimposed with its input features in the current layer to obtain the updated node features after propagation at that layer. In the multi-layer propagation chain of the hypergraph neural network, each layer performs the above-mentioned first-level attention aggregation, second-level attention fusion, and residual connection operations until the propagation of all layers is completed, resulting in the final updated node features. These features fully integrate their initial features with the higher-order correlation features of other related nodes in the hypergraph.

[0037] S440: In addition, to address the common modality loss issues in actual field data collection scenarios (such as satellite signal obstruction leading to no satellite remote sensing images, missing mask annotations, etc.), this step also includes an adaptive compensation step for missing modalities: when multiple modalities acquired in S100 are missing, a zero-vector imputation strategy is adopted for the feature dimension corresponding to the missing modality to ensure the uniformity of the input feature dimension; then, using the information propagation mechanism of the hypergraph neural network, based on the known modal features of neighboring nodes that have hyperedge connections with the current missing node, the fused feature vector of the current sample node is implicitly imputed to compensate for the feature information of the missing modality and improve the robustness of the model in modality loss scenarios.

[0038] S500: Input the updated node features into the classifier and output the species identification result of the target fungal sample.

[0039] In this embodiment, the final updated node features obtained in S400 are input into the pre-constructed multilayer perceptron (MLP) classification head. The classification head performs nonlinear transformation and classification mapping on the node features through a combination of fully connected layers and activation functions, outputting the probability distribution of the target fungal sample belonging to each species, and selecting the species category with the highest probability value as the final species identification result of the fungal sample.

[0040] like Figure 4 As shown, the training curves of this application after 100 iterations on the FungiTastic-Mini dataset include four curves: training loss, validation loss, training accuracy, and validation accuracy. The horizontal axis represents the number of training epochs, and the vertical axes represent the loss value and accuracy, respectively. It is clear from the figure that both the training loss and validation loss of the method in this application decrease rapidly with the increase of the number of training epochs, and tend to stabilize after the 10th training epoch, with the loss value fluctuating slightly around 0.1. The training accuracy and validation accuracy increase rapidly in tandem, stabilizing above 99% after the 10th epoch. This indicates that the model of the method in this application converges quickly and the training process is stable, with no overfitting.

[0041] like Figure 5As shown, the visualization results of the t-random neighborhood embedding tested on the FungiTastic-Mini dataset are presented. This visualization maps the high-dimensional fungal sample node features to a two-dimensional plane, with different colored points representing different fungal species. A total of 215 fungal samples are included. The visualization results show that the sample features of different species form clear clustering regions on the two-dimensional plane. Most samples are completely separated, and the boundaries between categories are clear. Only a very small number of samples show misclassification due to cross-category distribution. This intuitively demonstrates that the features extracted by the method in this application have good discriminative power and can effectively characterize the feature differences between different fungal species.

[0042] like Figure 6 As shown in the heatmap, the class performance of this application on the FungiTastic-Mini dataset is as follows: the horizontal axis represents fungal species categories (sorted by sample size), and the vertical axis represents model evaluation metrics, mainly including precision, recall, and F1 score. The color intensity of the heatmap represents the score of the metrics, with dark green indicating high scores and light green indicating low scores. The heatmap shows that the precision, recall, and F1 score of the vast majority of fungal categories are in the dark green high-score region. Only a few rare species show a slight decrease in score due to smaller sample sizes. This fully demonstrates the excellent feature extraction capability and classification stability of the model in the fungal species identification task, achieving accurate identification of fungal species with varying sample sizes.

[0043] Please refer to Figure 2 , Figure 2 This is a structural diagram of a fungal species accurate identification device based on a multimodal hypergraph neural network, as proposed in an embodiment of this application. Figure 2 As shown, this disclosure also provides a device for accurate fungal species identification based on a multimodal hypergraph neural network. The device includes: a data acquisition module 201, a feature extraction and projection module 202, a hypergraph construction module 203, a hypergraph neural network propagation module 204, and a classification output module 205; wherein, The data acquisition module 201 is used to acquire multiple modal data of the target fungal sample, including microscopic visual images, segmentation mask images, satellite remote sensing images, and text metadata. The feature extraction and projection module 202 is used to extract features from the multiple modal data respectively, and project the extracted multimodal features into a semantic space of a preset dimension to obtain a fused feature vector; The hypergraph construction module 203 is used to construct a hypergraph structure containing multiple hyperedges, using each fungal sample as a node and the fused feature vector as the initial feature of the node. The hypergraph neural network propagation module 204 is used to input the hypergraph structure into a pre-constructed hypergraph neural network, and to perform information propagation and node feature updating through the hypergraph neural network; and The classification output module 205 is used to input the updated node features into the classifier and output the species identification result of the target fungal sample.

[0044] This disclosure also provides an electronic device, please refer to... Figure 7 , Figure 7 This is a schematic diagram of an electronic device illustrated in an embodiment of this disclosure. For example... Figure 7 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus for communication. The memory 110 stores a computer program that can run on the processor 120 to implement the steps in the method for accurate identification of fungal species based on multimodal hypergraph neural networks disclosed in this embodiment.

[0045] The disclosed embodiments also provide a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by a processor of a computer device, enables the computer device to perform the steps in the fungal species accurate identification method based on a multimodal hypergraph neural network as described in the embodiments of this disclosure.

[0046] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0047] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0048] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0049] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0050] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0051] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0052] The above provides a detailed description of a method for accurate fungal species identification based on a multimodal hypergraph neural network. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for accurate identification of fungal species based on a multimodal hypergraph neural network, characterized in that, include: Acquire multiple modal data of the target fungal sample, including microscopic visual images, segmentation mask images, satellite remote sensing images, and text metadata; Feature extraction is performed on the various modal data respectively, and the extracted multimodal features are uniformly projected onto a semantic space of a preset dimension to obtain a fused feature vector; Using each fungal sample as a node and the fused feature vector as the initial feature of the node, a hypergraph structure containing multiple hyperedges is constructed. The hypergraph structure is input into a pre-constructed hypergraph neural network, through which information propagation and node feature updates are performed; The updated node features are input into the classifier, which outputs the species identification results of the target fungal sample.

2. The method for accurate fungal species identification based on multimodal hypergraph neural networks according to claim 1, characterized in that, Feature extraction is performed on the various modal data, including: The microscopic visual image is processed by an integrated Transformer framework consisting of DINOv2, BEIT and ViT to extract topological features, texture details and global morphological features in parallel. The extracted features are then stitched together and compressed to obtain the visual features. The segmented mask image is processed using a dual-path parallel processing method to extract mask features. The first path extracts deep spatial layout features of the segmented mask image using a convolutional neural network. The second path performs connected component analysis on the segmented mask image to extract geometric descriptor features, and then fuses the deep spatial layout features and the geometric descriptor features to obtain a mask fusion feature. The geometric descriptor features are at least one of area, perimeter, circularity, and aspect ratio. Environmental features were extracted from the satellite remote sensing images using a pre-trained ResNet50 network. Discrete features in the text metadata are embedded and encoded, while continuous features are standardized to obtain metadata features.

3. The method for accurate fungal species identification based on multimodal hypergraph neural networks according to claim 1, characterized in that, The construction involves a hypergraph structure with multiple hyperedges, and multiple construction strategies are employed for parallel processing: Cluster the fusion features of all nodes and construct a hyperedge for multiple nodes belonging to the same cluster; In at least one dimension of the fused feature vector, the interval is divided according to the quantile of the feature value in that dimension, and multiple nodes whose feature values ​​are located in the same quantile interval are constructed as a hyperedge; Multiple nodes are selected by random sampling and constructed into a superedge for feature transfer of rare species.

4. The method for accurate fungal species identification based on multimodal hypergraph neural networks according to claim 1, characterized in that, The information propagation and node feature update through the hypergraph neural network include a first-level attention aggregation: For each hyperedge, calculate the similarity between the features of each node within the hyperedge and the mean of the features of all its nodes, and determine the contribution weight of each node to the hyperedge based on the similarity. The features of each node within the hyperedge are weighted and summed based on the contribution weights to obtain the feature representation of the hyperedge.

5. The method for accurate fungal species identification based on multimodal hypergraph neural networks according to claim 4, characterized in that, The information propagation and node feature update via the hypergraph neural network also includes a second-level attention fusion: For the current node, obtain the feature representations of each hyperedge connected to the node; Calculate the attention coefficient between the feature representation of each hyperedge and the feature of the current node, and determine the contribution weight of each hyperedge to the update of the node based on the attention coefficient; The feature representations of each hyperedge are weighted and fused based on the contribution weights to obtain the aggregated features of the current node.

6. The method for accurate fungal species identification based on a multimodal hypergraph neural network according to claim 5, characterized in that, The information propagation and node feature update via the hypergraph neural network also includes residual connections: The aggregated features of the current node are superimposed with its input features in the current layer to obtain the updated node features; The first-level attention aggregation and the second-level attention fusion are repeatedly executed during the multi-layer propagation process.

7. The method for accurate fungal species identification based on multimodal hypergraph neural networks according to claim 1, characterized in that, It also includes adaptive compensation for missing modes: When a missing modality exists in the acquired multimodal data, the feature dimension corresponding to the missing modality is filled with a zero vector; and the information propagation mechanism of the hypergraph neural network is used to implicitly complete the fused feature vector of the current sample node based on the known modal features of the neighboring nodes that are connected to the current missing node by a hyperedge.

8. The fungal species accurate identification device based on a multimodal hypergraph neural network according to any one of claims 1-7, characterized in that, The device includes: The data acquisition module is used to acquire multiple modal data of the target fungal sample, including microscopic visual images, segmentation mask images, satellite remote sensing images, and text metadata. The feature extraction and projection module is used to extract features from the multiple modal data respectively, and project the extracted multimodal features into a semantic space of a preset dimension to obtain a fused feature vector; The hypergraph construction module is used to construct a hypergraph structure containing multiple hyperedges, using each fungal sample as a node and the fused feature vector as the initial feature of the node. A hypergraph neural network propagation module is used to input the hypergraph structure into a pre-constructed hypergraph neural network, and to perform information propagation and node feature updating through the hypergraph neural network; and The classification output module is used to input the updated node features into the classifier and output the species identification results of the target fungal sample.

9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method for accurate identification of fungal species based on a multimodal hypergraph neural network as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method for accurate identification of fungal species based on a multimodal hypergraph neural network as described in any one of claims 1 to 7.

Images