Spore and pollen identification method based on double model decision arbitration and multi-modal database association
By employing a dual-model decision arbitration method and linking it with a multimodal database, the problems of model decision transparency and multimodal knowledge base association in pollen identification were solved, achieving efficient and reliable pollen identification results and morphological interpretation, thus improving identification efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING INST OF GEOLOGY & PALAEONTOLOGY CAS
- Filing Date
- 2026-03-26
- Publication Date
- 2026-07-03
AI Technical Summary
Existing pollen identification technologies suffer from opaque model decision features, isolated results, lack of multimodal knowledge base association, and inability to resolve decision conflicts between models, resulting in low identification efficiency and results that are difficult for researchers to accept.
Parallel inference is performed using a global modeling model based on visual Transformer and a convolutional neural network model based on YOLO. An arbitration engine is used for dynamic weighting to output the final classification. The morphological feature text description is obtained by association with a multimodal database.
It improves the scientific credibility and practical effectiveness of pollen identification, enhances the robustness of identification of complex samples through a dynamic arbitration mechanism, provides morphological interpretation and professional knowledge support, and simulates an expert-level identification process.
Smart Images
Figure CN122336744A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent recognition technology, and in particular to a pollen recognition method based on dual-model decision arbitration and multimodal database association. Background Technology
[0002] Pollen fossils, as one of the most important categories in paleontology research, have extremely important scientific application value in petroleum exploration and environmental and climate evolution studies. Due to the tiny size and complex morphology of pollen individuals, traditional pollen identification relies heavily on manual observation under an optical microscope and experience-based classification. This traditional method is not only time-consuming and labor-intensive, making it difficult to meet the research needs of massive samples in the era of big data, but it is also highly susceptible to the influence of the identifyr's subjective experience, leading to significant biases in identification data from different sources. In recent years, with the rapid development of artificial intelligence technology, deep learning-based image recognition technology has been gradually introduced into the field of pollen identification. Current computer-aided identification methods mostly focus on optimizing a single neural network architecture to improve classification accuracy, or simply using a hard voting mechanism of multiple models for comprehensive judgment. This has improved the automation and intelligence level of pollen image processing to some extent.
[0003] However, existing deep learning-based pollen identification technologies still have many limitations that urgently need to be addressed. First, existing models often exhibit "black box" decision-making characteristics, only outputting a single category label and probability value, lacking morphological explanations of the decision-making basis. When faced with difficult pollen samples exhibiting subtle morphological differences and high similarity, domain experts struggle to trace the judgment logic, resulting in low industry acceptance of the identification results. Second, the outputs of existing identification systems are typically isolated, severely detached from the support of a vast and specialized theoretical framework of pollen morphology. After obtaining the identification labels, researchers still need to frequently interrupt their workflow to manually search literature and feature texts to verify the results, limiting overall identification efficiency. Furthermore, although some existing technologies attempt to introduce multi-model collaboration, they have failed to effectively resolve the problem of decision conflicts between models, lacking a dynamic arbitration mechanism capable of deeply perceiving model confidence and the degree of disagreement, and failing to achieve real-time intelligent association between underlying image features and multimodal knowledge bases. Summary of the Invention
[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.
[0005] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a pollen identification method based on dual-model decision arbitration and multimodal database association to solve the problems mentioned in the background art.
[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a pollen identification method based on dual-model decision arbitration and multimodal database association, comprising: The pollen image to be identified is obtained, and the pollen image is input into the independently trained first model and second model for parallel inference. The first candidate category list and its corresponding predicted probability, and the second candidate category list and its corresponding predicted probability are output. The first model is a global modeling model based on visual Transformer, and the second model is a convolutional neural network model based on YOLO. The first candidate category list and the second candidate category list are input into the arbitration engine to determine whether the highest confidence category predicted by the first model and the second model are consistent. If they are consistent, the highest confidence category is directly used as the final judgment category. If they are inconsistent, the first candidate category list and the second candidate category list are dynamically weighted and arbitrated to obtain the final judgment category. Based on the final determined category, a real-time correlation search is performed in the multimodal database to obtain the morphological feature text description corresponding to the category, and the final determined category and the morphological feature text description are simultaneously pushed to the interactive interface.
[0007] As a preferred embodiment of the pollen identification method based on dual-model decision arbitration and multimodal database association described in this invention, wherein: inputting the pollen image into the first model for inference includes: The pollen image is divided into a set of image blocks of a preset size that do not overlap; Each image patch is vectorized, and each image patch is projected to a feature embedding space of a preset dimension through a linear mapping function to obtain an embedding vector sequence. The embedded vector sequence is input into the multi-head self-attention module, and attention weights are dynamically allocated according to the correlation between different image patches to perform global self-attention modeling of the macroscopic morphology of pollen. The output of the multi-head self-attention module is input into a multi-layer Transformer encoding structure composed of a self-attention mechanism and a feedforward neural network for feature fusion transformation to generate a global feature vector describing the macroscopic morphological characteristics of pollen. Based on the global feature vector, the first candidate category list and its corresponding predicted probability are output through a classification mapping function.
[0008] As a preferred embodiment of the pollen identification method based on dual-model decision arbitration and multimodal database association described in this invention, wherein: inputting the pollen image into the second model for inference includes: By using a feature extraction operator consisting of convolution operations, normalization processing, and nonlinear activation functions, the input image is mapped layer by layer to extract a multi-scale feature mapping set with different spatial resolutions. The multi-scale feature mapping set is subjected to scale alignment and cross-scale aggregation processing, and local discrimination information at different resolutions is fused to construct local morphological feature vectors. Based on the local morphological feature vector, the second candidate category list and its corresponding predicted probability are output through the classification mapping function.
[0009] As a preferred embodiment of the pollen identification method based on dual-model decision arbitration and multimodal database association described in this invention, wherein: dynamic weighted arbitration is performed on the first candidate category list and the second candidate category list to obtain the final determined category, including: The categories in the first candidate category list and the second candidate category list are merged and deduplicated to construct a unified candidate category pool; For any candidate category in the candidate category pool, the product of the prediction probability of the candidate category by the first model and the weight coefficient of the first model is added to the product of the prediction probability of the candidate category by the second model and the weight coefficient of the second model to obtain the comprehensive arbitration score of the candidate category. The category with the highest overall arbitration score will be selected as the final judgment category.
[0010] As a preferred embodiment of the pollen identification method based on dual-model decision arbitration and multimodal database association described in this invention, wherein: according to the predictive certainty of the first model and the second model, weight coefficients are dynamically assigned to the first model and the second model respectively, and the predictive certainty is expressed as the magnitude of the predicted probability of the model for its respective highest confidence category.
[0011] As a preferred embodiment of the pollen identification method based on dual-model decision arbitration and multimodal database association described in this invention, wherein: weight coefficients are dynamically assigned to the first model and the second model respectively according to the predictive determinism of the first model and the second model, including: When the prediction probability of the first model for its highest confidence class is greater than that of the second model for its highest confidence class, increase the weight coefficient of the first model and decrease the weight coefficient of the second model accordingly. When the prediction probability of the second model for its highest confidence class is greater than that of the first model for its highest confidence class, the weight coefficient of the second model is increased and the weight coefficient of the first model is decreased accordingly.
[0012] As a preferred embodiment of the pollen identification method based on dual-model decision arbitration and multimodal database association described in this invention, the method includes: performing real-time association retrieval in the multimodal database according to the final determined category, obtaining the morphological feature text description corresponding to the category, and simultaneously pushing the final determined category and the morphological feature text description to the interactive interface, including: The name of the final determined category is used as the search term to query a text knowledge base containing pollen identification features, and the corresponding detailed morphological text description is used as the information push content. The identification results, key evidence, and information push content are presented side by side on the interactive interface.
[0013] As a preferred embodiment of the pollen identification method based on dual-model decision arbitration and multimodal database association described in this invention, the identification result includes the final judgment category and its corresponding confidence level or arbitration score, and the key evidence is the heat map of the most discriminative feature region generated and highlighted in the current pollen image through visualization technology.
[0014] As a preferred embodiment of the pollen identification method based on dual-model decision arbitration and multimodal database association described in this invention, when the final determination category is calculated by the dynamic weighted arbitration process, alternative reference content is also presented in the interactive interface. The alternative reference content includes the highest confidence category of another model that was not selected as the final determination category and its corresponding detailed morphological text description in the multimodal database.
[0015] As a preferred embodiment of the pollen identification method based on dual-model decision arbitration and multimodal database association described in this invention, the first candidate category list and the second candidate category list are the top K candidate categories obtained by sorting the first model and the second model in descending order of predicted probability; after the first model and the second model complete the inference, they also independently output a complete predicted probability distribution covering the entire category space for review and retention.
[0016] Compared with existing technologies, the beneficial effects of the invention are: 1. Real-time correlation between the output of the deep learning model and the multimodal knowledge base (RAG). By generating the most discriminative feature region heatmaps (key evidence) and simultaneously pushing standard morphological text descriptions, domain experts can intuitively compare the model's visual focal points with academic textual standards. This overcomes the shortcomings of traditional models that only output isolated labels, improving the scientific credibility and practical effectiveness of the recognition results in paleontology and geology.
[0017] 2. To address the challenges of complex morphology and minimal inter-class differences in pollen fossils, this invention employs a first model based on Visual Transformer (emphasizing global macroscopic contours and spatial proportions) and a second model based on YOLO (emphasizing local microscopic surface textures and pores) for parallel inference. This architecture balances feature extraction at different scales, avoiding the limitations of single-network architectures in receptive field or long-range dependency modeling.
[0018] 3. This invention abandons the traditional simple hard voting or arithmetic averaging mechanism in multi-model systems. When disagreements arise between two models, the arbitration engine of this invention can perceive the predictive certainty of the models and automatically allocate weights towards the model that has extracted stronger discriminative features and has higher confidence. This dynamic adaptive adjustment effectively avoids the neutralization of a highly certain correct judgment by the ambiguous judgment of another model, thus improving the robustness of identifying complex and incomplete geological samples.
[0019] 4. Furthermore, in scenarios of disagreement, this invention not only outputs the final arbitration result but also extracts the rejected high-confidence interference items as alternative reference content and pushes them along with the results. It perfectly simulates the workflow of senior experts in "eliminating falsehoods and comparing similar species," providing excellent fault tolerance and auxiliary comparison space for the final conclusion of similar and difficult species. Attached Figure Description
[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating the overall process of a pollen identification method based on dual-model decision arbitration and multimodal database association, as described in one embodiment of the present invention. Detailed Implementation
[0021] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.
[0022] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0023] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.
[0024] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.
[0025] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0026] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. Example 1
[0027] Reference Figure 1 This is the first embodiment of the present invention, which provides a pollen identification method based on dual-model decision arbitration and multimodal database association, including: S1. Obtain the pollen image to be identified, and input the pollen image into the independently trained first model and second model for parallel inference. Output the first candidate category list and its corresponding predicted probability, and the second candidate category list and its corresponding predicted probability. The first model is a global modeling model based on visual Transformer, and the second model is a convolutional neural network model based on YOLO.
[0028] It should be noted that, due to the extremely small size of pollen fossils (typically 20-50 micrometers), images acquired under a 400x optical microscope exhibit complex morphological structures, minimal inter-class differences, and diverse intra-class morphology. A single artificial intelligence model often struggles to simultaneously capture both macroscopic morphological proportions and microscopic decorative details. Therefore, this embodiment employs two fully complementary deep learning models (the first model and the second model) with complementary network structures and feature modeling methods. Both are trained independently on the same training dataset (containing labeled pollen microscopic images of multiple classes) and under supervised labeling conditions, but inference is performed in parallel.
[0029] Furthermore, the process of inputting the pollen image into the first model for inference is as follows: S101. Divide the pollen image into a set of image blocks of a preset size that do not overlap.
[0030] Specifically, the input is a pollen image to be identified. height ,width and number of channels (For example, an RGB image has 3 channels), denoted as To enable the model to model the macroscopic morphological features of pollen on a holistic scale and avoid the limitations of traditional convolutional networks (CNNs) in long-range dependency modeling due to their local receptive field, it is necessary to perform regular spatial block processing on the input image. This involves dividing the image into several sets of uniformly sized, non-overlapping image blocks. Each image patch Each corresponds to a basic observation unit of local pollen morphology (such as a corner of an air sac or the edge of the body). Through structured segmentation, while maintaining the integrity of local morphological information, a unified data input format is provided for subsequent global relationship modeling that breaks through spatial distance limitations.
[0031] S102. Vectorize each image block and project each image block to a feature embedding space of a preset dimension through a linear mapping function to obtain an embedding vector sequence.
[0032] Specifically, each image block Flattened into a one-dimensional vector, it is projected onto a fixed-dimensional feature embedding space through a learnable linear embedding mapping matrix to obtain the embedding vector: in, It is a linear embedding mapping matrix; This indicates a vectorized expansion operation.
[0033] It should be noted that the embedding vectors obtained by this operation can form an embedding vector sequence, thereby realizing the transformation from the original pixel space to the high-dimensional semantic feature space, so that the differences in pollen morphology at different locations can be compared on the same numerical scale.
[0034] S103. Input the embedded vector sequence into the multi-head self-attention module, dynamically allocate attention weights according to the correlation between different image patches, and perform global self-attention modeling of the macroscopic morphology of spores and pollen.
[0035] Specifically, let the embedded vector sequence be... A linear transformation is performed on the embedded vector sequence to generate the query vector matrix. Key vector matrix and numerical vector matrix The formula for calculating self-attention mechanism is: in, This is a scaling factor for the feature dimension, used to prevent the gradient from vanishing due to excessively large dot product results; The function is used to normalize the relevance scores into attention weights in the form of a probability distribution; For matrix The transpose of .
[0036] It should be noted that, through and Through dot product operations, the model can calculate the semantic correlation between any two image patches. In specific scenarios of pollen recognition, this mechanism can explicitly model the spatial symmetry and aspect ratio between "two oppositely developing air sacs" and "central body" in pine pollen, transcending physical pixel distances. A larger attention weight indicates that the model considers the collaborative features of these two regions to be more decisive in judging the overall morphology.
[0037] S104. Input the output of the multi-head self-attention module into the multi-layer Transformer encoding structure composed of self-attention mechanism and feedforward neural network for feature fusion transformation to generate a global feature vector describing the macroscopic morphological characteristics of spores and pollen.
[0038] Specifically, the global feature vector output after multiple encoding layers is: in, This represents a feature transformation function composed of a multi-layer self-attention mechanism and a feedforward network, used to generate a global representation describing the macroscopic morphological characteristics of pollen.
[0039] Furthermore, simultaneously, the process of inputting the pollen image into the second model for inference is as follows: S105. The input image is mapped layer by layer by a feature extraction operator consisting of convolution operation, normalization processing and nonlinear activation function to extract a multi-scale feature mapping set with different spatial resolutions.
[0040] Specifically, the same pollen image is input into a backbone network based on the YOLO architecture. Because pollen surfaces often have fine textures (such as granular, reticulated, and striped patterns) and pore structures of varying sizes, these highly discriminative local features differ significantly in scale. The model extracts multi-scale features using the following formula. : in, Indicates the feature scale index; ; The total number of multi-scale layers is set (usually 3 to 4 feature layers with different resolutions). For the first The layer's feature extraction operators include two-dimensional convolution operations (Conv2D), batch normalization, and nonlinear activations (such as SiLU or ReLU).
[0041] It should be noted that the significance of extracting multi-scale features is that shallow high-resolution feature maps can capture extremely small surface particle texture responses, while deep low-resolution feature maps can capture coarser pore structure boundaries, thereby constructing a multi-scale convolutional feature pyramid that covers various fine structures.
[0042] S106. Perform scale alignment and cross-scale aggregation on the multi-scale feature mapping set, fuse local discrimination information at different resolutions, and construct local morphological feature vectors.
[0043] Specifically, to eliminate the receptive field information bias that may result from a single scale, we use a cross-scale aggregation function to fuse features: in, This represents a cross-scale feature aggregation function (such as upsampling and fusion implemented through a Feature Pyramid Network (FPN) or a Path Aggregation Network (PANet). The resulting local morphological feature vector is obtained after global average pooling (GAP) following fusion. It can accurately depict key local structures that determine subtle differences in species classification, such as the arrangement of pores and the sharpness of air sac boundaries.
[0044] Furthermore, for both the first and second models, their final feature vectors Defined as: Global features are mainly used to distinguish macroscopic morphological differences at the genus and family level, while local features are used to distinguish subtle differences between species or similar species, such as ornamentation type, number of pits and grooves, and degree of air sac development.
[0045] S107, Classification Mapping and Candidate Category List Output.
[0046] Specifically, in extracting the global feature vector and local feature vectors Then, the first and second models generate probability distributions using their respective independent classification mapping functions. The classification mapping process is expressed as follows: in, and These are the weight matrix and bias vector of the m-th model classifier, respectively, which are used to linearly map the data in the high-dimensional feature space to the space of various pollen classification labels; The function transforms the mapping result into a probability distribution that sums to 1. .
[0047] It should be noted that after the inference is completed, the model can not only output a complete predicted probability distribution covering the entire pollen category space, but also sort the respective probability distributions in descending order and extract the highest probability values. Each category is divided into two groups, forming a first candidate category list and its corresponding predicted probability, and a second candidate category list and its corresponding predicted probability.
[0048] S2. Input the first candidate category list and the second candidate category list into the arbitration engine, and determine whether the highest confidence category predicted by the first model and the second model are consistent. If they are consistent, the highest confidence category is directly used as the final judgment category. If they are inconsistent, the first candidate category list and the second candidate category list are dynamically weighted and arbitrated to obtain the final judgment category.
[0049] It should be noted that, due to the existence of many morphologically very similar species (such as pollen from the genera *Pinus* and *Picea*), in actual pollen identification scenarios, the first model (ViT), which focuses on the global macroscopic morphology, and the second model (YOLO), which focuses on local microscopic ornamentation, may capture features from different perspectives, leading to discrepancies in their judgments. Existing multi-model fusion methods often employ simple arithmetic averaging or hard voting. This "one-size-fits-all" approach easily drowns out single-model judgments that capture crucial identification evidence on specific features (i.e., single model with extremely high confidence). Therefore, this embodiment introduces a dynamic arbitration engine based on "confidence-disagreement perception" to achieve intelligent judgment beyond simple voting. Its specific processing logic is as follows: First, the arbitration engine receives the first candidate category list and its corresponding predicted probabilities output by the first model, and the second candidate category list and its corresponding predicted probabilities output by the second model. Let the category with the highest predicted probability (Top-1) in the first candidate category list be... The corresponding probability is Let the category with the highest predicted probability (Top-1) in the second candidate category list be... The corresponding probability is .
[0050] Then, the arbitration engine first executes the consistency verification logic: determining... and Are they the same? If so... This indicates that the pollen sample provides a highly consistent identification direction to the model, both in terms of macroscopic overall outline proportions and microscopic local texture features. At this point, the current sample is considered to have extremely high discriminative certainty, and conflict resolution is unnecessary; the highest confidence class is directly used as the final classification class. If... This indicates a discrepancy between the two models in interpreting morphological features. In this case, the dynamic weighted arbitration process needs to be automatically initiated as follows: S201. Merge and deduplicate the categories in the first and second candidate category lists to construct a unified candidate category pool.
[0051] Specifically, let's say the first and second models are sorted by predicted probability from largest to smallest, and the outputs are ranked as follows: The candidate category sets are respectively and A unified arbitration candidate pool is constructed through union operations. : It should be noted that when model divergence occurs, the true pollen category is very likely not to be ranked at the highest confidence position in both models simultaneously, but it is usually included in their respective Top-K lists (e.g., the first model ranks second, and the second model ranks first). By constructing a unified candidate pool after merging and deduplication, it is possible to ensure that potential correct categories are not missed, thereby providing a complete candidate space for refined feature rescoring.
[0052] S202. Based on the predictive certainty of the first model and the second model, dynamically assign weight coefficients to the first model and the second model respectively. The predictive certainty is expressed as the magnitude of the model's predicted probability for its respective highest confidence class.
[0053] Specifically, in the expert logic of pollen identification, if an expert is extremely certain about their preferred conclusion (e.g., they have clearly observed evidence of a unique pore-groove structure for a particular genus or species), their opinion should be weighted. Similarly, this embodiment quantifies predictive certainty by using the absolute predictive probability of the model for its highest confidence category. The dynamic weighting coefficients of the first model are calculated. Dynamic weight coefficients of the second model The calculation formula is as follows: in, The first model is its highest confidence category. The predicted probability; For the second model, its highest confidence category The predicted probability.
[0054] It should be noted that when the prediction probability of the first model for its highest confidence class is significantly greater than the prediction probability of the second model for its highest confidence class (i.e., If so, increase the weight coefficient of the first model. And correspondingly reduce the weight coefficients of the second model. Conversely, the same applies. Through this dynamic adaptive adjustment mechanism, when a disagreement occurs, the arbitration scale will automatically tilt towards the model that has extracted stronger discriminative features and higher confidence, thereby effectively preventing a correct judgment with high certainty from being neutralized by a fuzzy judgment with low certainty from another model.
[0055] S203. For any candidate category in the candidate category pool, add the product of the prediction probability of the candidate category by the first model and the weight coefficient of the first model to the product of the prediction probability of the candidate category by the second model and the weight coefficient of the second model to obtain the comprehensive arbitration score of the candidate category.
[0056] Specifically, for the candidate pool any candidate category Its overall arbitration score The calculation formula is: in, and For the specific category, the first model and the second model are respectively used. The original predicted probability.
[0057] It is important to emphasize that, since both the first and second models independently output and retain the complete predicted probability distribution covering the entire class space after inference, even if a certain class... Even if a category exists only in the first candidate category list, the arbitration engine can still accurately retrieve its corresponding value from the complete predicted probability distribution preserved by the second model. This ensures the rigor and numerical continuity of cross-model weighted calculations.
[0058] S204. Select the category with the highest comprehensive arbitration score as the final judgment category.
[0059] Specifically, the arbitration engine iterates through and calculates the candidate pool. After calculating the scores for all categories, the final category determination is output using the following objective function. : It should be noted that, through the above-mentioned dynamic weighted arbitration process, this invention can not only solve the decision conflict problem when the dual-model parallel inference is performed, but also perfectly match the identification thinking of human experts at the underlying logic. That is, when faced with difficult or damaged pollen fossils, it can make a final decision by dynamically measuring the relative reliability of macroscopic contour clues and microscopic texture clues, thereby improving the robustness of the identification results of complex geological samples.
[0060] S3. Based on the final determined category, perform real-time correlation retrieval in the multimodal database to obtain the morphological feature text description corresponding to the category, and simultaneously push the final determined category and morphological feature text description to the interactive interface.
[0061] It should be noted that existing deep learning-based pollen identification technologies typically only output isolated category labels and probability values (i.e., "black box" decision-making). When faced with difficult samples with extremely similar morphologies, domain experts often find it difficult to trace the judgment logic and still need to manually consult literature to verify the standard morphology of the category, leading to workflow interruptions. Therefore, this embodiment constructs a real-time association mechanism of "image features - judgment result - text knowledge." Specifically, it includes the following sub-steps: S301. Using the name of the final determined category as the search term, perform a real-time query in a text knowledge base containing pollen identification characteristics to obtain the corresponding detailed morphological text description as the information push content.
[0062] Specifically, this invention pre-constructs a multimodal database (i.e., the RAG database, a retrieval-enhanced generative knowledge base). This database not only stores a massive number of Z-axis multifocal plane microscopic images of pollen, but also embeds a textual knowledge base of pollen identification characteristics rigorously compiled by paleontologists. When the arbitration engine outputs the final classification... (For example, "Pinus") after which, As the unique index keyword, the standard morphological text description corresponding to the genus and species is retrieved in real time from the text knowledge base through string exact matching or vectorized semantic retrieval technology (e.g., "has two well-developed air sacs, the body outline is elliptical, and the surface has fine granular texture").
[0063] It should be noted that this step aims to leverage pre-structured domain knowledge to endorse a single classification label, transforming isolated data outputs into knowledge entries with academic reference value, thereby constructing an intelligent assisted recognition system that is "instantly recognizable and interpretable".
[0064] S302. Using feature visualization technology, generate and highlight the most distinctive feature region heatmap corresponding to the final determined category in the current pollen image as key evidence.
[0065] Furthermore, to help users intuitively understand why the model arrives at this conclusion, this embodiment employs a gradient-based class activation mapping algorithm to perform reverse spatial tracing of the internal decision-making process of the deep network. Let the set of feature maps output by a key layer in the network (such as the last spatial convolutional layer in the second model) be... ,Include There are 1 channel, and the feature map of each channel is denoted as . The system first calculates the target classification. Predicted score (Original log-odds ratio before Softmax activation) for feature map The gradient is calculated and global average pooling (GAP) is performed to obtain the weight coefficients of that feature channel. : in, These are the pixel coordinates of the feature map in the spatial dimension; Indicates the first Each channel is located in The feature activation value; The total number of pixels contained in the feature map (i.e., width) high).
[0066] It should be noted that the parameters The physical quantization of the first Each deep semantic feature channel is crucial for the final identification of the category. The degree of importance or contribution.
[0067] Furthermore, the aforementioned weights are used to perform a linear weighted summation on all feature map channels, and a non-linear activation function is used to remove negative value interference, generating a two-dimensional spatial heatmap for this category. : in, The function of the (linear rectified function) is to retain only the pixel region response that has a positive effect on the target classification, and filter out the negative response that belongs to other categories or the background.
[0068] Furthermore, by using the bilinear interpolation algorithm... The image is upsampled to the size of the original input pollen image and overlaid with pseudo-color (such as Jet color maps) to form highlighted areas. This heatmap directly reflects the "key evidence" areas that the model focuses on when making decisions (such as precisely locating the connection between the air sac and the body or tiny pore structures).
[0069] S303. The identification results, key evidence, and information push content are presented side by side on the interactive interface; and when the final determination category is calculated by the dynamic weighted arbitration process, alternative reference content is also presented on the interactive interface.
[0070] Specifically, the following multimodal "decision packages" are synchronously assembled and pushed into the human-computer interaction interface through the front-end rendering engine: (1) Recognition result: Displays the final determined category And the confidence level of its calculation. If it is a consistency judgment, the original predicted probability is displayed; if it has undergone arbitration in S2, the comprehensive arbitration score calculated in S203 is displayed. .
[0071] (2) Key evidence: Present the current original pollen microscopic image generated in S302 and superimposed with the feature-focused heatmap, providing experts with the most direct visual identification basis.
[0072] (3) Information push content: The detailed morphological feature text description retrieved from S301 is automatically displayed in pop-up or collapsed mode. Experts can quickly judge the rationality of the model's conclusions by comparing whether the "highlighted areas of the heat map" match the "text description features".
[0073] (4) Alternative Reference Content: Special fault-tolerant design for complex divergent samples. If the final decision category is calculated by the dynamic weighted arbitration process of S2 (i.e., the initial Top-1 of the two models are inconsistent), the highest confidence category of the other model that was not selected as the final decision category will be automatically extracted (set as...). Similarly, it is used as a search term to query the corresponding detailed morphological text description in the multimodal database, and presented in the sidebar or bottom list of the interface as "alternative references".
[0074] It should be explained that, due to the compression deformation or partial damage of pollen in stratigraphic sediments, some similar species exhibit morphological overlap. By presenting alternative reference content, this invention not only informs users of the model's final biased conclusion but also provides experts with the rejected high-confidence interference items and their standard textual descriptions for comparison. Therefore, the mechanism of this invention perfectly simulates the rigorous identification workflow of senior identification experts in "eliminating falsehoods and comparing similar species," not only completely moving the solution away from simple single-label output but also enhancing its credibility and practical effectiveness in rigorous paleontological research.
[0075] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.
[0076] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0079] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0080] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A pollen identification method based on dual-model decision arbitration and multimodal database association, characterized in that, include: The pollen image to be identified is obtained, and the pollen image is input into the independently trained first model and second model for parallel inference. The first candidate category list and its corresponding predicted probability, and the second candidate category list and its corresponding predicted probability are output. The first model is a global modeling model based on visual Transformer, and the second model is a convolutional neural network model based on YOLO. The first candidate category list and the second candidate category list are input into the arbitration engine to determine whether the highest confidence category predicted by the first model and the second model are consistent. If they are consistent, the highest confidence category is directly used as the final judgment category. If they are inconsistent, the first candidate category list and the second candidate category list are dynamically weighted and arbitrated to obtain the final judgment category. Based on the final determined category, a real-time correlation search is performed in the multimodal database to obtain the morphological feature text description corresponding to the category, and the final determined category and the morphological feature text description are simultaneously pushed to the interactive interface.
2. The pollen identification method based on dual-model decision arbitration and multimodal database association as described in claim 1, characterized in that, Inputting the pollen image into the first model for inference includes: The pollen image is divided into a set of image blocks of a preset size that do not overlap; Each image patch is vectorized, and each image patch is projected to a feature embedding space of a preset dimension through a linear mapping function to obtain an embedding vector sequence. The embedded vector sequence is input into the multi-head self-attention module, and attention weights are dynamically allocated according to the correlation between different image patches to perform global self-attention modeling of the macroscopic morphology of pollen. The output of the multi-head self-attention module is input into a multi-layer Transformer encoding structure composed of a self-attention mechanism and a feedforward neural network for feature fusion transformation to generate a global feature vector describing the macroscopic morphological characteristics of pollen. Based on the global feature vector, the first candidate category list and its corresponding predicted probability are output through a classification mapping function.
3. The pollen identification method based on dual-model decision arbitration and multimodal database association as described in claim 1, characterized in that, The pollen image is input into the second model for inference, including: By using a feature extraction operator consisting of convolution operations, normalization processing, and nonlinear activation functions, the input image is mapped layer by layer to extract a multi-scale feature mapping set with different spatial resolutions. The multi-scale feature mapping set is subjected to scale alignment and cross-scale aggregation processing, and local discrimination information at different resolutions is fused to construct local morphological feature vectors. Based on the local morphological feature vector, the second candidate category list and its corresponding predicted probability are output through the classification mapping function.
4. The pollen identification method based on dual-model decision arbitration and multimodal database association as described in claim 1, characterized in that, The first candidate category list and the second candidate category list are dynamically weighted and arbitrated to obtain the final determined category, including: The categories in the first candidate category list and the second candidate category list are merged and deduplicated to construct a unified candidate category pool; For any candidate category in the candidate category pool, the product of the prediction probability of the candidate category by the first model and the weight coefficient of the first model is added to the product of the prediction probability of the candidate category by the second model and the weight coefficient of the second model to obtain the comprehensive arbitration score of the candidate category. The category with the highest overall arbitration score will be selected as the final judgment category.
5. The pollen identification method based on dual-model decision arbitration and multimodal database association as described in claim 4, characterized in that, Based on the predictive certainty of the first model and the second model, weight coefficients are dynamically assigned to the first model and the second model respectively. The predictive certainty is represented by the predicted probability of the model for its respective highest confidence class.
6. The pollen identification method based on dual-model decision arbitration and multimodal database association as described in claim 4, characterized in that, Based on the predictive determinism of the first model and the second model, weight coefficients are dynamically assigned to the first model and the second model respectively, including: When the prediction probability of the first model for its highest confidence class is greater than that of the second model for its highest confidence class, increase the weight coefficient of the first model and decrease the weight coefficient of the second model accordingly. When the prediction probability of the second model for its highest confidence class is greater than that of the first model for its highest confidence class, the weight coefficient of the second model is increased and the weight coefficient of the first model is decreased accordingly.
7. The pollen identification method based on dual-model decision arbitration and multimodal database association as described in claim 1, characterized in that, Based on the final determined category, a real-time correlation retrieval is performed in the multimodal database to obtain the morphological feature text description corresponding to the category, and the final determined category and the morphological feature text description are simultaneously pushed to the interactive interface, including: The name of the final determined category is used as the search term to query a text knowledge base containing pollen identification features, and the corresponding detailed morphological text description is used as the information push content. The identification results, key evidence, and information push content are presented side by side on the interactive interface.
8. The pollen identification method based on dual-model decision arbitration and multimodal database association as described in claim 7, characterized in that, The identification results include the final classification category and its corresponding confidence level or arbitration score, and the key evidence is the heat map of the most distinctive feature region generated and highlighted in the current pollen image through visualization technology.
9. The pollen identification method based on dual-model decision arbitration and multimodal database association as described in claim 7, characterized in that, When the final determination category is calculated by the dynamic weighted arbitration process, alternative reference content is also presented in the interactive interface. The alternative reference content includes the highest confidence category of another model that was not selected as the final determination category and its corresponding detailed morphological text description in the multimodal database.
10. The pollen identification method based on dual-model decision arbitration and multimodal database association as described in claim 1 or 4, characterized in that, The first candidate category list and the second candidate category list are the top K candidate categories obtained by the first model and the second model in descending order of predicted probability; after the inference is completed, the first model and the second model also independently output a complete predicted probability distribution covering the entire category space for review and retention.