Adaptive cross-modal feature mapping image recognition method based on zero-order learning

Abstract

The application provides an adaptive cross-modal feature mapping image recognition method based on zero-order learning, extracts fine-grained features of bird images through image preprocessing, patch encoding and position encoding; then constructs a semantic knowledge graph and extracts associated semantic features, finally maps the bird image features and semantic features into a feature space, optimizes the similarity probability of positive and negative sample pairs, and realizes bird image recognition; solves the problem that the existing method directly extracts features from the whole image and ignores the inconsistent importance of different regions in the image; the application obtains the semantic relationship between classes, between classes and attributes, and between attributes, extracts semantic features containing rich semantic information, and solves the problem that the existing artificially defined binary attribute vector cannot represent the relationship between different classes and different attributes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to an adaptive cross-modal feature mapping image recognition method based on zero-learning. Background Technology

[0002] With societal development, the demand for image classification and recognition is increasing. Traditional manual recognition alone is inefficient. In recent years, thanks to the growing availability of large-scale labeled image datasets (such as ImageNet and COCO) and the rapid development of deep learning technology, traditional supervised machine learning algorithms have achieved performance close to or even surpassing human capabilities in areas such as face recognition, object recognition, and classification. However, labeling image samples is time-consuming and labor-intensive, limiting traditional supervised machine learning algorithms to a level of "perceptual intelligence" and resulting in very limited training samples for image recognition. Furthermore, the data distribution differences between different modalities and domains mean that traditional models can only model given training image data, making it difficult to apply the trained model to images in unknown domains. Zero-step learning, even when image sample labeling information for the task is completely missing, can construct a common feature space for known and unknown domain images by finding correspondences between image data from different modalities. This enables knowledge reuse and transfer, improving the ability to solve the target task.

[0003] Zero-order learning first requires acquiring the visual and semantic features of an image, and then performing cross-modal data analysis. Existing attention-based models compress overall visual features to perform global embedding, directly using entangled region features for zero-order learning classification, which cannot effectively capture the subtle differences between various fine-grained classes. Traditional semantic feature representations utilize manually defined binary attribute vectors, but binary attributes have a significant drawback: 0 / 1 values ​​only indicate whether an individual or class possesses a certain attribute, but cannot represent the relationships between different attributes, which are helpful for individual or class identification. However, multimodal image data suffers from problems such as distributional heterogeneity, loss of cross-modal transformation information, and domain drift. Traditional alignment models are often affected by these factors when learning cross-modal mapping relationships and struggle to learn highly generalizable mapping functions. Summary of the Invention

[0004] The purpose of this invention is to provide an adaptive cross-modal feature mapping image recognition method based on zero-learning, in order to solve the problems of existing technologies requiring a large number of labeled training samples, having high requirements for the labeling of training image samples, low robustness of training results, and traditional models being able to only model for given training data, making it difficult to apply the trained model to cross-modal and unknown domain images.

[0005] To address the aforementioned technical problems, this invention provides an adaptive cross-modal feature mapping image recognition method based on zero-learning, characterized by comprising the following steps:

[0006] S1. Extract fine-grained visual features;

[0007] S2. Construct a semantic knowledge graph of birds and extract associated semantic features;

[0008] S3. Map bird image features and semantic features into the feature space, optimize the similarity probability of positive and negative sample pairs, and compare the data with positive and negative samples in the feature space respectively.

[0009] Furthermore, the specific steps for S1 to extract fine-grained visual features are as follows:

[0010] S11. Bird image preprocessing;

[0011] S12. Perform patch coding and position coding on the preprocessed bird images;

[0012] S13. Input the encoded vector into the visual feature extraction module and output fine-grained visual features.

[0013] Furthermore, the specific steps for preprocessing S11 bird images are as follows:

[0014] S111. Adjust image size: Adjust the size of the bird images to a uniform 224×224;

[0015] S112. Normalization: Normalizes the pixel values ​​of the image from 0 to 255 to be distributed between [0, 1].

[0016] S113. Bird image segmentation: Divide the input 224×224 image into 16×16 patches, resulting in 196 patches.

[0017] S114. Divide the dataset: Randomly divide the bird image dataset into a bird image training set and a bird image test set.

[0018] Specifically, in the S114 split dataset, the ratio of bird image training set to bird image test set is 8:2; and all 200 bird species are evenly distributed in the bird image training set and bird image test set.

[0019] Furthermore, the specific steps of S12 in performing patch encoding and position encoding on the preprocessed bird images are as follows:

[0020] Patch encoding: Each patch with a data format of [16,16,3] is mapped using a linear mapping to obtain a vector of length 768; then a classification vector for classification is inserted into the resulting vector, which is a trainable parameter of length 768; finally, it is concatenated with the vector previously generated from the image to obtain a vector with an encoding format of [197,768].

[0021] Positional encoding: Select a trainable parameter and accumulate it onto the original vector. The format of positional encoding is [197, 768]:

[0022]

[0023]

[0024] Where PE is the positional encoding, pos is the position of the patch in the image, i is the encoding position, and d is the position of the patch. model It is the length of the patch code.

[0025] Furthermore, S13 inputs the encoded vector into the visual feature extraction module and outputs fine-grained visual features. The specific steps are as follows:

[0026] Layered normalization is used to normalize each encoded vector, which is then fed into a multi-head self-attention module. This module assigns different weights to the input information based on the current query, aggregating the vector into query, key, and value. The key and value are paired; for a given query... Let represent a d-dimensional real vector. It is matched with k key vectors through inner product calculation. The resulting inner product is normalized using Softmax to obtain k weights. The output of the query's attention is then the weighted average of the value vectors corresponding to the k key vectors. A linear transformation is then performed on the query, key, and value to obtain matrices Q, K, and V.

[0027] Q = XW Q

[0028] K = XW K

[0029] V = XW V

[0030] Among them, W Q W K W V It is the weight matrix, and X is the input token vector. Represents N d-dimensional real vectors; for a matrix Their attention outputs can be calculated using matrices:

[0031]

[0032] in, It is a scaling factor to avoid the variance effect caused by the dot product, K T The matrix K is the transpose, and Softmax() is the activation function. The attention output is passed to the output of the multi-head self-attention module, and the outputs obtained from the multi-head self-attention are concatenated together. Finally, a linear transformation is used to obtain the final output.

[0033]

[0034] y represents the prediction result, and LN represents the layer normalization. This represents the 0th vector in the Lth layer.

[0035] Specifically, the multi-head self-attention uses six self-attention points applied to the input sequence, and finally concatenates the outputs obtained from the six different self-attention points together;

[0036] MultiHead(Q,K,V)=Concat(head1,head2,...,head6)W 0

[0037] Among them, W 0 Here, represents the weight matrix, MultiHead() is the multi-head self-attention function, and Concat() is the concatenation function. head1, head2, ..., head6 represent six self-attention heads.

[0038] Furthermore, the construction of the bird semantic knowledge graph in S2 includes three stages: knowledge extraction, knowledge fusion, and knowledge processing, as detailed below:

[0039] S211 Knowledge Extraction: Using rule-based and dictionary-based methods, bird entity concepts, attributes, and relationships between entities are extracted from the existing knowledge base, and on this basis, ontological knowledge representation is formed.

[0040] S212. Knowledge Integration: After acquiring new knowledge about birds, eliminate ambiguity between entities, relationships, attributes, and other referents and factual objects, so that knowledge from different sources can be standardized and integrated.

[0041] S213. Knowledge Processing: Conduct quality assessments on the integrated new knowledge and add the qualified parts to the knowledge base.

[0042] Furthermore, the specific method for extracting associated semantic features in S2 is as follows:

[0043] Based on the semantic knowledge graph G, its adjacency matrix A and degree matrix D are constructed; node information is transformed into vector X using an initialization method, and then a two-layer GCN is constructed to improve the quality of generated semantics. ReLU and Softmax are used as activation functions. The overall forward propagation formula is:

[0044]

[0045] in, To scale the adjacency matrix, ReLU is the activation function, X is the node feature vector, and W... (0) W is the weight matrix of the first layer of GCN, used to map the feature representations of nodes to the corresponding hidden states. (1) Z is the weight matrix of the second layer of GCN, which is used to map the hidden layer representation of the node to the corresponding output; the final output node vector Z is the semantic feature vector of the node.

[0046] Furthermore, S3 maps bird image features and semantic features into a feature space, and the specific method for optimizing the similarity probability of positive and negative sample pairs is as follows:

[0047] Image features In and semantic features Gn are mapped to three different feature spaces O1, O2, and O3 using three different mapping methods: semantic-to-visual space mapping, visual-to-semantic space mapping, and visual and semantic-to-third-party space mapping. Then, the mapped image and semantic features are divided into positive and negative sample pairs. Cross-entropy loss is used to optimize the similarity probability of positive and negative sample pairs, increasing the similarity of positive sample pairs and decreasing the similarity of negative sample pairs. Specifically:

[0048]

[0049] Among them, L I Let G represent the contrastive learning loss of image I, G+ be the corresponding semantic feature of image feature I, τ be the temperature hyperparameter, and C be the number of sample pairs. Finally, the feature representation of the sample is learned by comparing the data with positive and negative samples in the feature space.

[0050] The beneficial effects of this invention are

[0051] (1) The fine-grained visual feature extraction method based on Transformer of the present invention solves the problem that existing methods ignore the inconsistent importance of different regions in the image because they directly extract features from the whole image. By slicing the image and then using a self-attention mechanism to assign different weights to different regions of visual features, the method aims to highlight important regions and filter out useless regions, thereby extracting fine-grained, low-redundancy visual features.

[0052] (2) The semantic feature extraction method based on semantic knowledge graph of the present invention solves the problem that existing manually defined binary attribute vectors cannot represent the relationships between different classes and between different attributes. A semantic knowledge graph is constructed by using category semantic information, and then a graph convolutional neural network is used to aggregate the nodes in the semantic knowledge graph to fuse the information of surrounding nodes, thereby obtaining the semantic relationships between classes, between classes and attributes, and between attributes, and extracting semantic features containing rich semantic information.

[0053] (3) The cross-modal mapping method based on contrastive learning in this invention solves the problem of cross-modal alignment difficulties caused by a single perspective during training in existing zero-learning methods. By using three different mappings—semantic-to-visual space mapping, visual-to-semantic space mapping, and semantic and visual-to-third-party space mapping—multiple perspectives are formed for learning. This allows each learner to learn and analyze the correspondence between different modalities from different angles, thereby reducing the impact of noisy data alignment models and maximizing the extraction of effective information between different modalities. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a diagram of the Patch encoding and positional encoding structure;

[0056] Figure 2 This is a diagram of the Transformer encoder end structure;

[0057] Figure 3 This is a flowchart of the semantic knowledge graph construction process;

[0058] Figure 4 This is a flowchart of semantic feature extraction;

[0059] Figure 5 This is a schematic diagram of cross-modal feature mapping based on contrastive learning. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] my country boasts one of the world's richest bird resources, with diverse natural ecosystems fostering a myriad of bird species. However, due to overexploitation of natural resources, environmental pollution, and human hunting, many bird populations are facing a rapid decline. Because the physical characteristics of many wild birds are unfamiliar to humans, relying solely on manual identification requires over 20 days of field observation in the mountains each month—a laborious and resource-intensive process that severely limits observation and research. To effectively improve the scientific protection of wild animals and enhance their safety, this invention utilizes a feature mapping method for wildlife identification, providing a technological pathway for biodiversity conservation. The embodiment of this invention's zero-learning adaptive cross-modal feature mapping image recognition method will be validated on the CUB-200 dataset.

[0062] S1. A fine-grained visual feature extraction method based on Transformer

[0063] S11, Image Preprocessing

[0064] S111, Adjusting image size: The operation of adjusting the size of bird images. In the specific operation of this invention, the size of bird images is uniformly processed to 224×224.

[0065] S112. Normalization: In many image processing processes, image normalization is a necessary preprocessing step. Normalization means that the pixel values ​​of the image are distributed between 0 and 255 and are distributed between [0,1]. After processing, the convergence speed of model parameters can be accelerated. Here, we also perform normalization on bird images to enable the network parameters to converge quickly during training.

[0066] S113, Image Segmentation: For the granular visual feature extraction module, the input is required to be a vector sequence, i.e., a two-dimensional matrix. The input image (224×224) is divided into 16×16 patches, resulting in 196 patches.

[0067] S114. Dataset Division: The 11,788 bird images in the dataset are randomly divided into a bird image training set and a bird image test set in an 8:2 image ratio. The bird image training set is used to train the feature mapping model of this invention, and the bird image test set is used to evaluate the performance of the feature mapping model of this invention on specific bird image classification tasks. Specifically, 200 bird species are evenly distributed in the bird image training set and the bird image test set.

[0068] S12, Patch encoding, and positional encoding: such as Figure 1As shown, each patch has a data format of [16, 16, 3]. A linear mapping is used to map each patch to a vector of length 768. A class vector specifically for classification is inserted into this series of vectors. This class vector is a trainable parameter with the same data format as the other vectors, i.e., a vector of length 768. It is concatenated with the vectors previously generated from the image: Concat([1, 768], [196, 768]) —> [197, 768]. Position encoding uses a trainable parameter, directly accumulated into the original vector, and the position encoding format is also [197, 768].

[0069]

[0070]

[0071] Where PE is the positional encoding, pos is the position of the patch in the image, i is the encoding position, and d is the position of the patch. model It is the length of the patch code.

[0072] S13, Granularity Visual Feature Extraction Module: such as Figure 2 As shown, firstly, layer normalization is used to normalize each 768-dimensional vector, and then it is fed into the multi-head self-attention module. The attention mechanism aggregates information by assigning different weights to the input information according to the current query, involving three variables: query, key, and value. The key and value are paired, and for a given query vector... in Representing a d-dimensional real vector, it is used to match k key vectors (also d-dimensional, stacked together to form a matrix) through inner product calculation. The inner product is normalized using Softmax to obtain k weights. Therefore, the attention output for a query is the value vector (i.e., matrix) corresponding to the k key vectors. The weighted average of ( ). Q, K, V are all derived from a sequence containing N vectors. The matrix obtained through linear transformation:

[0073] Q = XW Q

[0074] K = XW K

[0075] V = XW V

[0076] Among them, W Q W K W VX is the weight matrix, and X is the input token vector.

[0077] For a series of N queries (i.e., matrices) in (Representing N d-dimensional real vectors), their attention outputs can be computed using matrices:

[0078]

[0079] in, It is a scaling factor to avoid the variance effect caused by the dot product, K T It is the transpose of matrix K, and Softmax() is the activation function.

[0080] In this invention, the multi-head self-attention mechanism uses six self-attention heads, that is, six self-attention heads are applied to the input sequence, and the outputs obtained from the six different self-attention heads are concatenated together, and finally the final output is obtained through linear transformation.

[0081] MultiHead(Q,K,V)=Concat(head1,head2,...,head6)W O

[0082]

[0083] Among them, W 0 Here, `head1` represents the weight matrix, `MultiHead()` is the multi-head self-attention function designed to transfer attention between different positions in the input, `Concat()` is the concatenation function designed to combine the outputs from different self-attention heads, and `head1`, `head2`, ..., `head6` represent the six self-attention heads. `y` represents the prediction result, and `LN` is the layer normalization. This represents the 0th vector in the Lth layer.

[0084] S2. A method for extracting related semantic features based on semantic knowledge graphs.

[0085] S21. Construct a semantic knowledge graph: such as Figure 3 As shown, this invention uses a bottom-up approach to construct a semantic knowledge graph, which mainly includes three stages: knowledge extraction, knowledge fusion, and knowledge processing.

[0086] S211 Knowledge Extraction: This invention uses rule-based and dictionary-based methods to extract bird entities (concepts), attributes, and relationships between entities from Wikipedia, and forms ontological knowledge representation based on this.

[0087] S212. Knowledge integration: After acquiring new knowledge about birds, due to the fact that some entities may have multiple expressions and a specific name may correspond to multiple different entities, it is necessary to eliminate the ambiguity between entities, relations, attributes and other referents and factual objects, so that knowledge from different sources can be standardized and integrated.

[0088] S213. Knowledge Processing: The newly integrated knowledge needs to undergo quality assessment before qualified parts are added to the knowledge base to ensure its quality. Based on the above three stages, a bird semantic knowledge graph G is constructed.

[0089] S22. Semantic Feature Extraction: Based on the semantic knowledge graph G, construct its adjacency matrix A and degree matrix D. Use word2vec or an initialization method to convert node information into vector X. Then, based on the characteristic of GCN to aggregate the neighbor and self-information of each node, construct a two-layer GCN to improve the quality of generated semantic features. The activation functions used are ReLU and Softmax, respectively. Figure 4 As shown, the overall forward propagation formula is:

[0090]

[0091] in, To scale the adjacency matrix, ReLU is the activation function, X is the node feature vector, and W... (0) W is the weight matrix of the first layer of GCN, used to map the feature representations of nodes to the corresponding hidden states. (1) Z is the weight matrix of the second layer of the GCN, used to map the hidden layer representation of a node to the corresponding output. Finally, we use the output node vector Z as the semantic feature vector of that node.

[0092] S3. Contrastive Learning-Based Cross-Modal Mapping Method

[0093] like Figure 5 As shown, image features I are first mapped using three different mapping methods: semantic-to-visual space mapping, visual-to-semantic space mapping, and visual and semantic-to-third-party space mapping. n and semantic features G n The features are mapped to three different feature spaces O1, O2, and O3 respectively. Then, the mapped image features and semantic features are divided into positive sample pairs (e.g., I1 and G1) and negative sample pairs (e.g., I1 and G2). Cross-entropy loss is used to optimize the similarity probability of positive and negative sample pairs, making the similarity of positive sample pairs increasingly higher and the similarity of negative sample pairs increasingly lower. Specifically:

[0094]

[0095] Among them, L IG represents the info NCE loss of image I. + Let be the semantic feature corresponding to image feature I, i.e., the positive sample, τ represent the temperature hyperparameter (a scalar), and C represent the number of sample pairs. Finally, the feature representation of the sample is learned by comparing the data with the positive and negative samples in the feature space.

[0096] The adaptive cross-modal feature mapping image recognition method based on zero-order learning of the present invention solves the problem that existing methods ignore the inconsistent importance of different regions in the image because they directly extract features from the whole image; the association semantic feature extraction method based on semantic knowledge graph solves the problem that existing manually defined binary attribute vectors cannot represent the relationship between different classes or between different attributes; and the cross-modal mapping method based on contrastive learning solves the problem of cross-modal alignment difficulties caused by a single perspective during training of existing zero-order learning methods.

[0097] Through a series of mathematical derivations and proofs, a qualitative theoretical analysis of the method proposed in this invention is provided. For algorithm testing, this invention uses commonly used benchmark datasets in zero-shot learning (CUB, AwA1, aPY, etc.) to comprehensively test and verify the algorithm from multiple aspects (including accuracy, stability, etc.). Two baseline methods, Baseline1 and Baseline2, are set up for the experiment. The effectiveness of semantic enhancement is demonstrated by adding a semantic knowledge graph-based semantic enhancement strategy to Baseline1, and the effectiveness of multi-view mapping is demonstrated by adding a contrastive learning-based feature mapping method to Baseline2. The relevant experimental results are shown in Tables 1 and 2, respectively.

[0098] Based on Tables 1 and 2, the ablation analysis leads to the following conclusions: First, by incorporating a semantic knowledge graph-based semantic enhancement method into Baseline 1, the experimental results are significantly improved compared to Baseline 1, indicating that the feature enhancement strategy based on semantic knowledge graphs can effectively improve the performance of the zero-learning model. Second, by adding a contrastive learning method to Baseline 2, the experimental results are significantly improved compared to Baseline 2, indicating that contrastive learning can effectively improve the performance of the zero-learning model. Therefore, this invention, by combining feature enhancement methods with a feature mapping method based on contrastive learning, will achieve better results and performance than the traditional zero-learning model. In the tables, U represents the accuracy rate for identifying unseen classes, S represents the accuracy rate for identifying seen classes, and H represents the harmonic mean.

[0099] Table 1. Impact of semantic augmentation based on semantic knowledge graph on the performance of the Baseline 1 model on different datasets.

[0100]

[0101] Table 2 shows the impact of the contrastive learning-based feature mapping method on the performance of the Baseline2 model on different datasets.

[0102]

[0103] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A zeroth-order learning based adaptive cross-modal feature mapping image recognition method, characterized in that, The method comprises the following steps: S1, a fine-grained visual feature extraction method based on a Transformer; The specific steps of S1 are as follows: S11, bird image preprocessing; S12, performing Patch coding and position coding on the preprocessed bird image; The specific steps are as follows: Patch coding: each data format [16, 16, 3] Patch is mapped to a vector with a length of 768 through linear mapping; then a classification vector for classification is inserted into the obtained vector, the classification vector is a trainable parameter with a length of 768; and the vector obtained from the picture is spliced together to obtain a vector with a coding format of [197, 768]; Position encoding: A trainable parameter is selected to accumulate on the original vector, the format of position encoding is [197, 768]: where, is the position encoding, pos is the position of the Patch in the image, i is the encoded position, is the length of the Patch encoding; S13, inputting the coded vector into a visual feature extraction module to output fine-grained visual features; S2, constructing a bird semantic knowledge graph and extracting associated semantic features; the specific method is as follows: Based on the semantic knowledge graph G, an adjacency matrix A and a degree matrix D are constructed; node information is converted into a vector X using an initialization method, then a 2-layer GCN is constructed to improve the quality of generated semantics, and ReLU and Softmax are used as activation functions, and the overall forward propagation formula is: wherein, is a scaled adjacency matrix, ReLU is an activation function, is a node feature vector, is a weight matrix of the first layer of the GCN, used for mapping the feature representation of a node to a corresponding hidden layer state, is a weight matrix of the second layer of the GCN, used for mapping the hidden layer representation of a node to a corresponding output; and the final output node vector Z is the semantic feature vector of the node. S3, mapping the bird image features and semantic features into a feature space, and optimizing the similarity probability of positive and negative sample pairs, and comparing the data with positive samples and negative samples in the feature space. 2.The zeroth-order learning based adaptive cross-modal feature mapping image recognition method according to claim 1, characterized in that, The specific steps of S11 bird image preprocessing are as follows: S111, adjusting the image size: uniformly adjusting the bird image size to 224x224 size; S112, normalization processing: making the value distribution of the image pixel value points between 0 and 255 between [0, 1]; S113, bird image segmentation: dividing the input 224x224 size image into 16x16 size Patches, and obtaining 196 Patches after division; S114, dividing the data set: randomly dividing the bird image data set into a bird image training set and a bird image test set. 3.The zeroth-order learning based adaptive cross-modal feature mapping image recognition method according to claim 2, characterized in that, In S114, the proportion of the bird image training set and the bird image test set in the data set is 8:2; and 200 bird species are evenly distributed in the bird image training set and the bird image test set. 4.The zeroth-order learning based adaptive cross-modal feature mapping image recognition method of claim 2, wherein, The specific steps of S13, inputting the coded vector into the visual feature extraction module to output fine-grained visual features, are as follows: Layered normalization is used to normalize each encoded vector, which is then fed into a multi-head self-attention module to assign different weights to the input information and aggregate the vectors based on the current query. and ;in and They are paired, for a given , express d A 1-dimensional real vector is matched by inner product calculation. k The inner product of the given key vectors is normalized using Softmax. k Each weight, then Its output of attention is k indivual key The vector corresponding The weighted average of the vectors; then... and Perform a linear transformation to obtain a matrix in, It is a weight matrix. It is the input token vector. Represents N d-dimensional real vectors; for a matrix Their attention outputs are calculated using matrices: in, It is a scaling factor to avoid the variance effect caused by the dot product. It is a matrix transpose, It is an activation function; it inputs the attention output into the multi-head self-attention module output, concatenates the outputs obtained from the multi-head self-attention, and finally obtains the final output through a linear transformation: y is the prediction result. LN It is layer normalization. This represents the 0th vector in the Lth layer.

5. The zeroth-order learning based adaptive cross-modal feature mapping image recognition method according to claim 4, characterized in that, The multi-head self-attention adopts 6 self-attention heads to apply on the input sequence, and finally the outputs of the 6 different self-attention heads are spliced together; wherein, is a weight matrix, is a multi-head self-attention function, is a splicing function, denotes 6 self-attention heads. 6.The zeroth-order learning based adaptive cross-modal feature mapping image recognition method of claim 1, wherein, The construction of the bird semantic knowledge graph in S2 includes three stages of knowledge extraction, knowledge fusion and knowledge processing, and the specific steps are as follows: S211, knowledge extraction: using a rule and dictionary-based method to extract bird entity concepts, attributes and mutual relationships between entities in an existing knowledge base, and forming an ontological knowledge expression on this basis; S212, knowledge fusion: after obtaining new bird knowledge, eliminating the ambiguity between entity, relationship, attribute reference and fact object, and standardizing and integrating knowledge from different sources; S213, knowledge processing: quality evaluation is performed on the fused new knowledge, and the part that passes the evaluation is added to the knowledge base.

7. The zeroth order learning based adaptive cross-modal feature mapping image recognition method according to claim 1, characterized in that, The specific method of S3, mapping the bird image features and semantic features into a feature space and optimizing the similarity probability of positive and negative sample pairs, is as follows: The image features and the semantic features are respectively mapped to three different feature spaces through three different mapping modes of semantic-to-visual space mapping, visual-to-semantic space mapping and visual and semantic-to-third-party space mapping and semantic features respectively Then, the mapped image features and semantic features are divided into positive sample pairs and negative sample pairs, and the similarity probability of the positive sample pairs and the negative sample pairs is optimized by using cross-entropy loss, so that the similarity of the positive sample pairs is higher and higher, and the similarity of the negative sample pairs is lower and lower, specifically as follows: Wherein, represents the contrastive learning loss of the image , is the corresponding semantic feature of the image feature , represents a temperature hyperparameter, C represents the number of sample pairs; finally, the feature representation of the sample is learned by comparing the data with the positive samples and the negative samples in the feature space respectively.

Images