Fine-grained image recognition method, system and device based on context information fusion

Abstract

This invention discloses a fine-grained image recognition method, system, and device based on context information fusion. The method includes: acquiring and preprocessing a multimodal fine-grained image dataset for training; constructing a fine-grained image recognition neural network, which includes a visual encoder, a function-based context information fusion path, a template-based context information fusion path, and an output convergence module. The two context information fusion paths respectively implement feature fusion and cross-modal interactive learning processes. The output convergence module is used to parse the multimodal representations after interaction enhancement, and a classifier generates prediction results corresponding to each modal representation; training the fine-grained image recognition neural network on the preprocessed image dataset; and finally, inputting the test fine-grained image to be classified and the matched context information into the trained fine-grained image recognition neural network to obtain the recognition result of the test fine-grained image.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a fine-grained image recognition method, system and device based on context information fusion. Background Technology

[0002] Fine-grained image recognition distinguishes species by capturing subtle visual differences (e.g., different subclasses of birds may appear similar but have subtle differences in beak, head shape, and feather morphology). With the development of deep neural networks, many techniques have been proposed, including those based on localization-classification subnetworks and those based on end-to-end feature encoding. However, these techniques only use visual information about the species' appearance, neglecting additional contextual information, including the geographic location and time information when the image was taken (e.g., latitude, longitude, and date). This limits the performance of models on large-scale fine-grained image datasets to some extent.

[0003] Therefore, on the one hand, more flexible and efficient context encoding methods are needed to define different representations of context information more comprehensively and completely; on the other hand, streamlined and efficient multimodal interaction structures are needed to effectively mine and fuse multimodal information while controlling computational costs, and to improve the discriminative ability of features. Summary of the Invention

[0004] The purpose of this invention is to solve the problems of existing technologies being unable to fully perform information interaction and limiting the representation of contextual information, and to provide a fine-grained image recognition method, system and device based on contextual information fusion.

[0005] This invention addresses different representation methods of contextual information by designing two representation fusion paths: function-based and template-based, to construct diverse and comprehensive multimodal interactions. After generating learnable contextual representations, the function-based representation fusion path extracts the aggregated and enhanced multimodal representations by constructing a global awareness encoder. Simultaneously, the template-based representation fusion path enables full cross-modal interaction between the semantic representations generated by dynamically generated templates that fill in contextual information and the aggregated and enhanced multimodal representations in a vision-context decoder.

[0006] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0007] In a first aspect, the present invention provides a fine-grained image recognition method based on contextual information fusion, comprising the following steps:

[0008] S1: Obtain the multimodal fine-grained image dataset used for training, and preprocess the images and context information in the image dataset to obtain fine-grained image and context information representations of uniform size;

[0009] S2: Construct a fine-grained image recognition neural network with dual-path multimodal context information fusion. The fine-grained image recognition neural network includes a visual encoder, a function-based context information fusion path, a template-based context information fusion path, and an output convergence module. The visual encoder includes a convolutional neural network, a convolutional layer, and a dimension transformation layer. The function-based context information fusion path fuses the context information representation and the visual features output by the visual encoder to obtain a combined multimodal representation. This combined multimodal representation is input into a global perception encoder for cross-modal interaction learning, outputting an aggregated and enhanced multimodal representation. The template-based context information fusion path fuses the context information representation and the aggregated and enhanced multimodal representation to obtain an aggregated and enhanced fused representation. This aggregated and enhanced fused representation is input into a vision-context decoder for cross-modal interaction learning, outputting an interaction-enhanced multimodal representation. The output convergence module parses the interaction-enhanced multimodal representation and inputs each parsed representation into a classifier to generate a prediction result corresponding to each modal representation.

[0010] S3: Feed the preprocessed image dataset into the fine-grained image recognition neural network, train the fine-grained image recognition neural network, and obtain the trained fine-grained image recognition neural network.

[0011] S4: Input the test fine-grained image to be classified and the matching context information into the trained fine-grained image recognition neural network to obtain the recognition result of the test fine-grained image.

[0012] Based on the above scheme, each step can be implemented in the following preferred manner.

[0013] As a preferred embodiment of the first aspect, in step S1, the method for preprocessing the images in the image dataset is as follows: after scaling, randomly flipping, and normalizing the images in the image dataset, fine-grained images of uniform size are obtained.

[0014] As a preferred embodiment of the first aspect, in step S1, the method for preprocessing the context information in the image dataset is as follows: first, normalize the longitude and latitude in the context information; then, normalize the date by dividing the date index in the context information by 365; concatenate the normalized longitude, latitude, and date belonging to the same context information as a context information vector; multiply the normalized longitude, latitude, and date by π and then calculate using the cosine function; multiply the normalized longitude, latitude, and date by π and then calculate using the sine function; concatenate the cosine function calculation result and the sine function calculation result together to obtain a one-dimensional tensor form of context information representation, which is used as the first context information representation.

[0015] As a preferred embodiment of the first aspect above, in step S1, the method for preprocessing the context information in the image dataset is as follows: obtain a template in the form of natural language, fill the template with the longitude, latitude and date in the context information, convert the context information into a filled template, and then input the filled template into a pre-trained language model. The tokenizer method of the pre-trained language model tokenizes the filled template to obtain a sequence-form context information representation, which is used as a second context information representation.

[0016] As a preferred embodiment of the first aspect, in the visual encoder of step S2, a fine-grained image is used as input. First, the high-dimensional convolutional feature map is output by the convolutional neural network. Then, the high-dimensional convolutional feature map is reduced in dimensionality by the convolutional layer to obtain a low-dimensional convolutional feature map. Finally, the low-dimensional convolutional feature map is input into the dimensionality transformation layer to obtain a serialized representation of the visual features.

[0017] As a preferred embodiment of the first aspect, in the function-based context information fusion path of step S2, the context feature encoder first performs a nonlinear transformation on the obtained first context information representation to obtain a nonlinearly transformed context information representation. The nonlinearly transformed context information representation and the visual features output by the visual encoder are then concatenated to obtain a combined multimodal representation. The global perception encoder calculates the combined multimodal representation and outputs the aggregated and enhanced multimodal representation.

[0018] As a preferred embodiment of the first aspect, in step S2, the template-based context information fusion path, a pre-trained language model is used to perform semantic extraction on the second context information representation to obtain a semantic context representation. The visual-context decoder calculates the aggregated and enhanced multimodal representation and the semantic context representation output by the function-based context information fusion path to output the interaction-enhanced multimodal representation.

[0019] As a preferred embodiment of the first aspect, in step S2, the output aggregation module parses the multimodal representation after interaction enhancement to obtain the context representation after interaction with global image information and the image representation after interaction with semantic context information. Then, the context representation after interaction with global image information is sent to the first classifier to obtain the first output result, and the image representation after interaction with semantic context information is sent to the second classifier to obtain the second output result.

[0020] Secondly, the present invention provides a fine-grained image recognition system based on contextual information fusion, comprising:

[0021] The data acquisition module is used to acquire the multimodal fine-grained image dataset used for training, and to preprocess the images and context information in the image dataset to obtain fine-grained image and context information representations of uniform size.

[0022] The model acquisition module is used to construct a fine-grained image recognition neural network with dual-path multimodal context information fusion. This fine-grained image recognition neural network includes a visual encoder, a function-based context information fusion path, a template-based context information fusion path, and an output aggregation module. The visual encoder includes a convolutional neural network, a convolutional layer, and a dimension transformation layer. The function-based context information fusion path fuses the context information representation and the visual features output by the visual encoder to obtain a combined multimodal representation. This combined multimodal representation is then input into a global perception encoder for cross-modal interaction learning, outputting an aggregated and enhanced multimodal representation. The template-based context information fusion path fuses the context information representation and the aggregated and enhanced multimodal representation to obtain an aggregated and enhanced fused representation. This aggregated and enhanced fused representation is then input into a vision-context decoder for cross-modal interaction learning, outputting an interaction-enhanced multimodal representation. The output aggregation module parses the interaction-enhanced multimodal representation and inputs each parsed representation into a classifier to generate a prediction result corresponding to each modal representation.

[0023] The model training module is used to feed the preprocessed image dataset into the fine-grained image recognition neural network, train the fine-grained image recognition neural network, and obtain the trained fine-grained image recognition neural network.

[0024] The result acquisition module is used to input the test fine-grained image to be classified and the matching context information into the trained fine-grained image recognition neural network to obtain the recognition result of the test fine-grained image.

[0025] Thirdly, the present invention provides a computer electronic device, including a memory and a processor;

[0026] The memory is used to store computer programs;

[0027] The processor is configured to, when executing the computer program, implement the fine-grained image recognition method based on context information fusion as described in any of the first aspects above.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] The function-based context information fusion path of this invention is consistent with previous multimodal fusion techniques, namely, visual representations are combined with other modal representations and fed into a multimodal encoder to achieve cross-modal interaction. Secondly, compared with the spatially compressed one-dimensional visual representations used in previous methods, the two-dimensional visual representations in the form of global region encoding can more fully interact with contextual representations across modalities.

[0030] The template-based context information fusion path of the present invention uses a well-initialized pre-trained model to extract semantic representations of context information, thereby providing better representation extraction for subsequent cross-modal interactions. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the method of the present invention;

[0032] Figure 2 This is a schematic diagram of the overall structure of the fine-grained image recognition neural network according to an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram of the global perception encoder and visual-context decoder structure in the fine-grained image recognition neural network of this invention; wherein (a) is a schematic diagram of the global perception encoder structure and (b) is a schematic diagram of the visual-context decoder structure.

[0034] Figure 4 This is a system block diagram of the present invention. Detailed Implementation

[0035] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0036] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0037] like Figure 1 As shown, in a preferred embodiment of the present invention, the fine-grained image recognition method based on context information fusion includes the following steps S1 to S4. The specific implementation process of each step will be described in detail below.

[0038] S1: Obtain the multimodal fine-grained image dataset used for training, and preprocess the images and context information in the image dataset to obtain fine-grained image and context information representations of uniform size.

[0039] It should be noted that in step S1 of the present invention, the method for preprocessing the images in the image dataset is as follows: after scaling, randomly flipping and normalizing the images in the image dataset, fine-grained images X of uniform size are obtained.

[0040] It should be noted that in step S1 of this invention, the method for preprocessing the context information in the image dataset is as follows: First, the longitude and latitude in the context information are normalized; then, the date index in the context information is divided by 365 to normalize the date; the normalized longitude, latitude, and date belonging to the same context information are concatenated into a context information vector; the normalized longitude, latitude, and date are multiplied by π and then calculated using the cosine function; the normalized longitude, latitude, and date are multiplied by π and then calculated using the sine function; the cosine function calculation result and the sine function calculation result are concatenated together to obtain a one-dimensional tensor representation of the context information. And it serves as the first contextual information representation.

[0041] It should be noted that in step S1 of this invention, the method for preprocessing the context information in the image dataset is as follows: obtain a template in natural language form, fill the template with the longitude, latitude and date in the context information, convert the context information into a filled template, and then input the filled template into a pre-trained language model. The tokenizer method of the pre-trained language model tokenizes the filled template to obtain a sequence-form context information representation w, which is used as the second context information representation.

[0042] In step S1 of this embodiment, four large-scale fine-grained image recognition datasets are acquired as training image datasets. These four datasets are iNaturalist 2017 (iNat17), iNaturalist2018 (iNat18), iNaturalist 2021Mini (iNat21 Mini), and iNaturalist 2021Full (iNat21Full). Most images in each dataset contain the geographic location and time information of the species, namely latitude, longitude, and date. Another multimodal image recognition dataset, YFCC100M-GEO100, is also acquired and used as a training image dataset. After acquiring these image datasets, the images and contextual information are preprocessed separately. Specifically, the image preprocessing method is as follows: the image size is scaled to 224×224; the scaled image is randomly horizontally flipped; the values ​​of the RGB channels of the flipped image are normalized to a value between 0 and 1; and finally, the preprocessed image is used as a uniformly sized, fine-grained image X. There are two methods for preprocessing the context information. Method 1: The latitude, longitude, and date are normalized to a value between 0 and 1 and then merged; sine and cosine functions are used simultaneously for calculation to obtain a one-dimensional tensor representation of the context information. (i.e., the first context information representation); Method 2: Convert the template into a natural language form, and then use the tokenizer method of the pre-trained model to tokenize it, so as to obtain the context information representation w in sequence form (i.e., the second context information representation).

[0043] Specifically, the template constructed in this embodiment has three slots, used to place longitude, latitude, and date, respectively. For example, for Figure 2 The context information entered is: (Longitude: 76.95714, Latitude: 8.53002, Date: 2018-01-21). The defined template is: "A species with date information of..." <date>,and geographic location of east longitude <lon>degree and northlatitude <lat>"degree" is described in Chinese as "the date information of a species". <x>The geographical location is east longitude <x>latitude and north latitude <x>"Degree". Among them, <date> 、 <lon>and <lat>The direction (east or west, north or south) in the template is extracted from the input context. The direction is adjusted based on whether the longitude and latitude are greater than zero. For empty context information, to maintain the integrity of the cross-modal interaction process, this embodiment uses the template "a species missing date information or geographical location," so that the cross-modal interaction process can proceed normally even if the sample lacks context information. After obtaining the context information in natural language form, the template is tokenized w = (w1, w2, ..., wk), where k is the maximum token length of the template.

[0044] S2: Construct a fine-grained image recognition neural network with dual-path multimodal context information fusion. The fine-grained image recognition neural network includes a visual encoder, a function-based context information fusion path, a template-based context information fusion path, and an output convergence module. The visual encoder includes a convolutional neural network, a convolutional layer, and a dimension transformation layer. The function-based context information fusion path represents the context information. The combined multimodal representation z is obtained by fusing the visual features v output by the visual encoder. e The combined multimodal representation z e The input is fed into a global perceptual encoder for cross-modal interactive learning, and the output is an aggregated and enhanced multimodal representation h. e The template-based context information fusion path combines the context information representation w with the aggregated and enhanced multimodal representation h. e The fusion process yields an aggregated and enhanced fusion representation, which is then input into a vision-context decoder for cross-modal interaction learning, outputting an interaction-enhanced multimodal representation T. The output convergence module parses the interaction-enhanced multimodal representation T and inputs each parsed representation into a classifier to generate a prediction result corresponding to each modal representation.

[0045] It should be noted that in the visual encoder of step S2, the fine-grained image X is used as input. First, the high-dimensional convolutional feature map is output by the convolutional neural network. Then, the high-dimensional convolutional feature map is reduced in dimensionality by the convolutional layer to obtain the low-dimensional convolutional feature map. Then, the low-dimensional convolutional feature map is input into the dimension transformation layer to obtain the serialized visual feature v.

[0046] In this invention, the convolutional neural network can be selected from ResNet50, ResNet101, or SK-Res2Net-101, such as... Figure 2 As shown, this embodiment uses ResNet50, ResNet101, or SK-Res2Net-101 as a convolutional neural network form for the visual encoder to extract image features. Specifically, given a uniformly sized, fine-grained image X, this embodiment uses the image features F (i.e., high-dimensional convolutional feature maps) extracted by the convolutional neural network before the global average pooling operation. Then, a linear transformation Pr and a matrix dimensionality transformation operation Re are performed on the image features F to make them consistent with the dimension of the context features.

[0047] v = Re(Pr(F))

[0048] Among them, the linear transformation Pr uses a 1×1 convolutional layer for dimensionality reduction; the matrix dimensionality transformation operation Re is used to transform the dimension of the convolutional feature map into the dimension of the sequence feature, which is implemented using a dimension transformation layer; v is the serialized visual feature, which needs to be input into the function-based context information fusion path later.

[0049] It should be noted that in step S2, the function-based context information fusion path aims to learn multimodal fusion between high-dimensional representations, and a global perception encoder is used to obtain the aggregated and enhanced multimodal representation. In step S2, the function-based context information fusion path first performs a nonlinear transformation on the obtained first context information representation to obtain a nonlinearly transformed context information representation; then, the nonlinearly transformed context information representation and the visual features output by the visual encoder are concatenated to obtain a combined multimodal representation; finally, the global perception encoder calculates the combined multimodal representation and outputs the aggregated and enhanced multimodal representation.

[0050] Specifically, in step S2, the function-based context information fusion path, the first context information obtained is first represented by the context feature encoder FCNet. Perform a nonlinear transformation to obtain the contextual information representation after the nonlinear transformation. c :

[0051]

[0052] Characterize the context information after nonlinear transformation. c The combined multimodal representation z is obtained by concatenating the visual features v output by the visual encoder with the combined multimodal representation z. e :

[0053] z e =Concat(v,l) c )

[0054] The global sensing encoder will combine the multimodal representation z e Perform calculations and output the aggregated and enhanced multimodal characterization h. e :

[0055] h e =Encoder(z e )

[0056] In this embodiment, as Figure 3 As shown in (a), the global perception encoder is a standard Transformer encoder, which includes a multi-head self-attention mechanism, a position feedforward network, layer normalization, and residual connections. The implementation of this model is existing technology and will not be described in detail here.

[0057] It should be noted that in step S2, the template-based context information fusion path, a pre-trained language model is used to extract semantics from the second context information representation to obtain a semantic context representation. The visual-context decoder calculates the aggregated and enhanced multimodal representation and the semantic context representation output by the function-based context information fusion path to output the interaction-enhanced multimodal representation.

[0058] Specifically, in the template-based context information fusion path, the pre-trained language model BERT is used to extract semantics from the second context information representation w, resulting in a semantic context representation s. BERT The visual-context decoder fuses the function-based contextual information of the path output into an aggregated and enhanced multimodal representation h. e and semantic context representations BERT The calculation is performed, and the output is the multimodal representation T after interaction enhancement:

[0059] T = Decoder(h) e ,s BERT ).

[0060] In this embodiment, as Figure 3 As shown in (b), the visual-context decoder is a standard Transformer decoder, including a multi-head cross-attention mechanism, a position feedforward network, layer normalization, and residual connections. The implementation of this model is existing technology and will not be described in detail here.

[0061] It should also be noted that, in addition to the BERT model mentioned above, the RoBERTa model can also be used as the pre-trained language model, and experimental results for applying these two pre-trained language models are given in the examples section. Of course, those skilled in the art can choose a suitable model according to actual needs, and no limitation is imposed in this invention.

[0062] It should be noted that in step S2, the output convergence module parses the enhanced multimodal representation T to obtain the context representation l after interaction with the global image information. ′ c and the image representation v after interaction with semantic context information ′ Then, the context representation after interaction with global image information is used. ′ c The image is fed into the first classifier to obtain the first output result, which is then used to represent the image after interaction with semantic context information. ′ The data is fed into the second classifier to obtain the second output result.

[0063] S3: Feed the preprocessed image dataset into the fine-grained image recognition neural network, train the fine-grained image recognition neural network, and obtain the trained fine-grained image recognition neural network.

[0064] It should be noted that in step S3 of the present invention, the preprocessed image dataset (including two kinds of contextual information representations, images and matching) in S1 is input into the fine-grained image recognition neural network constructed in S2 to train the fine-grained image recognition neural network and set the relevant training parameters.

[0065] In this embodiment, the convolutional neural network in the visual encoder can be selected from ResNet-50, ResNet-101, and SK-Res2Net-101 pre-trained on ImageNet as feature extraction architectures, and experimental results of applying these three convolutional neural networks are given in the embodiment section. The number of layers in the global perceptual encoder and the visual-context decoder are set to 1 and 2, respectively. A stochastic gradient descent optimizer with a momentum of 0.9 and a weight decay value of 1e-4 is used. The initial learning rates of the pre-trained language model and the remaining network structures are set to 1e-4 and 4e-2, respectively, and the training iteration batch size is 512. The fine-grained image recognition neural network is trained for a total of 90 iterations, with the learning rate set to linear warm-up for the first two iterations.

[0066] S4: Input the test fine-grained image to be classified and the matching context information into the trained fine-grained image recognition neural network to obtain the recognition result of the test fine-grained image.

[0067] To better demonstrate the specific implementation and technical effects of the present invention, the fine-grained image recognition method based on context information fusion shown in steps S1 to S4 of the above preferred implementation is applied to a specific example.

[0068] Example

[0069] The specific implementation process of the fine-grained image recognition method based on context information fusion used in this embodiment is as described above and will not be repeated here.

[0070] This invention evaluated the proposed method on four large-scale fine-grained image recognition datasets: iNaturalist 2017 (iNat17), iNaturalist 2018 (iNat18), iNaturalist 2021Mini (iNat21Mini), and iNaturalist 2021Full (iNat21 Full). Most images in each fine-grained image recognition dataset contain geographic location and time information for the species, namely latitude, longitude, and date. Experiments were also conducted on another multimodal image recognition dataset, YFCC100M-GEO100, and the top 100 categories were selected based on location sensitivity to form a fifth fine-grained image recognition dataset, YFCC100M-GEO100 (YFCC-GEO). This dataset contains 88,986 images, split into training and test sets with a sample ratio of 86,986:2,000. In the fifth dataset, the test set has 20 samples for each category. In addition, this embodiment also provides training / testing splitting methods for all the above datasets, as shown in Table 1.

[0071] Table 1. Training / test split of five large-scale fine-grained image recognition datasets

[0072]

[0073] The embodiments of this invention compare the method of this invention with the following two types of methods. The first type is an image-based method, and the second type is a multimodal method. In the two comparison methods, the baseline method is a method that uses only ResNet and a classifier. FixSENet is a data augmentation method for image recognition that uses different image resolutions in the training and testing phases. It is derived from the existing technical literature: TOUVRON H, VEDALDI A, DOUZEM, et al. Fixing the train-test resolution discrepancy[J]. Conference on Neural Information Processing Systems, 2019, 32. TransFG utilizes the internal self-attention weights of ViT to improve the accuracy of fine-grained recognition. It is derived from the existing technical literature: HE J, CHEN JN, LIU S, et al. Transfg: A transformer architecture for fine-grained recognition[C] / / AAAI Conference on Artificial Intelligence:36.2022:852-860.ConcatNet, GeoNet, and PriorsNet are all fine-grained image recognition methods based on a combination of contextual and image information. ConcatNet is derived from existing technical literature: TANG K, PALURI M, FEI-FEI L, et al. Improving image classification with location context [C] / / IEEE Conference on Computer Vision and Pattern Recognition. 2015:1008-1016; GeoNet is derived from existing technical literature: CHU G, POTETZ B, WANG W, et al. Geo-aware networks for fine-grained recognition [C] / / IEEE International Conference on Computer Vision Workshops. 2019:0-0; PriorsNet is derived from existing technical literature: MAC AODHA O, COLE E, PERONA P. Presence-only geographical priors for fine-grained image classification [C] / / IEEE International Conference on Computer Vision. 2019:9596-9606. EnsembleNet integrates the final result by multiplying features by relative classification probabilities. It is derived from existing technical literature: TERRY JCD, ROY HE, AUGUST T A. Thinking like a naturalist: Enhancing computer vision of citizen science images by harnessing contextual data[J].Methods in Ecology and Evolution,2020,11(2):303-315.Dynamic MLP proposes using a multilayer perceptron architecture to dynamically fuse one-dimensional image and contextual features, derived from existing literature: YANGL, LI X, SONG R, et al. Dynamic MLP for fine-grained image classification by leveraging geographical and temporal information [C] / / IEEE Conference on Computer Vision and Pattern Recognition. 2022:10945-10954. Unlike Dynamic MLP, this invention enables more comprehensive cross-modal interaction between image features and contextual features in function-based and template-based dual-path multimodal contextual information fusion.

[0074] This invention employs three feature extraction architectures: ResNet-50, ResNet-101, and SK-Res2Net-101, which are then used as convolutional neural networks in the visual encoder. After selecting the appropriate convolutional neural networks, BERT and RoBERTa models are used as pre-trained language models. Thus, a total of six model structures are obtained for implementing the method of this invention.

[0075] After obtaining the above six models, this embodiment of the invention comprehensively and fairly compared the proposed method with the six methods on the YFCC100M-GEO100 dataset (YFCC-GEO in Table 2), iNaturalist 2018 dataset (iNat18 in Table 2), iNaturalist2021Mini dataset (iNat21 Mini in Table 2), and iNaturalist 2021Full dataset (iNat21 Full in Table 2). The top-1 accuracy and top-5 accuracy results are shown in Table 2. It can be seen that the present invention is superior to the other methods compared. For the iNat18 dataset, with ResNet-50 as the backbone model, the present invention achieves a 17.9% higher top-1 accuracy and a 9.5% higher top-5 accuracy than the baseline method that only uses image information. Meanwhile, compared to Dynamic MLP, a context information fusion method based solely on functions, the top-1 accuracy of this invention is 4.0%, 3.8%, and 2.1% higher under the three feature extraction architectures of ResNet-50, ResNet-101, and SK-Res2Net-101, respectively. This indicates that the encoder-decoder multimodal fusion structure proposed in this invention, based on exploring different forms of context representation, can achieve state-of-the-art results on multiple datasets. Consistent conclusions are also obtained on the iNat21 Mini and iNat21 Full datasets. Using ResNet-50 / ResNet-101 as the backbone model, the top-1 accuracy of this invention is 13.2% and 11.4% higher than the image-based baseline method, respectively, on the iNat21 Mini dataset. On the challenging iNat21 Full dataset, this invention also achieves excellent performance, with top-1 accuracy improvements of over 3.0% and 2.4% compared to Dynamic MLP using ResNet-50 / ResNet-101, respectively.

[0076] The results on the iNaturalist 2017 (iNat17) dataset are shown in Table 3. The experimental results obtained by this invention are 5.8% higher than those of TransFG using higher image resolution, and are still superior to existing multimodal methods.

[0077] Table 2. Comparison of results from different methods on various datasets (top-1% and top-5%)

[0078]

[0079]

[0080] Table 3 shows the performance comparison on the iNat17 dataset (top-1%).

[0081]

[0082] To verify the effectiveness of the proposed function-based and template-based context information fusion paths, this embodiment decomposes the context information fusion paths for different types and conducts comparative experiments. To avoid changes in model structure and parameters affecting the experimental results and thus ablation analysis, this example maintains the overall network structure unchanged, i.e., retains the encoder-decoder structure. This invention ablates the context information fusion paths by removing two types (function-based and template-based), retaining one type (either only function-based or only template-based), and retaining both types (both function-based and template-based). The corresponding ablation experimental results are shown in Table 4. This embodiment uses ResNet-50 and ResNet-101 as convolutional neural networks in the visual encoder to verify the effectiveness of the proposed context information fusion paths and to demonstrate that stacked network structures do not necessarily improve performance.

[0083] Table 4. Ablation Experiment Results

[0084]

[0085] It should also be noted that the fine-grained image recognition method based on context information fusion in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a fine-grained image recognition system based on context information fusion, corresponding to the fine-grained image recognition method based on context information fusion provided in the above embodiments, such as... Figure 4 As shown, it includes:

[0086] The data acquisition module is used to acquire the multimodal fine-grained image dataset used for training, and to preprocess the images and context information in the image dataset to obtain fine-grained image and context information representations of uniform size.

[0087] The model acquisition module is used to construct a fine-grained image recognition neural network with dual-path multimodal context information fusion. This fine-grained image recognition neural network includes a visual encoder, a function-based context information fusion path, a template-based context information fusion path, and an output aggregation module. The visual encoder includes a convolutional neural network, a convolutional layer, and a dimension transformation layer. The function-based context information fusion path fuses the context information representation and the visual features output by the visual encoder to obtain a combined multimodal representation. This combined multimodal representation is then input into a global perception encoder for cross-modal interaction learning, outputting an aggregated and enhanced multimodal representation. The template-based context information fusion path fuses the context information representation and the aggregated and enhanced multimodal representation to obtain an aggregated and enhanced fused representation. This aggregated and enhanced fused representation is then input into a vision-context decoder for cross-modal interaction learning, outputting an interaction-enhanced multimodal representation. The output aggregation module parses the interaction-enhanced multimodal representation and inputs each parsed representation into a classifier to generate a prediction result corresponding to each modal representation.

[0088] The model training module is used to feed the preprocessed image dataset into the fine-grained image recognition neural network, train the fine-grained image recognition neural network, and obtain the trained fine-grained image recognition neural network.

[0089] The result acquisition module is used to input the test fine-grained image to be classified and the matching context information into the trained fine-grained image recognition neural network to obtain the recognition result of the test fine-grained image.

[0090] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the fine-grained image recognition method based on context information fusion provided in the above embodiments, which includes a memory and a processor;

[0091] The memory is used to store computer programs;

[0092] The processor is configured to implement, when executing the computer program, the fine-grained image recognition method based on context information fusion as described in the above embodiments.

[0093] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0094] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0095] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0096] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.< / lat> < / lon> < / date> < / x> < / x> < / x> < / lat> < / lon> < / date>

Claims

1. A fine-grained image recognition method based on contextual information fusion, characterized in that, Includes the following steps: S1: Obtain the multimodal fine-grained image dataset used for training, and preprocess the images and context information in the image dataset to obtain fine-grained image and context information representations of uniform size; S2: Construct a fine-grained image recognition neural network with dual-path multimodal context information fusion. The fine-grained image recognition neural network includes a visual encoder, a function-based context information fusion path, a template-based context information fusion path, and an output convergence module. The visual encoder includes a convolutional neural network, a convolutional layer, and a dimension transformation layer. The function-based context information fusion path fuses the context information representation and the visual features output by the visual encoder to obtain a combined multimodal representation. This combined multimodal representation is input into a global perception encoder for cross-modal interaction learning, outputting an aggregated and enhanced multimodal representation. The template-based context information fusion path fuses the context information representation and the aggregated and enhanced multimodal representation to obtain an aggregated and enhanced fused representation. This aggregated and enhanced fused representation is input into a vision-context decoder for cross-modal interaction learning, outputting an interaction-enhanced multimodal representation. The output convergence module parses the interaction-enhanced multimodal representation and inputs each parsed representation into a classifier to generate a prediction result corresponding to each modal representation. S3: Feed the preprocessed image dataset into the fine-grained image recognition neural network, train the fine-grained image recognition neural network, and obtain the trained fine-grained image recognition neural network. S4: Input the test fine-grained image to be classified and the matching context information into the trained fine-grained image recognition neural network to obtain the recognition result of the test fine-grained image; In step S1, the method for preprocessing the context information in the image dataset is as follows: obtain a template in natural language form, fill the template with the longitude, latitude and date in the context information, convert the context information into a filled template, and then input the filled template into the pre-trained language model. The tokenizer method of the pre-trained language model tokenizes the filled template to obtain a sequence-form context information representation, which is used as the second context information representation. In step S2, the template-based context information fusion path, a pre-trained language model is used to extract semantics from the second context information representation to obtain a semantic context representation. The visual-context decoder calculates the aggregated and enhanced multimodal representation and the semantic context representation output by the function-based context information fusion path to output the interactive enhanced multimodal representation.

2. The fine-grained image recognition method based on contextual information fusion as described in claim 1, characterized in that, In step S1, the method for preprocessing the images in the image dataset is as follows: after scaling, randomly flipping, and normalizing the images in the image dataset, fine-grained images of uniform size are obtained.

3. The fine-grained image recognition method based on contextual information fusion as described in claim 1, characterized in that, In step S1, the method for preprocessing the context information in the image dataset is as follows: First, the longitude and latitude in the context information are normalized. Then, the date index in the context information is divided by 365 to normalize the date. The normalized longitude, latitude, and date belonging to the same context information are concatenated into a context information vector. The normalized longitude, latitude, and date are multiplied by π and then calculated by the cosine function. The normalized longitude, latitude, and date are multiplied by π and then calculated by the sine function. The cosine function calculation result and the sine function calculation result are concatenated together to obtain a one-dimensional tensor form of context information representation, which is used as the first context information representation.

4. The fine-grained image recognition method based on contextual information fusion as described in claim 1, characterized in that, In the visual encoder of step S2, the fine-grained image is used as input. First, the high-dimensional convolutional feature map is output by the convolutional neural network. Then, the high-dimensional convolutional feature map is reduced in dimensionality by the convolutional layer to obtain the low-dimensional convolutional feature map. Then, the low-dimensional convolutional feature map is input into the dimension transformation layer to obtain the serialized visual features.

5. The fine-grained image recognition method based on contextual information fusion as described in claim 3, characterized in that, In step S2, the context information fusion path based on functions, the first context information representation obtained by the context feature encoder is nonlinearly transformed to obtain the nonlinearly transformed context information representation. The nonlinearly transformed context information representation and the visual features output by the visual encoder are concatenated to obtain the combined multimodal representation. The global perception encoder calculates the combined multimodal representation and outputs the aggregated and enhanced multimodal representation.

6. The fine-grained image recognition method based on contextual information fusion as described in claim 1, characterized in that, In step S2, the output convergence module parses the multimodal representation after interaction enhancement to obtain the context representation after interaction with global image information and the image representation after interaction with semantic context information. Then, the context representation after interaction with global image information is sent to the first classifier to obtain the first output result, and the image representation after interaction with semantic context information is sent to the second classifier to obtain the second output result.

7. A fine-grained image recognition system based on contextual information fusion, characterized in that, The system is used to implement the fine-grained image recognition method based on context information fusion as described in any one of claims 1 to 6.

8. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the fine-grained image recognition method based on context information fusion as described in any one of claims 1 to 6 when executing the computer program.

Images