Fine-grained image recognition method, system and equipment based on contextual information fusion

Abstract

The invention discloses a fine-grained image recognition method, system and equipment based on context information fusion. The method comprises the following steps: acquiring a multi-modal fine-grained image data set for training and preprocessing the multi-modal fine-grained image data set; a fine-grained image recognition neural network is constructed, the fine-grained image recognition neural network comprises a visual encoder, a function-based context information fusion path, a template-based context information fusion path and an output convergence module, and the two context information fusion paths achieve feature fusion and a cross-modal interactive learning process respectively; the output convergence module is used for analyzing the multi-modal representation after interaction enhancement and generating a prediction result corresponding to each modal representation by a classifier; and training a fine-grained image recognition neural network on the preprocessed image data set, and finally inputting a to-be-classified test fine-grained image and matched context information into the trained fine-grained image recognition neural network to obtain a recognition result of the test fine-grained image.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a fine-grained image recognition method, system and device based on context information fusion. Background Art

[0002] Fine-grained image recognition distinguishes sub-species by capturing subtle visual differences between species (e.g., different sub-species of birds have similar appearances, but have subtle differences in beak, head shape, and feather morphology). With the development of deep neural networks, many techniques have been proposed, including those based on localization-classification sub-networks and those based on end-to-end feature encoding. However, these techniques only use visual information about the appearance of the species, ignoring additional contextual information, including the geographic location and time information (e.g., longitude, latitude, and date) when the picture was taken. This limits the performance of the model on large-scale fine-grained image datasets to a certain extent.

[0003] Therefore, on the one hand, a more flexible and efficient context encoding method is needed to more comprehensively and completely define the different representation methods of context information; on the other hand, a streamlined and efficient multimodal interaction structure is needed to achieve effective mining and fusion of multimodal information while controlling computational costs, as well as improve the discriminative ability of features. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the prior art cannot fully carry out information interaction and at the same time limits the representation method of context information, and to provide a fine-grained image recognition method, system and device based on context information fusion.

[0005] Starting from different representation methods of context information, the present invention designs two representation fusion paths based on functions and templates to construct diversified and sufficient multimodal interactions. After generating a learnable context representation, the function-based representation fusion path extracts the multimodal representation after aggregation enhancement by constructing a global perception encoder. At the same time, the template-based representation fusion path combines the semantic representation generated by the dynamic template filled with context information and the multimodal representation after aggregation enhancement in the visual-context decoder for sufficient cross-modal interaction.

[0006] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a fine-grained image recognition method based on context information fusion, comprising the following steps:

[0008] S1: Obtain a multimodal fine-grained image dataset used for training, and preprocess the images and context information in the image dataset to obtain fine-grained image and context information representations of uniform size;

[0009] S2: Construct a fine-grained image recognition neural network with dual-path multimodal context information fusion, the fine-grained image recognition neural network includes a visual encoder, a function-based context information fusion path, a template-based context information fusion path and an output aggregation module; wherein the visual encoder includes a convolutional neural network, a convolutional layer and a dimension conversion layer; the function-based context information fusion path fuses the context information representation and the visual features output by the visual encoder to obtain a combined multimodal representation, which is input into the global perception encoder for cross-modal interactive learning, and an aggregated and enhanced multimodal representation is output; the template-based context information fusion path fuses the context information representation and the aggregated and enhanced multimodal representation to obtain an aggregated and enhanced fused representation, which is input into the visual-context decoder for cross-modal interactive learning, and an interactively enhanced multimodal representation is output; the output aggregation module is used to parse the interactively enhanced multimodal representation, and each of the parsed representations is input into a classifier to generate a prediction result corresponding to each modality representation;

[0010] S3: sending the preprocessed image data set into the fine-grained image recognition neural network, training the fine-grained image recognition neural network, and obtaining a trained fine-grained image recognition neural network;

[0011] S4: Input the test fine-grained image to be classified and the matching context information into the trained fine-grained image recognition neural network to obtain the recognition result of the test fine-grained image.

[0012] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0013] As a preferred embodiment of the first aspect, in step S1, the method for preprocessing the images in the image data set is: scaling, randomly horizontally flipping and normalizing the images in the image data set to obtain fine-grained images of uniform size.

[0014] As a preferred embodiment of the above-mentioned first aspect, in step S1, the method for preprocessing the context information in the image data set is: first normalize the longitude and latitude in the context information, then divide the index of the date in the context information by 365 to normalize the date, splice the normalized longitude, latitude and date belonging to the same context information as a context information vector, multiply the normalized longitude, latitude and date with pi respectively and then calculate by the cosine function, multiply the normalized longitude, latitude and date with pi respectively and then calculate by the sine function, splice the calculation results of the cosine function and the calculation results of the sine function together, obtain the context information representation in the form of a one-dimensional tensor and use it as the first context information representation.

[0015] As a preferred embodiment of the above-mentioned first aspect, in step S1, the method for preprocessing the context information in the image data set is: obtain a template in natural language form, fill in the longitude, latitude and date in the context information on the template, convert the context information into a filled-in template, and then input the filled-in template into a pre-trained language model, and the filled-in template is tokenized by the tokenizer method of the pre-trained language model to obtain a context information representation in a sequence form and use it as the second context information representation.

[0016] As a preferred embodiment of the above-mentioned first aspect, in the visual encoder of step S2, the fine-grained image is taken as input, and the convolutional neural network first outputs a high-dimensional convolutional feature map, and then the high-dimensional convolutional feature map is reduced in dimension through the convolutional layer to obtain a low-dimensional convolutional feature map after dimensionality reduction, and then the low-dimensional convolutional feature map is input into the dimension conversion layer to obtain a serialized visual feature representation.

[0017] As a preferred embodiment of the above-mentioned first aspect, in the function-based context information fusion path of step S2, the context feature encoder first performs a nonlinear transformation on the obtained first context information representation to obtain a context information representation after the nonlinear transformation, and the context information representation after the nonlinear transformation and the visual features output by the visual encoder are spliced ​​to obtain a combined multimodal representation, and the global perception encoder calculates the combined multimodal representation and outputs an aggregated enhanced multimodal representation.

[0018] As a preferred embodiment of the above-mentioned first aspect, in the template-based context information fusion path of step S2, a pre-trained language model is used to perform semantic extraction on the second context information representation to obtain a semantic context representation, and the visual-context decoder calculates the aggregated enhanced multimodal representation and the semantic context representation output by the function-based context information fusion path, and outputs an interactively enhanced multimodal representation.

[0019] As a preferred embodiment of the above-mentioned first aspect, in step S2, the output convergence module parses the multimodal representation after interaction enhancement to obtain the context representation after interaction with the global image information and the image representation after interaction with the semantic context information, and then sends the context representation after interaction with the global image information to the first classifier to obtain a first output result, and sends the image representation after interaction with the semantic context information to the second classifier to obtain a second output result.

[0020] In a second aspect, the present invention provides a fine-grained image recognition system based on context information fusion, comprising:

[0021] A data acquisition module is used to acquire a multimodal fine-grained image dataset used for training, and preprocess the images and context information in the image dataset to obtain fine-grained images and context information representations of uniform size;

[0022] A model acquisition module is used to construct a fine-grained image recognition neural network with dual-path multimodal context information fusion, wherein the fine-grained image recognition neural network includes a visual encoder, a function-based context information fusion path, a template-based context information fusion path and an output aggregation module; wherein the visual encoder includes a convolutional neural network, a convolutional layer and a dimension conversion layer; the function-based context information fusion path fuses the context information representation and the visual features output by the visual encoder to obtain a combined multimodal representation, which is input into the global perception encoder for cross-modal interactive learning, and an aggregated and enhanced multimodal representation is output; the template-based context information fusion path fuses the context information representation and the aggregated and enhanced multimodal representation to obtain an aggregated and enhanced fused representation, which is input into the visual-context decoder for cross-modal interactive learning, and an interactively enhanced multimodal representation is output; the output aggregation module is used to parse the interactively enhanced multimodal representation, and each of the parsed representations is input into a classifier to generate a prediction result corresponding to each modality representation;

[0023] A model training module is used to send the preprocessed image data set into the fine-grained image recognition neural network, train the fine-grained image recognition neural network, and obtain a trained fine-grained image recognition neural network;

[0024] The result acquisition module is used to input the test fine-grained image to be classified and the matching context information into the trained fine-grained image recognition neural network to obtain the recognition result of the test fine-grained image.

[0025] In a third aspect, the present invention provides a computer electronic device, comprising a memory and a processor;

[0026] The memory is used to store computer programs;

[0027] The processor is used to implement the fine-grained image recognition method based on context information fusion as described in any solution of the first aspect above when executing the computer program.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The function-based context information fusion path of the present invention is consistent with the previous multimodal fusion technology, that is, the visual representation is combined with other modal representations and fed into the multimodal encoder to achieve cross-modal interaction. Secondly, compared with the one-dimensional visual representation after spatial compression used in previous methods, the two-dimensional visual representation in the form of global region encoding can more fully interact with the context representation cross-modally.

[0030] The template-based context information fusion path of the present invention uses a well-initialized pre-trained model to extract the semantic representation of context information, thereby providing better representation extraction for subsequent cross-modal interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic flow diagram of the method of the present invention;

[0032] Figure 2 is a schematic diagram of the overall structure of a fine-grained image recognition neural network according to an embodiment of the present invention;

[0033] Figure 3 1 is a schematic diagram of the structure of a global perception encoder and a visual-context decoder in a fine-grained image recognition neural network according to an embodiment of the present invention; wherein (a) is a schematic diagram of the structure of a global perception encoder, and (b) is a schematic diagram of the structure of a visual-context decoder;

[0034] Figure 4 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0035] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0036] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0037] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned fine-grained image recognition method based on context information fusion includes the following steps S1 to S4. The specific implementation process is described in detail below.

[0038] S1: Obtain a multimodal fine-grained image dataset used for training, and preprocess the images and context information in the image dataset to obtain fine-grained image and context information representations of uniform size.

[0039] It should be noted that in step S1 of the present invention, the method for preprocessing the images in the image data set is: scaling, randomly horizontally flipping and normalizing the images in the image data set to obtain fine-grained images X of uniform size.

[0040] It should be noted that in step S1 of the present invention, the method for preprocessing the context information in the image data set is: first normalize the longitude and latitude in the context information, then divide the index of the date in the context information by 365 to normalize the date, splice the normalized longitude, latitude and date belonging to the same context information as a context information vector, multiply the normalized longitude, latitude and date with pi π respectively and then calculate by the cosine function, multiply the normalized longitude, latitude and date with pi π respectively and then calculate by the sine function, splice the calculation results of the cosine function and the calculation results of the sine function together, and obtain a one-dimensional tensor form of context information representation And it is represented as the first context information.

[0041] It should be noted that in step S1 of the present invention, the method for preprocessing the context information in the image data set is: obtain a template in natural language form, fill in the longitude, latitude and date in the context information on the template, convert the context information into a filled template, and then input the filled template into the pre-trained language model, and the filled template is tokenized by the tokenizer method of the pre-trained language model to obtain a context information representation w in sequence form as the second context information representation.

[0042] In step S1 of this embodiment, a total of 4 large-scale fine-grained image recognition datasets are obtained as image datasets used for training. The 4 image datasets are iNaturalist 2017 (iNat17), iNaturalist2018 (iNat18), iNaturalist 2021Mini (iNat21 Mini), and iNaturalist 2021Full (iNat21Full). Most of the images in each image dataset contain the geographic location and time information of the species, i.e., longitude and latitude and date. At the same time, another multimodal image recognition dataset YFCC100M-GEO100 was obtained and used as the image dataset used for training. After obtaining these image datasets, the images and context information therein are preprocessed respectively. Specifically, the image is preprocessed as follows: scale the image to 224×224, randomly flip the scaled image horizontally, normalize the values ​​of the three RGB channels of the flipped image to 0 to 1, and finally use the preprocessed image as a uniform fine-grained image X. There are two ways to preprocess the context information. Method 1: Normalize the longitude and latitude and date to 0 to 1 and merge them at the same time, use the sine and cosine functions to calculate them simultaneously, and obtain a one-dimensional tensor representation of the context information. (i.e., the first context information representation); Method 2: Convert it into a template in natural language form, and then use the tokenizer method of the pre-trained model to tokenize it to obtain the context information representation w in sequence form (i.e., the second context information representation).

[0043] In particular, the template constructed in this embodiment has three slots, one for longitude, one for latitude, and one for date. Figure 2 The context information entered in the template is: (Longitude: 76.95714, Latitude: 8.53002, Date: 2018-01-21), and the defined template is: "A species with date information of <date>,and geographic location of east longitude <lon>degree and northlatitude <lat>degree", its corresponding Chinese description is "the date information of a species is <x>, geographical location is east longitude <x>degrees and latitude <x>Degree". Among them, <date> 、 <lon>and <lat>It is extracted from the input context. The direction in the template (east or west, north or south) will be adjusted according to whether the longitude and latitude are greater than zero. For empty context information, in order to maintain the integrity of the cross-modal interaction process, this embodiment uses the "a species missing date information orgeographic location" template, so that even if the sample lacks context information, the cross-modal interaction process can proceed normally. After obtaining the context information in natural language form, the template is tokenized w = (w1, w2, ..., wk), where k is the maximum Token length of the template.

[0044] S2: Construct a fine-grained image recognition neural network with dual-path multimodal context information fusion, wherein the fine-grained image recognition neural network includes a visual encoder, a function-based context information fusion path, a template-based context information fusion path, and an output aggregation module; wherein the visual encoder includes a convolutional neural network, a convolutional layer, and a dimension conversion layer; the function-based context information fusion path represents the context information The visual features v output by the visual encoder are fused to obtain the combined multimodal representation z e , the combined multimodal representation z e Input into the global perception encoder for cross-modal interactive learning, and output the aggregated and enhanced multimodal representation h e ; The template-based context information fusion path combines the context information representation w and the aggregated enhanced multimodal representation h e The multimodal representation T after interaction enhancement is output; the output aggregation module is used to parse the multimodal representation T after interaction enhancement, and the parsed representations are input into a classifier to generate prediction results corresponding to each modality representation.

[0045] It should be noted that, in the visual encoder of step S2, the fine-grained image X is taken as input, and the convolutional neural network first outputs a high-dimensional convolutional feature map, and then the high-dimensional convolutional feature map is reduced in dimension through the convolutional layer to obtain a low-dimensional convolutional feature map after dimensionality reduction, and then the low-dimensional convolutional feature map is input into the dimension conversion layer to obtain a visual feature v represented in a serialized manner.

[0046] In the present invention, the convolutional neural network can be selected from ResNet50, ResNet101 or SK-Res2Net-101, such as Figure 2 As shown, this embodiment uses ResNet50, ResNet101 or SK-Res2Net-101 as a convolutional neural network form of the visual encoder to extract image features. Specifically, given a fine-grained image X of uniform size, this embodiment uses the image features F (i.e., high-dimensional convolutional feature maps) extracted by the convolutional neural network before the global average pooling operation. The image features F are then linearly transformed Pr and matrix dimension-changing operations Re are performed to keep them consistent with the dimensions of the context features:

[0047] v=Re(Pr(F))

[0048] Among them, the linear transformation Pr uses a 1×1 convolution layer for dimensionality reduction; the matrix dimension change operation Re is used to convert the convolution feature map dimension into the dimension of the sequence feature, which is implemented using the dimension conversion layer; v is the visual feature of the serialized representation, which needs to be input into the function-based context information fusion path later.

[0049] It should be noted that in the function-based context information fusion path of step S2, the purpose of this path is to learn the multimodal fusion between high-dimensional representations, and a global perception encoder is used to obtain an aggregated and enhanced multimodal representation. In the function-based context information fusion path of step S2, the context feature encoder first performs a nonlinear transformation on the first context information representation obtained to obtain a nonlinearly transformed context information representation; the nonlinearly transformed context information representation and the visual features output by the visual encoder are concatenated to obtain a combined multimodal representation; the global perception encoder calculates the combined multimodal representation and outputs an aggregated and enhanced multimodal representation.

[0050] Specifically, in the function-based context information fusion path in step S2, the context feature encoder FCNet first represents the first context information obtained Perform nonlinear transformation to obtain the context information representation l after nonlinear transformation c :

[0051]

[0052] The context information after nonlinear transformation is represented as l c It is concatenated with the visual features v output by the visual encoder to obtain the combined multimodal representation z e :

[0053] z e =Concat(v,l c )

[0054] The global perceptual encoder will combine the multimodal representation z e Calculate and output the aggregated enhanced multimodal representation h e :

[0055] h e =Encoder(z e )

[0056] In this embodiment, if Figure 3 As shown in (a), the global perception encoder is a standard Transformer encoder, including a multi-head self-attention mechanism, a position feedforward network, layer normalization and residual connection. The implementation method of this model belongs to the existing technology and will not be repeated here.

[0057] It should be noted that in the template-based context information fusion path of step S2, a pre-trained language model is used to perform semantic extraction on the second context information representation to obtain a semantic context representation, and the visual-context decoder calculates the aggregated enhanced multimodal representation and the semantic context representation output by the function-based context information fusion path, and outputs an interactively enhanced multimodal representation.

[0058] Specifically, in the template-based context information fusion path, the pre-trained language model BERT is used to perform semantic extraction on the second context information representation w to obtain the semantic context representation s BERT ; The visual-context decoder decodes the aggregated enhanced multimodal representation h output by the function-based context information fusion path e and semantic context representations BERT Calculate and output the interactively enhanced multimodal representation T:

[0059] T=Decoder(h e ,s BERT ).

[0060] In this embodiment, if Figure 3 As shown in (b), the visual-context decoder is a standard Transformer decoder, including a multi-head cross-attention mechanism, a position feedforward network, layer normalization and residual connection. The implementation method of this model belongs to the existing technology and will not be repeated here.

[0061] It should also be noted that in addition to the BERT model, the RoBERTa model can also be selected as the pre-trained language model, and the experimental results of applying these two pre-trained language models are given in the embodiment section. Of course, those skilled in the art can also select a suitable model according to actual needs, which is not limited in the present invention.

[0062] It should be noted that in step S2, the output aggregation module parses the interactively enhanced multimodal representation T to obtain the contextual representation l after interaction with the global image information. ′ c And the image representation v after interaction with semantic context information ′ , and then the context representation l after interacting with the global image information ′ c The image representation v after interaction with the semantic context information is sent to the first classifier to obtain the first output result. ′ Send it to the second classifier to get the second output result.

[0063] S3: sending the preprocessed image data set into the fine-grained image recognition neural network, training the fine-grained image recognition neural network, and obtaining a trained fine-grained image recognition neural network.

[0064] It should be noted that in step S3 of the present invention, the image data set preprocessed in S1 (including two context information representations of the image and the match) is input into the fine-grained image recognition neural network constructed in S2, the fine-grained image recognition neural network is trained, and relevant training parameters are set.

[0065] In this embodiment, the convolutional neural network in the visual encoder can select ResNet-50, ResNet-101 and SK-Res2Net-101 pre-trained on ImageNet as feature extraction architectures, and the experimental results of applying these three convolutional neural networks are given accordingly in the embodiment section. The number of layers of the global perception encoder and the visual-context decoder are set to 1 and 2 respectively. A stochastic gradient descent optimizer with a momentum of 0.9 is used, and the weight decay value is 1e-4. The initial learning rates of the pre-trained language model and the remaining network structures are set to 1e-4 and 4e-2, respectively, and the training iteration batch size is 512. The fine-grained image recognition neural network has a total training iteration round of 90, of which the learning rate of the first 2 rounds is set to linear warm-up.

[0066] S4: Input the test fine-grained image to be classified and the matching context information into the trained fine-grained image recognition neural network to obtain the recognition result of the test fine-grained image.

[0067] In order to better demonstrate the specific implementation and technical effect of the present invention, the fine-grained image recognition method based on context information fusion shown in steps S1 to S4 in the above preferred implementation is applied to a specific example.

[0068] Example

[0069] The specific implementation process of the fine-grained image recognition method based on context information fusion adopted in this embodiment is as described above and will not be repeated here.

[0070] The present invention evaluates the proposed method on four large-scale fine-grained image recognition datasets, including iNaturalist 2017 (iNat17), iNaturalist 2018 (iNat18), iNaturalist 2021Mini (iNat21Mini), and iNaturalist 2021Full (iNat21 Full). Most of the images in each fine-grained image recognition dataset contain the geographical location and time information of the species, i.e., longitude, latitude and date. The present invention also conducts experiments on another multimodal image recognition dataset YFCC100M-GEO100, and selects the top 100 categories according to location sensitivity to form the fifth fine-grained image recognition dataset YFCC100M-GEO100 (YFCC-GEO), which contains 88,986 images, and the number of samples divided into training set and test set is 86,986:2000. In the fifth dataset, each category of the test set corresponds to 20 samples. In addition, this embodiment also provides a training / testing segmentation method for all the above data sets, as shown in Table 1.

[0071] Table 1. Training / testing splits of five large-scale fine-grained image recognition datasets

[0072]

[0073] The embodiments of the present invention compare the method of the present invention with the following two methods respectively. Among them, the first category is an image-based method, and the second category is a multimodal method. In the two types of comparison methods, the baseline method is a method that only uses resnet and one classifier. FixSENet is a data enhancement method for image recognition, which uses different image resolutions in the training phase and the test phase. It is derived from the prior art document: TOUVRON H, VEDALDI A, DOUZEM, et al. Fixing the train-test resolution discrepancy [J]. Conference on Neural Information Processing Systems, 2019, 32. TransFG uses the internal self-attention weights of ViT to improve the accuracy of fine-grained recognition, which is derived from the prior art document: HE J, CHEN JN, LIU S, et al. Transfg: A transformer architecture for fine-grained recognition [C] / / AAAI Conference on Artificial Intelligence: 36.2022: 852-860.ConcatNet, GeoNet, and PriorsNet are all fine-grained image recognition methods based on the combination of contextual information and image information. Among them, ConcatNet comes from the existing technical literature: TANG K, PALURI M, FEI-FEI L, et al. Improving image classification with locationcontext[C] / / IEEE Conference on Computer Vision and Pattern Recognition.2015:1008-1016; GeoNet comes from the existing technical literature: CHU G, POTETZ B, WANG W, et al. Geo-awarenetworks for fine-grained recognition[C] / / IEEE International Conference onComputer Vision Workshops.2019:0-0; PriorsNet comes from the existing technical literature: MAC AODHA O, COLE E, PERONA P. Presence-only geographical priors for fine-grained imageclassification[C] / / IEEE International Conference on Computer Vision.2019:9596-9606. EnsembleNet integrates the final result by multiplying the features by the relative classification probability, which is derived from the existing technical literature: TERRY JCD, ROY HE, AUGUST T A. Thinking like a naturalist: Enhancing computer vision of citizen science images by harnessing contextual data[J]. Methods in Ecology and Evolution, 2020, 11(2): 303-315.Dynamic MLP proposes to use a multi-layer perceptron structure to dynamically fuse image and context features in the form of one-dimensional features, which is derived from the prior art literature: YANGL, LI X, SONG R, et al. Dynamic MLP for fine-grained image classification by leveraging geographical and temporal information [C] / / IEEE Conference on Computer Vision and Pattern Recognition. 2022: 10945-10954. Different from Dynamic MLP, the present invention makes the cross-modal interaction of image features and context features more comprehensive in the dual-path multimodal context information fusion based on function and template.

[0074] The embodiment of the present invention selects three feature extraction architectures, namely ResNet-50, ResNet-101 and SK-Res2Net-101, and uses these three models as convolutional neural networks in the visual encoder of the present invention. After selecting the convolutional neural network, the embodiment of the present invention uses the BERT model and the RoBERTa model as pre-trained language models. Thus, a total of 6 model structures are obtained for implementing the method of the present invention.

[0075] After obtaining the above six models, the embodiment of the present invention comprehensively and fairly compares the method proposed in the present invention with the six methods on the YFCC100M-GEO100 (YFCC-GEO in Table 2), iNaturalist 2018 (iNat18 in Table 2), iNaturalist2021Mini (iNat21 Mini in Table 2) and iNaturalist 2021Full (iNat21 Full in Table 2) datasets, and the top-1 accuracy and top-5 accuracy results are shown in Table 2. It can be seen that the present invention is superior to other compared methods. For the iNat18 dataset, when ResNet-50 is the backbone model, the present invention has a top-1 accuracy of 17.9% and a top-5 accuracy of 9.5% higher than the baseline method that only uses image information. At the same time, compared with the context information fusion method Dynamic MLP based only on functions, the top-1 accuracy of the present invention under the three feature extraction architectures of ResNet-50 / ResNet-101 / SK-Res2Net-101 is 4.0%, 3.8% and 2.1% higher than it respectively. This shows that the encoder-decoder multimodal fusion structure proposed by the present invention based on exploring different forms of context representation can achieve the most advanced results on multiple data sets. Consistent conclusions can also be obtained on the iNat21 Mini and iNat21 Full data sets. With ResNet-50 / ResNet-101 as the backbone model, on the iNat21 Mini data set, the top-1 accuracy of the present invention is 13.2% and 11.4% higher than the image-based baseline method respectively. On the challenging iNat21 Full data set, the present invention also achieved excellent performance. Compared with Dynamic MLP, the present invention using ResNet-50 / ResNet-101 achieved a significant improvement of top-1 accuracy of more than 3.0% and 2.4% respectively.

[0076] The results on the iNaturalist 2017 (iNat17) dataset are shown in Table 3. The experimental results achieved by the present invention are 5.8% higher than those of TransFG using higher image resolution, and are still superior to existing multimodal methods.

[0077] Table 2. Comparison of results of different methods on various datasets (top-1% and top-5%)

[0078]

[0079]

[0080] Table 3 Performance comparison on iNat17 dataset (top-1%)

[0081]

[0082] In order to verify the effects of the function-based context information fusion path and the template-based context information fusion path proposed by the present invention, this embodiment splits different types of context information fusion paths and conducts experimental comparisons. In order to avoid the influence of changes in model structure and parameters on the experimental results, and then affect the ablation analysis, this experimental example keeps the overall network structure unchanged, that is, retains the encoder-decoder structure. The present invention ablates the context information fusion path, removes two context information fusion paths (function-based context information fusion path and template-based context information fusion path), retains one context information fusion path (that is, only retains the function-based context information fusion path or only retains the template-based context information fusion path), and retains two context information fusion paths (that is, retains the function-based context information fusion path and the template-based context information fusion path at the same time), and the corresponding ablation experimental results are shown in Table 4. In the embodiment of the present invention, ResNet-50 and ResNet-101 are used as the convolutional neural networks in the visual encoder of the present invention, respectively, to verify the effectiveness of the context information fusion path proposed by the present invention, and at the same time verify that the stacked network structure cannot effectively improve the performance.

[0083] Table 4. Ablation experiment results

[0084]

[0085] It should also be noted that the fine-grained image recognition method based on context information fusion in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a fine-grained image recognition system based on context information fusion corresponding to the fine-grained image recognition method based on context information fusion provided in the above embodiment, such as Figure 4 As shown, it includes:

[0086] A data acquisition module is used to acquire a multimodal fine-grained image dataset used for training, and preprocess the images and context information in the image dataset to obtain fine-grained images and context information representations of uniform size;

[0087] A model acquisition module is used to construct a fine-grained image recognition neural network with dual-path multimodal context information fusion, wherein the fine-grained image recognition neural network includes a visual encoder, a function-based context information fusion path, a template-based context information fusion path and an output aggregation module; wherein the visual encoder includes a convolutional neural network, a convolutional layer and a dimension conversion layer; the function-based context information fusion path fuses the context information representation and the visual features output by the visual encoder to obtain a combined multimodal representation, which is input into the global perception encoder for cross-modal interactive learning, and an aggregated and enhanced multimodal representation is output; the template-based context information fusion path fuses the context information representation and the aggregated and enhanced multimodal representation to obtain an aggregated and enhanced fused representation, which is input into the visual-context decoder for cross-modal interactive learning, and an interactively enhanced multimodal representation is output; the output aggregation module is used to parse the interactively enhanced multimodal representation, and each of the parsed representations is input into a classifier to generate a prediction result corresponding to each modality representation;

[0088] A model training module is used to send the preprocessed image data set into the fine-grained image recognition neural network, train the fine-grained image recognition neural network, and obtain a trained fine-grained image recognition neural network;

[0089] The result acquisition module is used to input the test fine-grained image to be classified and the matching context information into the trained fine-grained image recognition neural network to obtain the recognition result of the test fine-grained image.

[0090] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the fine-grained image recognition method based on context information fusion provided in the above embodiment, which includes a memory and a processor;

[0091] The memory is used to store computer programs;

[0092] The processor is used to implement the fine-grained image recognition method based on context information fusion as in the above embodiment when executing the computer program.

[0093] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0094] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0095] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0096] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.< / lat> < / lon> < / date> < / x> < / x> < / x> < / lat> < / lon> < / date>

Claims

1. A fine-grained image recognition method based on context information fusion, characterized in that: The following steps are involved: S1: Obtain a multimodal fine-grained image dataset used for training, and preprocess the images and context information in the image dataset to obtain fine-grained image and context information representations of uniform size; S2: Construct a fine-grained image recognition neural network with dual-path multimodal context information fusion, the fine-grained image recognition neural network includes a visual encoder, a function-based context information fusion path, a template-based context information fusion path and an output aggregation module; wherein the visual encoder includes a convolutional neural network, a convolutional layer and a dimension conversion layer; the function-based context information fusion path fuses the context information representation and the visual features output by the visual encoder to obtain a combined multimodal representation, which is input into the global perception encoder for cross-modal interactive learning, and an aggregated and enhanced multimodal representation is output; the template-based context information fusion path fuses the context information representation and the aggregated and enhanced multimodal representation to obtain an aggregated and enhanced fused representation, which is input into the visual-context decoder for cross-modal interactive learning, and an interactively enhanced multimodal representation is output; the output aggregation module is used to parse the interactively enhanced multimodal representation, and each of the parsed representations is input into a classifier to generate a prediction result corresponding to each modality representation; S3: sending the preprocessed image data set into the fine-grained image recognition neural network, training the fine-grained image recognition neural network, and obtaining a trained fine-grained image recognition neural network; S4: Input the test fine-grained image to be classified and the matching context information into the trained fine-grained image recognition neural network to obtain the recognition result of the test fine-grained image.

2. The fine-grained image recognition method based on context information fusion as claimed in claim 1, characterized in that: In step S1, the method for preprocessing the images in the image data set is: resizing, randomly horizontally flipping, and normalizing the images in the image data set to obtain fine-grained images of uniform size.

3. The fine-grained image recognition method based on context information fusion according to claim 1, characterized in that: In step S1, the method for preprocessing the context information in the image data set is: first normalize the longitude and latitude in the context information, then divide the index of the date in the context information by 365 to normalize the date, splice the normalized longitude, latitude and date belonging to the same context information as a context information vector, multiply the normalized longitude, latitude and date with pi π respectively and then calculate by the cosine function, multiply the normalized longitude, latitude and date with pi π respectively and then calculate by the sine function, splice the calculation results of the cosine function and the calculation results of the sine function together, obtain the context information representation in the form of a one-dimensional tensor and use it as the first context information representation.

4. The fine-grained image recognition method based on context information fusion according to claim 1, characterized in that: In step S1, the method for preprocessing the context information in the image data set is: obtain a template in natural language form, fill in the longitude, latitude and date in the context information on the template, convert the context information into a filled template, and then input the filled template into a pre-trained language model, and the tokenizer method of the pre-trained language model tokenizes the filled template to obtain a context information representation in a sequence form and use it as the second context information representation.

5. The fine-grained image recognition method based on context information fusion as claimed in claim 1, characterized in that: In the visual encoder of step S2, the fine-grained image is taken as input, and the convolutional neural network first outputs a high-dimensional convolutional feature map, and then the high-dimensional convolutional feature map is reduced in dimension through the convolutional layer to obtain a low-dimensional convolutional feature map after dimensionality reduction, and then the low-dimensional convolutional feature map is input into the dimension conversion layer to obtain the visual features of the serialized representation.

6. The fine-grained image recognition method based on context information fusion as claimed in claim 3, characterized in that: In the function-based context information fusion path of step S2, the context feature encoder first performs a nonlinear transformation on the obtained first context information representation to obtain a context information representation after the nonlinear transformation, and concatenates the context information representation after the nonlinear transformation with the visual features output by the visual encoder to obtain a combined multimodal representation. The global perception encoder calculates the combined multimodal representation and outputs an aggregated enhanced multimodal representation.

7. The fine-grained image recognition method based on context information fusion as claimed in claim 4, characterized in that: In the template-based context information fusion path of step S2, a pre-trained language model is used to perform semantic extraction on the second context information representation to obtain a semantic context representation, and the visual-context decoder calculates the aggregated enhanced multimodal representation and the semantic context representation output by the function-based context information fusion path, and outputs an interactively enhanced multimodal representation.

8. The fine-grained image recognition method based on context information fusion as claimed in claim 1, characterized in that: In step S2, the output convergence module parses the interactively enhanced multimodal representation to obtain a contextual representation after interaction with the global image information and an image representation after interaction with the semantic context information, and then sends the contextual representation after interaction with the global image information to the first classifier to obtain a first output result, and sends the image representation after interaction with the semantic context information to the second classifier to obtain a second output result.

9. A fine-grained image recognition system based on context information fusion, characterized in that: include: A data acquisition module is used to acquire a multimodal fine-grained image dataset used for training, and preprocess the images and context information in the image dataset to obtain fine-grained images and context information representations of uniform size; A model acquisition module is used to construct a fine-grained image recognition neural network with dual-path multimodal context information fusion, wherein the fine-grained image recognition neural network includes a visual encoder, a function-based context information fusion path, a template-based context information fusion path and an output aggregation module; wherein the visual encoder includes a convolutional neural network, a convolutional layer and a dimension conversion layer; the function-based context information fusion path fuses the context information representation and the visual features output by the visual encoder to obtain a combined multimodal representation, which is input into the global perception encoder for cross-modal interactive learning, and an aggregated and enhanced multimodal representation is output; the template-based context information fusion path fuses the context information representation and the aggregated and enhanced multimodal representation to obtain an aggregated and enhanced fused representation, which is input into the visual-context decoder for cross-modal interactive learning, and an interactively enhanced multimodal representation is output; the output aggregation module is used to parse the interactively enhanced multimodal representation, and each of the parsed representations is input into a classifier to generate a prediction result corresponding to each modality representation; A model training module is used to send the preprocessed image data set into the fine-grained image recognition neural network, train the fine-grained image recognition neural network, and obtain a trained fine-grained image recognition neural network; The result acquisition module is used to input the test fine-grained image to be classified and the matching context information into the trained fine-grained image recognition neural network to obtain the recognition result of the test fine-grained image.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the fine-grained image recognition method based on context information fusion as described in any one of claims 1 to 8 when executing the computer program.

Images