Fine-grained image recognition method and system based on shared visual backbone network

Abstract

The invention relates to the technical field of image recognition and classification, in particular to a fine-grained image recognition method and system based on a shared visual backbone network. The method comprises the following steps: acquiring a fine-grained image data set; preprocessing the acquired image data; constructing a deep network model, including performing feature extraction by using the deep network model; object proposing and screening are carried out on the extracted feature map; performing feature pooling and information fusion on the screened features; performing global abstract extraction and gating context fusion on the fused features; projecting an object sequence in the gated and fused features to a feature space of a language model; training and optimizing the constructed deep network model; and performing fine-grained image classification by using the optimized model. According to the method, a parallel local feature extraction path is introduced, and a cross-attention fusion module is designed, so that the model can sense the global context and local discriminative details of the image at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition and classification, and in particular to a fine-grained image recognition method and system based on a shared visual backbone network. Background Art

[0002] In recent years, the field of artificial intelligence, particularly multimodal large language models, has made significant progress. These models, by combining a pre-trained visual feature extractor with a large language model, possess powerful general visual understanding and interaction capabilities, achieving outstanding performance in real-world tasks such as visual question answering, image description generation, and commonsense reasoning. The key to their success lies in the visual feature extractor's ability to refine an image into a set of global, contextually informed, and compact visual representations, which are then understood and processed by the language model.

[0003] However, despite the powerful capabilities of these large multimodal language models in general scenarios, their performance degrades significantly when faced with specific tasks that require domain expertise. This problem is particularly prominent in fine-grained visual recognition tasks. The goal of fine-grained visual recognition is to distinguish between highly visually similar subcategories within a larger category, such as different species of birds, different models of cars, or different types of aircraft. Completion of such tasks relies heavily on accurately identifying subtle, local, and discriminative visual cues in the image, such as the specific shape of a bird's crest or the distinctive style of a wheel's hub.

[0004] The fundamental reason why existing multimodal large language models perform poorly on such tasks lies in an inherent technical flaw: a "semantic alignment gap" between their visual representations and language concepts. Specifically, the single, holistic global visual features generated by the visual feature extractor in the model, while effective for understanding the macroscopic scene of the image, are inherently coarse-grained and lack the ability to accurately describe local details. When the language model is required to describe and distinguish subtle differences based on these features, this coarse-grained visual representation cannot provide sufficient basis for the textual concepts that describe fine features, resulting in the model only being able to generate generalized or even erroneous outputs when handling fine-grained tasks. Therefore, how to bridge this semantic gap and enhance the model's ability to perceive and express fine-grained visual features is a technical problem that urgently needs to be solved in the current field of multimodal technology. Summary of the Invention

[0005] In order to solve the technical problem in the prior art that the multimodal large language model has a "semantic alignment gap" when processing fine-grained visual recognition tasks due to its coarse visual representation granularity, and is therefore unable to generate accurate recognition results, the present invention provides a fine-grained image recognition method and system based on a shared visual backbone network.

[0006] In a first aspect, the present invention provides a fine-grained image recognition method based on a shared visual backbone network, which adopts the following technical solutions: A fine-grained image recognition method based on a shared visual backbone network, comprising: Obtain a fine-grained image dataset; Preprocessing the acquired image data; Constructing a deep network model, which includes: extracting features using the deep network model; proposing and filtering objects on the extracted feature maps; performing feature pooling and information fusion on the filtered features; performing global summary extraction and gated context fusion on the fused features; and projecting the object sequence in the gated fused features into the feature space of the language model. Train and optimize the constructed deep network model; Use the optimized model for fine-grained image classification.

[0007] Furthermore, the acquired image data is preprocessed, including formatting the original data set to integrate scattered data sources, wherein the data loading logic is defined by creating a data loading script; executing the data loading script, traversing the original data source, and generating a structured data set object according to the predefined logic; and in the process of generating the data set object, using memory mapping as the backend storage; and then normalizing the size and pixel value of the formatted image data, and performing image enhancement through cropping, flipping, erasing and fusion.

[0008] Furthermore, the feature extraction using the deep network model includes using a shared visual backbone network to perform deep forward propagation on the pre-processed single input image, wherein a visual encoder pre-trained on large-scale image-text data is selected as the backbone network, and the core hyperparameters of the network are set. For a batch size of Input image ,The feature extraction process includes the following steps: Step 1: Image segmentation and linear embedding, each image The spatial dimension is divided into non-overlapping image patches, and the flattened image patches are mapped to a dimension of feature space, thus obtaining the block embedding sequence ; Step 2: Input sequence construction, introducing learnable global context embedding , embedding the learnable position into the matrix Add element by element to the concatenated sequence to form the complete sequence of ViT encoder input , in, ; Furthermore, the feature extraction using the deep network model also includes step 3: Transformer encoder forward propagation; the constructed input sequence Input by Visual encoder composed of stacked Transformer layers ,go through After the depth nonlinear transformation of the layer, the model outputs the final feature sequence : ,in, Contains the depth encoding information of the input image; Step 4: Feature decoupling and output, from the encoder output sequence The two core outputs required for the subsequent process of the present invention are accurately separated: global context token , defined as: in It will be used in the subsequent feature fusion module for high-level query; spatial feature map , defined as: in .

[0009] Furthermore, the object proposal and screening of the extracted feature map includes converting the spatial feature map It is directly used as the input of the SAM mask decoder. The decoder densely samples query points on the feature map and autonomously generates thousands of candidate object masks. Then, the candidate object masks are screened out through a multi-stage screening process. A high-quality object, including the following steps: Step 1: Input SAM mask decoder , generates a file containing The set of original masks and the corresponding quality score set , expressed as ,in, , is a binary mask, , is the corresponding predicted IoU score; Step 2: Heuristic pre-filtering Each mask in Applying a filter function , expressed as ; Step 3: Non-maximum suppression, where the calculation formula of the mask intersection-over-union ratio is: ; Step 4: Top-K selection and bounding box generation from NMS output Among them, select the top masks, and get the final object mask set ,for Each mask in Calculate its minimum enclosing rectangle and get the corresponding bounding box , all bounding boxes constitute the final bounding box set .

[0010] Furthermore, the filtered features are subjected to feature pooling and information fusion, including using the RoIAlign algorithm to align each bounding box from the spatial feature map. The corresponding area is extracted and pooled to a preset, uniform spatial size. Then, global average pooling is applied to the feature map to compress its spatial dimension and generate a single feature vector for each object that represents its core appearance information. In order to eliminate the influence of image resolution on position information, the absolute coordinates of each bounding box are Divide by the width and height of the image to normalize it to The relative range of the normalized 4-dimensional coordinate vector is input into a multi-layer perceptron MLP specifically used to encode position information. The number of neurons in the output layer must be equal to the visual feature dimension D_v so that it can be fused with the appearance feature. Finally, the appearance feature vector of each object and its corresponding position encoding vector are fused by element-by-element addition.

[0011] Furthermore, the fused features are subjected to global summary extraction and gated context fusion, including using a cross attention layer to calculate the attention of the global context to all local objects, and generating a condensed global summary vector accordingly, wherein the global context token As the query vector, the initial object sequence As both the key vector and the value vector, let the input , through three independent learnable linear projection matrices , first transform the input into , then, according to the standard multi-head scaling dot product attention formula, the global summary vector is calculated ; Introducing lightweight gate control unit, Input Gated Projection Network , the output gate value , the value of each element is in the interval (0, 1), and finally the global summary information after gated adjustment is fused back into the original features of each object through element-by-element multiplication and addition, and the broadcast mechanism is used to complete the calculation.

[0012] Furthermore, the object sequence in the gated fusion feature is projected into the feature space of the language model, including respectively using a projection layer based on a multi-layer perceptron MLP and a projection layer based on a perceptron resampler to project the object sequence Projected into the feature space of the language model, where each input object feature vector is independently transformed in feature dimension through a combination of linear layers and nonlinear activation functions. The MLP consists of two linear layers and a GELU activation function, which is expressed as in, for: ; Introduce a fixed number of learnable latent query vectors , through the cross attention layer to actively extract the visual feature sequence from the input Extract and resample information from As the query Query in the cross attention layer, As both the key and the value, it is expressed as: , The final output visual token sequence This is the final product of the entire visual encoder.

[0013] Furthermore, the constructed deep network model is trained and optimized, including for a given training sample, the sample consists of an image and its corresponding answer text sequence consisting of T word units Token Composition, adopting the causal language modeling loss in the natural language generation task Calculation method: , in: is the t-th base word in the answer sequence, Represents the sequence of all previous benchmark words when predicting the t-th word , is a sequence of visual tokens, For all learnable parameters, The next word is predicted to be probability; then the entire complex model is directly fine-tuned end-to-end through a two-stage optimization training flow. The first stage is feature alignment pre-training, in which the parameters of the two main parts of the model are set to a frozen state, including the main parameters of the shared visual backbone network and the main parameters of the downstream large language model LLM. For each training sample, backpropagation is performed from the back end of the language model. When encountering the boundary of the frozen language model body, the gradient will stop propagating into it, and only propagate back to and update the parameters of the above-mentioned specified module to be trained; the second stage is end-to-end joint fine-tuning, which unfreezes all or part of the parameters of the shared visual backbone network and the large-scale language model frozen in the first stage to make them trainable. All trainable parameters of the entire model are jointly fine-tuned on the dataset of the fine-grained recognition task.

[0014] In the second aspect, a fine-grained image recognition system based on a shared visual backbone network includes: The data acquisition module is configured to acquire a fine-grained image dataset; A preprocessing module is configured to preprocess the acquired image data; The model building module is configured to build a deep network model, which includes: using the deep network model to extract features; performing object proposal and screening on the extracted feature map; performing feature pooling and information fusion on the filtered features; performing global summary extraction and gated context fusion on the fused features; and projecting the object sequence in the gated fused features into the feature space of the language model. The model training module is configured to train and optimize the constructed deep network model; The classification module is configured to perform fine-grained image classification using the optimized model.

[0015] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device for the fine-grained image recognition method based on a shared visual backbone network.

[0016] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded and executed by the processor to implement the fine-grained image recognition method based on a shared visual backbone network.

[0017] In summary, the present invention has the following beneficial technical effects: 1. Significantly improves the accuracy of fine-grained recognition and resolves the performance bottleneck of existing technologies.

[0018] The multimodal large language model in the background technology generally performs poorly on fine-grained tasks due to its coarse visual representation granularity. The present invention introduces a parallel local feature extraction path and designs a cross-attention fusion module, so that the model can simultaneously perceive the global context and local discriminative details of the image. Experimental verification results show that the average accuracy of the method of the present invention on multiple standard fine-grained recognition datasets reached 75.43%, which is a significant improvement compared to the 43.24% of the baseline model LLaVA1.5 and the 67.41% of the Qwen-VL-Chat model with stronger performance in the prior art. This quantitative result directly proves that the present invention effectively overcomes the performance defects of the prior art.

[0019] 2. Structurally, it bridges the "semantic gap" between vision and language, and enhances the intrinsic interpretability of the model. The core problem of the existing technology is that coarse-grained visual features cannot provide an effective basis for fine-grained language concepts. The "global query-local response" cross-attention fusion mechanism designed by the present invention solves this problem in theory. As a "query" for high-level semantic intent, the global feature vector can actively and specifically explore and extract the subtle visual clues that are most relevant to the current task from the "evidence pool" of the local feature library. This structure makes the enhanced features finally fed into the language model no longer a vague overall impression, but contains specific and focused local details, thereby providing a solid visual foundation for generating accurate fine-grained descriptions and enhancing the logic and interpretability of the model's decision-making process.

[0020] 3. It improves the versatility and deployment flexibility of the technical solution and lowers the application threshold. The shared visual backbone network feature extraction and fusion mechanism proposed in the present invention has good modularity and scalability in design. It can be used as a "plug-and-play" enhancement module, flexibly integrated into a variety of different existing multimodal large language model architectures without the need to make drastic modifications to the core components of these models. This design reduces the technical complexity and R&D costs of adapting general large models to professional fine-grained tasks, making it more convenient to apply to various professional fields that require expert-level visual perception, such as biometrics, industrial quality inspection, and medical image analysis. It has strong versatility and practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of a fine-grained image recognition method based on a shared visual backbone network according to embodiment 1 of the present invention; Figure 2 is another schematic diagram of a fine-grained image recognition method based on a shared visual backbone network according to embodiment 1 of the present invention; Figure 3 Schematic diagram of the experimental results of Example 1 of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below with reference to the accompanying drawings.

[0023] Example 1 Reference Figure 1 In this embodiment, a fine-grained image recognition method based on a shared visual backbone network includes: S1 obtaining fine-grained image data The execution of the method of the present invention first requires obtaining one or more fine-grained image datasets for model training, verification or actual recognition. The data acquisition methods include but are not limited to one or more of the following: 1. Loading from a preset database or dataset: directly loading one or more pre-built professional datasets for fine-grained visual recognition tasks from a local or remote storage device. 2. Collecting from the network: Targetedly capturing images in specific fields from public data sources on the Internet. 3. Real-time collection through dedicated equipment: In specific application scenarios, images that need to be fine-grained analyzed are directly acquired through image acquisition devices such as industrial cameras, high-power microscopes, digital cameras or scanners.

[0024] S2 processes the acquired image data This step aims to convert the image data obtained in S1 in the original file state into standardized, high-quality data suitable for efficient training and inference of large models.

[0025] S2.1 Formatting of image datasets To optimize data loading efficiency and standardize subsequent processing, this operation first formats the raw image dataset. Raw data typically exists as scattered files (for example, JPEG and PNG files stored in different folders) along with their corresponding annotations (such as category labels stored in text files or databases). The formatting operation consolidates these disparate information sources into a unified, structured dataset object.

[0026] In our implementation, the process of formatting the image dataset includes the following steps: 1. Define the data loading logic: Create a data loading script or configuration file. The core task of this script is to define the dataset metadata and reading logic. Specifically, it specifies the dataset features, such as an image feature named "image" and a class label feature named "label." It also defines how to generate samples from the original file path, that is, how to match and associate specific image files with their corresponding labels.

[0027] 2. Generate a structured dataset object: Execute the data loading script described above. This script traverses the original data source and generates a unified, structured dataset object based on predefined logic. This object encapsulates all samples into standardized records.

[0028] 3. Using efficient backend storage: When generating dataset objects, we use memory-mapped technology (for example, using formats like Apache Arrow) as the backend storage. This technology allows the system to efficiently access and operate on datasets without loading the entire dataset into physical memory all at once, enabling the processing of large datasets that far exceed memory capacity.

[0029] S2.2 Preprocessing of image data Before feature extraction, the image data extracted from the formatted dataset in S2.1 will undergo a series of carefully designed preprocessing operations. The purpose of these operations is to enhance the learning ability of the model and improve its generalization performance and robustness. The preprocessing techniques that can be used in the present invention include, but are not limited to, one or more combinations of the following: S 2.2.1 Size Normalization: Since vision models usually require fixed-size tensors as input, images of varying sizes in the dataset need to be uniformly processed into a preset fixed size (for example, 224x224 pixels or 448x448 pixels). Implementation methods include: Resizing: Directly resize the image to the target size, which may change the aspect ratio of the original image.

[0030] Padding: After scaling the image while maintaining its original aspect ratio, pad its edges (for example, with black pixels) to achieve the target size.

[0031] Cropping: Cropping a region of target size from an image, which can be center cropping, corner cropping, or random cropping.

[0032] S 2.2.2 Pixel Value Normalization: Linearly transform the pixel values ​​of an image from their original range (usually integers from 0-255) to a smaller standard range that is more suitable for neural network processing. For example: Scale to [0, 1]: Divide each pixel value by 255.

[0033] Normalization: Subtract the pixel mean of the entire dataset in each color channel and divide by the standard deviation.

[0034] S 2.2.3 Data Augmentation: Data augmentation technology can greatly expand the diversity of training samples without increasing the cost of additional data annotation, effectively suppressing model overfitting. The method of the present invention can use one or more of the following data augmentation methods: Random cropping: Randomly cropping an area from the original image and scaling it to the target size can effectively simulate the shape of objects at different scales and positions.

[0035] Random horizontal flip: Mirror the image horizontally with a certain probability.

[0036] Random Rotation: Randomly rotate the image within a given angle range.

[0037] Color jittering: Randomly change the brightness, contrast, saturation, and hue of an image within a certain range to improve the model's adaptability to lighting and color changes.

[0038] Random erasing: Randomly select a rectangular area on the image and fill it with random or fixed values ​​to simulate the scene where objects are partially occluded in the real world, forcing the model to focus on the global and other unoccluded features.

[0039] Mixup: Linearly superimpose two randomly selected images and their labels in a certain ratio to generate a new composite image.

[0040] CutMix: Crops a region from one image and pastes it onto another randomly selected image, with labels also fused in proportion to the area of ​​the mixed region.

[0041] S3 builds deep network models S3.1 How to build The deep network model proposed in this paper is built on a basic multimodal large language model and enhances its fine-grained perception capabilities by introducing specialized modules. In general, the model mainly consists of the following functional components: 1. Shared Visual Backbone Network: It replaces the single visual encoder in traditional models and its task is to extract two visual features of different granularities from the input image in parallel: a global feature representing the overall context and a set of local features containing multiple local details.

[0042] 2. Feature Fusion Module: This is another core innovation of the model. This module receives the shared visual backbone network features and, through a specially designed attention mechanism, intelligently incorporates the most discriminative local details into the global features, generating a richer and more comprehensive enhanced visual feature.

[0043] 3. Large Language Model: This part typically uses a pre-trained large-scale language model. It receives the enhanced visual features generated by the feature fusion module and uses them as visual evidence to understand the image content and generate the final fine-grained text description or classification results.

[0044] S3.2 Shared Visual Backbone Network and Feature Extraction This invention eliminates the computational redundancy and feature space inconsistency issues inherent in traditional dual-stream or multi-stream architectures, proposing an efficient feature extraction solution based on a shared visual backbone network. The core of this solution lies in performing a single deep forward propagation on a single input image preprocessed in step S2 to simultaneously and homogenously generate the two key visual representations required by subsequent modules: a global context token representing the global semantics of the image, and a spatial feature map that retains rich spatial information. This not only significantly improves computational efficiency but also ensures consistency and coherence between global and local information in the underlying representation.

[0045] In one example, the shared visual backbone network uses a Vision Transformer (ViT)-based architecture, with the following specifications and processing flow: S3.2.1 Backbone network configuration and initialization Architecture selection: Use a visual encoder pre-trained on large-scale image-text data as the backbone network, such as the visual part of the EVA-02-CLIP or SigLIP models. This ensures that the model has strong general visual prior knowledge.

[0046] Core parameters: To cover different implementation scenarios, the core hyperparameters of the network can be varied within an optimal range: Patch Size 14 or 16; Number of Transformer layers 12 to 24; Number of Heads 12 to 16; Feature Embedding Dimension 768, 1024, or 1280.

[0047] Weight initialization: The initial weights of the backbone network are obtained by loading the pre-trained model mentioned above. In the training process described in S4, these weights will be further fine-tuned to make the model better suited to the target fine-grained recognition task.

[0048] S3.2.2 Forward Propagation and Feature Generation Process For a batch size of Input image , its feature extraction process can be rigorously described as the following steps: Step 1: Image segmentation and linear embedding First, each image The spatial dimension is divided into non-overlapping image patches. Subsequently, a learnable linear projection layer (determined by the weight matrix Definition) maps these flattened image blocks to a feature space, thus obtaining the block embedding sequence .

[0049] Step 2: Input sequence construction Introducing a learnable global context embedding (Class Token) that is independent of the specific image content This embedding is concatenated into the sequence of block embeddings To enable the model to perceive the original spatial position of the image patch, a learnable position embedding matrix is added element by element to the concatenated sequence. Finally, the complete sequence that constitutes the input of the ViT encoder is It can be expressed by the following formula: , in, .

[0050] Step 3: Transformer encoder forward propagation The constructed input sequence Input by Visual encoder composed of stacked Transformer layers Each Transformer layer is mainly composed of a Multi-Head Self-Attention module and a Feed-Forward Network. After the depth nonlinear transformation of the layer, the model outputs the final feature sequence : , in, Contains the depth encoding information of the input image.

[0051] Step 4: Feature decoupling and output Finally, the output sequence from the encoder The two core outputs required for the subsequent process of the present invention are precisely separated: 1. Global context token Extract the feature vector at the first position (index 0) in the sequence. After multiple layers of self-attention information aggregation, this vector has been separated from any specific spatial position and forms a highly abstract and holistic semantic summary of the entire image content. It is defined as: , in It will be used in the subsequent feature fusion module for high-level intent "Query".

[0052] 2. Spatial Feature Map Extract all remaining feature vectors except the first position in the sequence. These vectors correspond to the original image blocks one by one, thus completely preserving the spatial layout information of the image and forming the feature basis for subsequent local object perception. It is defined as: , in In the specific implementation, its dimensions can be rearranged (reshaped) into a form with a two-dimensional spatial structure , to facilitate subsequent spatial alignment operations with segmentation masks (SegmentationMasks).

[0053] S3.3 Object Proposal and Screening This module inherits the spatial feature map generated by the shared visual backbone network in Section S3.2. Its core technical goal is to segment and filter a set of high-quality, diverse, and representative object-level regions from an image through a multi-stage, automated processing pipeline. This process aims to resolve the contradiction between the massive, redundant candidate regions generated by general segmentation models (such as SAM) and the high-quality, refined object sets required for downstream tasks. It is a key step in achieving the transition from macroscopic scene understanding to microscopic detail focus. In one example, the module mainly consists of two core components: object proposal generation and multi-stage screening.

[0054] 1. Object Proposal Generation Method. Model Selection: Use a large-scale segmentation model, such as the Mask Decoder part of the Segment Anything Model (SAM), as the object proposal generation engine. Working Mode: This invention adopts a fully automatic (Prompt-Free) working mode. Specifically, the spatial feature map output in Section S3.2 is The SAM mask decoder directly uses these points as input, without any external cues (such as points, boxes, or text). The decoder autonomously generates thousands of candidate object masks by densely sampling query points on the feature map. This automated design is the foundation of our invention's end-to-end, hands-free processing workflow.

[0055] 2. Multi-stage screening process. In order to extract the final In order to find high-quality objects, the present invention designs a cascade screening process, and each stage uses different strategies to optimize the candidate set. Heuristic Pre-filtering: This is the first coarse screening, which aims to quickly eliminate candidate masks that obviously do not meet the requirements. The screening criteria may include: Area threshold: Remove masks with an area that is too small (e.g., less than 0.1% of the total image area) or too large (e.g., greater than 80% of the total image area). The former is usually noise, and the latter lacks locality. Quality score threshold: SAM provides a predicted intersection-over-union (IoU) score for each generated mask. Masks with a score lower than a preset threshold (e.g., 0.80) are removed to ensure basic quality. Non-Maximum Suppression (NMS): This is the second fine screening, which aims to eliminate the high spatial redundancy between candidate masks. IoU calculation method: Mask IoU is used instead of the traditional bounding box IoU as the suppression criterion because the former can more accurately measure the degree of overlap between irregularly shaped objects. NMS threshold: Set a mask IoU threshold (e.g., 0.7). During the iteration process, the mask with the current highest quality score is retained, and all masks whose IoU with the remaining masks exceeds the threshold are removed. Top-K Selection: From the set of masks with high diversity after NMS processing, the final masks are sorted from high to low according to their quality scores (predicted IoU). A mask. is a configurable hyperparameter whose preferred range is 32 to 128 to strike a balance between preserving enough details and controlling computational cost.

[0056] The processing flow of this module can be formalized as follows: 1. Object Proposal Generation Will Input SAM mask decoder , generates a file containing The set of original masks and the corresponding quality score set .

[0057] , in, , is a binary mask. , is the corresponding predicted IoU score.

[0058] 2. Heuristic pre-filtering right Each mask in Applying a filter function This function returns true if and only if the mask satisfies all pre-set conditions (area, score).

[0059] , 3. Non-maximum suppression This process can be described by the following pseudo code: function Mask_NMS(Masks, Scores, IoU_threshold): / / Sort by score in descending order Sorted_Indices = argsort(Scores, descending=True) keep_masks = [] while Sorted_Indices is not empty: / / Select the mask with the highest current score current_idx=Sorted_Indices[0] Kept_Masks.add(Masks[current_idx]) / / Calculate its IoU with the remaining mask remaining_indices = Sorted_Indices[1:] ious=calculate_mask_iou(Masks[current_idx],Masks[remaining_indices]) / / Remove masks that overlap too much indices_to_remove = where(ious > IoU_threshold) Sorted_Indices = remove(remaining_indices, indices_to_remove) return Kept_Masks The calculation formula of mask intersection-over-union ratio is: , The sum operation here means counting the elements with a value of 1 in the binary matrix, and AND and OR are pixel-by-pixel logical operations.

[0060] 4. Top-K Selection and Bounding Box Generation Output from NMS Among them, select the top masks, and get the final object mask set .

[0061] for Each mask in Calculate its minimum enclosing rectangle and get the corresponding bounding box All bounding boxes constitute the final bounding box set .

[0062] S3.4 Location-aware object feature extraction The goal of this module is to generate a feature vector for each object selected in Section S3.3 that contains both its appearance and spatial location information. By decoupling these two types of information and then fusing them, we can provide a structured, information-rich sequence of local visual features for the subsequent language model, which is the basis for accurate language description of fine-grained features.

[0063] The input of this module is the spatial feature map generated in Section S3.2 and the final bounding box set generated in Section S3.3 The processing flow mainly includes three steps: feature pooling, position encoding and information fusion: 1. Feature pooling.

[0064] The RoIAlign (Region of Interest Align) algorithm is used. RoIAlign avoids coordinate quantization errors through bilinear interpolation and can more accurately align and pool object regions of any size onto a fixed-size feature map, which is crucial for preserving the fine spatial information required for fine-grained tasks. For each bounding box, RoIAlign is used to obtain the spatial feature map. The corresponding regions are extracted from the image and pooled to a preset, uniform spatial size (e.g., 7 × 7). Global Average Pooling (GAP) is then applied to this fixed-size feature map to compress its spatial dimensions and generate a single feature vector for each object that represents its core appearance information.

[0065] 2. Position encoding.

[0066] To eliminate the influence of image resolution on position information, the absolute coordinates of each bounding box are first Divide by the width and height of the image to normalize it to The normalized 4D coordinate vector is input into a multi-layer perceptron (MLP) specifically designed to encode position information. An example MLP structure is: Input layer: 4 neurons. Hidden layer: 1 to 2 fully connected layers, the number of neurons can be set to 256 or 512, and ReLU or GELU is used as the nonlinear activation function. Output layer: The number of neurons must be equal to the visual feature dimension , in order to merge with the appearance features.

[0067] 3. Information fusion.

[0068] The simplest and most efficient element-wise addition method is used to fuse the appearance feature vector of each object with its corresponding position encoding vector. This method is computationally very low-cost and has been proven to be very effective in injecting auxiliary information.

[0069] Let the input be the spatial feature map For the first objects The processing flow of this module can be formalized as follows: 1. Feature pooling.

[0070] make For the First, apply the RoIAlign operation, and then perform global average pooling to obtain the appearance feature vector : , in, The appearance features of all objects constitute the appearance feature sequence .

[0071] 2. Position encoding.

[0072] Bounding Box Normalize it and get .Will Input Position Encoding MLP Get position embedding : , in, .

[0073] MLP It can be expressed as: , in, are learnable weights and biases.

[0074] 3. Information fusion.

[0075] Add the appearance features and position features to get the final object feature vector : , all The final feature vector of the objects constitutes the output of this module, that is, the initial object sequence

[0076]

[0077] S3.5 Gated Context Fusion This module is the core innovation of the present invention in achieving fine-grained recognition, and aims to solve the problem of inaccurate semantic alignment caused by the separation or simple splicing of global and local information in the existing technology. This module receives two key inputs: the global context token generated in Section S3.2, which represents the overall high-level semantics of the image, and the global context token generated in Section S3.2, which represents the overall high-level semantics of the image. , and the initial object sequence generated by Section S3.4, which contains the appearance and position information of multiple objects .

[0078] Its core technical idea is to establish a dynamic information interaction mechanism of "global query-local response". In this mechanism, the global intention is represented by Rather than directly concatenating the local features, it acts as a "query," actively exploring the "evidence base" composed of all local object features and adaptively aggregating the most relevant global summary information for the current recognition task. This summary information is then intelligently and selectively fused back into each object feature through a learnable gating unit, generating a final object sequence that retains the accuracy of local details while enhancing global context. In one example, this module achieves this through three closely linked steps: cross-attention, gated generation, and gated fusion.

[0079] 1. Global summary extraction based on cross-attention.

[0080] First, a cross-attention layer is used to calculate the "attention" of the global context to all local objects, and a condensed global summary vector is generated based on this. Specifically, the global context token As the query vector (Query, Q), the initial object sequence As both the key vector (Key, K) and the value vector (Value, V). Let the input . Through three independent learnable linear projection matrices . First, transform the input into : , , , Then, according to the standard Multi-Head Scaled Dot-ProductAttention formula, the global summary vector is calculated : , in, is the dimension of the key vector, which is used to scale the dot product result to maintain the gradient stability. The output of this step It is considered as a summary that is "refined and focused on the global context based on all local object information."

[0081] 2. Gating Unit Generation In order to achieve adaptive control of global information, the present invention introduces a lightweight gating unit. This unit is based on the global summary vector generated in the previous step. , dynamically generates a control signal (i.e., gate value) for the subsequent fusion steps. Input a dedicated gated projection network with small number of parameters In one instance, It can be composed of a simple linear layer and a Sigmoid activation function. The calculation process is as follows: , in, and is a learnable parameter. Is the Sigmoid activation function. The gate value of the output , the value of each element is in the interval (0, 1), which can be understood as "what proportion of global summary information is allowed to pass through". In order to expand the scope of patent protection, More complex structures can also be used, such as MLP with multiple hidden layers, or using other activation functions such as Hard Sigmoid.

[0082] 3. Gated Fusion The final step is to fuse the gated global summary information back into the original features of each object. This step is implemented by element-by-element multiplication and addition, and the broadcasting mechanism is used to efficiently complete the calculation. Let the initial object sequence be , the global summary vector is , the gate value is The final object sequence after enhancement The calculation is as follows: , in, stands for element-wise multiplication (Hadamard Product). In this operation, the dimension is of and Will be automatically broadcast to match the dimensions of , thereby enhancing the features of each object in the sequence with the same gated global information. This gated fusion mechanism allows the model to learn, during end-to-end training, how much global context to inject into local features in different scenarios. For example, when recognizing an object that relies on the entire scene, the model may learn to generate a larger gating value; whereas, when recognizing an object with very distinct features, it may generate a smaller gating value, thus avoiding unnecessary interference from global information.

[0083] S3.6 Projection Layer for Aligned Language Models This module is the key bridge connecting the visual encoder and the downstream large-scale language model (LLM). Its core task is to convert the generated data in Section S3.5 and in the visual feature space (dimension ) , projected to the feature space of the language model (dimension is This step not only solves the problem of feature dimension mismatch between the two modalities, but more importantly, it achieves a preliminary alignment of visual features to language semantics through a learnable transformation, enabling the language model to "understand" and "process" this visual information.

[0084] The input of this module is the enhanced object sequence , the output is a set of vision tokens that can be directly used by the language model To cover a wider range of implementations, the present invention discloses the following two or more optional projection layer implementation solutions: 1. Multi-layer perceptron (MLP) based projection layer.

[0085] This is a direct and efficient implementation. By combining one or more linear layers with nonlinear activation functions, each input object feature vector is independently transformed in its feature dimension. In a preferred embodiment, the MLP can be composed of two linear layers and a GELU (Gaussian Error Linear Unit) activation function. Its mathematical expression is as follows: , in, The specific calculation process is: In this structure, the input linear layer Change the feature dimension from Map to an intermediate dimension and output linear layer Then map it to the final language model dimension .

[0086] 2. Projection layer based on Perceiver Resampler This is a more complex alternative implementation based on the attention mechanism, which is inspired by the Perceiver series model. It introduces a fixed number (k) of learnable latent query vectors (Latent Queries) These query vectors act as “information probes” and are used through a cross-attention layer to actively extract information from the input visual feature sequence. Specifically, As the query in the cross-attention layer, It serves as both a key and a value. The calculation process is as follows: , Compared with the point-by-point independent transformation of MLP, this solution uses the attention mechanism to comprehensively aggregate and reorganize all object features while projecting. This allows each visual token generated to contain contextual information associated with other object features, thereby providing a higher-level, pre-processed visual input for the language model.

[0087] Through any of the above schemes, the visual token sequence finally output by this module is This is the final product of the entire visual encoder. This sequence is aligned with the downstream language model in both feature dimension and semantic level. It can be directly concatenated with its own word embeddings and fed into the main body of the language model to perform subsequent fine-grained recognition and text generation tasks.

[0088] S4 trains and optimizes the constructed model This section details a systematic process for learning and optimizing the deep network model defined in Section S3 that incorporates the core mechanisms of this invention. The goal of this process is to fully optimize all learnable parameters of the model by training it on a dataset of relevant fine-grained image and text pairs, ultimately enabling the entire model to accurately understand discriminative visual details in the input image and generate fully corresponding, accurate text descriptions or classifications.

[0089] S4.1 Training Objectives and Loss Functions The core goal of the training process of the present invention is to maximize the conditional probability that the model generates its corresponding ground-truth text sequence given an input image. This is an end-to-end auto-regressive training process, and its training signal will be back-propagated from the final text output layer to every learnable component of the entire model, including but not limited to: the main parameters of the downstream language model, the projection layer in Section S3.6, the gated context fusion module in Section S3.5 (including cross-attention and gated units), the positional encoding MLP in Section S3.4, and part or all of the parameters of the shared visual backbone network in Section S3.2. To achieve the above goals, the present invention adopts the causal language modeling loss (Causal Language Modeling Loss) which is widely used in natural language generation tasks. For a given training sample, the sample consists of an image and its corresponding answer text sequence consisting of T tokens. Its loss is calculated as follows: , in: is the tth base word in the answer sequence. Represents the sequence of all previous benchmark words when predicting the t-th word . is the final sequence of visual tokens generated by the visual encoder of the present invention (Sections S3.2 to S3.6) for the input image. Represents all learnable parameters of the entire model. Is the language model given a visual token and above Under the condition of , the next word is predicted to be This probability is obtained by applying the Softmax function to the output of the language model in the vocabulary dimension. This loss function is essentially the sum of the cross-entropy losses calculated at each time step. It penalizes the model for incorrect predictions on any token, thereby driving the entire model to learn the precise mapping relationship between image content and text description.

[0090] S4.2 Two-stage optimization training process To ensure training stability and achieve optimal performance, this paper proposes a carefully designed two-stage optimization training process. This process aims to address the training instability issues that can arise from directly fine-tuning the entire complex model end-to-end, particularly the performance degradation caused by the noise input to the pre-trained language model by newly introduced, randomly initialized modules (such as the gated fusion module and projection layer).

[0091] Stage 1: Feature Alignment Pre-training The core goal of this stage is to train the newly introduced modules responsible for connection and fusion functions while keeping the strong prior knowledge of the large-scale pre-trained models (visual backbone network and language model) basically unchanged.

[0092] Parameter Freezing Strategy: In this phase, the parameters of two main parts of the model are set to a frozen state, that is, their weights are not updated during the back-propagation process. Specifically, this includes the main parameters of the shared visual backbone network described in Section S3.2, and the main parameters of the downstream large language model (LLM).

[0093] Training target parameters: The only goal of training is to optimize the parameters of the newly introduced, randomly initialized modules related to the core mechanism of the present invention. In a preferred embodiment, these parameters to be trained specifically include: the MLP ( ) parameters; all parameters of the gated context fusion module in Section S3.5 (including the three linear projection layers used to generate Q, K, V , and gate control unit ); and the parameters of the projection layer (either the MLP implementation or the Perceiver Resampler implementation) used to align the language model in Section S3.6.

[0094] Data Flow and Gradient Flow: For each training example, the forward propagation path of the data remains intact. However, the key point is that the gradient flow is restricted: the gradient calculated by the loss function defined in Section S4.1, when backpropagating from the language model backend, stops propagating into the frozen language model body when it encounters the boundary of the frozen language model body. Instead, the gradient is propagated back only to update the parameters of the specified training module.

[0095] Purpose of this stage: This stage can be understood as an "adapter" pre-training process. Its purpose is to enable these new "connectors" and "fuser" to quickly learn how to convert the features extracted by the visual backbone network into meaningful representations that the language model can "understand", thereby laying a good initial state for the next stage of joint fine-tuning of the entire model.

[0096] Phase 2: End-to-end joint fine-tuning After completing the first phase of feature alignment, the newly introduced modules have acquired preliminary and reasonable feature transformation capabilities. At this point, we enter the second phase, which involves global and coordinated optimization of the entire model.

[0097] Parameter unfreezing: All or part of the parameters of the shared visual backbone network and large-scale language model that were frozen in the first stage are unfrozen to make them trainable.

[0098] Joint fine-tuning: On the dataset of the fine-grained recognition task, all trainable parameters of the entire model are jointly fine-tuned using the loss function defined in Section S4.1. In this case, the gradient signal can flow unimpeded to every corner of the model.

[0099] Phase Objective: This phase enables the visual perception front-end, feature fusion mid-end, and language generation back-end to function as an integrated whole, adapting to the specific data distribution and inherent requirements of fine-grained tasks. For example, the language model can use gradient signals to "inform" the visual backbone network which types of visual patterns to focus on. Simultaneously, the gated fusion module will learn more sophisticated global-local information fusion strategies under the goal of global optimization, thereby achieving optimal performance for the entire system.

[0100] S4.3 Optimizer and Learning Rate Scheduling Strategy To ensure the efficiency and stability of the above training process, the present invention adopts the following optimizer and learning rate scheduling strategy: Optimizer: Uses the AdamW (Adam with Decoupled Weight Decay) optimizer. Compared to the traditional Adam optimizer, AdamW decouples the weight decay term from gradient updates, more effectively preventing model overfitting. This often results in better performance and stability when fine-tuning large pre-trained models.

[0101] Learning rate scheduling strategy: A cosine annealing strategy with warm-up is used to dynamically adjust the learning rate. This strategy consists of two stages: Warm-up phase: During a small number of iterations at the beginning of training (e.g., the first 10% of training steps), the learning rate is increased from a very small value (e.g., ) linearly increases to a preset base learning rate (such as This helps avoid "oscillations" caused by excessive learning rates in the early stages of training when model parameters are still unstable.

[0102] Annealing phase: After the warmup phase, the learning rate decays smoothly from the base learning rate to a minimum close to zero, following the shape of a cosine function. This smooth decay helps the model converge more stably to the local optimum of the loss function in the later stages of training.

[0103] S5 Using the model for fine-grained image classification This section describes how to deploy and utilize the trained deep network model proposed in this invention after completing the systematic training and optimization process described in S4 to perform efficient and accurate fine-grained recognition tasks on new images that have not been seen in the training phase.

[0104] S5.1 Model deployment and loading Model weight solidification and preservation: After the S4 training process is completely completed, the final model state containing all fully optimized learnable parameters (Θ) will be serialized and saved as one or more standard format model files (for example, PyTorch's .pth or .safetensors format).

[0105] Application Environment Loading: Within an application system or computing device providing the actual recognition service (such as a cloud server or edge computing device), an untrained deep network model instance, identical to the one used during training, is first constructed in memory according to the complete architecture defined in Section S3. The solidified model file is then loaded into this architecture, accurately loading the optimized parameters into the corresponding components of the model. Once loaded, the model is ready and can be switched to inference / evaluation mode, ready to receive and process recognition requests.

[0106] S5.2 Inference Process When a recognition request arrives, the system receives an input image to be recognized and performs a complete forward pass to obtain the final recognition result. This inference process is consistent with the data flow during training, but does not involve any backpropagation or parameter updates. The specific process is as follows: Step 1: Input image preprocessing The input images to be recognized are preprocessed exactly as described in Section S2.2. This includes: 1) size normalization, such as scaling, padding, or cropping to adjust the image to the fixed input size required for model training (e.g., 448 × 448 pixels); and 2) pixel value normalization, such as transforming pixel values ​​from the range [0, 255] to the range [0, 1] and normalizing them according to the mean and standard deviation used during training.

[0107] Step 2: Shared backbone network feature extraction The preprocessed image tensor is fed into the shared visual backbone defined in Section S3.2. After a ViT forward pass, two outputs are obtained in parallel: a global context token representing the global semantics of the image and a , and spatial feature maps that retain rich spatial information .

[0108] Step 3: Object Proposal and Screening The spatial feature map The input is sent to the object proposal and filtering module defined in Section S3.3. Through the fully automatic segmentation of the SAM mask decoder, heuristic pre-filtering, non-maximum suppression (NMS) based on mask IoU, and Top-K selection, a set of (k) high-quality and diverse object bounding boxes is finally generated. .

[0109] Step 4: Location-aware object feature extraction The spatial feature map and bounding box sets The two are input to the module defined in Section S3.4. For each bounding box, its appearance features are extracted through RoIAlign and global average pooling, and its normalized coordinates are encoded into position features through an MLP. After adding the two, the initial object sequence is obtained. .

[0110] Step 5: Gated Context Fusion The global context token and the initial object sequence The input is sent to the core innovation module defined in Section S3.5. The global summary is extracted by cross attention, the control signal is generated by the gate unit, and finally the enhanced final object sequence is generated through the gated fusion operation. .

[0111] Step 6: Projection and Alignment The enhanced object sequence Input to the projection layer defined in Section S3.6. Through structures such as MLP or PerceiverResampler, it is projected from the visual feature space to the language model feature space to obtain the final visual token sequence aligned with the language model .

[0112] Step 7: Autoregressive text generation The final visual token sequence This condition is then fed into a downstream large-scale language model. The language model then performs an autoregressive text generation process to generate a recognition result for the input image. This process typically employs strategies such as beam search, nucleus sampling, or greedy decoding to generate the most likely tokens one by one until an end-of-sequence token is encountered or a preset maximum length is reached. The resulting text sequence is the fine-grained recognition result for the input image.

[0113] Experimental verification: In order to objectively and quantitatively verify the effectiveness of the technical solution proposed in this invention and its beneficial effects, we conducted a series of comparative experiments. We selected a number of standard datasets recognized in the field of fine-grained visual recognition and tested the performance of the deep network model implemented in this invention with a variety of other mainstream and representative multimodal large language models in the current field under the same conditions. The evaluation indicator is the classification accuracy, that is, the percentage of image subcategories correctly identified by the model. Figure 3 As shown in FIG, by comparing the accuracy of the model of the present invention with other baseline models on these data sets, we can intuitively judge the superiority of the present invention in solving the fine-grained recognition problem.

[0114] Example 2 This embodiment provides a fine-grained image recognition system based on a shared visual backbone network, including: The data acquisition module is configured to acquire a fine-grained image dataset; A preprocessing module is configured to preprocess the acquired image data; The model building module is configured to build a deep network model, which includes: using the deep network model to extract features; performing object proposal and screening on the extracted feature map; performing feature pooling and information fusion on the filtered features; performing global summary extraction and gated context fusion on the fused features; and projecting the object sequence in the gated fused features into the feature space of the language model. The model training module is configured to train and optimize the constructed deep network model; The classification module is configured to perform fine-grained image classification using the optimized model.

[0115] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a fine-grained image recognition method based on a shared visual backbone network.

[0116] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded and executed by the processor to implement a fine-grained image recognition method based on a shared visual backbone network.

[0117] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A fine-grained image recognition method based on a shared visual backbone network, characterized in that: include: Obtain a fine-grained image dataset; Preprocessing the acquired image data; Building a deep network model, including using the deep network model for feature extraction; Perform object proposal and screening on the extracted feature maps; perform feature pooling and information fusion on the filtered features; perform global summary extraction and gated context fusion on the fused features; project the object sequence in the gated fused features into the feature space of the language model; Train and optimize the constructed deep network model; Use the optimized model for fine-grained image classification.

2. A fine-grained image recognition method based on a shared visual backbone network according to claim 1, characterized in that: The preprocessing of the acquired image data includes formatting the original data set to integrate the scattered data sources, wherein the data loading logic is defined by creating a data loading script; executing the data loading script, traversing the original data source, and generating a structured data set object according to the predefined logic; and in the process of generating the data set object, using memory mapping as the backend storage; and then normalizing the size and pixel value of the formatted image data, and then performing image enhancement through cropping, flipping, erasing and fusion.

3. The fine-grained image recognition method based on a shared visual backbone network according to claim 2, characterized in that: The feature extraction using the deep network model includes using a shared visual backbone network to perform deep forward propagation on a pre-processed single input image, wherein a visual encoder pre-trained on large-scale image-text data is selected as the backbone network, and the core hyperparameters of the network are set. For a batch size of Input image ,The feature extraction process includes the following steps: Step 1: Image segmentation and linear embedding, each image The spatial dimension is divided into non-overlapping image patches, and the flattened image patches are mapped to a dimension of feature space, thus obtaining the block embedding sequence ; Step 2: Input sequence construction, introducing learnable global context embedding , embedding the learnable position into the matrix Add element by element to the concatenated sequence to form the complete sequence of ViT encoder input , in, .

4. The fine-grained image recognition method based on a shared visual backbone network according to claim 3 is characterized in that: The feature extraction using the deep network model also includes step 3: Transformer encoder forward propagation; the constructed input sequence Input by Visual encoder composed of stacked Transformer layers ,go through After the depth nonlinear transformation of the layer, the model outputs the final feature sequence : ,in, Contains the depth encoding information of the input image; Step 4: Feature decoupling and output, from the encoder output sequence The two core outputs required for the subsequent process of the present invention are accurately separated: global context token , defined as: in It will be used in the subsequent feature fusion module for high-level query; spatial feature map , defined as: in .

5. The fine-grained image recognition method based on a shared visual backbone network according to claim 4 is characterized in that: The object proposal and screening of the extracted feature map includes converting the spatial feature map It is directly used as the input of the SAM mask decoder. The decoder densely samples query points on the feature map and autonomously generates thousands of candidate object masks. Then, the candidate object masks are screened out through a multi-stage screening process. A high-quality object, including the following steps: Step 1: Input SAM mask decoder , generates a file containing The set of original masks and the corresponding quality score set , expressed as ,in, , is a binary mask, , is the corresponding predicted IoU score; Step 2: Heuristic pre-filtering Each mask in Applying a filter function , expressed as ; Step 3: Non-maximum suppression, where the calculation formula of the mask intersection-over-union ratio is: ; Step 4: Top-K selection and bounding box generation from NMS output Among them, select the top masks, and get the final object mask set ,for Each mask in Calculate its minimum enclosing rectangle and get the corresponding bounding box , all bounding boxes constitute the final bounding box set .

6. The fine-grained image recognition method based on a shared visual backbone network according to claim 5, characterized in that: The filtered features are subjected to feature pooling and information fusion, including using the RoIAlign algorithm to align each bounding box from the spatial feature map. The corresponding area is extracted and pooled to a preset, uniform spatial size. Then, global average pooling is applied to the feature map to compress its spatial dimension and generate a single feature vector for each object that represents its core appearance information. In order to eliminate the influence of image resolution on position information, the absolute coordinates of each bounding box are Divide by the width and height of the image to normalize it to The relative range of the normalized 4-dimensional coordinate vector is input into a multi-layer perceptron MLP specifically used to encode position information. The number of neurons in the output layer must be equal to the visual feature dimension D_v so that it can be fused with the appearance feature. Finally, the appearance feature vector of each object and its corresponding position encoding vector are fused by element-by-element addition.

7. The fine-grained image recognition method based on a shared visual backbone network according to claim 6, characterized in that: The fused features are subjected to global summary extraction and gated context fusion, including using a cross attention layer to calculate the attention of the global context to all local objects, and generating a condensed global summary vector based on this, wherein the global context token As the query vector, the initial object sequence As both the key vector and the value vector, let the input , through three independent learnable linear projection matrices , first transform the input into , then, according to the standard multi-head scaling dot product attention formula, the global summary vector is calculated ; Introducing lightweight gate control unit, Input Gated Projection Network , the output gate value , the value of each element is in the interval (0, 1), and finally the global summary information after gated adjustment is fused back into the original features of each object through element-by-element multiplication and addition, and the broadcast mechanism is used to complete the calculation.

8. The fine-grained image recognition method based on a shared visual backbone network according to claim 7, characterized in that: The object sequence in the gated fusion feature is projected into the feature space of the language model, including respectively using a projection layer based on a multi-layer perceptron MLP and a projection layer based on a perceptron resampler to project the object sequence Projected into the feature space of the language model, where each input object feature vector is independently transformed in feature dimension through a combination of linear layers and nonlinear activation functions. The MLP consists of two linear layers and a GELU activation function, which is expressed as in, for: ; Introduce a fixed number of learnable latent query vectors , through the cross attention layer to actively extract the visual feature sequence from the input Extract and resample information from As the query Query in the cross attention layer, As both the key and the value, it is expressed as: ， The final output visual token sequence This is the final product of the entire visual encoder.

9. The fine-grained image recognition method based on a shared visual backbone network according to claim 8, characterized in that: The training and optimization of the constructed deep network model includes: for a given training sample, the sample consists of an image and its corresponding answer text sequence consisting of T word units Token Composition, adopting the causal language modeling loss in the natural language generation task Calculation method: ， in: is the t-th base word in the answer sequence, Represents the sequence of all previous benchmark words when predicting the t-th word , is a sequence of visual tokens, For all learnable parameters, The next word is predicted to be probability; then the entire complex model is directly fine-tuned end-to-end through a two-stage optimization training flow. The first stage is feature alignment pre-training, in which the parameters of the two main parts of the model are set to a frozen state, including the main parameters of the shared visual backbone network and the main parameters of the downstream large language model LLM. For each training sample, backpropagation is performed from the back end of the language model. When encountering the boundary of the frozen language model body, the gradient will stop propagating into it, and only propagate back to and update the parameters of the above-mentioned specified module to be trained; the second stage is end-to-end joint fine-tuning, which unfreezes all or part of the parameters of the shared visual backbone network and the large-scale language model frozen in the first stage to make them trainable. All trainable parameters of the entire model are jointly fine-tuned on the dataset of the fine-grained recognition task.

10. A fine-grained image recognition system based on a shared visual backbone network, characterized in that: include: The data acquisition module is configured to acquire a fine-grained image dataset; A preprocessing module is configured to preprocess the acquired image data; The model building module is configured to build a deep network model, which includes: using the deep network model to extract features; performing object proposal and screening on the extracted feature map; performing feature pooling and information fusion on the filtered features; performing global summary extraction and gated context fusion on the fused features; and projecting the object sequence in the gated fused features into the feature space of the language model. The model training module is configured to train and optimize the constructed deep network model; The classification module is configured to perform fine-grained image classification using the optimized model.

Images