Fine granularity identification method and system based on multi-mode large model collaborative distillation and application
By employing multimodal large-model collaborative distillation technology, the problems of high-cost annotation and insufficient generalization ability in fine-grained recognition are solved, achieving efficient fine-grained recognition without manual annotation and improving the recognition performance and deployment efficiency of the model in complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- RECONOVA TECH CO LTD
- Filing Date
- 2026-02-04
- Publication Date
- 2026-05-12
AI Technical Summary
Existing fine-grained recognition technologies rely on costly component-level or attribute-level data annotation, have insufficient generalization ability, and traditional models experience performance degradation in complex scenarios, while independent tasks increase storage and computing power burdens.
We adopt a multimodal large model collaborative distillation method, which reduces data annotation costs, enhances model generalization ability, and meets the requirements of lightweight deployment by co-training expert teacher networks and student networks, and utilizes cross-modal semantic alignment and attention distillation loss.
It achieves fine-grained recognition without manual annotation, improves the generalization accuracy and recognition performance of the model in complex scenarios, and meets the real-time and lightweight deployment requirements of actual systems.
Smart Images
Figure SMS_30 
Figure SMS_46 
Figure SMS_69
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a fine-grained recognition method, system and application based on multimodal large model collaborative distillation. Background Technology
[0002] Fine-grained Visual Classification (FGVC) is a significant research area in computer vision, aiming to distinguish highly similar subclasses within the same broad category, such as face recognition, vehicle model recognition, and biological species recognition. Current mainstream FGVC methods are based on deep convolutional neural networks or visual Transformers, employing local attention mechanisms and metric learning to enhance the model's ability to discriminate and perceive details in key regions.
[0003] However, existing fine-grained recognition technologies have significant drawbacks: First, fine-grained recognition heavily relies on component-level or attribute-level data annotation, which is extremely costly to obtain and has limited generalization ability. Second, traditional models based on convolutional neural networks or visual Transformers lack explicit semantic constraints, and their performance may significantly degrade when faced with complex scenarios or changes in data distribution. Furthermore, in practical applications, quality assessment, attribute analysis, and classification tasks typically employ independent models, which not only limit recognition accuracy due to the lack of feature sharing among multiple tasks but also increase the storage and computing power burden during edge deployment.
[0004] Therefore, how to solve the problem of the high dependence of traditional methods on component-level or attribute-level data annotation, how to solve the problem of insufficient generalization ability caused by the disconnect between traditional pure visual features and deep semantic information, and how to meet the real-time and lightweight deployment requirements of actual systems while improving performance by introducing large modal models have become key issues that current fine-grained recognition technology urgently needs to solve. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a fine-grained identification method, system and application based on multimodal large model collaborative distillation, which can reduce data annotation costs, enhance the generalization ability of the model in complex scenarios, and meet the real-time and lightweight requirements of actual system deployment.
[0006] To achieve the above objectives, the solution of the present invention is: A fine-grained recognition system based on multimodal large model collaborative distillation includes an expert teacher network module, a student network module, a training module, and an inference module; The expert teacher network module adopts a multimodal large model, which has at least visual understanding and language interaction capabilities, and has a built-in visual encoder and language encoder. The expert teacher network module is used to extract the local attention distribution map of the input image and generate deep semantic embedding features. The student network module is a single-modal visual model, which includes a backbone recognition network, a classification head, a quality assessment head, and an attribute analysis head. It is used to receive knowledge information and local perception capabilities transferred from the expert teacher network module during the training phase, and to realize fine-grained image recognition and comparison, image quality assessment, and fine-grained attribute analysis during the inference phase. The training module is used to transfer knowledge from the expert teacher network module to the student network module through cross-modal semantic alignment loss and attention distillation loss, and to train the student network module using the Adam optimization algorithm. The inference module is used to call the backbone recognition network in the student network module during the inference phase to complete fine-grained recognition-related tasks.
[0007] Furthermore, the multimodal large model uses the Qwen2.5-VL-7B multimodal large model, and the student network uses the Tiny-ViT-21M unimodal vision model.
[0008] A fine-grained identification method for multimodal large-model collaborative distillation based on the aforementioned fine-grained identification system includes the following steps: S1. Using a multimodal large model as an expert teacher network, the attention distribution map of the local input image is extracted through its visual encoder, and its language encoder is used to generate deep semantic embedding features for fine-grained images. S2. Construct a fine-grained student network that includes a backbone recognition network, a classification head, a quality assessment head, and an attribute analysis head; S3. During the training phase, knowledge information and local perception capabilities from the pre-trained multimodal large model are transferred to the student network through cross-modal semantic alignment loss and attention distillation loss. S4. During the inference phase, only the backbone recognition network built during the training phase is used to complete fine-grained image recognition and comparison, image quality assessment, and fine-grained attribute analysis.
[0009] Furthermore, in step S1, the multimodal large model serves as the expert teacher network T, and its visual feature output... for: ;in, For visual networks in large multimodal models; The number of local feature modules in the expert teacher network; The feature dimension of visual features; Input image; Semantic embedding features of multimodal large models for: ;in, For language model networks in multimodal large models, As a prompt word, The feature dimension of semantic embedding features; The input image.
[0010] Furthermore, in step S2, the student network is used for fine-grained image recognition, quality assessment, and attribute recognition tasks, and the output feature map of the student network... for: ;in, The number of local feature modules in the student network; Output the feature dimensions of the feature map for the student network; Visual embedding features of student networks ,in Visual embedding feature dimension for student networks; For quality assessment, the student network quality score branch output is: The core function of quality assessment is to perform multi-dimensional quality analysis on the input fine-grained images and output a normalized comprehensive quality score.
[0011] For fine-grained attribute recognition, the attribute output branches of the student network are: ,in For the size of the predefined attribute dictionary, for the first A type of attribute, if it exists, then If it does not exist, then .
[0012] Furthermore, in order to transfer the visual local perception capability in the multimodal large model in step S1 to the student network in step S2, a local perception distillation loss is proposed to align the feature map output of the student network with that of the visual network in the multimodal large model. Due to the multimodal large model visual feature output Student network output feature map The number of local feature modules does not match the feature dimension. For the matching number of local feature modules, average pooling is used; for the matching feature dimension, a linear projection matrix is introduced. , The feature dimension of visual features; mapping the student network feature output map to the visual feature space of a multimodal large model; Therefore, the local sensing distillation loss function for:
[0013] in Average pooling operation; , For visual networks in multimodal large models, The number of local feature modules in the expert teacher network. For visual features, the feature dimensions are... Input image; , The number of local feature modules in the student network; Output the feature dimensions of the feature map for the student network; To incorporate the logical knowledge from the multimodal large model into the student network, a cross-modal semantic alignment distillation loss is proposed to align the semantic embedding features of the multimodal large model with the visual embedding features of the student network. The semantic embedding features are... ,in, As a prompt word, For the input image, For language model networks in multimodal large models; A linear projection matrix is introduced. The visual embedding features of the student network are mapped to the latent embedding space of the language large model of the multimodal large model, and semantic alignment is performed using cosine similarity; therefore, the semantic alignment distillation loss is used. for:
[0014] Visual embedding features for student networks; A quality assessment module is introduced, utilizing a multimodal large model to perform multi-dimensional decomposition and evaluation of fine-grained image quality. For fine-grained image quality assessment, prompt words are used to evaluate fine-grained recognition samples, with the output format being JSON: {"Clarity": x1, "Illumination": y1, "Occlusion": z1}, where x1 is the clarity score; y1 is the illumination score; and z1 is the occlusion score. The original scores for the three dimensions are then parsed from the JSON. ， And mapped to ,Right now: ; in, =x1； =y1； =z1； Then, a quality score for the expert teacher network synthesis is defined using weighted coefficients. for: ; in These are weighted parameters, and they satisfy... ; Quality assessment loss for: ; in, The overall quality score output by the student network quality assessment header.
[0015] Regarding fine-grained attribute analysis, a multimodal large model is used to transform unstructured visual features into structured multi-label semantic features, and cue words are used to describe the fine-grained visual attributes of the target object. Define a global property dictionary , scale The output of the multimodal large model is mapped to a Two-dimensional vector Regarding the first A type of attribute, if it exists, then If it does not exist, then Attribute prediction multi-label classification loss for:
[0016] in, The size of the predefined attribute dictionary; It is the sigmoid activation function. Its function is to convert the attribute prediction values output by the student network. Mapped to the [0,1] interval, transformed into "the first "The probability of the existence of a certain attribute"; Showing from the first attribute to the second These attributes are accumulated over all predefined fine-grained attributes; The first output for expert teachers network The actual label for this attribute, with a value of 0 or 1; Output branch for student network attributes to the first The original predicted value of the attribute; It is a logarithmic function used to... Sigmoid The probability value after activation is logarithmically transformed to amplify the loss when the predicted value deviates from the true label. In summary, the total loss function for network optimization is:
[0017] in As a constant, based on the above loss function, during the training process in step S3, the Adam optimization algorithm is used for backpropagation, and the expert teacher network is fixed. The parameters are used only to train and optimize the student network. The parameters are determined, and the final training results are obtained.
[0018] Further, in step S1, the process of extracting the attention distribution map is as follows: the visual encoder of the multimodal large model performs hierarchical feature extraction on the input image, calculates the association weight between each pixel and its surrounding pixels through a self-attention mechanism, and generates a two-dimensional attention weight matrix with dimensions corresponding to the resolution of the input image, which is the local attention distribution map.
[0019] Furthermore, in step S3, when training using the Adam optimization algorithm, the initial learning rate, weight decay coefficient, and training iterations are set to 200-300 rounds, with the learning rate decayed every 30-50 rounds and the decay coefficient being 0.5.
[0020] Furthermore, in step S4, the reasoning process for fine-grained recognition is as follows: first, the score output by the quality assessment head... Filter images, when If the value is ≥0.5, the subsequent recognition process is executed; otherwise, it is judged as a low-quality image and the recognition failure result is output. For images that pass the screening, the backbone recognition network outputs the classification result and attribute vector, and outputs the final recognition result by combining the classification confidence and attribute matching degree.
[0021] An application of a fine-grained recognition method based on multimodal large model collaborative distillation is disclosed, which applies the fine-grained recognition method based on multimodal large model collaborative distillation to any scenario in pedestrian re-identification, face recognition, vehicle model recognition, or biological species recognition.
[0022] After adopting the above scheme, the fine-grained recognition method, system and application based on multimodal large model collaborative distillation of the present invention automatically generates semantic descriptions of local features of objects and attention maps through multimodal large models, realizing the ability to guide the fine-grained recognition network to adaptively discriminate regions of interest without manual annotation, which greatly reduces the threshold and cost of data preparation.
[0023] This invention introduces a cross-modal semantic alignment mechanism through a multimodal large model, mapping visual features to the language space of the large model, which significantly enhances the model's understanding ability in complex scenarios and improves the model's generalization accuracy.
[0024] This invention adopts a training-coupled, inference-independent architecture. During the training phase, it leverages a multimodal large model for knowledge empowerment, while during the inference phase, it completely decouples the large model, retaining only the main network structure for fine-grained recognition, thus achieving the requirements of real-time deployment and lightweight design in practical systems. Detailed Implementation
[0025] To further explain the technical solution of the present invention, the present invention will be described in detail below through specific embodiments.
[0026] This invention discloses a fine-grained recognition system based on multimodal large model collaborative distillation, which includes an expert teacher network module, a student network module, a training module, and an inference module.
[0027] The expert teacher network module employs a multimodal large model to extract local attention distribution maps of the input image and generate deep semantic embedding features; The student network module comprises a backbone recognition network, a classification head, a quality assessment head, and an attribute analysis head. During the training phase, it receives knowledge and local perception capabilities transferred from the expert teacher network module. During the inference phase, it performs fine-grained image recognition and comparison, image quality assessment, and fine-grained attribute analysis. The backbone recognition network is the core feature extraction module of the student network module, responsible for extracting general and discriminative visual features from the input image, providing basic feature support for subsequent tasks such as classification, quality assessment, and attribute analysis. In this invention, the backbone recognition network adopts the Tiny-ViT-21M model. The classification head, quality assessment head, and attribute analysis head are all "task branch modules" built based on the output features of the backbone recognition network in the student network, each corresponding to a different specific task. Essentially, they are network substructures designed for specific tasks. The core task of the classification head is to achieve "fine-grained classification," that is, distinguishing highly similar subclasses within the same major category (such as different vehicle models or different pedestrians). Its function is to map the visual features extracted by the backbone network to the classification space, output the predicted probabilities of each category, and ultimately complete the recognition and comparison. The core task of the quality assessment head is to score the "fine-grained recognition applicability" of the input image. Its function is to output a comprehensive quality score normalized to the [0,1] interval based on backbone network features. The scoring criteria include dimensions such as image sharpness, lighting conditions, and occlusion degree, used to determine whether the image meets the quality requirements for fine-grained recognition. The core task of the attribute analysis head is to extract fine-grained attributes of targets in the image (such as pedestrian gender, hairstyle, clothing color, vehicle brand logo, body color, etc.), and its output is a... 3D binary vector The size of the attribute dictionary is preset, and each element in the vector corresponds to the "existence" or "non-existence" of an attribute (e.g., 1 indicates existence, 0 indicates non-existence).
[0028] The training module is used to transfer knowledge from the expert teacher network module to the student network module through cross-modal semantic alignment loss and attention distillation loss, and the student network module is trained using the Adam optimization algorithm; The inference module is used to call the backbone recognition network in the student network module during the inference phase to complete fine-grained recognition-related tasks.
[0029] This invention also discloses a fine-grained identification method based on multimodal large-model collaborative distillation, which includes the following steps: S1. Using a multimodal large model as an expert teacher network, the visual encoder extracts the local attention distribution map of the input image, and the language encoder generates deep semantic embedding features for fine-grained images. The process of extracting the attention distribution map is as follows: the visual encoder of the multimodal large model performs hierarchical feature extraction on the input image, calculates the association weight between each pixel and its surrounding pixels through a self-attention mechanism, and generates a two-dimensional attention weight matrix with dimensions corresponding to the resolution of the input image, which is the local attention distribution map. S2. Construct a fine-grained student network that includes a backbone recognition network, a classification head, a quality assessment head, and attribute analysis. S3. During the training phase, knowledge information and local perception capabilities from the pre-trained multimodal large model are transferred to the student network through cross-modal semantic loss and attention distillation loss. S4. During the inference phase, only the backbone recognition network built during the training phase is used for fine-grained image recognition and comparison. This includes image quality assessment and fine-grained attribute analysis.
[0030] The multimodal large model of this invention is a multimodal large model with at least visual understanding and language interaction capabilities, and it has a built-in visual encoder and language encoder. The multimodal large model of this invention uses Qwen2.5-VL-7B as the expert teacher network. T Qwen2.5-VL-7B is a typical large-scale visual-language multimodal model. Its architecture includes independent visual encoders (processing images) and language encoders (processing text), and achieves semantic association between the two through a cross-modal alignment mechanism. It represents a mature application of large-scale multimodal models. The Qwen2.5-VL-7B multimodal model can respond to cross-modal instructions (such as image + text prompts) and realize the transformation from "visual content to language semantics".
[0031] Multimodal large model visual feature output for:
[0032] in, For visual networks in large multimodal models; The number of local feature modules in the expert teacher network; The feature dimension of visual features; The semantic embedding features of a multimodal large model are:
[0033] in, For language model networks in multimodal large models, As a prompt word, The feature dimension of semantic embedding features; Input image; The student network described is a pure visual unimodal model. The student network in this invention uses Tiny-ViT-21M, a highly efficient and small-scale visual Transformer model that balances model performance and deployment efficiency. In this invention, Tiny-ViT-21M is used for fine-grained image recognition, quality assessment, and attribute recognition tasks. The output feature map of the student network is as follows:
[0034] in, The number of local feature modules in the student network; Output feature dimensions of the feature map for the student network. Visual embedding features of student networks ,in Visual embedding feature dimension for student networks.
[0035] For quality assessment, the student network quality score branch output is: The core function of quality assessment is to perform multi-dimensional quality analysis on the input fine-grained images and output a normalized comprehensive quality score.
[0036] For fine-grained attribute recognition, the attribute output branches of the student network are: ,in For the size of the predefined attribute dictionary, for the first A type of attribute, if it exists, then If it does not exist, then .
[0037] To transfer the visual local perception capabilities from a multimodal large model to the student network, this invention proposes a local perception distillation loss to align the feature map outputs of the student network and the visual network in the multimodal large model. This loss is based on the visual feature outputs of the multimodal large model. Student network output feature map The number of local feature modules does not match the feature dimension. To address the mismatch in the number of local feature modules, this invention uses average pooling. To address the mismatch in the feature dimension, this invention introduces a linear projection matrix. This maps the student network feature output map to the visual feature space of a multimodal large model. Therefore, the local perceptual distillation loss function... for:
[0038] in Average pooling operation.
[0039] To incorporate logical knowledge from a multimodal large model into the student network, this invention proposes a cross-modal semantic alignment distillation loss to align the semantic embedding features of the multimodal large model with the visual embedding features of the student network. The semantic embedding features of this invention are... ,in The English prompt used is "describe the image, including appearance, attribute, and state in 20 words." This invention introduces a linear projection matrix. This method maps the visual embedding features of the student network to the latent embedding space of the language large model in a multimodal large model, and uses cosine similarity for semantic alignment. Therefore, the semantic alignment distillation loss... for:
[0040] This invention introduces a quality assessment module, utilizing a multimodal large model to perform multi-dimensional decomposition and evaluation of fine-grained image quality. For fine-grained image quality assessment, the prompt used in this invention is: "As an image assessment expert, evaluate the fine-grained recognition samples and assign scores from 0 to 10 for the following aspects: 1. Clarity: sharpness of edges, blur or defocus. 2. Lighting: overexposure, low light or strong shadows. 3. Occlusion: Visibility. Output format in JSON: {'clarity': x1, 'lighting': y1, 'occlusion': z1}" x1 "Light": y1 "Covering": z1})",in, x1 Rate the clarity; y1 Rate the illumination; z1 To occlude the scoring. Parse the raw scores from the three dimensions of the JSON. and mapped to ,Right now:
[0041] in, =x1； =y1； =z1。
[0042] Then, a quality score for the expert teacher network synthesis is defined using weighted coefficients. for:
[0043] in These are weighted parameters, and they satisfy... .
[0044] In summary, quality assessment loss for:
[0045] in, The overall quality score output by the student network quality assessment header.
[0046] Regarding fine-grained attribute analysis, this invention also utilizes a multimodal large model to transform unstructured visual features into structured multi-label semantic features, thereby enhancing the student network's ability to perceive local details. For attribute analysis, the prompt used in this invention is "Describe the fine-grained visual attributes of the object, including color, texture, material, and structure. Output format: {'color': [...], 'texture': [...], 'material': [...], 'structure': [...]}". A global attribute dictionary is defined. , scale The output of the multimodal large model is mapped to a Two-dimensional vector Regarding the first A type of attribute, if it exists, then If it does not exist, then Therefore, attribute prediction multi-label classification loss. for:
[0047] in, The size of the predefined attribute dictionary; It is the sigmoid activation function. Its function is to convert the attribute prediction values output by the student network. Mapped to the [0,1] interval, transformed into "the first The probability of the existence of a certain attribute.
[0048] Showing from the first attribute to the second These attributes are accumulated over all predefined fine-grained attributes; The first output for expert teachers network The actual label for this attribute, with a value of 0 or 1; Output branch for student network attributes to the first The original predicted value of the attribute; It is a logarithmic function used to... Sigmoid The activation probability values are logarithmically transformed to amplify the loss when the predicted value deviates from the true label, thereby enhancing the sensitivity of the loss function to prediction errors and guiding the model to converge quickly.
[0049] In summary, the total loss function for network optimization is:
[0050] in The loss function is constant. Based on the above loss function, the Adam optimization algorithm is used for backpropagation during training, while the expert teacher network is fixed. The parameters are used only to train and optimize the student network. The parameters are determined, and the final training results are obtained.
[0051] When training with the Adam optimization algorithm, set the initial learning rate, weight decay coefficient, and training iterations of 200-300 rounds. Perform learning rate decay every 30-50 rounds with a decay coefficient of 0.5.
[0052] In step S4, the reasoning process for fine-grained recognition is as follows: First, the score output by the quality assessment head is used... Filter images, when If the value is ≥0.5, the subsequent recognition process is executed; otherwise, it is judged as a low-quality image and the recognition failure result is output. For images that pass the screening, the backbone recognition network outputs the classification result and attribute vector, and outputs the final recognition result by combining the classification confidence and attribute matching degree.
[0053] The fine-grained recognition method based on multimodal large model collaborative distillation of this invention can be applied to various fine-grained recognition scenarios such as pedestrian re-identification, face recognition, vehicle model recognition, and biological species recognition, and has good versatility and practicality. This invention is illustrated using pedestrian re-identification as an example.
[0054] A private dataset was used as the training data, which contains 28,358 people and 242,584 pedestrian digital images.
[0055] First, the multimodal large model Qwen2.5-VL-7B was used as the expert teacher network. Tiny-ViT-21M was used as the student network. For attribute recognition, the attribute dictionary is defined as shown in Table 1, with a total of 73 attribute states.
[0056] Table 1 shows the pedestrian re-identification attribute groups and their corresponding fine-grained values.
[0057] Then, the loss function of the fine-grained identification method based on multimodal large model collaborative distillation and the Adam optimization method are used to evaluate the student network. The parameters were optimized.
[0058] This invention uses two publicly available test sets, Market-1501 and DukeMTMC-reID, to compare the results with four other existing person re-identification models in terms of Rank-1 accuracy and Mean Average Precision (mAP). Table 2 shows the comparison results on the Market-1501 test set.
[0059]
[0060] Table 3 shows the comparison results on the test set DukeMTMC-reID.
[0061]
[0062] According to the test results in Tables 2 and 3, the fine-grained recognition method based on multimodal large model collaborative distillation of the present invention exhibits excellent performance in pedestrian re-identification tasks, which can effectively improve the accuracy of pedestrian recognition, while meeting the requirements of lightweight and real-time deployment in actual applications.
[0063] This invention automatically generates semantic descriptions of local features of objects and attention maps through a multimodal large model, enabling fine-grained recognition networks to adaptively identify regions of interest without manual annotation, greatly reducing the threshold and cost of data preparation.
[0064] This invention introduces a cross-modal semantic alignment mechanism through a multimodal large model, mapping visual features to the language space of the large model, which significantly enhances the model's understanding ability in complex scenarios and improves the model's generalization accuracy.
[0065] This invention adopts a training-coupled, inference-independent architecture. During the training phase, it leverages a multimodal large model for knowledge empowerment, while during the inference phase, it completely decouples the large model, retaining only the main network structure for fine-grained recognition, thus achieving the requirements of real-time deployment and lightweight design in practical systems.
[0066] The above embodiments are not intended to limit the product form and style of the present invention. Any appropriate changes or modifications made by those skilled in the art should be considered as not departing from the patent scope of the present invention.
Claims
1. A fine-grained identification system based on multimodal large-model collaborative distillation, characterized in that: It includes an expert teacher network module, a student network module, a training module, and a reasoning module; The expert teacher network module adopts a multimodal large model, which has at least visual understanding and language interaction capabilities, and has a built-in visual encoder and language encoder. The expert teacher network module is used to extract the local attention distribution map of the input image and generate deep semantic embedding features. The student network module is a single-modal visual model, which includes a backbone recognition network, a classification head, a quality assessment head, and an attribute analysis head. It is used to receive knowledge information and local perception capabilities transferred from the expert teacher network module during the training phase, and to realize fine-grained image recognition and comparison, image quality assessment, and fine-grained attribute analysis during the inference phase. The training module is used to transfer knowledge from the expert teacher network module to the student network module through cross-modal semantic alignment loss and attention distillation loss, and to train the student network module using the Adam optimization algorithm. The inference module is used to call the backbone recognition network in the student network module during the inference phase to complete fine-grained recognition-related tasks.
2. The fine-grained identification system based on multimodal large-model collaborative distillation as described in claim 1, characterized in that: The multimodal large model uses the Qwen2.5-VL-7B multimodal large model, and the student network uses the Tiny-ViT-21M unimodal vision model.
3. A fine-grained identification method for multimodal large-model collaborative distillation based on the fine-grained identification system of multimodal large-model collaborative distillation as described in claim 1 or 2, characterized in that, Includes the following steps: S1. Using a multimodal large model as an expert teacher network, the attention distribution map of the local input image is extracted through its visual encoder, and its language encoder is used to generate deep semantic embedding features for fine-grained images. S2. Construct a fine-grained student network that includes a backbone recognition network, a classification head, a quality assessment head, and an attribute analysis head; S3. During the training phase, knowledge information and local perception capabilities from the pre-trained multimodal large model are transferred to the student network through cross-modal semantic alignment loss and attention distillation loss. S4. During the inference phase, only the backbone recognition network built during the training phase is used to complete fine-grained image recognition and comparison, image quality assessment, and fine-grained attribute analysis.
4. The fine-grained identification method based on multimodal large model collaborative distillation as described in claim 3, characterized in that: In step S1, the multimodal large model serves as the expert teacher network T, and its visual feature output... for: ;in, For visual networks in large multimodal models; The number of local feature modules in the expert teacher network; The feature dimension of visual features; Input image; Semantic embedding features of multimodal large models for: ;in, For language model networks in multimodal large models, As a prompt word, The feature dimension of semantic embedding features; The input image.
5. The fine-grained identification method based on multimodal large-model collaborative distillation as described in claim 3, characterized in that, In step S2, the student network is used for fine-grained image recognition, quality assessment, and attribute recognition tasks, and the output feature map of the student network... for: ;in, The number of local feature modules in the student network; Output the feature dimensions of the feature map for the student network; Visual embedding features of student networks ,in Visual embedding feature dimension for student networks; For quality assessment, the student network quality score branch output is: The core function of quality assessment is to perform multi-dimensional quality analysis on the input fine-grained images and output a normalized comprehensive quality score. For fine-grained attribute recognition, the attribute output branches of the student network are: ,in For the size of the predefined attribute dictionary, for the first A type of attribute, if it exists, then If it does not exist, then .
6. The fine-grained identification method based on multimodal large model collaborative distillation as described in claim 3, characterized in that: To transfer the visual local perception capability from the multimodal large model in step S1 to the student network in step S2, a local perception distillation loss is proposed to align the feature map outputs of the student network and the visual network in the multimodal large model. Due to the output of multimodal large model visual features Student network output feature map The number of local feature modules does not match the feature dimension. For the matching number of local feature modules, average pooling is used; for the matching feature dimension, a linear projection matrix is introduced. , The feature dimension of visual features; mapping the student network feature output map to the visual feature space of a multimodal large model; Therefore, the local sensing distillation loss function for: in Average pooling operation; , For visual networks in multimodal large models, The number of local feature modules in the expert teacher network. For visual features, feature dimensions Input image; , The number of local feature modules in the student network; Output the feature dimensions of the feature map for the student network; To incorporate the logical knowledge from the multimodal large model into the student network, a cross-modal semantic alignment distillation loss is proposed to align the semantic embedding features of the multimodal large model with the visual embedding features of the student network. The semantic embedding features are... ,in, As a prompt word, For the input image, For language model networks in multimodal large models; A linear projection matrix is introduced. The visual embedding features of the student network are mapped to the latent embedding space of the language large model of the multimodal large model, and semantic alignment is performed using cosine similarity; therefore, the semantic alignment distillation loss is used. for: Visual embedding features for student networks; A quality assessment module is introduced, utilizing a multimodal large model to perform multi-dimensional decomposition and evaluation of fine-grained image quality. For fine-grained image quality assessment, prompt words are used to evaluate fine-grained recognition samples, with the output format being JSON: {"Clarity": x1, "Illumination": y1, "Occlusion": z1}, where x1 is the clarity score; y1 is the illumination score; and z1 is the occlusion score. The original scores for the three dimensions in the JSON are parsed. ， And mapped to ,Right now: ； in, =x1； =y1； =z1； Then, a quality score for the expert teacher network synthesis is defined using weighted coefficients. for: ； in These are weighted parameters, and satisfy... ; Quality assessment loss for: ； in, The overall quality score output by the student network quality assessment header; Regarding fine-grained attribute analysis, a multimodal large model is used to transform unstructured visual features into structured multi-label semantic features, and cue words are used to describe the fine-grained visual attributes of the target object. Define a global property dictionary , scale The output of the multimodal large model is mapped to a Two-dimensional vector Regarding the first A type of attribute, if it exists, then If it does not exist, then Attribute prediction multi-label classification loss for: in, The size of the predefined attribute dictionary; It is the sigmoid activation function. Its function is to convert the attribute prediction values output by the student network. Mapped to the [0,1] interval, transformed into "the first "The probability of the existence of a certain attribute"; Showing from the first attribute to the second These attributes are accumulated over all predefined fine-grained attributes; The first output for expert teachers network The actual label for this attribute, with a value of 0 or 1; Output branch for student network attributes to the first The original predicted value of the attribute; It is a logarithmic function used to... Sigmoid The probability value after activation is logarithmically transformed to amplify the loss when the predicted value deviates from the true label. In summary, the total loss function for network optimization is: in As a constant, based on the above loss function, during the training process in step S3, the Adam optimization algorithm is used for backpropagation, and the expert teacher network is fixed. The parameters are used only to train and optimize the student network. The parameters are determined, and the final training results are obtained.
7. The fine-grained identification method based on multimodal large model collaborative distillation as described in claim 3, characterized in that: In step S1, the process of extracting the attention distribution map is as follows: the visual encoder of the multimodal large model performs hierarchical feature extraction on the input image, calculates the association weight between each pixel and its surrounding pixels through the self-attention mechanism, and generates a two-dimensional attention weight matrix with dimensions corresponding to the resolution of the input image, which is the local attention distribution map.
8. The fine-grained identification method based on multimodal large model collaborative distillation as described in claim 3, characterized in that: In step S3, when training with the Adam optimization algorithm, the initial learning rate and weight decay coefficient are set, the number of training iterations is 200-300 rounds, and the learning rate is decayed once every 30-50 rounds with a decay coefficient of 0.
5.
9. The fine-grained identification method based on multimodal large model collaborative distillation as described in claim 3, characterized in that: In step S4, the reasoning process for fine-grained recognition is as follows: First, the score output by the quality assessment head is used... Filter images, when If the value is ≥0.5, the subsequent recognition process is executed; otherwise, it is judged as a low-quality image and the recognition failure result is output. For images that pass the screening, the backbone recognition network outputs the classification result and attribute vector, and outputs the final recognition result by combining the classification confidence and attribute matching degree.
10. An application of a fine-grained identification method based on multimodal large-model collaborative distillation, characterized in that, The fine-grained recognition method based on multimodal large model collaborative distillation as described in any one of claims 3-8 can be applied to any scenario in pedestrian re-identification, face recognition, vehicle model recognition, or biological species recognition.