Open vocabulary pedestrian attribute recognition method based on lightweight multi-modal large model

Abstract

The invention relates to the technical field of pedestrian attribute recognition, and provides an open vocabulary pedestrian attribute recognition method based on a lightweight multi-modal large model, which comprises the following steps: selecting a pre-trained multi-modal large language model; constructing an instruction fine-tuning data set for pedestrian attribute recognition based on a unified recognition architecture of a multi-modal large language model; and performing instruction fine tuning on the multi-modal large language model by using the instruction fine tuning data set: obtaining visual feature representation by using a double-strategy screening and double-layer information compensation mechanism of an importance perception pruning method, inputting the visual feature representation and a text instruction into the multi-modal large language model, and taking the output of the multi-modal large language model as an attribute recognition result. According to the method, the calculated amount and the video memory overhead can be remarkably reduced on the premise of keeping high accuracy, meanwhile, the strict requirement of an open vocabulary pedestrian attribute recognition task for the high accuracy can be perfectly met, and the method is not limited by a predefined and fixed text template.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pedestrian attribute recognition, and particularly relates to an open-vocabulary pedestrian attribute recognition method based on a lightweight multi-modal large model. BACKGROUND

[0002] Pedestrian attribute recognition (PAR) is a core task and a subtask in the field of computer vision, aiming to automatically analyze and identify the fine-grained semantic attributes of pedestrians, such as gender, age, clothing, etc., from surveillance images or videos. It is of great significance to intelligent security and pedestrian re-identification. In key places such as airports, train stations, and subway stations, individuals with specific characteristics can be automatically identified and focused on, achieving proactive defense. In scenarios such as shopping malls, banks, and hotels, service robots can provide more targeted voice, expressions, and interaction methods by identifying the gender, age, and other attributes of users. Traditional mainstream methods treat pedestrian attribute recognition as a multi-label classification problem, based on convolutional neural network architecture to simultaneously predict all predefined attributes. To overcome the limitations of early methods that ignore the semantic correlation between attributes, subsequent research began to introduce text modality information. For example, the VTB (Visual-Textual Baseline) method combines visual Transformer and language model BERT (Bidirectional Encoder Representations from Transformers), constructing PAR as a multi-modal learning task, using the context semantics of text to enhance the representation ability of visual features. The PromptPAR (Prompt Pedestrian Attribute Recognition) method further utilizes the prior knowledge of the CLIP (Contrastive Language-Image Pre-Training) model, generating more discriminative feature representations through region-aware prompt learning technology.

[0003] Open vocabulary refers to a system or model that is not limited to a pre-defined vocabulary when processing tasks, and can understand or generate new words or rare words beyond the defined vocabulary range in the training set. The goal is to break through the limitation of closed vocabulary models that can only handle a limited set, enabling the model to handle infinite language phenomena in the open world in actual applications.

[0004] Multimodal large models (MLLMs) are a class of general artificial intelligence models that can understand, reason, and generate multiple modal data in a unified framework through large-scale pre-training and cross-modal alignment. There are two problems in the field of pedestrian attribute recognition for general multimodal large models: first, general MLLMs lack knowledge modeling for the pedestrian field, making it difficult to accurately understand and distinguish fine-grained attribute features. Second, the visual encoder of general MLLMs generates a large number of visual tokens when processing high-resolution images, resulting in high computational complexity and slow reasoning speed, making it difficult to meet real-time monitoring requirements.

[0005] In recent years, with the strong ability of large language models in semantic understanding and reasoning, researchers have begun to explore the introduction of LLMs into the PAR task. For example, LLM-PAR (Large Language Model Pedestrian Open-Attribute Recognition) attempts to integrate MLLMs as a reasoning engine into the PAR framework, aiming to leverage their ability to capture semantic logical relationships between attributes to assist in optimizing the learning of visual features and enhancing the consistency of prediction results. This work is similar to the direction of the present invention, both focusing on utilizing the advanced cognitive abilities of MLLMs, but it still solves the problem within the predefined attribute set.

[0006] At the same time, in order to break through the limitations of the predefined attribute set, open-vocabulary pedestrian attribute recognition has become an important research direction, aiming to enable models to recognize and describe attribute categories that have not been seen during the training phase, thereby better adapting to the complexity and diversity of the real world. The strong image-text alignment ability learned by the CLIP model through large-scale pre-training makes it possible to achieve this goal. For example, the POAR (Pedestrian Open-Attribute Recognition) method first expands the predefined attribute labels in the dataset (such as "long sleeve") through a fixed natural language template (such as "This person is wearing a [attribute]"). Then, the text encoder of CLIP is used to encode these sentences into text feature vectors, while the image encoder of CLIP is used to encode the input pedestrian image into image feature vectors. Finally, by calculating the similarity between the image features and all text features, the most matching text description is selected as the recognition result, thus achieving attribute prediction. This method provides a feasible path to achieve open-vocabulary capability by constructing the pedestrian attribute recognition task as an image-text retrieval problem. It is a similar solution that the present invention needs to compare and surpass in terms of open-vocabulary capability, but its performance is limited by the predefined and fixed text template, lacking deep semantic understanding and flexible generation capabilities.

[0007] To solve the performance and efficiency trade-off problem of multimodal large language models (MLLMs) in vertical field deployment, various technical paths have emerged in the industry, which can be broadly divided into training-time optimization and inference-time optimization.

[0008] In terms of training-time optimization, a representative scheme is the large-small model co-training framework, which connects a large model and a small model through a shared multimodal encoder and a projection layer. It innovatively adopts a bidirectional distillation strategy: first, the lightweight small model guides the large model to quickly align multimodal information through forward knowledge distillation (KD); then, the rich knowledge base of the large model is used to enhance the performance of the small model through reverse KD loss. In terms of inference-time optimization, there is a training-free visual token pruning method that cleverly uses the attention score distribution generated by the [CLS] token and image block token across multiple layers in the visual encoder to evaluate and filter important visual tokens, and only inputs these tokens into the subsequent language model for inference. Pruning is a model compression and acceleration technique that removes redundant or unimportant parameters in neural networks to reduce model size and computational complexity while ensuring performance. In the study of MLLMs pruning, existing technologies include two paradigms: independent of the text side and related to the text side.

[0009] The existing technologies mainly have the following disadvantages: (1) As shown in (a) of Figure 1 , the closed model (a model trained based on a predefined set) cannot recognize attributes outside the predefined set. (2) As shown in (b) of Figure 1 , the POAR method implemented based on CLIP is severely limited by the predefined and fixed text templates, lacking the necessary understanding and flexible generation capabilities. (3) Using general MLLMs lacks sufficient domain expertise in pedestrian attributes and faces the challenge of huge storage overhead. (4) The method of simply selecting important tokens based on attention scores often has a single perspective, making it difficult to ensure spatial coverage of the entire pedestrian region in the image and easily missing key local features. This approach can cause a significant decline in the performance of the fine-tuned MLLMs, making it difficult to compress while preserving the key information required for the task. (5) The large-small model co-training requires high training resources and has limited optimization capabilities for specific tasks. SUMMARY

[0010] In view of the technical problems that existing pedestrian attribute recognition methods are limited by pre-defined and fixed text templates, lack of deep semantic understanding and flexible generation capability, the present application provides an open-vocabulary pedestrian attribute recognition method based on a lightweight multi-modal large model, which can realize automatic recognition and semantic description of pedestrian attributes in large-scale video monitoring scenes, assist security agencies to efficiently complete target analysis, and significantly reduce investigation time and resource waste. In view of the characteristics of the pedestrian attribute recognition task, the general multi-modal large model is structurally optimized and task-adapted to significantly improve the reasoning efficiency while maintaining strong semantic understanding capability. Moreover, the Lite-InstructPAR model of the present application can still recognize attributes outside the pre-defined set, and the Lite-InstructPAR model is not limited by pre-defined and fixed text templates.

[0011] In order to achieve the above purpose, the technical scheme of the present application is implemented as follows: an open-vocabulary pedestrian attribute recognition method based on a lightweight multi-modal large model, comprising the following steps:

[0012] Step S1: selecting a pre-trained multi-modal large language model;

[0013] Step S2: constructing an instruction fine-tuning data set for pedestrian attribute recognition based on the unified recognition architecture of the multi-modal large language model;

[0014] Step S3: instruction fine-tuning of the multi-modal large language model using the instruction fine-tuning data set: obtaining visual feature representation using the double-strategy screening and double-layer information compensation mechanism of the importance-aware pruning method, inputting the visual feature representation and the text instruction into the multi-modal large language model, and the output of the multi-modal large language model being the attribute recognition result.

[0015] Preferably, the pre-trained multi-modal large language model is an LLaVA-1.5 model, which includes a visual encoder and a backbone network, the visual encoder converts an input image into a visual feature token sequence, and the backbone network processes the fused multi-modal information by following the instructions;

[0016] The pedestrian attribute recognition instruction fine-tuning data set guides the multi-modal large language model to learn the semantic expression and visual mapping relationship related to pedestrian attributes through instruction-based task, and enhances the understanding and reasoning capability of the model for open attributes.

[0017] Preferably, the method for constructing the instruction fine-tuning data set for pedestrian attribute recognition comprises constructing "instruction-description" and constructing "question-answer";

[0018] In the construction of the "instruction-description" process, the domain knowledge is systematically summarized into several core categories, and based on the structured knowledge system, the prompt words that can guide the multi-modal large language model to generate accurate descriptions are constructed; when constructing "question-answer", each attribute corresponds to a pre-defined question template and multiple answer templates, and the question template and the answer template are instantiated according to the specific attribute label.

[0019] Preferably, the main part of the prompt word comprises: task description, attribute disintegration and expression requirement;

[0020] The content of the task description includes core instructions and boundary restrictions;

[0021] The attribute disintegration decomposes the pedestrian features into multiple structured dimensions; the expression requirement guides the model to generate natural and fluent language description, which can be based on the existing description to make reasonable inference;

[0022] All generated sample pairs are manually reviewed to eliminate samples that are not logical, unreasonable or unnatural;

[0023] The labels annotated by the standard dataset are injected into the randomly selected question template and answer template to form a "question-answer" sample pair;

[0024] The instruction fine-tuning dataset includes two data types: instruction-description and multi-round question and answer. Each sample included in the instruction-description is composed of an instruction and a corresponding description text generated by the multi-modal large model GPT-4o mini from the pedestrian image. Each sample included in the multi-round question and answer is composed of a set of labels annotated by the standard dataset injected into the randomly selected question template and answer template to form multiple question-answer pairs.

[0025] Preferably, the method for fine-tuning the multi-modal large language model in step S3 is:

[0026] S31: The visual encoder of the multi-modal large language model extracts the visual features of the input target pedestrian image to obtain a visual feature sequence, and calculates the attention distribution matrix of the image block token through the self-attention mechanism;

[0027] S32: Normalizing the attention distribution matrix to obtain a token importance sequence;

[0028] S33: Based on the token importance sequence, the double-strategy screening and double-layer information compensation mechanism of the importance-aware pruning method are used to obtain the visual feature representation;

[0029] S34: The visual feature representation is mapped to the embedding space of the multi-modal large language model through a trainable projection layer to obtain the visual semantic feature;

[0030] S34: projecting the input instruction using a multi-modal large language model to obtain text features, and performing feature extraction on the text features and visual semantic features to generate natural language response features.

[0031] Preferably, the double-strategy screening of the importance-aware pruning method includes Top-K global screening and GSB mechanism. The Top-K global screening processes the token importance sequence to obtain the top K most important image block tokens, which form a key visual token set. The GSB mechanism reshapes the image block token sequence in the visual feature sequence into a two-dimensional feature map and divides it into a grid. The token with the highest attention score in each grid is mapped back to the original image block token sequence to obtain a global index, and a token set after spatial balancing is constructed.

[0032] The double-layer information compensation mechanism includes set union and semantic clustering. The set union is to take the union set of the key visual token set and the token set after spatial balancing to obtain a visual token set.

[0033] The semantic clustering is to cluster all image block tokens in the visual token set according to semantic similarity to obtain semantic clustering clusters. The tokens in the semantic clustering clusters are weighted and fused to obtain locally enhanced key tokens. The locally enhanced key tokens and the residual tokens of all pruned image block tokens form the pruned and compressed visual feature representation.

[0034] Preferably, a target pedestrian image is processed by a visual encoder to obtain a visual feature sequence , the visual feature sequence includes an [CLS] token and an image block token sequence composed of N image block tokens , wherein represents the visual feature representation of each image block, respectively; the self-attention module of the visual encoder linearly transforms all tokens to generate a query matrix, a key matrix and a value matrix, calculates the dot product of the query matrix and the key matrix, and normalizes by softmax to obtain an attention distribution matrix ;

[0035] The elements in the i-th row of the attention distribution matrix are normalized to obtain the importance score of the i-th image block token , and the importance scores of all image block tokens form a token importance score sequence ;

[0036] The Top-K global filtering mechanism is implemented as follows: the importance scores of image patch tokens in the token importance scoring sequence are sorted in descending order, and a fixed number of K most important image patch tokens are obtained according to a preset compression ratio, thereby obtaining a set of key visual tokens. ;

[0037] The GSB mechanism is implemented by: taking the image block token sequence output by the visual encoder. Reconstructed into a two-dimensional feature map of size H×H, where Then the two-dimensional feature map is divided into a G×G grid, and each grid contains Image block tokens, and For each grid, based on the obtained importance score sorting results, the token with the highest importance score in the grid is selected as the representative token. The row and column number of the grid where the selected token is located in each grid and the local offset of the token in the grid are calculated and mapped back to its global index in the original image patch token sequence.

[0038] Preferably, the semantic clustering cluster is implemented by: calculating all tokens obtained by the visual encoder and the set of visual tokens. The cosine similarity between the key vectors contained within; for each important token that is retained. Based on cosine similarity, the T most relevant tokens are identified, including both retained and pruned tokens, forming a semantic cluster. Semantic clusters are formed based on the attention scores of each image patch token. The tokens within the token are weighted and merged to enhance the semantic representation of the central token;

[0039] All pruned tokens And not included in the visual token set The included tokens are pruned redundant tokens. The pruned redundant tokens are weighted and merged according to their own importance weights to form residual tokens.

[0040] Preferably, the spatially balanced token set for: Where X represents the global index;

[0041] The visual token set ;

[0042] The fused i-th visual feature representation ;in, , Representing visual token sets respectively The visual features represented by the i-th and j-th image block tokens, , respectively represent the attention scores corresponding to the jth and kth image patch token;

[0043] The visual feature representation of the residual token is:

[0044]

[0045] wherein, , represents the visual feature represented by the cropped image patch token, , represent the attention scores corresponding to the jth and kth image patch token;

[0046] The visual feature representation is jointly composed of the locally enhanced key tokens and the globally preserved residual token: , and M represents the number of finally selected image patch tokens.

[0047] Preferably, the visual feature representation is mapped to the embedding space of the multi-modal large language model through a trainable projection matrix to obtain the optimized visual semantic feature .

[0048] On the text side, the last two layers of the multi-modal large language model are fine-tuned using LoRA. In the self-attention module and the feed-forward network module of the last two Transformer decoders of the multi-modal large language model, a low-rank bypass update is injected to the original weight matrix, so that the multi-modal large language model learns task-specific knowledge.

[0049] Given a dataset or user input instruction , the instruction is projected into a text feature by a tokenizer , the obtained dimension-aligned visual semantic feature is concatenated with the text feature , and input into the backbone network. The next token is gradually predicted through the autoregressive generation method until a complete natural language answer is generated .

[0050] ​The present application provides a comprehensive solution: by constructing a high-quality instruction fine-tuning dataset for the pedestrian attribute field, and adapting the general MLLMs to the field, the model can deeply understand and flexibly generate diversified natural language descriptions related to pedestrian attributes, thereby breaking through the limitations of fixed templates, achieving accurate identification and description of unknown attributes in an open vocabulary scenario, and significantly improving the generalization performance of the model. At the same time, to overcome the computational bottleneck faced by MLLMs in pedestrian attribute recognition, the fundamental breakthrough of the present application lies in the proposed importance-aware pruning (IATP) compression method. Unlike previous co-training and pruning schemes, the core advantage of IATP is to achieve the collaborative optimization of efficient compression and information preservation; through the unique double-strategy screening and double-layer information compensation mechanism, while significantly reducing the number of visual tokens of the input large language model (LLM) and greatly improving the reasoning speed, the key features of the pedestrian region can still be completely preserved, thereby ensuring that the core performance of the model does not decrease but increases. At the same time, it can be used as a plug-and-play module, without the need for retraining, and with low resource requirements.

[0051] Compared with the prior art, the present application first introduces a multi-modal large language model into the open vocabulary pedestrian attribute recognition task, and first proposes an efficient importance-aware pruning method. By constructing a high-quality instruction dataset, the model's understanding of fine-grained attribute semantics and open vocabulary generation ability are significantly enhanced, and the dependence on fixed templates in previous methods is eliminated. Through the designed double-strategy token screening mechanism, the most critical visual information is preferentially retained while ensuring complete spatial coverage of the pedestrian region. Then, an information compensation mechanism is introduced to weight and fuse the semantics of the pruned tokens and inject them into the remaining tokens, thereby significantly reducing the number of visual tokens while effectively avoiding information loss. Relying on this lightweight pruning and compensation process, the model can significantly reduce the computational load and memory overhead while maintaining high accuracy, and still perfectly meet the stringent requirements of high accuracy for open vocabulary pedestrian attribute recognition tasks. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0053] Figure 1The existing work provided for the examples of the present application is compared with the present work, and the figure shows the performance comparison of the model trained based on the predefined attribute set and the Lite-InstructPAR model of the present application when facing attributes outside the predefined attribute set, and the figure shows the performance comparison of the POAR trained based on the predefined and fixed text template and the Lite-InstructPAR model of the present application when facing non-predefined and non-fixed text.

[0054] Figure 2 The model architecture diagram provided for the examples of the present application is shown.

[0055] Figure 3 The instruction data construction flow diagram provided for the examples of the present application is shown.

[0056] Figure 4 The attention distribution diagram provided for the examples of the present application is shown.

[0057] Figure 5 The Top-K optimal compression rate diagram provided for the examples of the present application is shown.

[0058] Figure 6 The screening token effect diagram provided for the examples of the present application is shown.

[0059] Figure 7 The model performance effect diagram provided for the examples of the present application is shown. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0061] As Figure 2As shown, an open vocabulary pedestrian attribute recognition method based on a lightweight multi-modal large model, the core innovation is to first introduce a multi-modal large language model into the open vocabulary pedestrian attribute recognition task, and propose an efficient importance-aware pruning method, which effectively improves the inference speed of the model. The application first constructs an instruction fine-tuning prompt word by using domain knowledge, and generates a natural language description and a unified question and answer data set by means of a multi-modal large model GPT-4o mini; then, the LLaVA-1.5 model is fine-tuned using the data set. In order to minimize the computational overhead caused by the increase of token number in the multi-modal large language model, a novel importance-aware pruning (Importance-Aware Token Pruning, IATP) method is introduced, which selectively retains the visual tokens with the largest amount of information according to the attention distribution, thereby ensuring robust performance without sacrificing speed or scalability. The execution process of the application is as shown in Figure 2 As shown, specifically includes the following steps:

[0062] Step S1: screening a pre-trained multi-modal large language model.

[0063] The LLaVA-1.5 model is selected as the base model for fine-tuning. The model architecture core includes two parts: the visual encoder adopts CLIP-ViT-L / 14, which is used to convert the input image into a visual feature token sequence; the backbone network of the multi-modal large language model adopts Vicuna-7B, which has excellent instruction following ability, and is responsible for processing the fused multi-modal information. This combination of characteristics makes it particularly suitable as a base model for instruction fine-tuning of multi-modal tasks.

[0064] The LLaVA-1.5 model is a pre-trained large model, and the application fine-tunes the pre-trained large model by constructing an instruction data set (supervised fine-tuning), so that it changes from a general large model to a vertical large model facing the pedestrian attribute recognition field.

[0065] Step S2: constructing an instruction fine-tuning data set for pedestrian attribute recognition based on a unified recognition architecture of a multi-modal large language model.

[0066] The application focuses on the construction of an open vocabulary pedestrian attribute recognition framework, i.e. a unified recognition architecture based on multi-modal large language models (MLLMs), and designs a pedestrian attribute recognition instruction fine-tuning data set. The model learns the semantic expression and visual mapping relationship related to pedestrian attributes through instruction-based task guidance, thereby enhancing the model's understanding and reasoning ability for open attributes.

[0067] The specific construction process is as shown in Figure 3As shown, it is divided into two parts: "instruction-description" and "question-answer". In the process of constructing "instruction-description", the domain knowledge is systematically summarized into several core categories: basic attributes (such as age, body type, etc.), clothing details (such as top, bottom, shoes, etc.), carrying items (such as backpack, handbag, mobile phone, etc.), prominent accessories (such as hat, glasses, etc.), behavior state and inferred identity role. Based on this knowledge system, the iterative optimization and design of comprehensive prompt words are laid a foundation. Based on the above structured knowledge system, the invention constructs a prompt word that can guide the large language model to generate accurate descriptions, which is used to batch construct high-quality "instruction-description" training data. Specifically, the prompt word consists of three main parts: (1) task description, which aims to define the boundaries and goals of the task for the model. Its content includes core instructions (explicitly requiring the generation of descriptions of pedestrians) and boundary restrictions (describing each attribute according to the refined attributes). (2) Attribute decomposition, which systematically decomposes the pedestrian features into multiple structured dimensions, including specific attributes in the above six categories (such as gender, top type and color, backpack style, behavior posture, etc.), ensuring complete and hierarchical coverage of the description. (3) Expression requirements, guiding the model to generate natural and fluent language descriptions, and under certain circumstances, making reasonable inferences based on existing descriptions. By combining the prompt word with a large language model (such as GPT-4o mini), "instruction-description" sample pairs can be generated on a large scale, significantly expanding the size of the dataset. To ensure data quality, all generated samples are strictly reviewed by humans, and samples that do not conform to logic, are unreasonable or unnatural are removed. When constructing "question-answer", each attribute (such as carrying a backpack, wearing a jacket, age less than 30, etc.) corresponds to a pre-defined question template and multiple answer templates. The question template and the answer template are instantiated according to the specific attribute label, specifically, the label annotated by the standard dataset is injected into the randomly selected question template and answer template to form a question-answer pair. Taking the "carrying a backpack attribute" as an example, the question template is: "Please carefully observe the pedestrian in the image, does the pedestrian carry a backpack?" The corresponding answer templates include "Yes, the pedestrian carries a backpack" or "No, the pedestrian does not carry a backpack", etc. Through this systematic template instantiation process, the original classification label data is efficiently converted into structured (question, answer) pairs, and a large-scale, fine-grained pedestrian attribute question-answer dataset is constructed. This method not only maintains the semantic accuracy of the original annotation, but also enhances the coverage and robustness of the dataset in the instruction learning task.

[0068] The final instruction fine-tuning dataset contains two types of data: instruction-description and multi-turn question-answering. Each sample contained in the instruction-description consists of an instruction and a corresponding description text generated by the multi-modal large model GPT-4o mini from the pedestrian image. Each sample contained in the multi-turn question-answering is composed of multiple question-answer pairs by injecting randomly selected questions and answer templates into the label set annotated by the standard dataset.

[0069] Step S3: Instruction fine-tuning of the multi-modal large language model using the instruction fine-tuning dataset constructed in step S2 above, using the double-strategy screening and double-layer information compensation mechanism of the importance-aware pruning method to obtain visual feature representation, inputting the visual feature representation and the text instruction into the multi-modal large language model, and the output of the multi-modal large language model being the attribute recognition result.

[0070] The constructed high-quality instruction dataset enables the multi-modal large language model to deeply master the domain knowledge of pedestrian attributes and learn to follow instructions for open-ended description and question-answering. Instruction fine-tuning is a large language model fine-tuning method that trains on a large-scale "instruction-response" dataset to enable the model to better follow human instructions and complete open-ended tasks. The instruction fine-tuning process is shown in Figure 3 , and the main steps are as follows:

[0071] Step S3-1: The visual encoder of the multi-modal large language model extracts features from the input target pedestrian image to obtain a visual feature sequence, and calculates the attention distribution matrix of the image block tokens through a self-attention mechanism.

[0072] A target pedestrian image is processed by the frozen visual encoder to obtain a visual feature sequence . The visual feature sequence contains a [CLS] token (i.e. ) and image block tokens (i.e. ), representing the visual feature representation of each image block, and N represents the number of image blocks. In the self-attention module of the visual encoder, all input tokens (including the [CLS] token and the image block tokens) are first transformed linearly to generate three matrices: the query matrix Q, the key matrix K, and the value matrix V. The correlation between tokens is calculated by the dot product of the query vector in the query matrix Q and the key vector in the key matrix K, and is normalized by softmax to obtain the attention weight. The complete attention output calculation formula is:

[0073] ,

[0074] where denotes matrix transposition, This represents the dimension of the key vector. The query matrix Q, key matrix K, and value matrix V are obtained by projecting the input token sequence onto the independent learnable weight matrix. The query matrix Q comes from the [CLS] tokens, and the key matrix K and value matrix V come from all image patch tokens. This is achieved through the attention distribution matrix. It can effectively characterize the semantic dependencies between the [CLS] token and each image tile token, thereby quantifying the importance of different tokens.

[0075] This invention optimizes the structure of the visual encoder by adding an importance-aware pruning module. While maintaining high-resolution feature representation capabilities, it reduces the number of visual tokens, lowering computational complexity and memory overhead, thereby achieving efficient real-time inference. Ultimately, while preserving the open semantic understanding capabilities of the large model, it achieves a unified optimization of lightweight design, high accuracy, and low latency, providing an efficient and reliable technical solution for the practical deployment of pedestrian attribute recognition in real-time scenarios such as intelligent security and video retrieval.

[0076] Step S3-2: Adjust the attention distribution matrix Normalization yields a preliminary token importance sequence. .

[0077] The input is the attention distribution matrix output from step S3-1. First, the data is normalized to obtain the importance score of each image patch token. The calculation method is as follows:

[0078] ,

[0079] in, This represents the attention weight of the i-th image patch token (as a query) on the j-th image patch token (as a key). This represents the importance score of the i-th image block token.

[0080] Obtain a preliminary token importance score sequence. .like Figure 4 As shown, observations based on this sequence reveal a severe sparsity in the attention distribution between the [CLS] token and the image patch tokens, indicating that not all tokens are necessary as input to the language model. Directly inputting this sequence into the projection layer would significantly increase the computational burden due to information redundancy. To address this issue, this invention introduces an Importance-Aware TokenPruning (IATP) module.

[0081] Step S3-3: Use Top-K global filtering to filter the token importance sequence The process involves filtering to obtain the top K most important image block tokens, which form a key visual token set. .

[0082] The Top-K global selection mechanism obtains the importance score sequence of each image patch token to the [CLS] token after normalizing the attention distribution matrix in step S3-2. Then, the token importance sequence reflects the contribution of each token in the visual encoder to the overall semantic representation. To reduce computational complexity while ensuring semantic integrity, a Top-k global filtering mechanism was designed. First, the token importance scores are sorted in descending order:

[0083] ,

[0084] in, This represents the importance score of the i-th sorted token. A fixed number K is obtained based on a preset compression ratio. The top K most important image patch tokens are selected, forming the key visual token set.

[0085] .

[0086] like Figure 5 As shown in the figure, experiments have verified that the optimal performance balance is achieved when K=0.5N. In summary, the Top-K global filtering mechanism ensures that key visual information is fully preserved even when the number of tokens is significantly compressed, thereby maximizing the fidelity of semantic information.

[0087] Steps S3-4: Using the GSB mechanism, the image patch token sequences in the key visual token set are reshaped into two-dimensional feature maps and divided into grids. A global index is obtained that maps the token with the highest attention score in each grid back to the original image patch token sequence, thus constructing a spatially balanced token set. .

[0088] The GSB (Grid-based Spatial Balancing) mechanism obtains the set of key visual tokens after completing the Top-K global filtering in step S3-3. This set primarily preserves the semantic information of globally salient regions. However, in pedestrian attribute recognition tasks, local details (such as clothing texture, accessories, and carried items) are equally important. Relying solely on Top-K global filtering can easily lead to the accidental deletion of some key local features due to low global attention scores. To address this, this invention proposes the GSB mechanism, aiming to reduce computational complexity while complementing the Top-K global filtering mechanism, maintaining the spatial balance and semantic integrity of visual features. The core idea of ​​GSB is to divide the visual feature map into a G×G regular grid and select the token with the highest attention score within each grid for retention, thereby achieving uniform spatial coverage and diverse preservation of key features. Specifically, the image block token sequence output by the visual encoder is used to... Reconstructed into a two-dimensional feature map of size H×H, where Then, the two-dimensional feature map is divided into a G×G grid, with each grid containing... Image block tokens, and This partitioning ensures uniform coverage of the image space, providing a foundation for local feature preservation. For each grid, based on the attention ranking result obtained in step S3-3, the token with the highest attention score within that grid is selected as the representative token, thus preventing key local regions from being ignored by the Top-K global filtering mechanism. Finally, the selected token position within each grid is mapped back to its global index in the original image patch sequence (i.e., the absolute position index of the image patch corresponding to the token among N image patches) by calculating the row and column index of the grid and the local offset of the token within the grid. A spatially balanced token set is then constructed. The calculation method is as follows:

[0089] ,

[0090] Here, X represents the corresponding global index. This set is much smaller in number than the original token set, but it is more balanced in spatial coverage, effectively balancing the preservation of local features with the representation of global information.

[0091] Step S3-5: Set the key visual tokens Token set after spatial balance Taking the union of the sets yields the final set of visual tokens. .

[0092] Next, the results obtained from the two filtering mechanisms in steps S3-3 and S3-4 are combined to construct the final visual token set. .Right now .

[0093] like Figure 6The reserved human body region is more complete, such as the details of the hand. This design not only effectively captures the global saliency features, but also significantly enhances the spatial coverage of the whole human body region, realizing the organic unity of global semantic expression and local detail reservation.

[0094] Step S3-6: Clustering all image block tokens in the visual token set according to semantic similarity to obtain semantic clustering clusters, and performing weighted fusion on the tokens in the semantic clustering clusters to obtain locally enhanced key tokens. The locally enhanced key tokens and the residual tokens of all pruned tokens constitute the pruned and compressed visual feature representation.

[0095] obtaining a visual token set , the IATP module does not use the conventional direct discard strategy, but through an information compensation mechanism, the semantic information that may be lost in the pruning process is recaptured and fused, so that the feature is enhanced while being compressed. Specifically, first, the cosine similarity between all tokens obtained by the visual encoder and the key vectors contained in is calculated to quantify the semantic correlation strength. For each important token that is reserved, the T most relevant tokens (including both the reserved and pruned tokens) are found according to the semantic correlation strength, i.e. the cosine similarity, to form a semantic clustering cluster . Subsequently, the tokens in the semantic clustering cluster are weighted and fused according to the attention scores of the tokens obtained in step 3-2 , so as to enhance the semantic representation of the central token:

[0096] ,

[0097] wherein, denotes the i-th fused visual feature representation, , denote the visual features represented by the i-th and j-th image block tokens in the visual token set , , denote the attention scores corresponding to the j-th and k-th image block tokens. This mechanism effectively compensates for the loss of features that may be caused by pruning.

[0098] In addition, to minimize information loss, all pruned tokens that are not included in the visual token set , i.e. the pruned (deleted) redundant tokens, are weighted and fused according to their own importance weights to form a residual token:

[0099]

[0100] wherein, denotes the visual feature representation of all pruned tokens after fusion, 、 denotes the visual feature represented by the pruned image patch token, 、 denotes the attention score corresponding to the jth, kth image patch token.

[0101] The residual token serves as a supplement to global information, and together with the enhanced important tokens, forms the compressed visual feature representation. The final IATP module outputs the final visual feature representation which is composed of the locally enhanced key tokens and the globally preserved residual tokens, and , M denotes the number of final selected image patch tokens.

[0102] Step S3-7: Map the visual feature representation to the embedding space of the multi-modal large language model through a trainable projection layer to obtain visual semantic features.

[0103] Subsequently, the visual feature representation is mapped to the embedding space of the large language model through a trainable projection matrix , which effectively maps the visual feature space to the embedding space of the large language model, and the optimized visual feature representation is calculated as follows: . Wherein, denotes the visual semantic features conforming to the language embedding space dimension of the large language model.

[0104] Step S3-8: Project the input instruction using the multi-modal large language model to obtain text features. After concatenating the text features and the visual semantic features, perform feature extraction to generate natural language response features.

[0105] On the text side, LoRA (Low-Rank Adaptation) is used to efficiently fine-tune the parameters of the last two layers of the multi-modal large language model to better adapt to the visual-linguistic alignment task. Specifically, in the self-attention module and the feedforward network module of the last two Transformer decoder layers of the model, a low-rank bypass update is injected into the original weight matrix , where B and A are trainable low-rank matrices. The forward propagation calculation becomes , x denotes the input feature vector of the layer, so as to greatly reduce the number of trainable parameters while enabling the model to learn task-specific knowledge. Given a dataset or user input instruction , the instruction Projection to the corresponding text features The visual semantic features obtained in steps S3-7, after dimension alignment, conform to the dimensions of the large language model. Text features The concatenated data is fed into the backbone network of Vicuna-7B. The final model uses an autoregressive generation method to progressively predict the next token based on the current input and the already generated content, until a complete natural language answer is generated. .

[0106] This invention constructs open-vocabulary pedestrian attribute recognition as a multimodal visual-language understanding and generation task. Unlike traditional methods that output predefined attribute labels, the result of this method is a coherent piece of natural language text. It leverages the inherent open-world knowledge and language generation capabilities learned on large-scale corpora by large language models. When encountering a query for a new attribute (an attribute not predefined in the standard dataset), the model does not invoke a specific "classifier," but rather generates or evaluates the likelihood of the text description on the fly based on its general knowledge and understanding of the current visual context. The diverse "instruction-description" and "question-answer" data constructed during the instruction fine-tuning stage are precisely to stimulate and guide this open-vocabulary generation and understanding capability of the model.

[0107] Five evaluation metrics—mA, Acc, Prec, Recall, and F1—are used, and two testing methods are designed to evaluate the model performance of this invention: cross-domain fine-tuning testing and model question-answering testing. Specifically, in the cross-domain fine-tuning test, the model (denoted as model_A) is first trained on the instruction data of dataset A, and then used to extract visual features from dataset B, and a classification layer is trained based on these features. In the model question-answering evaluation (test), model_A is also fine-tuned on dataset A, but directly uses the question-answering data from dataset B for end-to-end evaluation. The former uses visual features to additionally train a classifier to test feature discriminative power and domain generalization, while the latter requires no additional training or adaptation and directly performs end-to-end inference to evaluate the interaction capabilities of the complete multimodal system, i.e., instruction understanding and natural language generation capabilities.

[0108] like Figure 7As shown, the test results show that the model of the present application can accurately understand the diversified questioning methods, can handle direct attribute queries (such as "Is the person in the picture a man?") and can deal with descriptive questions (such as "Please briefly describe the attributes of the person in the picture" or "Please describe the attributes of the person in the picture in detail") with a 40% reduction in computational load. Secondly, the model exhibits strong knowledge transfer ability, not only can accurately identify attribute categories within the training set, but also has certain inference ability for unseen attribute descriptions. In addition, while maintaining professional domain knowledge, the model still has the original general visual language understanding ability and can handle a variety of questions not included in the instruction data set, such as "Overall, does the person's dress style look casual, like going for a coffee or a simple walk?" and "What color is the person's hair?" and the like.

[0109] The present application designs structured instruction templates and question and answer templates in detail, generates high-quality "instruction-description" and "question-answer" sample pairs through a large language model, significantly improves the model's understanding ability of fine-grained attribute semantics, and enables the model to have real open vocabulary understanding and generation ability for pedestrian attribute recognition tasks. The important importance-aware pruning method can greatly reduce the token number of the input LLM, while achieving low performance loss and small resource consumption, still perfectly meeting the strict requirement of high accuracy of open vocabulary pedestrian attribute recognition task.

[0110] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for recognizing pedestrian attributes using an open vocabulary based on a lightweight multimodal large model, characterized in that, Includes the following steps: Step S1: Select a pre-trained multimodal large language model; Step S2: Based on the unified recognition architecture of multimodal large language model, construct the instruction fine-tuning dataset for pedestrian attribute recognition; Step S3: Use the instruction fine-tuning dataset to fine-tune the multimodal large language model: Use the dual-strategy screening and dual-layer information compensation mechanism of the importance-aware pruning method to obtain visual feature representations, input the visual feature representations and text instructions into the multimodal large language model, and the output of the multimodal large language model is the attribute recognition result.

2. The open-vocabulary pedestrian attribute recognition method based on a lightweight multimodal large model according to claim 1, characterized in that, The pre-trained multimodal large language model is the LLaVA-1.5 model, which includes a visual encoder and a backbone network. The visual encoder converts the input image into a sequence of visual feature tokens, and the backbone network processes the fused multimodal information by virtue of its instruction-following ability. The pedestrian attribute recognition instruction fine-tuning dataset guides a multimodal large language model to learn the semantic expressions and visual mapping relationships related to pedestrian attributes through instructional tasks, thereby enhancing the model's understanding and reasoning ability regarding open attributes.

3. The open-vocabulary pedestrian attribute recognition method based on a lightweight multimodal large model according to claim 1 or 2, characterized in that, The method for constructing a fine-tuning dataset for pedestrian attribute recognition includes: constructing "instruction-description" and constructing "question-answer"; In the process of constructing the "instruction-description" framework, domain knowledge is systematically categorized into several core categories. Based on the structured knowledge system, prompt words that can guide the multimodal large language model to generate accurate descriptions are constructed. These prompt words are then combined with the large language model to generate "instruction-description" sample pairs on a large scale. When constructing the "question-answer" framework, each attribute corresponds to a predefined question template and multiple answer templates. The question templates and answer templates are instantiated based on specific attribute tags.

4. The open-vocabulary pedestrian attribute recognition method based on a lightweight multimodal large model according to claim 3, characterized in that, The main components of the prompt words include: task description, attribute deconstruction, and expression requirements; The task description includes core instructions and boundary constraints; Attribute deconstruction breaks down pedestrian features into multiple structured dimensions; expression requirements guide the model to generate natural and fluent language descriptions, and can make reasonable inferences based on existing descriptions; All generated sample pairs are manually reviewed to remove samples that are illogical, unreasonable, or unnatural. The labels from the standard dataset are injected into randomly selected question and answer templates to form a "question-answer" sample pair; The instruction fine-tuning dataset includes two data types: instruction-description and multi-turn question-answering. Each sample in the instruction-description category consists of an instruction and a corresponding descriptive text generated by the multimodal large model GPT-4o mini from a pedestrian image. Each sample in the multi-turn question-answering category consists of multiple question-answer pairs formed by injecting a set of labels labeled by a standard dataset into randomly selected question and answer templates.

5. The open-vocabulary pedestrian attribute recognition method based on a lightweight multimodal large model according to claim 3, characterized in that, The method for fine-tuning the multimodal large language model in step S3 is as follows: S31: The visual encoder of the multimodal large language model extracts visual feature sequences from the input target pedestrian image and calculates the attention distribution matrix of image block tokens through a self-attention mechanism; S32: Normalize the attention distribution matrix to obtain the token importance sequence; S33: Based on the token importance sequence, visual feature representation is obtained by using the dual-strategy screening and dual-layer information compensation mechanism of the importance-aware pruning method; S34: Visual semantic features are obtained by mapping the visual feature representation to the embedding space of a multimodal large language model through a trainable projection layer; S34: The input instructions are projected using a multimodal large language model to obtain text features. The text features are then concatenated with the visual semantic features for feature extraction, generating natural language response features.

6. The open-vocabulary pedestrian attribute recognition method based on a lightweight multimodal large model according to any one of claims 1-5, characterized in that, The dual-strategy screening of the importance-aware pruning method includes Top-K global screening and GSB mechanism. Top-K global screening processes the token importance sequence to obtain the top K most important image block tokens, which form a key visual token set. The GSB mechanism reshapes the image block token sequence in the visual feature sequence into a two-dimensional feature map and divides it into grids. It then maps the token with the highest attention score in each grid back to the original image block token sequence to obtain a global index, and constructs a spatially balanced token set. The two-layer information compensation mechanism includes union set and semantic clustering. The union set is obtained by taking the union set of key visual tokens with the spatially balanced token set. Semantic clustering involves clustering all image patch tokens in the visual token set based on semantic similarity to obtain semantic clusters. The tokens within the semantic clusters are then weighted and fused to obtain locally enhanced key tokens. The visual features of the locally enhanced key tokens and the residual tokens of all pruned image patch tokens constitute the pruned and compressed visual feature representation.

7. The open-vocabulary pedestrian attribute recognition method based on a lightweight multimodal large model according to claim 6, characterized in that, One The target pedestrian image is processed by a visual encoder to obtain a visual feature sequence. Visual feature sequence Contains a [CLS] token A sequence of image block tokens consisting of N image block tokens ,in, These represent the visual feature representations of each image patch. The self-attention module of the visual encoder performs a linear transformation on all tokens to generate a query matrix, a key matrix, and a value matrix. It then calculates the dot product of the query matrix and the key matrix, and normalizes the result using softmax to obtain the attention distribution matrix. ; Attention distribution matrix The importance score of the i-th image patch token is obtained by normalizing the elements in the i-th row. The importance scores of all image block tokens constitute the token importance score sequence. ; The Top-K global filtering mechanism is implemented as follows: the importance scores of image patch tokens in the token importance scoring sequence are sorted in descending order, and a fixed number of K most important image patch tokens are obtained according to a preset compression ratio, thereby obtaining a set of key visual tokens. ; The GSB mechanism is implemented by: taking the image block token sequence output by the visual encoder. Reconstructed into a two-dimensional feature map of size H×H, where Then the two-dimensional feature map is divided into a G×G grid, and each grid contains Image block tokens, and For each grid, based on the obtained importance score sorting results, the token with the highest importance score in the grid is selected as the representative token. The row and column number of the grid where the selected token is located in each grid and the local offset of the token in the grid are calculated and mapped back to its global index in the original image patch token sequence.

8. The open-vocabulary pedestrian attribute recognition method based on a lightweight multimodal large model according to claim 7, characterized in that, The semantic clustering cluster is implemented by calculating all tokens obtained by the visual encoder and the set of visual tokens. The cosine similarity between the key vectors contained within; for each important token that is retained. Based on cosine similarity, the T most relevant tokens are identified, including both retained and pruned tokens, forming a semantic cluster. Semantic clusters are formed based on the attention scores of each image patch token. The tokens within the token are weighted and merged to enhance the semantic representation of the central token; All pruned tokens And not included in the visual token set The included tokens are pruned redundant tokens. The pruned redundant tokens are weighted and merged according to their own importance weights to form residual tokens.

9. The open-vocabulary pedestrian attribute recognition method based on a lightweight multimodal large model according to claim 8, characterized in that, The spatially balanced token set for: Where X represents the global index; The visual token set ; The fused i-th visual feature representation ;in, , Representing visual token sets respectively The visual features represented by the i-th and j-th image block tokens, , Let represent the attention scores corresponding to the j-th and k-th image block tokens, respectively; The visual features of the residual token are represented as follows: in, , The visual features represented by the tokens indicating the clipped image blocks. , This represents the attention score corresponding to the j-th and k-th image block tokens; Visual feature representation It consists of a locally enhanced key token and a globally preserved residual token: M represents the number of image block tokens that are ultimately selected.

10. The open-vocabulary pedestrian attribute recognition method based on a lightweight multimodal large language model according to any one of claims 7-9, characterized in that, Visual feature representation By mapping the visual feature space to the embedding space of a multimodal large language model using a trainable projection matrix, optimized visual semantic features are obtained. ; On the text side, LoRA is used to fine-tune the parameters of the last two layers of the multimodal large language model. In the self-attention module and feedforward network module of the last two Transformer decoders of the multimodal large language model, a low-rank side-channel update is injected into the original weight matrix, enabling the multimodal large language model to learn task-specific knowledge. Given a dataset or user-input instructions The instructions are processed by a word segmenter. Projection as text features The obtained dimension-aligned visual semantic features Text features The concatenated input is fed into the backbone network, where it is used to progressively predict the next token through an autoregressive generation method until a complete natural language response is generated. .

Images