A pedestrian re-identification method and device based on a multi-modal adapter and a medium

Abstract

The present application belongs to the technical field of data processing, and particularly relates to a pedestrian re-identification method based on a multi-modal adapter, a device and a medium. The method comprises the following steps: collecting pedestrian re-identification data, including image data of pedestrians and corresponding text descriptions; performing data enhancement on the pedestrian re-identification data, the data enhancement being used for processing image data lacking text descriptions to generate text descriptions corresponding to the image data; performing text enhancement on the text descriptions based on a multi-modal large language model; constructing a multi-modal adapter to generate text adaptation embedding and image adaptation embedding, and superimposing the text adaptation embedding and the image adaptation embedding with original text embedding and image embedding to obtain text fusion features and image fusion features; and performing pedestrian re-identification prediction by using the text fusion features and the image fusion features to realize pedestrian re-identification. The present application constructs a multi-modal adapter, has fewer trainable parameters, enhances the adaptation capability of the model to a target domain, and significantly reduces the calculation efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to a pedestrian re-identification method based on a multi-modal adapter, a device and a medium. BACKGROUND

[0002] Existing text-to-image person re-identification (T2IReID) models have achieved good performance on datasets such as CUHK-PEDES, ICFG-PEDES, and RSTPReid. However, these datasets belong to the supervised single-domain paradigm, where the distribution of the training set and the test set is the same. When applied to the cross-domain few-shot T2IReID scenario, the target domain usually has a large number of unlabeled samples and a small number of labeled samples. In this case, when the trained model is migrated to the "unseen" target domain, the retrieval performance of the model usually decreases significantly. This is because the domain difference is an inevitable problem in cross-domain learning and model migration, directly affecting the performance of T2IReID models in actual deployment. Therefore, the present application focuses on the cross-domain few-shot pedestrian re-identification scenario and explores effective strategies to solve the difference problem between different domains.

[0003] Currently, domain adaptation re-identification research mainly aims to enable models trained on labeled source domains to effectively adapt to unlabeled samples in the target domain. The three main methods include pseudo-label estimation, intermediate feature alignment, and GAN-based style transfer. Although these methods alleviate the challenges of cross-domain tasks to some extent, there are still some limitations. First, data augmentation based on GAN and other methods is not always reliable and may result in generated data that does not match the actual distribution of the target domain, thereby affecting the adaptability of the model. Second, they usually rely on complex training processes that require a large amount of computing and storage resources, especially when dealing with large-scale data, existing methods are difficult to efficiently cope with resource and time constraints.

[0004] In summary, there is an urgent need for a pedestrian re-identification method based on a multi-modal adapter, a device and a medium to solve the problems existing in the prior art. SUMMARY

[0005] The present application aims to provide a pedestrian re-identification method based on a multi-modal adapter, a device and a medium, and the specific technical solutions are as follows:

[0006] A pedestrian re-identification method based on a multi-modal adapter, comprising the following steps:

[0007] S1: Collect pedestrian re-identification data, including image data of pedestrians and text descriptions corresponding to these images;

[0008] S2: Perform data augmentation on the pedestrian re-identification data in S1. Data augmentation is used to process image data that lacks text description and generate the corresponding text description for the image data.

[0009] S3: Text enhancement based on a multimodal large language model for text description;

[0010] S4: Construct a multimodal adapter to generate text-adapted embeddings and image-adapted embeddings, and superimpose them with the original text embeddings and image embeddings to obtain text fusion features and image fusion features;

[0011] S5: Use text fusion features and image fusion features to predict pedestrian re-identification and achieve pedestrian re-identification.

[0012] Optionally, in S2, the data augmentation process is as follows:

[0013] A large language model is invoked to generate prompts with static text descriptions. The prompts include pedestrian posture, clothing features, carried items, and template constraints. The template constraints are used to limit the text length and style.

[0014] Based on prompts, a multimodal large language model is used to generate text descriptions corresponding to image data.

[0015] Optionally, in S3, the text enhancement process is as follows:

[0016] Construct natural language instructions, invoke the language generation model, and use the natural language instructions to generate enhanced templates;

[0017] Based on the enhanced template, a multimodal large language model is invoked to enhance the text description;

[0018] The expression for generating the enhanced template is as follows:

[0019] ;

[0020] in, Indicates the first m An enhanced template; t is the input, which includes image data or text description; s represents a natural language instruction; The language generation model GPT-4o is represented.

[0021] Optionally, in S4, the multimodal adapter includes an embedded downsampler, a masked multi-head attention network, and a linear network layer;

[0022] The embedding downsampling unit is set in a masked multi-head attention network to obtain image embeddings and text cue embeddings, and to reduce the embedding dimension.

[0023] Masked multi-head attention networks include mask matrices. and a mask attention calculation, the mask matrix is used to guide cross-modal information interaction, wherein , I and T respectively represent the number of image data and the number of text description;

[0024] The linear network layer comprises a first up-sampling linear layer , a non-linear activation function GELU and a second up-sampling linear layer , the first up-sampling linear layer and the second up-sampling linear layer are respectively provided with a corresponding first weight matrix and a second weight matrix , wherein represents the feature dimension.

[0025] Optionally, in S4, the calculation process of the mask multi-head attention network is as follows:

[0026] The mask matrix is calculated as follows:

[0027] ;

[0028] wherein , is the number of images, is the number of text descriptions, represents positive interaction, represents negative interaction, ensuring that the text embedding only focuses on image information;

[0029] The mask matrix is applied to the mask attention calculation through an addition operation, and the expression of the mask attention calculation is as follows:

[0030] ;

[0031] wherein , and are a query matrix, a key matrix and a value matrix respectively, represents the key vector dimension, represents matrix transposition, represents mask attention.

[0032] Optionally, in S4, the calculation process of the multi-modal adapter is as follows:

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] ;

[0038] wherein, denotes the input of the multi-modal adapter, denotes the sum of the number of text embeddings and image embeddings, denotes the number of samples in a batch, is the embedding dimension; denotes the embedding down-sampler; denotes the output of the embedding down-sampler; denotes the masked multi-head attention network; denotes the output of the masked multi-head attention network; denotes the output of the first up-sampling linear layer ; denotes the output of the non-linear activation function GELU; denotes the output of the multi-modal adapter.

[0039] Optionally, in S4, the output of the multi-modal adapter is split into text adaptation embedding and image adaptation embedding , after embedding dimension normalization, weighted fusion with the original text embedding and image embedding, to obtain text fusion features and image fusion features:

[0040] ;

[0041] ;

[0042] wherein, denotes the text fusion features, denotes the image fusion features, is a hyper-parameter, denotes the original text embedding, denotes the original image embedding.

[0043] Optionally, in S5, the parameters of the multi-modal adapter are optimized during the model training process, and the training loss includes an identity loss and a contrastive learning loss, and the calculation expression of the identity loss is as follows:

[0044] ;

[0045] wherein, and denote the first column and the second column of the classification weight matrix , respectively, denotes an identity tag, and denote the first and the second element in the bias vector, respectively.

[0046] In addition, the present application also includes a computer device comprising a memory and a processor;

[0047] The memory is used to store a computer program executable on the processor;

[0048] The processor is used to implement the steps of the pedestrian re-identification method as described above when the computer program is executed.

[0049] In addition, the present application also includes a computer readable storage medium, the computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the pedestrian re-identification method as described above.

[0050] The technical scheme of the present application has the following beneficial effects:

[0051] In the data preprocessing stage of the method, a multi-modal large language model is used to generate diversified text descriptions to make up for the lack of target domain data. In addition, a lightweight multi-modal adapter is also designed, which uses a multi-head attention mechanism to jointly adapt text and image features, generates adapted feature representations, and superimposes them with original features. The adapter has fewer trainable parameters, which enhances the model's adaptation to the target domain while significantly reducing computational efficiency.

[0052] In addition to the purposes, features and advantages described above, the present application has other purposes, features and advantages. The present application will be further described in detail below with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0054] Figure 1 is a step flow chart of the pedestrian re-identification method in the preferred embodiment of the present application;

[0055] Figure 2 is a data processing flow chart of the pedestrian re-identification method in the preferred embodiment of the present application;

[0056] ​​​Figure 3 is a schematic diagram of generating instruction prompts in preferred embodiments of the present application;

[0057] Figure 4(a) is a text feature map extracted by a pre-trained model in preferred embodiments of the present application;

[0058] Figure 4(b) is a text feature map extracted by the method of the present embodiments. DETAILED DESCRIPTION

[0059] In order to enable persons skilled in the art to better understand the present application scheme, the present application will be further described in detail below in combination with the drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor fall within the scope of protection of the present application.

[0060] As shown in Figure 1 and Figure 2 , the present embodiments provide a pedestrian re-identification method based on a multi-modal adapter, including the following steps:

[0061] S1: Collecting pedestrian re-identification data, including image data of pedestrians and text descriptions corresponding to the images.

[0062] In the present embodiments, the source domain and the target domain are represented by and respectively, and the goal of the method of the present embodiments is to mine other related information of pedestrians in the target domain by using the knowledge of the target domain and the source domain . The source domain and the target domain are represented by data pairs and , wherein, is the image-text pair of the pedestrian, is the corresponding identity information, , and , i.e. the source domain and the target domain do not share the same label space. And and are sampled from two different probability distributions and , wherein . The target domain is composed of a large number of pedestrian image-text pairs without specific identity information, and these data do not contain corresponding labels. In addition, there is a support set composed of annotated pedestrian data , wherein .

[0063] S2: Data augmentation is performed on the pedestrian re-identification data in S1, and the data augmentation is used to process image data lacking of text description to generate text description corresponding to the image data.

[0064] In S2, as shown in Figure 3 , the process of data augmentation is as follows:

[0065] The prompt instruction for calling a large language model to generate static text description includes pedestrian posture, clothing features, carried items, and template constraints, and the template constraints are used to limit the length and style of the text;

[0066] Based on the prompt instruction, a multi-modal large language model is used to generate text description corresponding to the image data.

[0067] S3: Text augmentation is performed on the text description based on the multi-modal large language model.

[0068] In S3, the process of text augmentation is as follows:

[0069] A natural language instruction is constructed, a language generation model is called, and the natural language instruction is used to generate an augmented template;

[0070] Based on the augmented template, a multi-modal large language model is called to perform text augmentation on the text description;

[0071] The expression of the augmented template is as follows:

[0072] ;

[0073] Among them, represents the m th augmented template; t is the input, which includes image data or text description; s represents a natural language instruction; represents a language generation model GPT-4o.

[0074] S4: A multi-modal adapter is constructed to generate text adaptation embedding and image adaptation embedding, and to superimpose the original text embedding and image embedding to obtain text fusion features and image fusion features.

[0075] In S4, the multi-modal adapter includes an embedding down-sampler, a masked multi-head attention network, and a linear network layer;

[0076] The embedding down-sampler is arranged in the masked multi-head attention network to obtain image embedding and text prompt embedding, and to reduce the embedding dimension;

[0077] The masked multi-head attention network includes a mask matrix and a mask attention calculation, the mask matrix is used to guide cross-modal information interaction, wherein , I and T respectively represent the number of image data and the number of text descriptions;

[0078] The linear network layer comprises a first up-sampling linear layer , a non-linear activation function GELU and a second up-sampling linear layer , the first up-sampling linear layer and the second up-sampling linear layer are respectively provided with corresponding first weight matrix and second weight matrix , wherein, represents the feature dimension.

[0079] Further, the calculation process of the mask multi-head attention network is as follows:

[0080] The mask matrix is calculated as follows:

[0081] ;

[0082] wherein, , is the number of images, is the number of text descriptions, represents positive interaction, represents negative interaction, ensuring that the text embedding only focuses on image information;

[0083] The mask matrix is applied to the mask attention calculation through an addition operation, and the expression of the mask attention calculation is as follows:

[0084] ;

[0085] wherein, , and are respectively a query matrix, a key matrix and a value matrix, represents the key vector dimension, represents matrix transposition, represents mask attention.

[0086] Further, the calculation process of the multi-modal adapter is as follows:

[0087] ;

[0088] ;

[0089] ;

[0090] ;

[0091] ;

[0092] wherein, denotes the input of the multi-modal adapter, denotes the sum of the number of text embeddings and image embeddings, denotes the number of samples in a batch, is the embedding dimension; denotes the embedding down-sampler; denotes the output of the embedding down-sampler; denotes the masked multi-head attention network; denotes the output of the masked multi-head attention network; denotes the output of the first up-sampling linear layer ; denotes the output of the non-linear activation function GELU; denotes the output of the multi-modal adapter.

[0093] Further, the output of the multi-modal adapter is split into a text adaptation embedding and an image adaptation embedding , which are normalized by the embedding dimension and fused with the original text embedding and image embedding to obtain a text fusion feature and an image fusion feature:

[0094] ;

[0095] ;

[0096] wherein, denotes the text fusion feature, denotes the image fusion feature, is a hyper-parameter, denotes the original text embedding, denotes the original image embedding.

[0097] S5: Perform pedestrian re-identification prediction using the text fusion feature and the image fusion feature to realize pedestrian re-identification.

[0098] In S5, the parameters of the multi-modal adapter are optimized during the model training process, and the training loss includes an identity loss and a contrastive learning loss. The calculation expression of the identity loss is as follows:

[0099] ;

[0100] wherein, and denote the classification weight matrix the first column and the second column, denote the identity tag of the and denote the first element and the second element in the bias vector, respectively.

[0101] In this embodiment, after the model training is completed, the process of pedestrian re-identification from text to image is implemented as follows: first, the feature extraction network trained is used to obtain the fused feature representation of the query text description and the candidate images in the image library, respectively, and the features are normalized to stabilize the similarity calculation. Then, the cosine similarity is used to calculate the similarity between the query text and each candidate image, the candidate images are sorted according to the similarity, and the top-ranked images are selected as the final recognition result.

[0102] In order to evaluate the effectiveness of this embodiment in the text retrieval based re-identification task, this embodiment uses a benchmark model for experimental comparison:

[0103] First of all, the data set, this embodiment uses 3 mainstream text-to-image data sets for experiments, and the 3 data sets are CHUK-PEDES, ICFG-PEDES and RSTPReid.

[0104] CUHK-PEDES is the first available data set for text-image personal retrieval, which contains 40206 images and 80412 text descriptions of 13003 pedestrians, and each image is equipped with 2 text descriptions. The data set is divided into training set, validation set and test set, wherein the training set contains 11003 pedestrians, 34054 images and 68108 descriptions of the training set; the validation set contains 1000 pedestrians, 3078 images and 6156 descriptions; the test set contains 1000 pedestrians, 3074 images and 6148 descriptions.

[0105] All images in ICFG-PEDES come from MSMT17, and it contains more identities and text descriptions than CHUK-PEDES. ICFG-PEDES contains 54522 text descriptions and 54522 images of 4102 pedestrians, i.e. each image has a corresponding text description. The data set is divided into training set and test set, wherein the training set contains 3102 pedestrians, 34674 image-text pairs, and the test set contains the remaining 1000 pedestrians, 19848 image-text pairs.

[0106] The images in RSTPReid are also collected from MSMT17, including 20505 images of 4101 pedestrians and 41010 text descriptions. Each person has 5 images captured by 15 different cameras, and each image has 2 corresponding text descriptions. The training set contains 3701 pedestrians, and the validation set and test set of the embodiment contain 200 pedestrians respectively.

[0107] Quantitative analysis:

[0108] In order to verify the effectiveness of the proposed framework in the few-shot cross-domain scene, the embodiment performs experiments on the domain generalization task. Hereinafter, the CUHK-PEDES dataset is abbreviated as "C", and the ICFG-PEDES dataset is abbreviated as "I". The proposed framework is compared with other two baseline strategies:

[0109] Strategy 1: Taking the I->C task as an example, this strategy means that the model is directly searched on the CUHK-PEDES target domain after pre-training on the ICFG-PEDES dataset, and the evaluation results are obtained. Therefore, the Rank-1 and mAP indicators are unchanged regardless of the number of samples.

[0110] Strategy 2: Taking the I->C task as an example, the model trained on the ICFG-PEDES dataset is used as the pre-trained model in the embodiment. During the entire training process, the model only learns on the support set CUHK-PEDES, adopts the early stopping strategy, sets the patience parameter to 10 rounds, the complete training round is 60, the learning rate is 1e−5, and the batch size is 64.

[0111] Strategy 3: Taking the I->C task as an example, the model trained on the ICFG-PEDES dataset is used as the pre-trained model in the embodiment. This strategy is the adapter strategy proposed by the present application, which uses the Adam optimizer to train the model, sets the learning rate to 1e-5, the batch size to 64, and the momentum to 0.5. The early stopping strategy is adopted, the patience parameter is set to 10 rounds, the total training round is set to 60, and in the preprocessing stage, 3 templates are randomly selected from Table 1 for text description diversity enhancement.

[0112] The strategies 1, 2 and 3 are experimentally verified, and the results are shown in Table 1. The number of samples in the table represents the number of pedestrians in the target domain support set. Note that the present application only has one image and its corresponding text description for each pedestrian. By analyzing the data in the table, it can be found that when the source domain, target domain and sample size are fixed, the method of the embodiment achieves better performance on the test set compared with strategies 1 and 2.

[0113] Table 1 Cross-dataset generalization ability table

[0114]

[0115] For example, when the ICFG-PEDES dataset is taken as the source domain, the CUHK-PEDES dataset is taken as the target domain, and the number of support set samples is 20. As can be seen from the table, the Rank-1 of strategy 1 is 33.48%, and the mAP index is 31.56%, which indicates that there is a large difference between the source domain and the target domain. At the same time, due to the too small number of support sets, the model is difficult to effectively align the cross-domain in the learning process, and the performance of strategy 2 which only trains through the support set is also not ideal, and the Rank-1 and mAP indexes are also relatively low, only 36.15% and 47.53%. However, with the increase of the number of support set samples used for training, the performance of strategy 2 has been significantly improved. Strategy 3 is the method proposed in the present application. Compared with strategy 2, the Rank-1 of strategy 3 reaches 45.38%, and the mAP index is 56.61%, which is increased by 1.65% and 1.79% respectively, which indicates that the strategy of the present application has obvious effectiveness. In addition, it can also be seen from the table that the effect of the number of samples on the effect of strategy 3 method, the less the number of samples, the more obvious the effect of strategy 3 data enhancement and multi-modal adapter. For example, when the number of support set samples is 20, the Rank-1 is increased by 2.2%, and the mAP is increased by 2.12%; when the number of samples is increased to 50, the Rank-1 is increased by 2.06%, and the mAP is increased by 2.29%; when the number of support set samples is further increased to 100, the Rank-1 is increased by 1.65%, and the mAP is increased by 1.79%. Therefore, through the comparison of the experimental results, the effectiveness of the proposed method is verified. In the case of only a small number of samples in the target domain, the model can only use a limited number of samples for training, which is easy to cause overfitting. Unlike directly global fine-tuning, the adapter introduced in the embodiment of the present application adapts to the original features by learning a small number of trainable parameters. Since the adapter does not change the core information of the original features, it can effectively reduce the risk of overfitting while preserving the pre-training knowledge.

[0116] As shown in Table 1, compared with other methods, the method of the present embodiment has improved in C->I and I->C tasks. Compared with the CFAM method, the Rank-1 is increased by 4.9% in the C->I task, and the Rank-1 is increased by 4.99% in the I->C task. These experimental results prove that the cross-domain generalization ability of the method of the present embodiment is competitive.

[0117] Table 2 Comparison of computing resource consumption of different strategies

[0118]

[0119] As shown in Table 2, it can be seen that in the case of training the model by default using single-precision float32, the computing resource consumption after introducing the adapter will be greatly reduced, the trainable parameter amount is only about 0.1% of the parameter amount of the fully fine-tuned CLIP visual and language encoder, and the GPU memory consumption is also greatly reduced, which shows that the method proposed in the embodiment can significantly reduce the consumption of computing resources while ensuring the performance by fine-tuning the adapter only.

[0120] Qualitative analysis:

[0121] Fig. 4(a) shows the text features extracted by the pre-trained model in the method of the embodiment, and Fig. 4(b) shows the text features extracted by the method of the embodiment. Different colors in the figures represent different pedestrian IDs, and 15 pedestrian IDs are randomly selected as test samples. As can be seen from Fig. 4(a) and Fig. 4(b), the pedestrian features extracted by the method of the embodiment are more closely clustered, the boundaries between categories are more clear, and the separability of the features is significantly improved. In contrast, the original feature distribution is relatively loose, and there is a lot of overlap between categories, which may affect the classification performance. After processing by the method of the embodiment, the sample points of the same category are more concentrated, and the interval between different categories is increased, which is more conducive to the expression of features for downstream tasks. Overall, the method proposed improves the discriminability of the features, and through the action of the adapter, the features of the target domain can be better separated and distinguished, thereby significantly enhancing the cross-domain adaptation capability of the model.

[0122] In addition, the embodiment also includes a computer device including a memory and a processor;

[0123] The memory is used to store a computer program that can run on the processor;

[0124] The processor is used to implement the steps of the pedestrian re-identification method as described above when the computer program is executed.

[0125] In addition, the embodiment also includes a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the pedestrian re-identification method as described above.

[0126] For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present application. The one or more modules / units can be a series of computer program instruction segments that can complete a specific function, which are used to describe the execution process of the computer program in the computer device.

[0127] The computer device can be a mobile phone, a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The computer device can include, but is not limited to, a processor, a memory, and the like. For example, the computer device can further include an input / output device, a network access device, a bus, and the like.

[0128] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor is a control center of the computer device, and is connected to various parts of the computer device through various interfaces and lines.

[0129] The memory can be used to store the computer program and / or the module, and the processor can realize the computer program by running or executing the computer program and / or the module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required for a function (such as a sound playing function, an image playing function, and the like), and the like. The data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, and the like), and the like. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0130] The modules / units integrated in the computer device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can realize the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. that can carry the computer program code.

[0131] In addition, the embodiment of the present application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the pedestrian re-identification method.

[0132] The embodiment provides a pedestrian re-identification method, in the data preprocessing stage, a multi-modal large language model is used to generate diversified text descriptions to make up for the problem of insufficient target domain data. In addition, the present application also designs a lightweight multi-modal adapter, which uses a multi-head attention mechanism to jointly adapt text and image features, generates adapted feature representations, and superimposes the original features. The adapter has fewer trainable parameters, which enhances the model's ability to adapt to the target domain while significantly reducing computational efficiency.

[0133] It should be noted that the above-described device embodiments are only schematic, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

Claims

1. A pedestrian re-identification method based on a multimodal adapter, characterized in that, Includes the following steps: S1: Collect pedestrian re-identification data, including pedestrian image data and corresponding text descriptions; S2: Perform data augmentation on the pedestrian re-identification data in S1. Data augmentation is used to process image data that lacks text description and generate the corresponding text description for the image data. S3: Text enhancement based on a multimodal large language model for text description; S4: Construct a multimodal adapter to generate text-adapted embeddings and image-adapted embeddings, and superimpose them with the original text embeddings and image embeddings to obtain text fusion features and image fusion features; The multimodal adapter includes an embedded downsampler, a masked multi-head attention network, and a linear network layer; The embedding downsampling unit is set in a masked multi-head attention network to obtain image embeddings and text cue embeddings, and to reduce the embedding dimension. Masked multi-head attention networks include mask matrices. And mask attention calculation, wherein the mask matrix is ​​used to guide cross-modal information interaction, wherein , I and T These represent the quantity of image data and the quantity of text description, respectively. The linear network layer includes a first upsampled linear layer arranged sequentially. GELU nonlinear activation function and second upsampling linear layer First upsampling linear layer Second upsampling linear layer Each has a corresponding first weight matrix. Second weight matrix ,in, Indicates the feature dimension; S5: Use text fusion features and image fusion features to predict pedestrian re-identification and achieve pedestrian re-identification.

2. The pedestrian re-identification method according to claim 1, characterized in that, In S2, the data augmentation process is as follows: A large language model is invoked to generate prompts with static text descriptions. The prompts include pedestrian posture, clothing features, carried items, and template constraints. The template constraints are used to limit the text length and style. Based on prompts, a multimodal large language model is used to generate text descriptions corresponding to image data.

3. The pedestrian re-identification method according to claim 2, characterized in that, In S3, the text enhancement process is as follows: Construct natural language instructions, invoke the language generation model, and use the natural language instructions to generate enhanced templates; Based on the enhanced template, a multimodal large language model is invoked to enhance the text description; The expression for generating the enhanced template is as follows: ； in, Indicates the first m An enhanced template; t is the input, which includes image data or text description; s represents a natural language instruction; The language generation model GPT-4o is represented.

4. The pedestrian re-identification method according to claim 3, characterized in that, In S4, the computation process of the masked multi-head attention network is as follows: Calculate the mask matrix: ； in, , For the number of images, The number of text descriptions, Indicates positive interaction. This indicates a negative interaction, ensuring that the text embedding focuses only on image information; The mask matrix is ​​applied to mask attention calculation using an addition operation. The expression for mask attention calculation is as follows: ； in, , and These are the query matrix, key matrix, and value matrix, respectively. Represents the dimension of the key vector. Indicates matrix transpose. This indicates masked attention.

5. The pedestrian re-identification method according to claim 4, characterized in that, In S4, the calculation process for the multimodal adapter is as follows: ； ； ； ； ； in, This represents the input of the multimodal adapter. This represents the sum of the number of text embeddings and image embeddings. This indicates the number of samples in a batch. For the embedded dimension; Indicates an embedded downsampling; This indicates the output of the embedded downsampler; This indicates a masked multi-head attention network; This represents the output of a masked multi-head attention network. Indicates the first upsampling linear layer The output; This represents the output of the nonlinear activation function GELU; This indicates the output of the multimodal adapter.

6. The pedestrian re-identification method according to claim 5, characterized in that, In S4, the output of the multimodal adapter Split into text adaptation embedding Image adaptation embedding After embedding dimension normalization, the text embedding and image embedding are weighted and fused together to obtain text fusion features and image fusion features: ； ； in, Indicates text fusion features, Represents image fusion features, For hyperparameters, This indicates the original text embedding. This represents the original image embedding.

7. The pedestrian re-identification method according to claim 6, characterized in that, In S5, the parameters of the multimodal adapter are optimized during model training. The training loss includes identity loss and contrastive learning loss. The expression for calculating the identity loss is as follows: ； in, and These represent the classification weight matrices respectively. The Column and number List, express Identity tags and These represent the first and second elements in the bias vector, respectively. Element and the element.

8. A computer device, characterized in that, Including memory and processor; The memory is used to store computer programs that can run on the processor; When the processor executes the computer program, it implements the steps of the pedestrian re-identification method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the pedestrian re-identification method as described in any one of claims 1 to 7.

Images