Pedestrian description generation method based on multi-model local description integration

Abstract

The invention discloses a pedestrian description generation method based on multi-model local description integration, and relates to the technical field of image data processing. Comprising the following steps: acquiring a pedestrian image to be labeled, inputting the image and a preset instruction into a plurality of image-to-text models, and generating local text description of each attribute of the pedestrian; extracting image and text features by utilizing an image-text matching model, calculating the matching degree of the image and the text through cosine similarity, and screening out attribute description which is most matched with the image from a plurality of descriptions; combining the screened local descriptions by adopting a large language model to generate a smooth text description of the whole pedestrian; and training the image-text matching model through the pedestrian image and the text data set, and optimizing the image-text matching model by using a comparison loss function and a matching loss function until the total loss function is converged. According to the method, the attribute phrases of the pedestrians are generated through the cross-modal large model, so that the description of the attribute phrases of the pedestrians is richer and more diversified, and the generated description is prevented from tending to be the same.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a pedestrian description generation method based on multi-model local description integration. Background Art

[0002] The text-based pedestrian retrieval task is to find images of specific pedestrians that match a given text description from a large number of pedestrian images captured by the camera. In the field of text-based pedestrian retrieval, image data is relatively easy to obtain and can be automatically captured by surveillance cameras or other devices, while detailed and high-quality pedestrian text descriptions usually need to be manually written or annotated, which takes a lot of time and manpower. With the development of vision-language cross-modal pre-trained large models, the model can better understand the complex scenes in the image with its huge parameter scale and powerful computing power, thereby generating a more accurate text description of the image. However, due to the domain-specific and fine-grained characteristics of pedestrian text descriptions, the direct application of these existing models to the pedestrian image description task is not ideal. At present, the more common methods for pedestrian description generation are the generation method based on predefined pedestrian attribute combination and the generation method based on cross-modal large models. The generation method based on predefined pedestrian attribute combination sets the attribute set of common pedestrians in advance, then uses the pre-trained model to select the most appropriate phrase in the pedestrian attribute set, and finally generates a pseudo-text description after combining all attributes.

[0003] However, although the predefined pedestrian attributes cover common pedestrian appearance features, when faced with more complex and diverse pedestrian scenes, the predefined attributes alone cannot fully express the differences between individuals and it is difficult to obtain diverse pedestrian descriptions. The generation method based on the cross-modal large model generates pedestrian descriptions through short instructions. Generally, a description of the image can be generated through a single round of dialogue, but due to the ambiguity of the pedestrian image itself, some of the generated descriptions may be irrelevant to the objective content of the image.

[0004] Currently, the common pedestrian description generation methods can be summarized as: generation methods based on predefined pedestrian attribute combinations and generation methods based on cross-modal large models, but there are still many problems, such as: generation methods based on predefined pedestrian attribute combinations, because in actual scenarios, the attributes of pedestrians are changeable, the predefined attribute combinations cannot cover all potential pedestrian characteristics, and the generated descriptions are easily restricted by predefined templates, lacking diversity and accuracy in detail descriptions; when the cross-modal large model is used to directly generate the overall description of the pedestrian image, it is impossible to generate all the pedestrian attributes in the image at one time, and some more common pedestrian attribute descriptions are often omitted. At the same time, the background information in the image may interfere with the model, causing the generated description to contain content that is not related to the pedestrian. Summary of the invention

[0005] In view of the problems existing in the above background technology, the present invention is proposed.

[0006] Therefore, the problem to be solved by the present invention is how to generate attribute phrases of pedestrians through a large cross-modal model, describe the attributes of pedestrians through multiple graph-to-text models, screen the generated results, eliminate possible erroneous descriptions, and improve the accuracy of text descriptions.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: In the first aspect, an embodiment of the present invention provides a method for generating pedestrian descriptions based on the integration of multi-model local descriptions, which includes obtaining a pedestrian image to be annotated, inputting the image and preset instructions into multiple image-text models, and generating local text descriptions of various attributes of the pedestrian; using an image-text matching model to extract image and text features, calculating the matching degree between the image and the text by cosine similarity, and screening out the attribute description that best matches the image from multiple descriptions; using a large language model to merge the screened local descriptions to generate a fluent text description of the pedestrian as a whole; training the image-text matching model using pedestrian image and text data sets, and optimizing it using contrast loss and matching loss functions until the overall loss function converges.

[0008] As a preferred solution of the pedestrian description generation method based on multi-model local description integration described in the present invention, the method of using a large language model to merge the screened local descriptions to generate a fluent text description of the pedestrian as a whole includes the following steps: performing a fluent text description of the pedestrian as a whole; extracting attribute phrases from the pedestrian's text, weighting them in combination with bidirectional attention, and constructing a masked phrase modeling loss function.

[0009] As a preferred solution of the pedestrian description generation method based on multi-model local description integration of the present invention, the fluent text description of the pedestrian as a whole includes: , = , , Among them, LLM is a large language model. is the final combined text description, , ,..., Represents the attributes of pedestrian images, is the i-th cross-modal large model, is the instruction prompt designed for the jth attribute, I is the image, K is the number of attributes in the pedestrian image, For the selected attributes A local text description of .

[0010] As a preferred solution of the pedestrian description generation method based on multi-model local description integration of the present invention, the construction of the mask phrase modeling loss function includes the following steps: And the image represents F I As the input of the cross-modal encoder, the cross-attention layer internal calculation is performed; according to the bidirectional attention mechanism, the features of each image block are weighted to obtain the weighted global representation; the image is calculated and phrases The coarse similarity between them; bidirectional attention weighted local alignment of local attributes; and design of mask phrase modeling loss function.

[0011] As a preferred solution of the pedestrian description generation method based on multi-model local description integration of the present invention, wherein: the cross-attention layer internal calculation includes: , , in, and H is the number of heads, , , is the linear projection matrix, yes The transpose of , A attention relation matrix describes the similarity relation between phrase tokens and image patches.

[0012] As a preferred solution of the pedestrian description generation method based on multi-model local description integration of the present invention, wherein: the weighting of the features of each image block according to the bidirectional attention mechanism includes: , in reflects the weight of the j-th image block to the input phrase, is the input j-th image block feature, is the number of image patches of the input image.

[0013] As a preferred solution of the pedestrian description generation method based on multi-model local description integration of the present invention, the training loss of the image-text matching model is: , in, is the set of phrases in the input text.

[0014] In the second aspect, an embodiment of the present invention provides a pedestrian description generation system based on the integration of multi-model local descriptions, which includes: an image acquisition module, used to obtain pedestrian images to be annotated; a description generation module, used to input images and preset instructions into multiple image-text models to generate local text descriptions of various attributes of pedestrians; a feature extraction and matching module, used to extract image and text features using an image-text matching model, calculate the matching degree between the image and the text by cosine similarity, and screen out the attribute description that best matches the image from multiple descriptions; a description integration module, used to merge the screened local descriptions using a large language model to generate a fluent text description of the pedestrian as a whole; a model training module, used to train the image-text matching model using pedestrian images and text data sets, and optimize the model using contrast loss and matching loss functions until the overall loss function converges.

[0015] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the method for generating pedestrian descriptions based on the integration of multi-model local descriptions as described in the first aspect of the present invention are implemented.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, the steps of the method for generating pedestrian descriptions based on the integration of multi-model local descriptions as described in the first aspect of the present invention are implemented.

[0017] The beneficial effects of the present invention are as follows: the present invention generates attribute phrases of pedestrians through a cross-modal large model, making the description of pedestrian attribute phrases richer and more diverse, avoiding the convergence of generated descriptions and failing to reflect the differences in details between pedestrians; compared with the traditional generation method based on a cross-modal large model, the present invention describes pedestrian attributes through multiple graph-to-text models, and screens the generated results to exclude possible erroneous descriptions, thereby improving the accuracy of the text description. The present invention is a solution to the problem of the diversity of pedestrian description generation in the past and the difficulty of ensuring the accuracy of global descriptions generated by a single model, making the pedestrian attributes in the generated text richer, excluding some text descriptions that are not related to the pedestrians themselves, and improving the accuracy of the text description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1The process of pedestrian description generation method based on the integration of multi-model local descriptions.

[0020] Figure 2 Schematic diagram of the training process of the pedestrian image-text matching model.

[0021] Figure 3 Schematic diagram of the global loss of training the image-text matching model.

[0022] Figure 4 Schematic diagram of the local loss of training the image-text matching model.

[0023] Figure 5 Generate structure graphs for local descriptions.

[0024] Figure 6 is the fusion triplet loss hyperparameter Graph of experimental data. DETAILED DESCRIPTION

[0025] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0027] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0028] Example 1 Reference Figure 1 to Figure 6 , which is the first embodiment of the present invention, and provides a method for generating pedestrian description based on the integration of multi-model local descriptions, comprising: S1: Obtain the pedestrian image to be annotated, input the image and preset instructions into multiple image-to-text models, and generate local text descriptions of various attributes of the pedestrian.

[0029] S1.1: Obtain the pedestrian image to be annotated.

[0030] S1.2: Input pedestrian images and pre-designed instruction prompts into n image-to-text models, and obtain n texts for K attributes: , , in, is the i-th cross-modal large model, is the instruction prompt designed for the jth attribute, I is the pedestrian image, , ,..., Represents the attributes of pedestrian images, which can be different aspects of pedestrian appearance or behavior characteristics, such as gender, top style and color, etc.

[0031] S2: Use the image-text matching model to extract image and text features, calculate the matching degree between image and text through cosine similarity, and select the attribute description that best matches the image from multiple descriptions.

[0032] S2.1: Build a pedestrian image-text matching model.

[0033] Preferably, Figure 2 The figure shows the training process of the pedestrian image-text matching model. The constructed pedestrian image-text matching model structure is as follows: Figure 3 shown.

[0034] Specifically, the architecture is based on the transformer model, with image and text encoders, corresponding momentum models and a cross-modal encoder. Specifically, ViT is used as the image encoder, the first 6 layers of BERT are used as the text encoder, and the last 6 layers are used as the cross-modal encoder.

[0035] S2.2: Obtain a pedestrian image and text dataset, and extract image and text features for each pair of pedestrian image and text.

[0036] Construct image-text loss function, image-text matching loss function and triplet loss function.

[0037] Here, the image-text contrast loss is used ( ) and image-text matching loss ( ) for global alignment and fuses the triplet loss ( ) to further enhance the global alignment and thus enhance the difference between positive and negative text-image pairs.

[0038] The specific calculation steps are as follows: Image-text pair The image features obtained by the image encoder and the text in , text input text encoder to get text features : , , Among them, d represents the feature dimension of the image or text after encoding , is the number of image patches in the input image and is the number of tokens of the input text.

[0039] Specifically, by calculating the cosine similarity of text embedding and image embedding to obtain their rough similarity, the rough similarity between image and text can be calculated: , in, represents the norm, (or ) is to (or ) is mapped to a linear projection of a representation with lower dimensionality, (or ) represents the global representation of text (or image).

[0040] During training, the latest images and text from the kinetic encoder The global representation is stored in two queues and In , the image-to-text and text-to-image similarities are computed by combining the representations from the original encoder and the paired momentum encoder as follows: , , in, Measure the global representation of the image and the A global representation of a text (in ), similar to , is a learnable temperature parameter, exp( ) represents an exponential operation with a constant e as the base.

[0041] Therefore, the image-text contrast loss Defined as: , in, , ∈ Indicates the label relationship between image and text, the value of image-text pair is 1, while the value of negative pair is 0. Represents the function for calculating the cross entropy loss.

[0042] In this way, a contrast loss between image and text is constructed, which increases the ability of the pedestrian image-text matching model to distinguish between different pairs.

[0043] Afterwards, and Input the cross-modal encoder to obtain the image-text fusion representation: ; On this basis, using the linear projection matrix W O Compute the true similarity between an image and text: , in Represents the similarity between the global image representation and text after fusion with the cross-modal encoder.

[0044] Based on the fine similarity, the image-text matching loss can be expressed as: , in, Indicates whether the image-text pair is a positive number pair, 1 indicates a positive number, and 0 indicates a negative number.

[0045] Typically, negative examples are sampled from the training batch.

[0046] Although the image-text matching loss is effective in facilitating the training of cross-modal encoders, it often prioritizes individual pairs without considering the contrast between pairs.

[0047] Specifically, with negative and( , T) compared to Should output higher ,in In represents the text that does not match image I, Same reason.

[0048] To satisfy this, in addition to the baseline and In addition, a fused triplet loss is added to emphasize the gap between positive and negative samples: , in, is the minimum margin between positive and negative pairs, represents max(·,0), and experiments show that Taking 0.6 has the best effect. The specific experimental data are as follows Figure 6 As shown, the R@k metric represents the probability that at least one matching person's image appears in the top k candidate list given a query description. It can be seen that here The best effect is achieved when the value is 0.6.

[0049] S3: A large language model is used to merge the selected local descriptions to generate a fluent text description of the pedestrian as a whole.

[0050] S3.1: Preferably, a fluent text description of the pedestrian as a whole is as follows: , Among them, LLM is a large language model. A textual description of the final merger.

[0051] Attribute phrases are extracted from pedestrian texts, weighted by combining bidirectional attention, and a masked phrase modeling loss function is constructed.

[0052] S3.2: Extract attribute phrases from pedestrian text, combine bidirectional attention for weighting, and construct a masked phrase modeling loss function.

[0053] Here, in order to improve the ability of local attribute matching, the NLTK natural language processing library is used to extract attribute phrases, and the gradient is used as backward attention weighting to assist the local alignment of pedestrian images and local pedestrian attribute phrases.

[0054] The phrase represents And the image represents F I As the input of the cross-modal encoder, the calculation process inside the cross-attention layer is: , , in, and H is the number of heads, , , is the linear projection matrix, Q, K, and V represent the query vector, key vector, and value vector in the Transformer architecture, respectively. yes The transpose of , A attention relation matrix describes the similarity relation between phrase tokens and image patches.

[0055] The forward attention of the representation of the jth image patch on the global features of the phrase is: , in, is the first row and jth column of A, Fa is the flag for forward attention, represents the forward attention weight of the j-th image patch to the phrase.

[0056] The predicted score for the masked word label is: , in, is a linear projection, represents the relationship of each image patch to the masked token in the phrase of A.

[0057] The backward attention based on the back-propagated gradient of s is expressed as: , in, Represents an image block The backward attention weights for the predicted scores s of the masked word labels are obtained by backpropagating the gradients To calculate, is an image block No. dimensional eigenvalue, Represents a linear projection matrix No. elements.

[0058] Finally, two-way attention is defined as: , in, and They are and The average value on the attention head, norm is a function that normalizes the weights.

[0059] According to the bidirectional attention mechanism, the features of each image block are weighted to obtain the weighted global representation: , in reflects the weight of the j-th image block to the input phrase, is the number of image patches of the input image.

[0060] therefore, , which is the sum of the representations of all weighted image patches, can also be regarded as a phrase-guided global image representation. The representation of the image patch corresponding to the phrase is highlighted, while the representations of the others are weakened.

[0061] Therefore, the image can be calculated and phrases The rough similarity between: , in, The norm represented by (resp. ) is to (or ) is a linear projection that maps to a representation with lower dimensionality.

[0062] In general, the bidirectional attention weighted local alignment loss for an image-phrase pair is: , In addition to the bidirectional attention-weighted local alignment for local attributes, a masked phrase modeling loss function is also proposed.

[0063] Specifically, in the input phrase A masking operation is performed on the phrase by randomly replacing one word in the phrase with the token [MASK]. The masked phrase is represented as ,Will Input into the image-text matching model and calculate the final fusion representation: , in The phrase you entered The number of tokens after passing through the text encoder, represents the text representation of the masked phrase, where represents the global representation of the masked phrase, and It represents the local representation of each text token.

[0064] Then, a multi-layer perceptron (MLP) classifier is used to predict the original tokens in the phrase via an MPM loss on the image-phrase pair: , in, is the one-hot vocabulary distribution of the jth token, is the size of the vocabulary.

[0065] S4: Train the image-text matching model using pedestrian image and text datasets, and optimize the model using contrast loss and matching loss functions until the overall loss function converges.

[0066] Preferably, the training loss of the entire model is: , in, is the set of phrases in the input text.

[0067] During the specific training, AdamW was used as the optimizer and the initial learning rate was set to 1×10 -5 And the learning rate is adjusted using the cosine learning rate decay method.

[0068] Furthermore, this embodiment also provides a pedestrian description generation system based on multi-model local description integration, including: The image acquisition module is used to obtain the pedestrian images to be annotated; the description generation module is used to input the images and preset instructions into multiple image-text models to generate local text descriptions of the pedestrian's attributes; the feature extraction and matching module is used to extract image and text features using the image-text matching model, calculate the matching degree between the image and the text by cosine similarity, and select the attribute description that best matches the image from multiple descriptions; the description integration module is used to merge the selected local descriptions using a large language model to generate a fluent text description of the pedestrian as a whole; the model training module is used to train the image-text matching model using pedestrian image and text datasets, and optimize the model using contrast loss and matching loss functions until the overall loss function converges.

[0069] This embodiment also provides a computer device, which is suitable for the pedestrian description generation method based on the integration of multi-model local descriptions, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the pedestrian description generation method based on the integration of multi-model local descriptions proposed in the above embodiment.

[0070] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0071] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating pedestrian description based on the integration of multi-model local descriptions as proposed in the above embodiment.

[0072] In summary, the present invention generates attribute phrases of pedestrians through a cross-modal large model, making the description of pedestrian attribute phrases richer and more diverse, avoiding the convergence of generated descriptions and failing to reflect the differences in details between pedestrians; compared with the traditional generation method based on a cross-modal large model, the present invention describes pedestrian attributes through multiple graph-to-text models, and screens the generated results to exclude possible erroneous descriptions, thereby improving the accuracy of the text description. The present invention is a solution to the problem of the diversity of pedestrian description generation in the past and the difficulty of ensuring the accuracy of global descriptions generated by a single model, making the pedestrian attributes in the generated text richer, excluding some text descriptions that are not related to the pedestrians themselves, and improving the accuracy of the text description.

[0073] Example 2 The second embodiment of the present invention provides a method for generating pedestrian descriptions based on the integration of multi-model local descriptions. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0074] Taking the unlabeled pedestrian dataset LUPerson as an example, the operation steps are as follows: The unlabeled dataset LUPerson is selected, which contains a large number of unlabeled pedestrian images; The pedestrian image I and the instruction prompts designed for the K attributes of the pedestrian are respectively input into n cross-modal large models. Specifically, the pedestrian image I and the instruction prompts designed for the jth (1 j K) command prompts designed for each attribute Input to the i-th (1 i n) A large cross-modal model In the above example, we get the text description of the jth attribute of the pedestrian image I by the model. , then for the jth attribute, n different text descriptions can be generated ; Given a pair of input images and their associated attribute text , through the image encoder to the input image Process it to extract its feature representation and get the image representation ,in, represents the global representation of the image, and represents the representation of the image block. The text encoder will target the attribute Each text Encode and generate text representation ,in, represents the global representation of the text, and The corresponding text tag; Then, the similarity between image and text embedding is estimated as in and is a linear transformation used to embed the global and Mapped to a standardized low-dimensional representation, the similarity between the text features and image features generated by each model is calculated to obtain , by calculating and summing the cosine similarity value Sort in descending order to define the overall match score, then use the sorted index argsort( ) For the generated n pairs The text descriptions of the attributes are sorted. According to this sorting, the attribute description text that is found last and ranked first is selected as the attribute The final description is recorded as = .

[0075] Select the attribute description text Combine the attribute name A with the corresponding attribute description text T A Combinations and prompt words The conditional probability is maximized to generate the most likely text description. In this process, the model predicts the probability distribution of the next word word by word ,in Is to generate text The words, is generated before Combine multiple local attribute description texts into a fluent and complete pedestrian text description.

[0076] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A pedestrian description generation method based on multi-model local description integration, characterized by: include, Obtain pedestrian images to be annotated, input the images and preset instructions into multiple image-to-text models, and generate local text descriptions of various attributes of pedestrians; The image and text matching model is used to extract image and text features, the matching degree between image and text is calculated by cosine similarity, and the attribute description that best matches the image is selected from multiple descriptions; A large language model is used to merge the selected local descriptions to generate a fluent text description of the pedestrian as a whole. The image-text matching model is trained using pedestrian image and text datasets, and optimized using contrast loss and matching loss functions until the overall loss function converges.

2. The method for generating pedestrian description based on integration of multiple model local descriptions as claimed in claim 1, characterized in that: The method of using a large language model to merge the selected local descriptions to generate a fluent text description of the pedestrian as a whole includes the following steps: Provide a fluent text description of the pedestrian as a whole; Attribute phrases are extracted from pedestrian texts, weighted by combining bidirectional attention, and a masked phrase modeling loss function is constructed.

3. The method for generating pedestrian description based on integration of multiple model local descriptions as claimed in claim 2, characterized in that: The fluent text description of the pedestrian as a whole includes: ， = ， ， Among them, LLM is a large language model. is the final combined text description, , ,..., Represents the attributes of pedestrian images, is the i-th cross-modal large model, is the instruction prompt designed for the jth attribute, I is the image, and K is the number of attributes in the pedestrian image.

4. The method for generating pedestrian description based on integration of multiple model local descriptions as claimed in claim 3, characterized in that: The construction of the mask phrase modeling loss function comprises the following steps: The phrase represents And the image represents F I As the input of the cross-modal encoder, it performs internal calculations across the attention layer; According to the bidirectional attention mechanism, the features of each image block are weighted to obtain a weighted global representation; Computing images and phrases The rough similarity between Bidirectional attention weighted local alignment of local attributes; Designing a loss function for masked phrase modeling.

5. The method for generating pedestrian description based on integration of multiple model local descriptions as claimed in claim 4, characterized in that: The internal calculations of the cross-attention layer include: ， ， in, and H is the number of heads, , , is the linear projection matrix, yes The transpose of , A attention relation matrix describes the similarity relation between phrase tokens and image patches.

6. The method for generating pedestrian description based on integration of multiple model local descriptions as claimed in claim 5, characterized in that: The weighting of the features of each image block according to the bidirectional attention mechanism includes: ， in reflects the weight of the j-th image block to the input phrase, is the feature of the j-th image block of the input, is the number of image patches of the input image.

7. The method for generating pedestrian description based on integration of multiple model local descriptions as claimed in claim 6, characterized in that: The training loss of the image-text matching model is: ， in, is the set of phrases in the input text.

8. A pedestrian description generation system based on multi-model local description integration, based on the pedestrian description generation method based on multi-model local description integration according to any one of claims 1 to 7, characterized in that: Also includes, An image acquisition module, used to obtain pedestrian images to be labeled; The description generation module is used to input images and preset instructions into multiple image-to-text models to generate local text descriptions of various attributes of pedestrians; The feature extraction and matching module is used to extract image and text features using the image-text matching model, calculate the matching degree between the image and the text through cosine similarity, and select the attribute description that best matches the image from multiple descriptions; The description integration module is used to merge the selected local descriptions using a large language model to generate a fluent text description of the pedestrian as a whole; The model training module is used to train the image-text matching model using pedestrian image and text datasets, and optimize the model using contrast loss and matching loss functions until the overall loss function converges.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for generating pedestrian description based on the integration of multi-model local descriptions as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating pedestrian description based on the integration of multi-model local descriptions as described in any one of claims 1 to 7 are implemented.

Images