Pedestrian re-identification method and device and medium

Abstract

The invention relates to the technical field of computer vision, in particular to a pedestrian re-identification method, which comprises the following steps of S1, collecting pedestrian re-identification data; s2, constructing a scene graph according to the pedestrian re-identification data; s3, constructing an image-text multi-modal model architecture which comprises a domain self-adaption module and a task self-adaption module; s4, training the image-text multi-modal model architecture based on prompt learning and scene graph knowledge to obtain an image-text alignment model; s5, inputting a scene graph and a text into the image-text alignment model, and outputting structural knowledge embedding; s6, the structure knowledge is embedded and combined with the original text, and fusion features are obtained; and S7, realizing pedestrian re-identification prediction by adopting the fusion features. According to the method, alignment of a downstream data field and an original training data field of the CLIP model is achieved through prompt learning, semantic negative sampling and a scene graph analyzer are further introduced, scene graph knowledge is used as input, and the obtaining capacity of the model for structured knowledge is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a pedestrian re-identification method, device and medium. Background Art

[0002] Large-scale visual language pre-training models provide new solutions for the text-to-image person re-identification task. Some works propose to use pre-trained models to learn visual semantic knowledge, achieve explicit global alignment or discover finer-grained local alignment, thereby improving re-identification performance. Although these methods have made significant progress, there are still some problems in practical applications. For example, the CLIP model has the same defects as the bag-of-words model, which makes it difficult to capture the fine-grained semantics in sentences, and the CLIP model has limitations in understanding the structural information of the text and cannot accurately distinguish ambiguous expressions in the text. Therefore, it is still unknown whether these existing CLIP-based text-to-image person re-identification models have the ability to understand the fine-grained semantics of sentences, that is, the ability to understand object attributes and the relationship between objects.

[0003] In summary, there is an urgent need for a pedestrian re-identification method, device and medium to solve the problems in the prior art. Summary of the invention

[0004] The present invention aims to provide a pedestrian re-identification method, device and medium, and the specific technical solution is as follows: A pedestrian re-identification method comprises the following steps: S1: Collect person re-identification data; S2: Construct scene graph based on person re-identification data; S3: Build a multimodal model architecture for images and texts, including a domain adaptation module and a task adaptation module; S4: Based on prompt learning and scene graph knowledge, the image-text multimodal model architecture is trained to obtain the image-text alignment model; S5: Input the scene graph and text into the image-text alignment model, and output structural knowledge embedding; S6: embed the structural knowledge and combine it with the original text to obtain fusion features; S7: Use fusion features to achieve pedestrian re-identification prediction.

[0005] Optionally, in S1, the pedestrian re-identification data includes an image and text, and the image and the text corresponding to the image constitute an image-text pair.

[0006] Optionally, in S2, the scene graph includes a triple structure consisting of objects, attributes and relationships.

[0007] Optionally, in S3, the domain adaptation module is provided with a text encoder and a visual encoder, and the task adaptation module is provided with a text encoder, a visual encoder, a hard negative sample generator and a scene graph parser; The text encoder is used to process text description information and extract text representation; The visual encoder is used to process pedestrian image data and extract image representation; The scene graph parser is used to parse the text representation into a scene graph; The hard negative sample generator is used to process image-text pairs and scene graphs generated by text representation to generate semantic negative samples; The image-text pair consists of a set of corresponding image representations and text representations.

[0008] Optionally, in S4, the image-text multimodal model architecture is trained based on the cue learning and scene graph knowledge, as follows: Call the domain adaptation module to add learnable domain-specific prompt words to the text description information, fix the parameters of the text encoder and visual encoder, process the pedestrian re-identification data through the text encoder and visual encoder respectively, and extract the text representation and image representation; The task adaptation module is called to fix the learned domain-specific prompt words, adjust the parameters of the text encoder and visual encoder during the training process, and introduce the scene graph parser and hard negative sample generator to obtain the image-text alignment model.

[0009] Optionally, in S5, structural knowledge embedding is obtained by integrating text and scene graph, expressed as follows: ; ; in, represents the triple encoding function, represents triple embedding, represents a triple, , and They represent head entity embedding, relation entity embedding, and tail entity embedding respectively; Represents structural knowledge embedding, Represents multiple transformer layers.

[0010] Optionally, in S6, the structural knowledge is embedded and combined with the original text representation to obtain a fusion feature , the process is as follows: ; in, represents the original text representation, represents a hyperparameter.

[0011] Optionally, the domain adaptation module is trained with the following loss: ; in, represents the domain adaptation training loss, represents the contrastive loss from text to image, Represents image to text comparison file; The loss for training the task adaptation module is as follows: ; in, represents the task adaptive training loss, Indicates ID loss, represents the negative sample learning loss; The expression of the ID loss function is as follows: ; in, represents the values ​​in the underlying target distribution, Indicates category ID prediction logarithm, Represents the total number of instances.

[0012] In addition, the present invention also provides a computer device, including a memory and a processor; The memory is used to store a computer program executable on the processor; The processor is used to implement the steps of the above-mentioned pedestrian re-identification method when executing the computer program.

[0013] In addition, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the pedestrian re-identification method as described above are implemented.

[0014] The application of the technical solution of the present invention has the following beneficial effects: The present invention provides a pedestrian re-identification method, device and medium. The method of the present invention first constructs a two-stage training strategy. In the first stage, the present invention uses prompt learning to optimize learnable prompts to achieve alignment of downstream data domains with the original training data domain of the CLIP model, thereby promoting domain adaptation. In the second stage, the present invention fine-tunes the CLIP model while further proposing semantic negative sampling and a scene graph knowledge encoder scene graph encoder, using scene graph knowledge as input to enhance the model's ability to acquire structured knowledge in the second stage.

[0015] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flowchart of the steps of the pedestrian re-identification method in a preferred embodiment of the present invention.

[0018] Figure 2 It is a schematic diagram of CLIP similarity scores of the model of the present invention and other models (CUHK-PEDES dataset).

[0019] Figure 3 It is a schematic diagram of CLIP similarity scores of the model of the present invention compared with other models (ICFG-PEDES-dataset). DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0021] See also Figure 1 , Figure 2 and Figure 3 , Figure 1 is a flowchart of the steps of a pedestrian re-identification method in a preferred embodiment of the present invention; Figure 2 It is a schematic diagram of comparative experimental results of the prediction ability of the model of the present invention; Figure 3 It is a schematic diagram of the semantic similarity scores of the model of the present invention and other models.

[0022] like Figure 1 As shown, this embodiment provides a pedestrian re-identification method, comprising the following steps: S1: Collect person re-identification data.

[0023] Specifically, the pedestrian re-identification data includes an image and text, and the image and the text corresponding to the image constitute an image-text pair. The image may be image data of a pedestrian, and the text may be detailed text description information corresponding to the image data.

[0024] S2: Constructing scene graph based on person re-identification data.

[0025] The scene graph includes a triple structure consisting of objects, attributes, and relationships. In this embodiment, detailed semantic information, including objects, attributes of objects, and relationships between objects, is essential for understanding structured knowledge. The information in the triple structure is also critical for cross-modal learning, which aims to enhance the joint representation of vision and language.

[0026] Specifically, given a text sentence Parsed into a scene graph, represented as ,in yes The collection of objects in (in is a set of relationship nodes) and a set of hyperedges representing the actual relationship between objects. is a set of attribute pairs, where Is a collection of attribute nodes associated with an object.

[0027] S3: Build a multimodal model architecture for images and text, including a domain adaptation module and a task adaptation module.

[0028] In this embodiment, the domain adaptation module is provided with a text encoder and a visual encoder, and the task adaptation module is provided with a text encoder, a visual encoder, a difficult negative sample generator and a scene graph parser; The text encoder is used to process text description information and extract text representation; The visual encoder is used to process pedestrian image data and extract image representation; The scene graph parser is used to parse the text representation into a scene graph; The hard negative sample generator is used to process image-text pairs and scene graphs generated by text representation to generate semantic negative samples; The image-text pair consists of a set of corresponding image representations and text representations.

[0029] It should be noted that the process of visual encoder extracting image representation is as follows: Given an image , first Split into a series fixed-size, non-overlapping image patches, where Indicates the image resolution, Indicates the number of channels, Represents the size of the block; Mapping a sequence of chunks into a sequence of visual tokens via a trainable linear projection , each visual token is represented by a non-overlapping local token or learnable [CLS] embedding; The visual markers will be input to the visual encoder In order to learn the correlation between each image block, each image block has a corresponding output value to represent the characteristics of the image block; Using linear projection method, Mapped to the joint text-image embedding space and used as the global image representation The final output of the image encoder can be expressed as ,in Indicates the number of image tags.

[0030] It should be noted that the text encoder The process of extracting text representation is as follows: Use lowercase bytes to tokenize the code, with a vocabulary size of 49152; The beginning and end of the text description are marked by [SOS] and [EOS] markers, indicating the start and end of the sequence; Mark the text enter and learn the relevance of each tag through a masked self-attention mechanism; Using linear projection method, Mapped to the joint text-image embedding space and used as the global text representation ; The output of the text encoder is finally represented as ,in Indicates the number of text tags.

[0031] Furthermore, the process of generating semantic negative samples by the hard negative sample generator is as follows: Given an image-text pair and by text Generated scene graph , considering that the scene corresponding to the ReID task is centered on people, the triples in the scene graph can be represented as (person, relation, object). Then the semantic negative sample It can be generated by object exchange. Specifically, given any two pairs of object relationships in the scene graph, and , its object exchange can be formally described as: ; in, is a function that swaps sentence components, Indicates a person-centered subject. Relational words, , Represents an object. Similarly, for any attribute pair in the scene graph and , It is generated by exchanging attributes, as shown below: .

[0032] Furthermore, the process of the scene graph parser parsing the text representation into a scene graph is as follows: The two types of structured knowledge, relation pairs and triples, are treated as triples, and the conjunction "is" is added to the relation pairs to achieve a unified representation. For example, the attribute-object relation pair In this way, it will be processed as a triple In this way, we can get a set of triples ,in, Represent the head entity, relationship entity and tail entity respectively. Each triple in , this example uses the word segmenter and vocabulary embedding from BERT to get the embedding of each entity , , , and finally the triple embedding of each entity is obtained through the following encoding function: ; in, represents the triple encoding function, represents triple embedding, represents a triple, , and They represent the head entity embedding, the relationship entity embedding and the tail entity embedding respectively. This embodiment distinguishes the position information of the triples by the structure of the triples. For example, two triples and , although they have the same elements but in different orders, the final embedding results obtained by a simple accumulation method will be the same, which will affect the model's ability to distinguish these triplets. Therefore, by introducing a triplet encoding function, this embodiment can ensure that the model can effectively recognize and process different triplet structures. The triples can be processed into Finally, the triples are fed into multiple transformer layers to obtain a scene graph representation.

[0033] S4: Based on prompt learning and scene graph knowledge, the image-text multimodal model architecture is trained to obtain the image-text alignment model.

[0034] Furthermore, based on the hint learning and scene graph knowledge, the image-text multimodal model architecture is trained as follows: Call the domain adaptation module to add learnable domain-specific prompt words to the text description information, fix the parameters of the text encoder and visual encoder, process the pedestrian re-identification data through the text encoder and visual encoder respectively, and extract the text representation and image representation; The task adaptation module is called to fix the learned domain-specific prompt words, adjust the parameters of the text encoder and visual encoder during the training process, and introduce the scene graph parser and hard negative sample generator to obtain the image-text alignment model.

[0035] Furthermore, this embodiment adds learnable domain-specific cues to the input sequence of the text encoder to learn how to adapt to the domain gap between downstream data and the original CLIP training data. The domain-specific cues are defined as continuous ID-specific learnable vectors whose dimensions are consistent with the joint text-image embedding space of the transformer block.

[0036] It should be noted that position embedding will not be added to the domain-specific prompt word, so the insertion position is not important. This embodiment directly adds the domain-specific prompt word to the end of the text sequence. Specifically, the domain-specific prompt word in this embodiment can be expressed as The superscript of each domain-specific cue word indicates that it is added to the text encoder, and the subscript Indicates the length of the text prompt word, domain specific prompt word is added directly to the end of the text sequence, so the text token becomes This embodiment adds domain-specific prompt words to the [EOS] tag, so that the model can introduce additional context or guidance information while maintaining the integrity of the original sentence. This embodiment adapts to new domain changes by adjusting the input text sequence without adjusting other model parameters by learning only text prompts without learning visual prompts.

[0037] In the stage of calling the domain adaptation module for training, this embodiment fixes the image encoder and text encoder The parameters of are optimized only for learnable domain-specific cues. Represents a batch ( images) The index of the image. is the [CLS] tag embedding of image features, and For the [EOS] tag embedding of the corresponding text features, this embodiment uses the contrast loss between text and image and Specifically, this embodiment maps and Get the final text representation respectively and the final image representation , where the subscript Represents the pedestrian ID. Contrastive loss of text to image It can be expressed as the following formula: ; in, represents the cosine similarity function that calculates the distance between text and image, Indicates that all The index set of relevant positive samples, Indicates the number of elements in this set. It should be noted that This batch includes all The indices of samples with the same label.

[0038] In this embodiment, the contrast loss from image to text can be expressed by the above expression, which will not be described again here.

[0039] S5: Input the scene graph and text into the graph-text alignment model, and output structural knowledge embedding.

[0040] Specifically, structural knowledge embedding is obtained by integrating text and scene graph, and the expression is as follows: ; in, Represents structural knowledge embedding, Represents multiple transformer layers.

[0041] S6: Embed the structural knowledge and combine it with the original text to obtain fused features.

[0042] Specifically, the structural knowledge is embedded and combined with the original text representation to obtain the fusion feature , the process is as follows: ; in, represents the original text representation, represents a hyperparameter.

[0043] Optionally, the domain adaptation module is trained with the following loss: ; in, represents the domain adaptation training loss, represents the contrastive loss from text to image, Represents image to text comparison file; The loss for training the task adaptation module is as follows: ; in, represents the task adaptive training loss, Indicates ID loss, represents the negative sample learning loss; The expression of the ID loss function is as follows: ; in, represents the values ​​in the underlying target distribution, Indicates category ID prediction logarithm, Represents the total number of instances.

[0044] This embodiment uses the contrast loss mentioned above Learn better feature representation. In addition, this embodiment introduces an additional contrastive learning objective to improve the performance of structured representation. Specifically, the contrastive learning objective in this embodiment is to With corresponding description (positive samples) and push the image further away With negative samples The distance between them is used to learn a sufficient representation, which is expressed by the following formula: ; in, Representing images Description with negative samples The distance between Representing images With corresponding description The distance between is a boundary hyperparameter.

[0045] S7: Use fusion features to achieve pedestrian re-identification prediction.

[0046] After the model training is completed, the prediction stage implements text-to-image person re-identification through the following steps. First, the trained feature extraction network is used to obtain the fused feature representation for the query text description and the candidate images in the gallery, and these features are normalized to stabilize the similarity calculation. Then, the cosine similarity is used to calculate the similarity between the query text and each candidate image, and the candidate images are sorted according to the similarity, and the top-ranked images are selected as the final recognition results.

[0047] In order to evaluate the effectiveness of the proposed method in the task of pedestrian re-identification, the present invention conducted experimental comparisons: First of all, the dataset. This experiment used three mainstream text-to-image datasets: CHUK-PEDES, ICFG-PEDES, and RSTPReid.

[0048] CUHK-PEDES is the first available dataset for text-image person retrieval. It contains 40,206 images of 13,003 pedestrians and 80,412 text descriptions, with 2 text descriptions for each image. It follows the official protocol. The dataset is divided into training, validation and test sets, where the training set contains 34,054 images of 11,003 pedestrians and 68,108 descriptions; the validation set contains 3,078 images of 1,000 pedestrians and 6,156 descriptions; the test set contains 3,074 images of 1,000 pedestrians and 6,148 descriptions.

[0049] All images in ICFG-PEDES come from MSMT17‎, which contains more identities and text descriptions than CHUK-PEDES. ICFG-PEDES contains 54,522 text descriptions and 54,522 images of 4,102 pedestrians, that is, each image has a corresponding text description. The dataset is divided into a training set and a test set, where the training set contains 34,674 image-text pairs of 3,102 pedestrians, and the test set contains 19,848 image-text pairs of the remaining 1,000 pedestrians.

[0050] The images in RSTPReid are also collected from MSMT17, including 20,505 images of 4,101 pedestrians and 41,010 text descriptions. Each person has 5 images captured by 15 different cameras, and each image has 2 corresponding text descriptions. The training set contains 3,701 pedestrians, and the validation set and test set each contain 200 pedestrians.

[0051] Quantitative analysis: As shown in Tables 1, 2, and 3, the experimental results of the proposed algorithm and the existing text-to-image person re-identification algorithm on three widely used benchmark datasets (CUHK-PEDES dataset, ICFG-PEDES dataset, and RSTPReiD dataset). The methods that do not use the CLIP model as a pre-training model are marked as "w / o CLIP" in the pre-training column of the table, and the methods that use the CLIP model as a pre-training model are marked as "w / CLIP". The baseline model uses InfoNCE loss‎ to directly fine-tune CLIP globally.

[0052] Table 1 Comparison of experimental results of the proposed algorithm and other text-to-image person re-identification algorithms on the CUHK-PEDES dataset

[0053] Note: Bold fonts indicate the best results in each column. “-” indicates that the authors did not report SRCT: From the paper A simple and robust correlation filtering method for text-based person search. AXM-Net: From the paper Axm-net: Implicit cross-modal feature alignment for person re-identification. IVT: from the paper See finer, see more: Implicit modality alignment for text-based person retrieval. ASAMN: From the paper Improving inconspicuous attributes modeling for personsearch by language. TransTPS: From the paper Multi-Granularity Matching Transformer for Text-Based Per-son Search. MGCN: From the paper Text-to-Image Person Re-identification Based onMultimodal Graph Convolutional Network. SAL: From the paper Cross-Modal Semantic Alignment Learning for Text-Based Person Search. EESSO from the paper EESSO: Exploiting Extreme and Smooth Signals via Om-ni-frequency learning for Text-based Person Retrieval. CFine: From the paper Clip-driven fine-grained text-image person re-identification. VGSG: From the paper VGSG: Vision-Guided Semantic-Group Network for Text-Based Person Search. CFAM: From the paper UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity. Table 2 Comparison of experimental results of the proposed algorithm with other text-to-image person re-identification algorithms on the ICFG-PEDES dataset

[0054] Table 3 Comparison of experimental results of the proposed algorithm and other text-to-image person re-identification algorithms on the RSTPReid dataset

[0055] The method of the present invention achieved high Rank1 and mAP on three challenging TIReID datasets. In particular, compared with the existing most advanced methods, the model of the present invention achieved the best prediction performance, which illustrates the effectiveness of the model.

[0056] Qualitative analysis: like Figure 2 and Figure 3As shown in the figure, the predicted semantic similarity scores of the proposed algorithm and other CLIP-based text-to-image person re-identification algorithms on these cases are shown, and experiments are conducted on the CUHK-PEDES and ICFG-PEDES datasets. The Image case column in the figure is the input image, the Caption column is the aligned and unaligned text descriptions corresponding to the input image, "√" and "×" respectively indicate whether the image and text description correspond, and the CLIP score column is the normalized CLIP score, which represents the semantic similarity of the image-text pair.

[0057] The results show that the text-to-image person re-identification model that does not introduce structured scene graph knowledge cannot correctly distinguish the semantic similarity between the text description and the given image after two attributes or objects are swapped, which shows that it lacks the ability to capture structured semantics. In contrast, the model of the present invention shows a high sensitivity to changes in fine-grained semantics. The retrieval results show that the model of the present invention can effectively distinguish between aligned and unaligned text descriptions with a large margin, which shows that the model fine-tuned based on scene graph knowledge has certain structured knowledge.

[0058] In addition, this embodiment also discloses a computer device, including a memory and a processor; The memory is used to store a computer program executable on the processor; The processor is used to implement the steps of the above-mentioned pedestrian re-identification method when executing the computer program.

[0059] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the computer device.

[0060] The computer device may be a computing device such as a mobile phone, a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may include, but is not limited to, a processor and a memory. For example, the computer device may also include an input / output device, a network access device, a bus, etc.

[0061] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0062] The memory can be used to store the computer program and / or module, and the processor implements the computer program by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0063] Wherein, if the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0064] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned pedestrian re-identification method are implemented.

[0065] The embodiments of the present invention provide a pedestrian re-identification method, device and medium. The method of the present invention first constructs a two-stage training strategy. In the first stage, the present invention uses prompt learning to optimize learnable prompts to achieve alignment of the downstream data domain with the original training data domain of the CLIP model, thereby promoting domain adaptation. In the second stage, the present invention fine-tunes the CLIP model while further proposing semantic negative sampling and a scene graph knowledge encoder scene graph encoder, using scene graph knowledge as input to enhance the model's ability to acquire structured knowledge in the second stage.

[0066] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A pedestrian re-identification method, characterized in that: The following steps are involved: S1: Collect person re-identification data; S2: Construct scene graph based on person re-identification data; S3: Build a multimodal model architecture for images and texts, including a domain adaptation module and a task adaptation module; S4: Based on prompt learning and scene graph knowledge, the image-text multimodal model architecture is trained to obtain the image-text alignment model; S5: Input the scene graph and text into the image-text alignment model, and output structural knowledge embedding; S6: embed the structural knowledge and combine it with the original text to obtain fusion features; S7: Use fusion features to achieve pedestrian re-identification prediction.

2. The pedestrian re-identification method according to claim 1, characterized in that: In S1, the pedestrian re-identification data includes an image and text, and the image and the text corresponding to the image constitute an image-text pair.

3. The pedestrian re-identification method according to claim 2, characterized in that: In S2, the scene graph includes a triple structure consisting of objects, attributes and relationships.

4. The pedestrian re-identification method according to claim 3, characterized in that: In S3, the domain adaptation module is provided with a text encoder and a visual encoder, and the task adaptation module is provided with a text encoder, a visual encoder, a hard negative sample generator and a scene graph parser; The text encoder is used to process text description information and extract text representation; The visual encoder is used to process pedestrian image data and extract image representation; The scene graph parser is used to parse the text representation into a scene graph; The hard negative sample generator is used to process image-text pairs and scene graphs generated by text representation to generate semantic negative samples; The image-text pair consists of a set of corresponding image representations and text representations.

5. The pedestrian re-identification method according to claim 4, characterized in that: In S4, the image-text multimodal model architecture is trained based on prompt learning and scene graph knowledge. The process is as follows: Call the domain adaptation module to add learnable domain-specific prompt words to the text description information, fix the parameters of the text encoder and visual encoder, process the pedestrian re-identification data through the text encoder and visual encoder respectively, and extract the text representation and image representation; The task adaptation module is called to fix the learned domain-specific prompt words, and the parameters of the text encoder and visual encoder are adjusted during the training process. The scene graph parser and hard negative sample generator are introduced to obtain the image-text alignment model.

6. The pedestrian re-identification method according to claim 5, characterized in that: In S5, structural knowledge embedding is obtained by integrating text and scene graph, and the expression is as follows: ； ； in, represents the triple encoding function, represents triple embedding, represents a triple, , and They represent head entity embedding, relation entity embedding, and tail entity embedding respectively; Represents structural knowledge embedding, Represents multiple transformer layers.

7. The pedestrian re-identification method according to claim 6, characterized in that: In S6, the structural knowledge is embedded and combined with the original text representation to obtain the fusion feature , the process is as follows: ； in, represents the original text representation, Represents a hyperparameter.

8. The pedestrian re-identification method according to claim 7, characterized in that: The loss for training the domain adaptation module is as follows: ； in, represents the domain adaptation training loss, represents the contrastive loss from text to image, Represents image to text comparison file; The loss for training the task adaptation module is as follows: ； in, represents the task adaptive training loss, Indicates ID loss, represents the negative sample learning loss; The expression of the ID loss function is as follows: ； in, represents the values ​​in the underlying target distribution, Indicates category ID prediction logarithm, Represents the total number of instances.

9. A computer device, characterized in that: including memory and processor; The memory is used to store a computer program executable on the processor; The processor is configured to implement the steps of the pedestrian re-identification method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the pedestrian re-identification method according to any one of claims 1 to 8 are implemented.

Images