Interactive pedestrian re-identification method

Abstract

The invention discloses an interactive pedestrian re-identification method, which belongs to the technical field of data processing, and comprises the following steps: S1, collecting a pedestrian picture data set, and extracting a representative picture set by using a retriever according to the pedestrian picture data set and dialogue history; s2, on the basis of the representative picture set, generating a new question by using a questioner; and S3, according to the new question and the dialogue history, generating a final retrieval result. The method effectively solves the problem that the user is difficult to completely and accurately describe due to lack of interaction between the user and a model in current text-based pedestrian re-identification, deeply mines information in user memory, helps the user to provide more discriminative description, solves the problems of redundancy and insufficient representativeness of candidate pictures, and improves the user experience. Therefore, the proposed problem can better induce the user to provide key information with higher discrimination, and the retrieval performance in a real scene is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to an interactive pedestrian re-identification method. Background Art

[0002] The goal of text-based person re-identification (T-ReID) is to retrieve a target person from an image database based on a textual description of their clothing and actions. T-ReID has a wide range of applications, such as tracking criminal suspects from surveillance footage based on eyewitness accounts and finding lost children or elderly people in shopping malls based on descriptions. Current text-based person re-identification methods assume that the user (i.e., the witness) can provide a clear and complete description in one go, which is difficult to achieve in reality. Furthermore, current research in text-based person re-identification focuses primarily on establishing correspondences between fine-grained features in text and images (e.g., descriptions of clothing color and style and the corresponding body parts in the image). Existing methods primarily rely on adding auxiliary supervisory signals. T-ReID models can only passively retrieve information based on user input and lack the ability to interact with the user in real time. Their test datasets and application scenarios implicitly assume that the provided text description is accurate and complete. However, in actual situations, the initial description provided by the user (i.e., witness) is often not as complete and accurate as the annotation in the dataset, and some key features and details may be omitted, resulting in the description being unable to effectively reflect the user's potential memory information.

[0003] Currently, several interactive image and video retrieval technologies exist. Conversational interaction has become a mainstream approach due to its close proximity to human communication and its generalizability in open scenarios. These technologies continuously collect and refine the user's search intent through multiple rounds of dialogue, with the model asking questions and the user answering them. These techniques typically require the selection of candidate images as conditions to generate more targeted questions. Existing interactive retrieval methods primarily focus on general application scenarios, lacking the ability to perceive fine-grained information in images and the domain-specific knowledge required for person re-ID tasks. Summary of the Invention

[0004] In order to solve the above problems, the present invention proposes an interactive pedestrian re-identification method.

[0005] The technical solution of the present invention is: an interactive pedestrian re-identification method comprises the following steps:

[0006] S1. Collect a pedestrian image dataset and use a retriever to extract a representative image set based on the pedestrian image dataset and conversation history.

[0007] S2, based on the representative image set, use the questioner to generate new questions;

[0008] S3. Generate final retrieval results based on new questions and historical conversations.

[0009] Furthermore, S1 includes the following sub-steps:

[0010] S11. Collect pedestrian image dataset and extract coarse-grained descriptions;

[0011] S12, using the coarse-grained description as an initial description, and generating a fine-grained description based on the initial description;

[0012] S13, encoding the conversation history and the images in the pedestrian image dataset, and calculating the similarity between each conversation history and each image after encoding;

[0013] S14, extracting k pictures with the highest similarity as a candidate picture set; where k represents the number of candidate pictures;

[0014] S15, calculating the representativeness score of each candidate image;

[0015] S16. Use the fine-grained description to train a retriever, and use the trained retriever to extract c candidate images with the highest representative scores as a representative image set; where c represents the number of representative images.

[0016] Furthermore, in S13, the conversation history and the image similarity The calculation formula is:

[0017]

[0018] Where, I i represents the i-th picture, represents the t-th round of dialogue history, z t represents the characteristics of the t-th round of dialogue history, f i Represents the features of the i-th picture, f j represents the features of the j-th image, sim(·) represents the cosine similarity function, exp(·) represents the natural exponential power function, and m represents the number of images in the pedestrian image dataset.

[0019] Furthermore, in S15, the representative score w of the candidate image is calculated as follows:

[0020] w=softmax(φ(f c ,z t ));

[0021] In the formula, softmax(·) represents the activation function, f c Represents the characteristics of the candidate image, z t represents the features of the t-th round of dialogue history, and φ(·) represents the filter.

[0022] Furthermore, in S16, candidate images are randomly sampled according to their representative scores to complete the retriever training;

[0023] The expression of the sampling distribution of random sampling is:

[0024]

[0025] Where, represents a representative image set, represents the probability of a candidate image being selected in a single sampling using the representative score, w represents the representative score of the candidate image, S ordered represents the arrangement of representative image sets, represents the possible permutations of all elements in the representative picture set, p(S ordered |w) represents the random order sampling of the candidate images according to their representative scores. ordered probability.

[0026] Furthermore, S2 includes the following sub-steps:

[0027] S21, adding unasked questions in the question set to the conversation history, and determining whether the questions added to the conversation history improve the retrieval ranking of the target task image in the representative image set;

[0028] S22. Improve the search ranking of the target task image based on each question and determine the optimal question;

[0029] S23. Use the optimal question to train the questioner, and use the trained questioner to generate new questions.

[0030] Furthermore, in S22, the optimal problem The expression is:

[0031]

[0032] Where, represents the target question for the t-th round of interaction, represents a set of problems, Indicates the questions that have been asked in the previous t-1 rounds, I gt represents the target task image, T represents the initial description given by the user, A1 represents the answer given by the user to the first question, and A t―1 represents the user's answer to the t-1th question, represents the answer corresponding to the optimal question, and rank(·) represents the retrieval ranking of the target task image.

[0033] Furthermore, in S23, the expression for training the questioner is:

[0034]

[0035] Where, represents the negative log-likelihood loss function, Indicates that the questioner is based on the t-1 round of dialogue history And the corresponding candidate set images Output The probability of represents the optimal problem.

[0036] Furthermore, in S3, the next round of dialogue is generated based on the new question and dialogue history until the dialogue round limit is reached and the final retrieval result is generated.

[0037] The beneficial effects of the present invention are: the present invention effectively solves the problem in current text-based pedestrian re-identification that the lack of interaction between users and models makes it difficult for users to provide complete and accurate descriptions. It can deeply mine the information in the user's memory, help users provide more discriminative descriptions, and solve the problems of redundancy and lack of representativeness of candidate images, so that the questions raised can better induce users to provide more discriminative key information, thereby improving retrieval performance in real scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flowchart of the interactive person re-identification method. DETAILED DESCRIPTION

[0039] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0040] like Figure 1 As shown, the present invention provides an interactive pedestrian re-identification method, comprising the following steps:

[0041] S1. Collect a pedestrian image dataset and use a retriever to extract a representative image set based on the pedestrian image dataset and conversation history.

[0042] S2, based on the representative image set, use the questioner to generate new questions;

[0043] S3. Generate final retrieval results based on new questions and historical conversations.

[0044] Interactive person re-identification (PRI) uses a multi-round question-and-answer system to conduct real-time interaction with the user. It primarily consists of a retriever and a questioner. Similar to traditional text-based PRI models, the retriever encodes the conversation description and person image into a feature space and retrieves matching person images from a database. The questioner asks the user discriminative questions based on the conversation history and candidate images provided by the retriever. After the user responds, the retriever re-searches based on the updated conversation history and then proceeds to the next round of interactive questioning until the target person is accurately identified or the round limit is reached.

[0045] In this embodiment of the present invention, S1 includes the following sub-steps:

[0046] S11. Collect pedestrian image dataset and extract coarse-grained descriptions;

[0047] S12, using the coarse-grained description as an initial description, and generating a fine-grained description based on the initial description;

[0048] S13, encoding the conversation history and the images in the pedestrian image dataset, and calculating the similarity between each conversation history and each image after encoding;

[0049] S14, extracting k pictures with the highest similarity as a candidate picture set; where k represents the number of candidate pictures;

[0050] S15, calculating the representativeness score of each candidate image;

[0051] S16. Use the fine-grained description to train a retriever, and use the trained retriever to extract c candidate images with the highest representative scores as a representative image set; where c represents the number of representative images.

[0052] In an embodiment of the present invention, a pre-trained MLLM model is first used to annotate coarse-grained short descriptions and fine-grained descriptions based on images and prompt words. The initial description provided by the witness is simulated, and the description mainly includes the overall appearance of the person, for example: "A woman wearing a long black coat, a scarf, and carrying a bag." Based on the initial description, a fine-grained detailed description is generated to simulate all the characteristics of the person in the witness's impression and annotate the details of the person, including clothing color, pattern, accessories, shoe type, hair color and length, etc.: "The woman is wearing a long black coat, a gray scarf, and a large handbag in gray and purple. Her shoes are black flats with a leopard print pattern."

[0053] LLM (Large Language Model) is used to remove details not mentioned in the initial description from the fine-grained description, and specific prompts are used to split the supplementary description into non-overlapping sub-descriptions, each of which focuses on a unique attribute while ensuring there is no redundancy or omission.

[0054] To simulate real conversation scenarios, the present invention has designed a variety of question formats. For each sub-description, a question format is randomly selected, and then the LLM is used to generate questions and corresponding answers based on the prompt word and sub-description. Question types include: descriptive questions (50%), true or false questions (40%), and multiple-choice questions (10%). For example, a descriptive question: Please describe the style of her pants? A true or false question: Do her shoes have any special patterns or designs? A multiple-choice question: What color is her hair? A) Medium length; B) Long and brown; C) Short and curly; D) Unsure.

[0055] In the embodiment of the present invention, in S13, the conversation history and the picture similarity The calculation formula is:

[0056]

[0057] Where, I i represents the i-th picture, represents the t-th round of dialogue history, z t represents the characteristics of the t-th round of dialogue history, f i Represents the features of the i-th picture, f j represents the features of the j-th image, sim(·) represents the cosine similarity function, exp(·) represents the natural exponential power function, and m represents the number of images in the pedestrian image dataset.

[0058] In this embodiment of the present invention, in S15, the formula for calculating the representative score w of the candidate image is:

[0059] w=softmax(φ(f c ,z t ));

[0060] In the formula, softmax(·) represents the activation function, f c Represents the characteristics of the candidate image, z t represents the features of the t-th round of dialogue history, and φ(·) represents the filter.

[0061] In the embodiment of the present invention, in S16, candidate images are randomly sampled according to their representative scores to complete the retriever training;

[0062] The expression of the sampling distribution of random sampling is:

[0063]

[0064] Where, represents a representative image set, represents the probability of a candidate image being selected in a single sampling using the representative score, w represents the representative score of the candidate image, S ordered represents the arrangement of representative image sets, represents the possible permutations of all elements in the representative picture set, p(S ordered |w) represents the random order sampling of the candidate images according to their representative scores. ordered probability.

[0065] In this embodiment of the present invention, S2 includes the following sub-steps:

[0066] S21, adding unasked questions in the question set to the conversation history, and determining whether the questions added to the conversation history improve the retrieval ranking of the target task image in the representative image set;

[0067] S22. Improve the search ranking of the target task image based on each question and determine the optimal question;

[0068] S23. Use the optimal question to train the questioner, and use the trained questioner to generate new questions.

[0069] In the embodiment of the present invention, in S22, the optimal problem The expression is:

[0070]

[0071] Where, represents the target question for the t-th round of interaction, represents a set of problems, Indicates the questions that have been asked in the previous t-1 rounds, I gt represents the target task image, T represents the initial description given by the user, A1 represents the answer given by the user to the first question, and A t―1 represents the user's answer to the t-1th question, represents the answer corresponding to the optimal question, and rank(·) represents the retrieval ranking of the target task image.

[0072] In the embodiment of the present invention, in S23, the expression for training the questioner is:

[0073]

[0074] Where, represents the negative log-likelihood loss function, Indicates that the questioner is based on the t-1 round of dialogue history And the corresponding candidate set images Output The probability of represents the optimal problem.

[0075] In the embodiment of the present invention, in S3, the next round of dialogue is generated based on the new question and the dialogue history until the dialogue round limit is reached, and the final retrieval result is generated.

[0076] In this embodiment of the present invention, to evaluate the effectiveness of the present invention, a portion of the labeled data was divided into a test set and interactive person re-ID was performed. During the test, LLM was used to simulate users answering questions. To demonstrate the superiority of the present invention, two performance comparisons were conducted:

[0077] Comparison with other interactive retrieval schemes: The present invention is compared with three other interactive techniques in the scenario of incomplete initial description, namely, the heuristic questioner (SimIRV), the conversational interactive retrieval method (ChatIR), and the plug-and-play interactive method (PlugIR). The recall rate (R@k), mean average precision (mAP), and logarithmic best ranking integral (BRI) are used. The experimental results are shown in Table 1.

[0078] Table 1

[0079]

[0080] As can be seen from Table 1, compared with other methods, the present invention has a significant improvement in conversations of different numbers of rounds, and can accurately perceive key information and raise targeted questions in practical applications.

[0081] Integration with Text-Based Person Re-ID Methods: This paper applies the proposed method to a traditional text-based person re-ID dataset, using manually annotated data as initial descriptions, conducting interactive questioning, and performing weighted fusion of the retrieval results. This method is integrated into IRRA, RDE and compared with CFINE, APTM, RaSa, and AUL. The experimental results are shown in Table 2.

[0082] Table 2

[0083]

[0084]

[0085] As can be seen from Table 2, interactive questioning can supplement the missing information in manual annotation and significantly improve the retrieval performance of the existing T-ReID method. This shows the advantages of interactive person re-identification in practical application scenarios. Through continuous iteration and refinement of descriptions, the retrieval effect is effectively improved.

[0086] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. An interactive person re-identification method, characterized in that: The following steps are involved: S1. Collect a pedestrian image dataset and use a retriever to extract a representative image set based on the pedestrian image dataset and conversation history. S2, based on the representative image set, use the questioner to generate new questions; S3. Generate the final retrieval results based on the new question and conversation history.

2. The interactive person re-identification method according to claim 1, characterized in that The S1 includes the following sub-steps: S11. Collect pedestrian image dataset and extract coarse-grained descriptions; S12, using the coarse-grained description as an initial description, and generating a fine-grained description based on the initial description; S13, encoding the conversation history and the images in the pedestrian image dataset, and calculating the similarity between each conversation history and each image after encoding; S14, extracting k pictures with the highest similarity as a candidate picture set; where k represents the number of candidate pictures; S15. Calculate the representativeness score of each candidate image; S16. Use the fine-grained description to train a retriever, and use the trained retriever to extract c candidate images with the highest representative scores as a representative image set; where c represents the number of representative images.

3. The interactive person re-identification method according to claim 2, characterized in that In S13, the conversation history and the image similarity The calculation formula is: Where, I i represents the i-th picture, represents the t-th round of dialogue history, z t represents the characteristics of the t-th round of dialogue history, f i Represents the features of the i-th picture, f j represents the features of the j-th image, sim(·) represents the cosine similarity function, exp(·) represents the natural exponential power function, and m represents the number of images in the pedestrian image dataset.

4. The interactive person re-identification method according to claim 2, characterized in that In S15, the representative score w of the candidate image is calculated as follows: w=softmax(φ(f c ,z t )); In the formula, softmax(·) represents the activation function, f c Represents the characteristics of the candidate image, z t represents the features of the t-th round of dialogue history, and φ(·) represents the filter.

5. The interactive person re-identification method according to claim 2, characterized in that In S16, the candidate images are randomly sampled according to their representative scores to complete the retriever training; The expression of the sampling distribution of the random sampling is: Where, represents a representative image set, represents the probability of a candidate image being selected in a single sampling using the representative score, w represents the representative score of the candidate image, S ordered represents the arrangement of representative image sets, represents the possible permutations of all elements in the representative picture set, p(S ordered |w) represents the random order sampling of the candidate images according to their representative scores. ordered probability.

6. The interactive person re-identification method according to claim 1, characterized in that The S2 includes the following sub-steps: S21. Add unasked questions in the question set to the conversation history, and determine whether the questions added to the conversation history improve the retrieval ranking of the target task image in the representative image set; S22. Improve the search ranking of the target task image based on each question and determine the optimal question; S23. Use the optimal question to train the questioner, and use the trained questioner to generate new questions.

7. The interactive person re-identification method according to claim 6, characterized in that: In S22, the optimal problem The expression is: Where, represents the target question for the t-th round of interaction, represents a set of problems, Indicates the questions that have been asked in the previous t-1 rounds, I gt represents the target task image, T represents the initial description given by the user, A1 represents the answer given by the user to the first question, and A t―1 represents the user's answer to the t-1th question, represents the answer corresponding to the optimal question, and rank(·) represents the retrieval ranking of the target task image.

8. The interactive person re-identification method according to claim 6, characterized in that In S23, the expression for training the questioner is: Where, represents the negative log-likelihood loss function, Indicates that the questioner is based on the t-1 round of dialogue history and the corresponding candidate set images Output The probability of represents the optimal problem.

9. The interactive person re-identification method according to claim 1, characterized in that In S3, the next round of dialogue is generated based on the new question and the dialogue history until the dialogue round limit is reached and the final retrieval result is generated.

Images