SAM-based single-lens pedestrian re-identification method
Through the SAM-based single-lens pedestrian re-identification method, the segmentation ability of the SAM model is used to obtain the pedestrian mask and match it, which solves the problems of high hardware equipment requirements and low recognition number of traditional methods, and realizes accurate identification and tracking of pedestrians.
Patent Information
- Application Number
- CN202411647614.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-04-25
Smart Images
Figure CN119888778A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent video analysis, and in particular to a single-shot pedestrian re-identification method based on SAM. Background Art
[0002] With the rapid development of economy and the mobility of population, especially in big cities, there are more densely populated areas, which are sensitive areas where security problems frequently occur. Therefore, these sensitive areas have also become hidden worries in urban construction. In order to prevent various thefts, robberies and fights in key sensitive areas, the relevant departments have installed a large number of cameras at relevant locations for real-time monitoring of these areas. However, due to the long distance or clarity problems, it is not possible to accurately locate and track the suspects after an emergency occurs, and the expected effect cannot be achieved.
[0003] Traditional pedestrian re-ID methods are relatively simple, all of which involve importing video images from multiple lenses. They have high requirements for hardware equipment and the number of recognitions is also low.
[0004] Therefore, how to provide a single-shot pedestrian re-identification method based on SAM to solve the problems existing in the existing technology is of great significance for its application. Summary of the invention
[0005] In view of this, the purpose of this application is to provide a single-shot pedestrian re-identification method based on SAM to solve the problem that traditional pedestrian re-id methods are relatively simple, all of which are to import video images from multiple shots, have high requirements for hardware equipment, and have a low number of recognitions.
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] A single-shot pedestrian re-identification method based on SAM includes the following steps:
[0008] S1: Use human-computer interaction to obtain the segmentation mask of pedestrians in the reference image based on the SAM model;
[0009] S2: Use the above image mask to perform scale-adaptive fine-tuning training on SAM;
[0010] S3: Based on the image and mask acquisition in S1, obtain the pixel coordinates of the positioning information map and the anchor point of interest;
[0011] S4: Combine the pixel coordinates of the anchor point of interest and the fine-tuned SAM to obtain the mask of the target pedestrian in other images;
[0012] S5: Matching is performed based on the above mask and target tracking results to obtain the ID of the pedestrian after re-identification.
[0013] Preferably, the specific steps of S1 are as follows: first, an input image containing the target pedestrian is prepared, and the point prompt mode is entered through parameter setting. Subsequently, the point prompt information is input into the SAM model. The model will accurately predict the target pedestrian based on the input point information and output the corresponding mask.
[0014] Preferably, the SAM model stands for Segment Anything Model, and the SAM model can achieve accurate positioning and segmentation of target objects in an image through a point prompt mode.
[0015] Preferably, the specific steps of S2 are as follows: a fine-tuning training method is introduced to obtain SAM-F, SAM-F first uses position prior information to output masks of three different scales, respectively denoted as M1, M2 and M3, on this basis, two learnable mask weights are designed, denoted as w1 and w2, and the final mask output is calculated by weighted summation.
[0016] Preferably, the mask calculation formula in S2 is:
[0017] M=w1*M1+w2*M2+(1-w1-w2)*M3;
[0018] Among them, w1 and w2 are both initialized to 1 / 3.
[0019] Preferably, the specific steps of S3 are as follows:
[0020] S3.1: Based on the reference image I given by the user R and mask M R , obtain a positioning information map in SAM to guide the test image I T The position of the target pedestrian in the image is extracted using the SAM image encoder Enc. R and I T The visual features of F T =Enc(I T ), F R =Enc(I R )
[0021] Among them, F T ,
[0022] S3.2: Using reference masks The features of foreground pixels in the visual concept are cropped from FR to obtain a set of n local features:
[0023] in Represents space vector multiplication;
[0024] S3.3: By calculation and the test image features F T The cosine similarity between them is used to calculate n confidence maps for each foreground pixel i. The formula is as follows:
[0025] in F T and has been pixel-wise L2 normalized. Each S i Represents the distribution probability of different local parts of the target in the test image, such as the head, torso or legs of a pedestrian;
[0026] S3.4: Based on S3.3, average pooling is used to aggregate all n local features to obtain the overall confidence map of the target object:
[0027]
[0028] By combining the confidence of each foreground pixel, S can take into account the visual appearance of different object parts and obtain a relatively comprehensive position estimate;
[0029] S3.4: Select the two points with the highest and lowest confidence values in S, denoted as P h and P l , P h represents the most likely center position of the target object, and then P l It represents the most likely background position, P h and P l are treated as positive and negative dot cues and serve as the input to the dot cue encoder.
[0030] Preferably, the specific steps of S4 are as follows: the positive and negative points obtained in S3 are used as point prompt marks of the SAM-F encoder. In this way, SAM-F tends to segment the adjacent areas near the positive points and discard the surrounding areas of the negative points on the image, thereby obtaining the segmentation mask of the target pedestrian in the test image.
[0031] Preferably, the specific steps of S5 are as follows: by calculating the intersection-and-union ratio of the minimum circumscribed rectangle of the segmentation mask obtained in S4 and the target tracking detection frame, if it exceeds a set threshold, it is determined to be a target ID;
[0032] The calculation formula is:
[0033] Compared with the prior art, the present invention has the following beneficial effects:
[0034] The present invention is based on the single-shot pedestrian re-identification method of Segment Anything Model (SAM), which utilizes the advantages of the SAM model in image segmentation to perform pixel-level segmentation on pedestrians, thereby extracting pedestrian features and realizing accurate identification and tracking of pedestrians. The SAM model is a powerful segmentation model that can obtain the mask of the target object through the point prompt mode and accurately segment the pedestrians in the image. By utilizing the powerful segmentation ability of the SAM model, accurate identification and tracking of pedestrians in the image can be realized, providing favorable support for research and application in related fields.
[0035] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application so that it can be implemented in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings as follows.
[0036] Based on the detailed description of the specific embodiments of the present application in combination with the accompanying drawings below, those skilled in the art will become more aware of the above and other objects, advantages and features of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale.
[0038] Figure 1 It is the overall flow chart of the present invention;
[0039] Figure 2 It is a flow chart of SAM mask segmentation under the S1 midpoint recommendation model of the present invention;
[0040] Figure 3 This is a flow chart of SAM-F training in S2 of the present invention;
[0041] Figure 4 The flowchart for obtaining the SAM-F anchor point of interest in S3 of the present invention;
[0042] Figure 5 This is a test picture segmentation flow chart in S4 of the present invention;
[0043] Figure 6This is a matching flow chart in S5 of the present invention. DETAILED DESCRIPTION
[0044] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiment of the present application, rather than all of the embodiments. In the following description, specific details such as specific configuration and components are provided only to help fully understand the embodiments of the present application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. In addition, for clarity and brevity, the description of known functions and structures is omitted in the embodiment.
[0045] In addition, the present application may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplicity and clarity, and does not in itself indicate the relationship between the various embodiments and / or settings discussed.
[0046] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" in this article describes another type of association object relationship, indicating that there can be two relationships. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship.
[0047] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusions.
[0048] See also Figure 1-6 The present invention provides a technical solution of a single-shot pedestrian re-identification method based on SAM, which includes the following steps:
[0049] S1: Use human-computer interaction to obtain the segmentation mask of pedestrians in the reference image based on the SAM model. The specific steps of S1 are as follows: First, prepare an input image containing the target pedestrian and enter the point prompt mode through parameter settings. Then, input the point prompt information into the SAM model. The model will accurately predict the target pedestrian based on the input point information and output the corresponding mask.
[0050] The full name of SAM model is Segment Anything Model. The SAM model can accurately locate and segment the target objects in the image through the point prompt mode.
[0051] S2: Use the above image mask to perform scale-adaptive fine-tuning training on SAM;
[0052] The specific steps of S2 are as follows: a fine-tuning training method is introduced to obtain SAM-F. SAM-F first uses the position prior information to output masks of three different scales, denoted as M1, M2 and M3 respectively. On this basis, two learnable mask weights are designed, denoted as w1 and w2, and the final mask output is calculated by weighted summation.
[0053] The mask calculation formula in S2 is:
[0054] M=w1*M1+w2*M2+(1-w1-w2)*M3;
[0055] Among them, w1 and w2 are both initialized to 1 / 3.
[0056] Based on the image and mask acquisition in S1, the pixel coordinates of the positioning information map and the anchor point of interest are obtained; the specific steps of S3 are as follows:
[0057] S3.1: Based on the reference image I given by the user R and mask M R , obtain a positioning information map in SAM to guide the test image I T The position of the target pedestrian in the image is extracted using the SAM image encoder Enc. R and I T The visual features of F T =Enc(I T ), F R =Enc(I R );
[0058] Among them, F T ,
[0059] S3.2: Using reference masks The features of foreground pixels in the visual concept are cropped from FR to obtain a set of n local features:
[0060] in Represents space vector multiplication;
[0061] S3.3: By calculation and the test image features F TThe cosine similarity between them is used to calculate n confidence maps for each foreground pixel i. The formula is as follows:
[0062] in F T and has been pixel-wise L2 normalized. Each S i Represents the distribution probability of different local parts of the target in the test image, such as the head, torso or legs of a pedestrian.
[0063] S3.4: Based on S3.3, average pooling is used to aggregate all n local features to obtain the overall confidence map of the target object:
[0064]
[0065] By combining the confidence of each foreground pixel, S can take into account the visual appearance of different object parts and obtain a relatively comprehensive position estimate;
[0066] S3.5: Select the two points with the highest and lowest confidence values in S, denoted as P h and P h , P h represents the most likely center position of the target object, and then P l It represents the most likely background position, P h and P l are treated as positive and negative dot cues and serve as the input to the dot cue encoder.
[0067] S4: Combine the pixel coordinates of the anchor point of interest and the fine-tuned SAM to obtain the mask of the target pedestrian in other pictures; the specific steps of S4 are as follows: the positive and negative points obtained in S3 are used as point prompt marks of the SAM-F encoder. In this way, SAM-F will tend to segment the adjacent areas near the positive points and discard the surrounding areas of the negative points on the image, thereby obtaining the segmentation mask of the target pedestrian in the test picture.
[0068] S5: Based on the above mask and target tracking results, matching is performed to obtain the ID of the pedestrian after re-identification. The specific steps of S5 are as follows: by calculating the intersection and union ratio of the minimum circumscribed rectangle of the segmentation mask obtained in S4 and the target tracking detection frame, if it exceeds the set threshold, it is determined to be the target ID;
[0069] The calculation formula is:
[0070] The above description is only the preferred embodiment of the present invention, which does not limit the protection scope of the present invention. For those skilled in the art, the present invention can be modified and varied in various ways. Within the spirit and principle of the present invention, any change, modification, replacement, integration and parameter change of these embodiments by conventional substitution or capable of achieving the same function without departing from the principle and spirit of the present invention shall fall within the protection scope of the present invention.
Claims
1. A single-shot pedestrian re-identification method based on SAM, characterized by: The following steps are involved: S1: Use human-computer interaction to obtain the segmentation mask of pedestrians in the reference image based on the SAM model; S2: Use the above image mask to perform scale-adaptive fine-tuning training on SAM; S3: Based on the image and mask acquisition in S1, obtain the pixel coordinates of the positioning information map and the anchor point of interest; S4: Combine the pixel coordinates of the anchor point of interest and the fine-tuned SAM to obtain the mask of the target pedestrian in other images; S5: Matching is performed based on the above mask and target tracking results to obtain the ID of the pedestrian after re-identification.
2. The single-shot pedestrian re-identification method based on SAM as claimed in claim 1, characterized in that: The specific steps of S1 are as follows: first, prepare an input image containing the target pedestrian, and enter the point prompt mode through parameter settings. Then, input the point prompt information into the SAM model. The model will accurately predict the target pedestrian based on the input point information and output the corresponding mask.
3. The single-shot pedestrian re-identification method based on SAM as claimed in claim 2, characterized in that: The SAM model stands for Segment Anything Model. The SAM model can accurately locate and segment the target object in the image through a point prompt mode.
4. The single-shot pedestrian re-identification method based on SAM as claimed in claim 3, characterized in that: The specific steps of S2 are as follows: a fine-tuning training method is introduced to obtain SAM-F. SAM-F first uses position prior information to output masks of three different scales, denoted as M1, M2 and M3 respectively. On this basis, two learnable mask weights are designed, denoted as w1 and w2, and the final mask output is calculated by weighted summation.
5. The single-shot pedestrian re-identification method based on SAM as claimed in claim 4, characterized in that: The mask calculation formula in S2 is: M=W1*M1+W2*M2+(1-W1-W2)*M3; Among them, w1 and w2 are both initialized to 1 / 3.
6. The single-shot pedestrian re-identification method based on SAM as claimed in claim 5, characterized in that: The specific steps of S3 are as follows: S3.1: Based on the reference image I given by the user R and mask M R , obtain a positioning information map in SAM to guide the test image I T The position of the target pedestrian in the image is extracted using the SAM image encoder Enc. R and I T The visual features of F T =Enc(I T ), F R =Enc(I R ); Among them, F T , S3.2: Using reference masks The features of foreground pixels in the visual concept are cropped from FR to obtain a set of n local features: in Represents space vector multiplication; S3.3: By calculation and the test image features F T The cosine similarity between them is used to calculate n confidence maps for each foreground pixel i. The formula is as follows: in F T and has been pixel-wise L2 normalized. Each S i Represents the distribution probability of different local parts of the target in the test image, such as the head, torso or legs of a pedestrian; S3.4: Based on S3.3, average pooling is used to aggregate all n local features to obtain the overall confidence map of the target object: By combining the confidence of each foreground pixel, S can take into account the visual appearance of different object parts and obtain a relatively comprehensive position estimate; S3.5: Select the two points with the highest and lowest confidence values in S, denoted as P h and P l , P h represents the most likely center position of the target object, and then P l It represents the most likely background position, P h and P l are treated as positive and negative dot cues and serve as the input to the dot cue encoder.
7. The single-shot pedestrian re-identification method based on SAM as claimed in claim 6, characterized in that: The specific steps of S4 are as follows: the positive and negative points obtained in S3 are used as point prompt marks of the SAM-F encoder. In this way, SAM-F tends to segment the adjacent areas near the positive points and discard the surrounding areas of the negative points on the image, thereby obtaining the segmentation mask of the target pedestrian in the test image.
8. The single-shot pedestrian re-identification method based on SAM as claimed in claim 7, characterized in that: The specific steps of S5 are as follows: by calculating the intersection-and-union ratio of the minimum circumscribed rectangle of the segmentation mask obtained in S4 and the target tracking detection frame, if it exceeds a set threshold, it is determined to be a target ID; The calculation formula is: