Automatic method for updating feature database of pedestrian re-identification
By adopting the pedestrian re-identification feature library update automation method in pedestrian re-identification technology, the problem of pedestrian ID not unique in multi-camera environment is solved, and the stability and rapid identification and tracking capabilities of pedestrian feature library are realized, which is suitable for fugitive tracking and theft Detection and other applications.
Patent Information
- Application Number
- CN202210382096.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-04-12
AI Technical Summary
Traditional pedestrian re-identification technology has ID jump problems in the case of multiple cameras and multiple pedestrians, and the missed tracking or missing tracking is serious, resulting in the pedestrian's ID not unique among multiple videos.
The pedestrian re-identification feature library update automation method is used to obtain pedestrian characteristics under multiple cameras, deduplication and comparison, and use maximum Kalman filter tracking and IOU comparison calculation to establish and update the pedestrian feature library to ensure that each pedestrian obtains Unique ID.
It realizes the stability of the pedestrian feature library in a multi-camera environment, and can quickly identify and track the same pedestrians, and can accurately track even if pedestrians leave and then enter. It is suitable for fugitive tracking and theft detection applications.
Smart Images

Figure CN114863475B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of pedestrian re-identification, and in particular to an automatic method for updating a pedestrian re-identification feature library. Background Art
[0002] Person re-identification technology refers to the technology of finding the same person in other cameras given a person in front of a camera. This technology is used in areas such as fugitive tracking and unmanned supermarkets.
[0003] The current field of pedestrian re-identification technology generally uses two methods: IOU and pedestrian feature extraction. However, the traditional IOU method is only applicable to single-camera tracking, and pedestrian feature extraction has not achieved reliable results, often resulting in mistracking or missed tracking. This leads to a more serious problem of pedestrian ID jitter in the case of multiple cameras and multiple pedestrians, that is, the pedestrian ID is not unique in multiple videos. Summary of the invention
[0004] The purpose of the present invention is to more accurately identify pedestrians in the video captured by the camera, and then quickly identify and track the pedestrians, and to provide an automated method for updating the pedestrian re-identification feature library.
[0005] In order to achieve the above-mentioned object of the invention, the embodiment of the present invention provides the following technical solutions:
[0006] The method for automatically updating a pedestrian re-identification feature library includes the following steps:
[0007] Step S1: Obtain the current frame images from all cameras and identify the pedestrian images in the current frame images; input all the identified pedestrian images into the feature extraction network to extract pedestrian features from the pedestrian images;
[0008] Step S2: When extracting pedestrian features, determine whether the current frame image meets the maximum Kalman filter tracking frame number. If not, perform Kalman filter tracking on the pedestrian features of the current frame, and establish or update a pedestrian database for each camera; if satisfied, establish a pedestrian feature library after comparing and removing duplicates based on the pedestrian databases of each camera; the maximum Kalman filter tracking frame number is an integer greater than 1.
[0009] Furthermore, the step of acquiring the current frame image from all cameras and identifying the pedestrian image in the current frame image includes:
[0010] Get the current frame image V(t)={v 1 (t),v 2 (t),...,v M (t)}, where v1 (t) represents the current frame image under the first camera, v M (t) represents the current frame image under the Mth camera, t represents the time of the current frame, and t is an integer greater than or equal to 1;
[0011] According to the current frame image V(t) under M cameras, pedestrians in the current frame image are identified, and all identified pedestrians are cut out to form a pedestrian image P(t) = {p 1 (t),p 2 (t),...,p M (t)}, where p 1 (t) represents the pedestrian image in the current frame image under the first camera, p M (t) represents the pedestrian image in the current frame image under the Mth camera, and the pedestrian image is presented in the form of a region frame.
[0012] Furthermore, when extracting pedestrian features, determining whether the current frame image meets the maximum Kalman filter tracking frame number, if not, performing Kalman filter tracking on the pedestrian features of the current frame, and establishing or updating the pedestrian database for each camera, includes:
[0013] If the current frame t is less than or equal to the maximum Kalman filter tracking frame number T, the current frame t does not meet the maximum Kalman filter tracking frame number, and the Kalman filter tracking is performed on the area frame of the pedestrian image P(t) in the current frame image;
[0014] Determine whether the current frame t is equal to 1. If the current frame t is equal to 1, establish a corresponding pedestrian database for each camera;
[0015] If the current frame t is greater than 1 and less than or equal to the maximum Kalman filter tracking frame number T, the corresponding pedestrian database under each camera is updated.
[0016] Furthermore, the step of performing Kalman filtering tracking on the area frame of the pedestrian image P(t) in the current frame image includes:
[0017] If the current frame t is equal to 1, a pedestrian database is established according to the region frame of the pedestrian image P(t), and the pedestrian database includes the region frame of the pedestrian image P(t) and the pedestrian features corresponding to the region frame;
[0018] If the current frame is greater than 1 and less than or equal to the maximum Kalman filter tracking frame number T, the Kalman filter tracking infers the predicted position of the area box in the current frame pedestrian image P(t) based on the area box of the previous frame pedestrian image P(t-1), and compares the actual area box of the current frame pedestrian image P(t) with the inferred area box predicted position.
[0019] Furthermore, if the above conditions are met, the step of establishing a pedestrian feature library after comparing and removing duplicates based on the pedestrian databases of each camera includes:
[0020] If the current frame t is greater than the maximum Kalman filter tracking frame number T, then the current frame t satisfies the maximum Kalman tracking frame number, and at the same time, it is determined whether the current frame t is equal to T+1;
[0021] If the current frame t is equal to T+1, the pedestrian database of each camera is compared and deduplicated to establish a unified pedestrian feature library, and the same pedestrian features in different pedestrian databases are assigned unique IDs in the pedestrian feature library;
[0022] If the current frame t is greater than T+1, the pedestrian feature extracted from the current frame t is matched with the pedestrian feature library. If the match is successful, the pedestrian feature is added to the pedestrian feature library and a corresponding unique ID is assigned to it to update the pedestrian feature library;
[0023] If the match is unsuccessful, an IOU comparison calculation is performed on the pedestrian feature. If the pedestrian feature is successfully matched with the pedestrian feature library after the IOU comparison calculation, the pedestrian feature is added to the pedestrian feature library and a corresponding unique ID is assigned to it. If the pedestrian feature is still not successfully matched with the pedestrian feature library after the IOU comparison calculation, the pedestrian feature is added to the pedestrian feature library and a new unique ID is assigned to it to update the pedestrian feature library.
[0024] Furthermore, while updating the pedestrian feature library, the pedestrian database under the corresponding camera is also updated.
[0025] Furthermore, if the matching is unsuccessful, the pedestrian feature is subjected to IOU comparison calculation, and if the pedestrian feature is successfully matched with the pedestrian feature library after the IOU comparison calculation, the pedestrian feature is added to the pedestrian feature library and a corresponding unique ID is assigned to it, including:
[0026] A function is fitted to the data of the current frame, and the region box i where the pedestrian may stay in the current frame is predicted by the function. The predicted region box i is used to perform an intersection operation with the actual obtained pedestrian image region box j:
[0027]
[0028] Where S ij交 represents the intersection of the predicted region box i and the actually acquired pedestrian image region box j, S ij并 Represents the union of the predicted region box i and the actual obtained pedestrian image region box j, IOU ij It represents the ratio of intersection to union. If the overlap area of two area boxes is larger, the IOU is ij The closer it is to 1, the closer it is to 0;
[0029] According to IOU ij The cosine distance D is calculated:
[0030]
[0031] in represents the cosine similarity between region box i and region box j, and D represents the cosine distance between region box i and region box j;
[0032] The cosine distance D is flipped: D`=1-D;
[0033] The distance threshold △D` is pre-set by the Hungarian algorithm. When D`≤△D`, the pedestrian feature successfully matches the pedestrian feature library. The pedestrian feature is then added to the pedestrian feature library and assigned a corresponding unique ID.
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] The present invention obtains pedestrian features in multiple frame videos from multiple cameras, removes duplicates and compares them to obtain a stable pedestrian feature library, which is suitable for applications such as fugitive tracking by public investigative agencies and theft detection in unmanned supermarkets.
[0036] This solution combines all the videos collected by the cameras and obtains a stable pedestrian feature library that stores the same pedestrian features. Even if the pedestrian leaves and enters again, it can be quickly identified and tracked. Therefore, this solution can not only more accurately identify pedestrians in the videos collected by the cameras, but also quickly identify and track the pedestrians afterwards. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0038] Figure 1 This is a flow chart of the pedestrian re-identification method of the present invention;
[0039] Figure 2 A schematic diagram of a pedestrian image area frame under multiple cameras according to an embodiment of the present invention. DETAILED DESCRIPTION
[0040] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.
[0041] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance, or implying any such actual relationship or order between these entities or operations.
[0042] Example:
[0043] The present invention is achieved through the following technical solutions: Figure 1 As shown, the automatic method for updating the pedestrian re-identification feature library includes the following steps:
[0044] Step S1: Obtain the current frame images from all cameras and identify the pedestrian images in the current frame images; input all identified pedestrian images into the feature extraction network to extract pedestrian features from the pedestrian images.
[0045] Assume that there are 8 cameras in total, that is, M=8, and obtain 10 identical continuous frame images from the 8 cameras, that is, t=10, such as the 10 continuous frame images of the first camera starting from 0 o'clock, the 10 continuous frame images of the second camera starting from 0 o'clock, and so on, the 10 continuous frame images of the eighth camera starting from 0 o'clock.
[0046] The 10 consecutive frames of images under the first camera are v 1 (t)={v 1 (1),v 1 (2),...,v 1 (10)}, where v 1 (1) represents the first frame image under the first camera, v 1 (10) represents the 10th frame image under the first camera. For example, the 10 consecutive frame images under the eighth camera are v 8 (t)={v 8 (1),v8 (2),...,v 8 (10)}, where v 8 (1) represents the first frame image under the 8th camera, v 8 (10) represents the 10th frame image under the 8th camera.
[0047] Then the current frame image obtained from M cameras is V(t)={v 1 (t),...,v i (t),...,v M (t)}, where v 1 (t) represents the current frame image under the first camera, v i (t) represents the current frame image under the i-th camera, v M (t) represents the current frame image under the Mth camera, t represents the time of the current frame, and t is an integer greater than or equal to 1.
[0048] Then, pedestrian images are recognized for each frame image under all cameras. For example, pedestrians in the current frame image V(t) under M cameras are recognized, and all recognized pedestrians are cut out to form a pedestrian image P(t) = {p 1 (t),...,p i (t),...,p M (t)}, where p 1 (t) represents the pedestrian image in the current frame image under the first camera, p i (t) represents the pedestrian image in the current frame image under the i-th camera, p M (t) represents the pedestrian image in the current frame image under the Mth camera (also called the pedestrian image in the current frame under the Mth camera), and the pedestrian image is presented in the form of a region frame. And the pedestrian image p in the current frame under each camera i (t) may contain one or more pedestrians, or no pedestrians.
[0049] Then the pedestrian image P(t) is input into the feature extraction network to obtain the pedestrian features C(t)={c 1 (t),...,c i (t),...,c M (t)}, where c i (t) represents the pedestrian image p of the current frame under the i-th camera i (t) The corresponding pedestrian features. If the current frame pedestrian image p i (t) contains one or more pedestrians, then the pedestrian feature c i (t) also corresponds to one or more pedestrian features; if the current frame pedestrian image p iIf there are no pedestrians in (t), then the pedestrian feature c i There are also no pedestrian features in (t).
[0050] Step S2: When extracting pedestrian features, determine whether the current frame image meets the maximum Kalman filter tracking frames. If not, perform Kalman filter tracking on the pedestrian features of the current frame, and establish or update the pedestrian database for each camera; if so, establish a pedestrian feature library after comparing and removing duplicates according to the pedestrian databases of each camera; the maximum Kalman filter tracking frames is an integer greater than 1.
[0051] The maximum Kalman filter tracking frames T is preset to form stable pedestrian features for the current frame t within the maximum Kalman filter tracking frames T.
[0052] When extracting pedestrian features C(t) from the current frame pedestrian image P(t), determine the relationship between the current frame t and the maximum Kalman filter tracking frames T. There are cases where the current frame does not meet the maximum Kalman filter tracking frames and cases where the current frame meets the maximum Kalman filter tracking frames, which are described separately below.
[0053] (1) When not meeting:
[0054] If t ≤ T, it means that the current frame t does not meet the maximum Kalman filter tracking frames T, then perform Kalman filter tracking on the bounding box of the current frame pedestrian image P(t).
[0055] When it has been determined that t ≤ T and t = 1, establish corresponding pedestrian databases for each camera. For example, there are 8 pedestrian databases corresponding to 8 cameras. And when performing Kalman filter tracking on the bounding box of the current frame pedestrian image P(t), since it is the first frame, directly establish a pedestrian database according to the bounding box of the pedestrian image P(t). The pedestrian database includes the bounding box of the pedestrian image P(t) and the pedestrian features corresponding to the bounding box.
[0056] If 1 < t ≤ T, update the corresponding pedestrian databases under each camera. And when performing Kalman filter tracking on the bounding box of the current frame pedestrian image P(t), predict the predicted position of the bounding box in the current frame pedestrian image P(t) based on the bounding box of the previous frame pedestrian image P(t - 1), and at the same time compare the actual bounding box of the current frame pedestrian image P(t) with the predicted position of the predicted bounding box.
[0057] It is easy to understand that the established pedestrian database contains the bounding box of the pedestrian image, pedestrian features, and unique ID.
[0058] When meeting:
[0059] If t>T, it means that the current frame t meets the maximum Kalman filter tracking frame number T, and then the pedestrian feature library is established after comparison and deduplication based on the pedestrian database of each camera.
[0060] When t>T is determined, if t=T+1, the pedestrian database of each camera is compared and duplicated to establish a unified pedestrian feature database. Assume that the area frame of the pedestrian image P(t) of the current frame of 8 cameras is as follows: Figure 2 As shown in the figure, it can be seen that the area frames of adjacent cameras will have overlapping parts. For example, the overlapping part of the area frame of the first camera and the area frame of the second camera is area a. Therefore, a comparison and deduplication operation is performed to remove the overlapping area frame of the first camera or the overlapping part of the area frame of the second camera, and then the pedestrian databases corresponding to the eight cameras are unified into a pedestrian feature library. And a unique ID is given to the same pedestrian features in different pedestrian data in the pedestrian feature library, because as the number of frames progresses, a pedestrian may enter the shooting area of another camera from the shooting area of this camera, so in the case of multiple frames, the pedestrian feature will exist in one pedestrian database, and the pedestrian feature may also exist in another pedestrian database. Therefore, this is the process of comparison and deduplication.
[0061] It is easy to understand that the established pedestrian feature library contains the area frame of the pedestrian image, the pedestrian features, and the unique ID.
[0062] If t>T+1, the pedestrian feature C(t) extracted from the current frame t is matched with the pedestrian feature in the pedestrian feature library. If the match is successful, the extracted pedestrian feature C(t) is added to the pedestrian feature library and a corresponding unique ID is assigned to it to update the pedestrian feature library and the pedestrian database. If the match is unsuccessful, an IOU comparison calculation is required.
[0063] The IOU comparison calculation is to fit the data of the current frame into a function. As the number of frames increases, the region box prediction of the pedestrian image P(t) of the tth frame is performed each time. That is, the region box where the pedestrian may stay in the current frame can be predicted by fitting the function, and the predicted region box i is used to perform an intersection operation with the actual obtained pedestrian image region box j:
[0064]
[0065] Where S ij交 represents the intersection of the predicted region box i and the actually acquired pedestrian image region box j, S ij并 Represents the union of the predicted region box i and the actual obtained pedestrian image region box j, IOU ij Represents the ratio of intersection to union. If the overlap area of two region boxes is larger, the IOU ijThe closer it is to 1, the greater the degree of successful matching; conversely, the closer it is to 0, the smaller the degree of successful matching.
[0066] In order to more accurately determine whether the match is successful, calculate the cosine distance D of the two area boxes:
[0067]
[0068] in represents the cosine similarity between region box i and region box j, and D represents the cosine distance between region box i and region box j.
[0069] But according to common sense, the higher the degree of overlap between the two, the closer the distance between them is to 0, so when IOU ij The closer it is to 1, the closer the distance between the two area boxes is to 0. In order to conform to common sense, the cosine distance D calculated by IOU comparison is flipped: D`=1-D. At this time, the size of D` can directly indicate the distance between the two area boxes. When D` is closer to 0, it means that the degree of overlap between the two area boxes is higher.
[0070] The distance threshold △D` is pre-set through the Hungarian algorithm. When D`≤△D`, it means that the IOU comparison calculation is successfully matched with the pedestrian feature library. At this time, the pedestrian feature C(t) is added to the pedestrian database and assigned a corresponding unique ID; when D`>△D`, it means that the IOU comparison calculation is unsuccessful and the pedestrian feature library is matched, which means that the pedestrian has just entered the camera shooting area. The pedestrian feature is added to the pedestrian feature library and assigned a new unique ID to update the pedestrian feature library and the pedestrian database at the same time.
[0071] In the traditional solution, when only the pedestrian image is tracked by Kalman filtering, when the pedestrian exits the shooting range of the current camera and then enters the camera again, the pedestrian features of the pedestrian image are no longer recognized and need to be re-identified and calculated. However, this solution combines the videos collected by all cameras and obtains a stable pedestrian feature library that stores the same pedestrian features. Even if the pedestrian leaves and enters again, it can be quickly identified and tracked. Therefore, this solution can not only more accurately identify pedestrians in the video collected by the camera, but also quickly identify and track the pedestrian afterwards.
[0072] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A method for automatically updating a pedestrian re-identification feature library, characterized in that: The following steps are involved: Step S1: Obtain the current frame images from all cameras and identify the pedestrian images in the current frame images; input all the identified pedestrian images into the feature extraction network to extract pedestrian features from the pedestrian images; Step S2: when extracting pedestrian features, determine whether the current frame image meets the maximum Kalman filter tracking frame number. If not, perform Kalman filter tracking on the pedestrian features of the current frame, and establish or update a pedestrian database for each camera; if satisfied, establish a pedestrian feature library after comparing and removing duplicates based on the pedestrian databases of each camera; the maximum Kalman filter tracking frame number is an integer greater than 1; The step of determining whether the current frame image satisfies the maximum Kalman filter tracking frame number when extracting pedestrian features, and if not, performing Kalman filter tracking on the pedestrian features of the current frame, and establishing or updating a pedestrian database for each camera, includes: If the current frame t is less than or equal to the maximum Kalman filter tracking frame number T, the current frame t does not meet the maximum Kalman filter tracking frame number, and the Kalman filter tracking is performed on the area frame of the pedestrian image P(t) in the current frame image; Determine whether the current frame t is equal to 1. If the current frame t is equal to 1, establish a corresponding pedestrian database for each camera; If the current frame t is greater than 1 and less than or equal to the maximum Kalman filter tracking frame number T, then the corresponding pedestrian database under each camera is updated; The step of performing Kalman filtering tracking on the area frame of the pedestrian image P(t) in the current frame image includes: If the current frame t is equal to 1, a pedestrian database is established according to the region frame of the pedestrian image P(t), and the pedestrian database includes the region frame of the pedestrian image P(t) and the pedestrian features corresponding to the region frame; If the current frame is greater than 1 and less than or equal to the maximum Kalman filter tracking frame number T, the Kalman filter tracking infers the predicted position of the area box in the current frame pedestrian image P(t) based on the area box of the previous frame pedestrian image P(t-1), and compares the actual area box of the current frame pedestrian image P(t) with the inferred area box predicted position; If the above conditions are met, the steps of establishing a pedestrian feature database after comparing and removing duplicates based on the pedestrian databases of each camera include: If the current frame t is greater than the maximum Kalman filter tracking frame number T, then the current frame t satisfies the maximum Kalman tracking frame number, and at the same time, it is determined whether the current frame t is equal to T+1; If the current frame t is equal to T+1, the pedestrian database of each camera is compared and deduplicated to establish a unified pedestrian feature library, and the same pedestrian features in different pedestrian databases are assigned unique IDs in the pedestrian feature library; If the current frame t is greater than T+1, the pedestrian feature extracted from the current frame t is matched with the pedestrian feature library. If the match is successful, the pedestrian feature is added to the pedestrian feature library and a corresponding unique ID is assigned to it to update the pedestrian feature library; If the match is unsuccessful, the pedestrian feature is subjected to IOU comparison calculation. If the pedestrian feature is successfully matched with the pedestrian feature library after IOU comparison calculation, the pedestrian feature is added to the pedestrian feature library and a corresponding unique ID is assigned to it. If the pedestrian feature is still unsuccessful after IOU comparison calculation, the pedestrian feature is added to the pedestrian feature library and a new unique ID is assigned to it to update the pedestrian feature library. If the matching is unsuccessful, the pedestrian feature is subjected to IOU comparison calculation. If the pedestrian feature is successfully matched with the pedestrian feature library after the IOU comparison calculation, the pedestrian feature is added to the pedestrian feature library and a corresponding unique ID is assigned to the pedestrian feature, including: A function is fitted to the data of the current frame, and the region box i where the pedestrian may stay in the current frame is predicted by the function. The predicted region box i is used to perform an intersection operation with the actual obtained pedestrian image region box j: Where S ij交 represents the intersection of the predicted region box i and the actually acquired pedestrian image region box j, S ij并 Represents the union of the predicted region box i and the actual obtained pedestrian image region box j, IOU ij It represents the ratio of intersection to union. If the overlap area of two area boxes is larger, the IOU is ij The closer it is to 1, the closer it is to 0; According to IOU ij The cosine distance D is calculated: in represents the cosine similarity between region box i and region box j, and D represents the cosine distance between region box i and region box j; The cosine distance D is flipped: D`=1-D; The distance threshold △D` is pre-set by the Hungarian algorithm. When D`≤△D`, the pedestrian feature successfully matches the pedestrian feature library. The pedestrian feature is then added to the pedestrian feature library and assigned a corresponding unique ID.
2. The method for automatically updating a person re-identification feature library according to claim 1, characterized in that: The step of acquiring the current frame image from all cameras and identifying the pedestrian image in the current frame image includes: Get the current frame image V(t)={v1(t),v2(t),...,v M (t)}, where v1(t) represents the current frame image under the first camera, v M (t) represents the current frame image under the Mth camera, t represents the time of the current frame, and t is an integer greater than or equal to 1; According to the current frame image V(t) under M cameras, the pedestrians in the current frame image are identified, and all the identified pedestrians are cut out to form a pedestrian image P(t)={p1(t),p2(t),...,p M (t)}, where p1(t) represents the pedestrian image in the current frame image under the first camera, p M (t) represents the pedestrian image in the current frame image under the Mth camera, and the pedestrian image is presented in the form of a region frame.
3. The method for automatically updating a pedestrian re-identification feature library according to claim 1, characterized in that: While updating the pedestrian feature library, the pedestrian database under the corresponding camera is also updated.
