A multi-modal pedestrian re-identification method, system, electronic device and storage medium

Abstract

The application discloses a multi-modal pedestrian re-identification method and system, an electronic device and a storage medium, relates to the technical field of pedestrian re-identification, and obtains a video to be re-identified and a probe request frame of Wi-Fi 802.11 in a preset time period, so as to determine a plurality of video sequences and probe request sequences; the matching degrees of any video sequence and each probe request sequence are calculated respectively; the probe request sequence corresponding to the maximum matching degree matched with the video sequence is determined as a matched probe request sequence; if a stop condition is met, the multi-modal feature vector of a pedestrian in the video sequence is determined according to the video sequence and the matched probe request sequence; if not, two to-be-updated matched probe request sequences are determined as a to-be-updated sequence group; all probe request sequences are updated based on all to-be-updated sequence groups, and the matching degrees of any video sequence and each probe request sequence are calculated respectively until the stop condition is met. The application improves the accuracy of pedestrian re-identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pedestrian re-identification, and in particular to a multi-modal pedestrian re-identification method and system, an electronic device and a storage medium. BACKGROUND

[0002] Pedestrian re-identification refers to matching pedestrian images or videos under different cameras to find target pedestrians who may appear at different times and in different places. Due to its great application prospects in intelligent security, video monitoring and the like, pedestrian re-identification has become a research focus in the field of computer vision. With the development of deep learning technology, pedestrian re-identification methods based on visual information have achieved certain results. Such methods extract a feature vector of a pedestrian from original image or video data through a deep neural network, and calculate a feature similarity to re-identify the pedestrian. However, an inherent defect of the method based on visual information is that it is susceptible to changes in the appearance of a pedestrian. Due to different shooting locations and times of different pedestrian images or videos, there are differences in the perspective, illumination, posture and clothing of a pedestrian, and there is also a great deviation in the feature information of the same pedestrian. Meanwhile, in a real scene, a pedestrian is usually in an environment with large pedestrian flow and complex background, and it is difficult to avoid occlusion of a pedestrian part.

[0003] On the other hand, with the gradual normality of pedestrians carrying wireless network devices such as smart phones, perception of pedestrians through wireless signals has gradually emerged and has been applied to indoor positioning, crowd estimation and the like. Among them, Wi-Fi 802.11 probe request frames have attracted widespread attention due to their high coverage and low acquisition cost. A device enabled with Wi-Fi functionality periodically broadcasts an 802.11 frame called a probe request to search for nearby available wireless networks. The probe request frame contains information such as a physical (Media Access Control, MAC) address as a device identifier and an RSSI indicating a received signal strength. Unlike visual information, the Wi-Fi 802.11 probe request frame is not affected by changes in the appearance of a pedestrian, and can match and re-identify the wireless signal of a smart phone carried by a pedestrian in the case of poor visual monitoring conditions. However, to prevent malicious tracking of devices, some smart phones enable a MAC address randomization function, which makes the MAC address no longer able to serve as a unique identifier of a device, bringing great challenges to matching and re-identification based on probe request frames, and making the accuracy of pedestrian re-identification low. SUMMARY

[0004] The present application aims to provide a multi-modal pedestrian re-identification method and system, an electronic device and a storage medium, which improve the accuracy of pedestrian re-identification.

[0005] To achieve the above object, the present application provides the following solutions:

[0006] A multi-modal pedestrian re-identification method, comprising:

[0007] obtaining a to-be-re-identified video and a plurality of Wi-Fi 802.11 probe request frames within a preset time period;

[0008] inputting the to-be-re-identified video into a multi-target detection model to obtain a plurality of video sequences; each video frame in the video sequence contains a pedestrian and a bounding box, and the multi-target detection model is obtained by training a ByteTrack network using a plurality of videos and a plurality of corresponding video sequences;

[0009] determining a plurality of probe request sequences according to each probe request frame by using a probe request association algorithm;

[0010] respectively calculating the matching degrees of any video sequence and each probe request sequence;

[0011] determining the probe request sequence corresponding to the maximum matching degree among all matching degrees matching any video sequence as a matching probe request sequence;

[0012] determining whether a stop condition is met to obtain a first determination result; the stop condition is whether the number of to-be-updated matching probe request sequences is less than 2 or reaches a preset update number; the to-be-updated matching probe request sequence is a matching probe request sequence corresponding to a matching degree greater than a matching degree threshold;

[0013] if the first determination result is yes, determining a multi-modal feature vector of a pedestrian in a video sequence according to any video sequence and a corresponding matching probe request sequence to realize re-identification of the pedestrian;

[0014] if the first determination result is no, determining two to-be-updated matching probe request sequences as one to-be-updated sequence group; updating all probe request sequences based on all to-be-updated sequence groups, returning to "respectively calculating the matching degrees of any video sequence and each probe request sequence", until the stop condition is met.

[0015] Optionally, determining a plurality of probe request sequences according to each probe request frame by using a probe request association algorithm, comprising:

[0016] dividing all probe request frames within the preset time period into a plurality of batches of probe request frames;

[0017] grouping each batch of probe request frames according to MAC addresses to obtain a plurality of groups of probe request frames;

[0018] determining a similarity probability between any two groups of probe request frames in all groups of probe request frames in any current batch as a first similarity probability;

[0019] determining whether any of the first similarity probabilities is greater than the similarity probability threshold, to obtain a second determination result;

[0020] If the second determination result is yes, merging two groups of probe request frames with the first similarity probability greater than the similarity probability threshold into one group of probe request frames to obtain updated groups of probe request frames, replacing the groups of probe request frames with the updated groups of probe request frames, and returning to "determining the similarity probability between any two groups of probe request frames in all groups of probe request frames", until all first similarity probabilities are less than or equal to the similarity probability threshold, to obtain the probe request frames updated multiple times in the current batch as the target probe request frames of the current batch; the updated probe request frames in the current batch include one or more groups of probe request frames;

[0021] If the second determination result is no, determining all groups of probe request frames in the current batch as the target probe request frames of the current batch;

[0022] determining the similarity probability between the target probe request frames of any two batches as a second similarity probability;

[0023] determining whether any of the second similarity probabilities is greater than the similarity probability threshold, to obtain a third determination result;

[0024] If the third determination result is yes, merging the target probe request frames of two batches with the second similarity probability greater than the similarity probability threshold into the target probe request frames of one batch to obtain updated target probe request frames of multiple batches, replacing the target probe request frames of multiple batches with the updated target probe request frames of multiple batches, and returning to "determining the similarity probability between the target probe request frames of any two batches as a second similarity probability", until all second similarity probabilities are less than or equal to the similarity probability threshold, to obtain the target probe request frames of multiple batches updated multiple times as the target probe request frames of the target batch;

[0025] If the third determination result is no, determining the target probe request frames of all batches as the target probe request frames of the target batch;

[0026] sorting all groups of target probe request frames of all target batches according to timestamps, to obtain multiple probe request sequences.

[0027] Optionally, the matching degrees of any video sequence and each probe request sequence are calculated respectively, including:

[0028] determining any of the video sequences and any of the probe request sequences as a sequence group, to obtain multiple sequence groups;

[0029] calculate similarity between video sequences and probe request sequences in each of the sequence groups; the similarity includes distance similarity and visual similarity;

[0030] For any sequence group, determine matching degree of video sequences and probe request sequences in the sequence group according to the similarity corresponding to the sequence group.

[0031] Optionally, the process of calculating distance similarity between video sequences and probe request sequences in any current sequence group includes:

[0032] calculate near-end coincidence degree of video sequences and probe request sequences in the current sequence group;

[0033] far-end coincidence degree between video sequences and probe request sequences in the current sequence group;

[0034] calculate the distance similarity according to the near-end coincidence degree and the far-end coincidence degree.

[0035] Optionally, the process of calculating visual similarity between video sequences and probe request sequences in any current sequence group includes:

[0036] input the video sequence into a feature vector determination model to obtain a video feature vector; the feature vector determination model is obtained by training an initial network using a plurality of training video sequences and corresponding video feature vectors, and the initial network is AP3D, PSTA or SINet;

[0037] determine a probe request feature vector according to all distance similarities corresponding to probe request sequences in the current sequence group;

[0038] calculate the cosine distance between the video feature vector and the probe request feature vector to obtain the visual similarity.

[0039] A multi-modal pedestrian re-identification system, comprising:

[0040] an original data acquisition module configured to acquire a video to be re-identified and a plurality of Wi-Fi 802.11 probe request frames within a preset time period;

[0041] a video sequence determination module configured to input the video to be re-identified into a multi-target detection model to obtain a plurality of video sequences; each video frame in the video sequence contains a pedestrian and a bounding box, and the multi-target detection model is obtained by training a ByteTrack network using a plurality of videos and corresponding video sequences;

[0042] a probe request sequence determination module configured to determine a plurality of probe request sequences according to the probe request frames by using a probe request association algorithm;

[0043] The matching degree determination module is configured to calculate the matching degree of any video sequence and each probe request sequence respectively;

[0044] The matching probe request sequence determination module is configured to determine the probe request sequence corresponding to the maximum matching degree among all matching degrees matched with any video sequence as the matching probe request sequence;

[0045] The judgment module is configured to judge whether a stop condition is met to obtain a first judgment result; the stop condition is whether the number of to-be-updated matching probe request sequences is less than 2 or reaches a preset update number; the to-be-updated matching probe request sequence is a matching probe request sequence with a corresponding matching degree greater than a matching degree threshold;

[0046] The re-identification module is configured to, if the first judgment result is yes, determine a multi-modal feature vector of a pedestrian in a video sequence according to any video sequence and a corresponding matching probe request sequence to realize re-identification of the pedestrian.

[0047] The update iteration module is configured to, if the first judgment result is no, determine two to-be-updated matching probe request sequences as one to-be-updated sequence group; update all probe request sequences based on all to-be-updated sequence groups, and return to the step of calculating the matching degree of any video sequence and each probe request sequence until the stop condition is met.

[0048] An electronic device includes:

[0049] One or more processors;

[0050] A storage device having one or more programs stored thereon;

[0051] When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-modal pedestrian re-identification method as described above.

[0052] A storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the multi-modal pedestrian re-identification method as described above.

[0053] According to the embodiments of the present application, the following technical effects are provided:

[0054] This invention discloses a multimodal pedestrian re-identification method, system, electronic device, and storage medium. It acquires videos to be re-identified and multiple Wi-Fi 802.11 detection request frames within a preset time period, thereby determining multiple videos and detection request sequences. The matching degree between any video sequence and each detection request sequence is calculated. The detection request sequence with the highest matching degree to the video sequence is determined as the matched detection request sequence. If a stopping condition is met, the multimodal feature vector of the pedestrian in the video sequence is determined based on the video sequence and the matched detection request sequence, achieving pedestrian re-identification. If not, two matching detection request sequences to be updated are determined as a set of sequences to be updated. All detection request sequences are updated based on all sets of sequences to be updated, and the process returns "calculate the matching degree between any video sequence and each detection request sequence" until the stopping condition is met. This invention improves the accuracy of pedestrian re-identification. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Fig. 1 This is a schematic diagram of the multimodal pedestrian re-identification method provided in Embodiment 1 of the present invention;

[0057] Fig. 2 A logic diagram for multimodal pedestrian re-identification;

[0058] Fig. 3 This is a schematic diagram illustrating the principle of matching video sequences with detection request sequences. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The purpose of this invention is to provide a multimodal pedestrian re-identification method, system, electronic device, and storage medium, aiming to improve the accuracy of pedestrian re-identification.

[0061] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Example 1

[0063] As shown in the figure, the multi-modal pedestrian re-identification method in the embodiment includes: Figs. 1-3

[0064] Step 101: Determine a plurality of video sequences and a plurality of probe request sequences. Specifically, it includes:

[0065] Step 1011: Obtain a to-be-re-identified video and a plurality of Wi-Fi 802.11 probe request frames in a preset time period.

[0066] Step 1012: Input the to-be-re-identified video into a multi-target detection model to obtain a plurality of video sequences; each video frame in the video sequence contains a pedestrian and a bounding box, and the multi-target detection model is obtained by training the ByteTrack network using a plurality of videos and a plurality of corresponding video sequences.

[0067] Step 1013: Determine a plurality of probe request sequences according to each probe request frame using a probe request association algorithm.

[0068] As an optional implementation, step 1013 includes:

[0069] Divide all probe request frames in the preset time period into a plurality of batches of probe request frames.

[0070] Group each batch of probe request frames according to the MAC address to obtain a plurality of groups of probe request frames.

[0071] Determine the similarity probability between any two groups of probe request frames in all groups of probe request frames in any current batch as the first similarity probability.

[0072] Specifically, taking a probe request frame group g and a probe request frame group h as an example, the determination process of the similarity probability between the two groups of probe request frames (such as g and h) includes:

[0073] Determine the centroid of the feature vectors of all probe request frames in the probe request frame group g and the probe request frame group h, and take the centroid of the feature vectors as the center feature vector f g and f h of the corresponding probe request frame group.

[0074] Calculate the similarity probability C(g, h) of f g and f h .

[0075] ​The determination process of the similarity probability is automatically realized by inputting the probe request frame group g and the probe request frame group h into the probe request partition model. The probe request partition model adopts a twin network structure, the input is two probe request frames, and the output is the predicted probability that the probe request frame is sent by the same wireless network device, that is, the similarity probability. A fully connected deep neural network is used as the feature extractor of the probe request partition model to encode the information vector of the probe request frame into a feature vector, and the output probability is predicted through the distance between the feature vectors of the two probe request frames. The probe request frames sent by the wireless network device without enabling the MAC address randomization function are collected as the training data set. The MAC address contained in such probe request frames is the real address of the device, which is used as the training true label.

[0076] It is judged whether any first similarity probability is greater than the similarity probability threshold value, to obtain a second judgment result.

[0077] If the second judgment result is yes, the two probe request frame groups whose first similarity probability is greater than the similarity probability threshold value are merged into one probe request frame group to obtain updated multiple probe request frame groups, the multiple probe request frame groups are replaced by the updated multiple probe request frame groups, and the step of “calculating the similarity probability between any two probe request frame groups in all probe request frame groups” is returned until all first similarity probabilities are less than or equal to the similarity probability threshold value, to obtain the multiple updated probe request frames in the current batch as the target probe request frames of the current batch; the updated probe request frames in the current batch include one or more probe request frame groups.

[0078] Specifically, in any current batch, for all every two probe request frame groups g and h, C(g, h) is arranged in descending order and checked in turn, if g and h are not marked as merged, and C(g, h) is greater than the similarity probability threshold value θ, then g and h are merged and marked as merged.

[0079] If the second judgment result is no, all probe request frame groups in the current batch are determined as the target probe request frames of the current batch.

[0080] The similarity probability between the target probe request frames of any two batches is determined as a second similarity probability.

[0081] It is judged whether any second similarity probability is greater than the similarity probability threshold value, to obtain a third judgment result.

[0082] If the third determination result is yes, the two batches of target probe request frames with the second similarity probability greater than the similarity probability threshold are merged into one batch of target probe request frames to obtain updated multiple batches of target probe request frames, the multiple batches of probe request frames are replaced by the updated multiple batches of target probe request frames, and the determination of the similarity probability between any two batches of target probe request frames as the second similarity probability is returned until all the second similarity probabilities are less than or equal to the similarity probability threshold, and the multiple batches of target probe request frames after multiple updates are obtained as the target batch of target probe request frames.

[0083] If the third determination result is no, all the batches of target probe request frames are determined as the target batch of target probe request frames.

[0084] All the multiple groups of target probe request frames of the target batch are sorted according to timestamps to obtain multiple probe request sequences.

[0085] Step 102: The matching degrees of any video sequence and each probe request sequence are calculated respectively.

[0086] As an optional implementation, step 102 includes:

[0087] Any video sequence and any probe request sequence are determined as a sequence group to obtain multiple sequence groups.

[0088] The similarity between the video sequence and the probe request sequence in each sequence group is calculated. The similarity includes distance similarity and visual similarity.

[0089] For any sequence group, the matching degrees of the video sequence and the probe request sequence in the sequence group are determined according to the similarity corresponding to the sequence group.

[0090] Specifically, before step 102, it further includes filtering the probe request frames, specifically including:

[0091] There are a large number of wireless network devices in an indoor environment, and these devices will all send probe request frames and form probe request sequences. Most of these devices are not carried by pedestrians, such as notebook computers, wireless printers, etc. The probe request sequences of these devices will interfere with pedestrian re-identification. If the length |G| of the probe request sequence G is greater than a threshold value δ L , and satisfies where |R| is the number of wireless access points collecting probe request frames, P is a probe request frame in G, P.r k is the received signal strength (RSSI) of P collected by the kth wireless access point, P.r kIf the average value is not found, then G is considered to have been sent by a device not carried by the pedestrian, and G is removed from the set of detection request sequences.

[0092] As an optional implementation, the process of calculating the distance similarity between video sequences and probe request sequences in any current sequence group includes:

[0093] Calculate the near-end overlap between the video sequences and the probe request sequences in the current sequence group.

[0094] The degree of remote overlap between video sequences and probe request sequences in the current sequence group.

[0095] Distance similarity is calculated based on near-end overlap and far-end overlap.

[0096] Specifically, using video sequence V i and detection request sequence G j For example, video sequence V i Recorded as Among them, F i,k and t i,k Given the k-th video frame and its corresponding timestamp, the distance similarity calculation process includes:

[0097] For video sequence V i Each video frame F i,k The bounding box is a rectangle. The midpoint of the lower boundary of the bounding box is taken as the pixel coordinates of the pedestrian in the video frame, and then converted into the pedestrian's position in the frame using homography transformation. i,k The real ground coordinates of the person at any given time (homography is a coordinate transformation algorithm in computer vision that converts the coordinates of any point in one plane coordinate system to those in another by pre-calibrating the coordinates of four reference points). After obtaining the real ground coordinates of the pedestrian in each video frame, the coordinates and timestamp sequence are interpolated to obtain the coordinates at time t. i,1 to The actual ground coordinates of a pedestrian at any time t within the time period

[0098] For video sequence V i With probe request sequence G j The overlapping portion of their time can be represented as a sequence. Where Pt is the timestamp of the probe request frame P, and d t,k Let t be the pedestrian's actual ground coordinates. Euclidean distance between the coordinates of the kth wireless access point. Both the video frames in the video sequence and the probe request frames in the probe request sequence are sorted by timestamps. If the video sequence is from t1 to t2 and the probe request sequence is from t3 to t4, then take the intersection of the intervals [t1, t2] and [t3, t4], and the video frames and probe request frames in the two sequences whose timestamps fall within the intersection are the overlapping part in time.

[0099] V i Proximity of the near end with G j The calculation formula is: where t is time, r is RSSI, d is distance, δ d and δ w are distance threshold and RSSI threshold respectively.

[0100] V i Proximity of the far end with G j The calculation formula is:

[0101] Distance similarity of the video sequence V i and the probe request sequence G j The calculation formula is where f is a monotone non-decreasing function on [0, 1] as a normalization term of sequence length.

[0102] As an optional implementation, the process of calculating the visual similarity between the video sequence and the probe request sequence in any current sequence group includes:

[0103] inputting the video sequence into a feature vector determination model to obtain a video feature vector; the feature vector determination model is obtained by training an initial network using a plurality of training video sequences and corresponding video feature vectors, and the initial network is AP3D, PSTA or SINet.

[0104] determining a probe request feature vector according to all distance similarities corresponding to the probe request sequences in the current sequence group.

[0105] Specifically, for the probe request sequence G j , the calculation formula of the probe request feature vector is: where V i is the video feature vector of V

[0106] ​​​​​The cosine distance of the video feature vector and the probe request feature vector is calculated to obtain a visual similarity.

[0107] Specifically, the video sequence V i and the probe request sequence G j are matched to obtain a visual similarity. G (·,·) is the cosine distance of two feature vectors.

[0108] Specifically, the video sequence V i and the probe request sequence G j are matched to obtain a matching degree Φ i,j . wherein, α is a weight, 0<α<1. Thus, the probe request sequence matched with the video sequence V i is determined as wherein, mv i =argmax i (Φ i,j ); and the video sequence matched with the probe request sequence G j is determined as wherein, mp j =argmax j (Φ i,j ).

[0109] Step 103: determining the probe request sequence corresponding to the maximum matching degree in all matching degrees matched with any video sequence as a matched probe request sequence.

[0110] Specifically, the matched probe request sequence of the video sequence V i is determined as wherein, mv i =argmax i (Φ i,j ); and the video sequence with the maximum matching degree matched with the probe request sequence G j is determined as wherein, mp j =argmax j (Φ i,j ).

[0111] Step 104: judging whether a stop condition is met to obtain a first judgment result.

[0112] wherein, the stop condition is whether the number of matched probe request sequences to be updated is less than 2 or reaches a preset updating number; and the matched probe request sequence to be updated is a matched probe request sequence with a matching degree greater than a matching degree threshold.

[0113] ​Step 105: If the first judgment result is yes, then determine the multimodal feature vector of the pedestrian in the video sequence based on any video sequence and the corresponding matching detection request sequence, so as to realize the re-identification of the pedestrian.

[0114] Specifically, video sequence V i The multimodal feature vector f of pedestrians i The calculation formula is:

[0115]

[0116] in, For video sequence V i Matching the probe request sequence is Distance similarity between them; To match the probe request sequence is The feature vector of the detection request.

[0117] Step 106: If the first judgment result is negative, then the two matching probe request sequences to be updated are determined as a sequence group to be updated. All probe request sequences are updated based on all sequence groups to be updated. Return to step 102 until the stopping condition is met.

[0118] Specifically, if the probe request sequence G j satisfy Then for G j Perform sequence generation and updates. The sequence group to be updated includes the matching probe request sequence G. i and G j For example, if the similarity probability between the two is ρ i,j The updated similarity probability is in, for With G i The degree of matching between them for With G j The degree of matching between them for and Multimodal feature vector distance, To match the detection request sequence G i The video sequence with the highest matching degree among the matched video sequences. To match the detection request sequence G j The video sequence with the highest matching degree among the matched video sequences.

[0119] Example 2

[0120] The multimodal pedestrian re-identification system in this embodiment includes:

[0121] An original data acquisition module is configured to acquire a to-be-reidentified video and a plurality of probe request frames of Wi-Fi 802.11 within a preset time period.

[0122] A video sequence determination module is configured to input the to-be-reidentified video into a multi-target detection model to obtain a plurality of video sequences, each video frame in the video sequence containing a pedestrian and a bounding box, and the multi-target detection model being obtained by training a ByteTrack network using a plurality of videos and a plurality of corresponding video sequences.

[0123] A probe request sequence determination module is configured to determine a plurality of probe request sequences according to each probe request frame using a probe request association algorithm.

[0124] A matching degree determination module is configured to calculate a matching degree between any video sequence and each probe request sequence.

[0125] A matching probe request sequence determination module is configured to determine, as a matching probe request sequence, a probe request sequence corresponding to a maximum matching degree among all matching degrees of any video sequence.

[0126] A judgment module is configured to determine whether a stop condition is met to obtain a first judgment result, the stop condition being whether a number of to-be-updated matching probe request sequences is less than 2 or reaches a preset number of updates, and the to-be-updated matching probe request sequence being a matching probe request sequence corresponding to a matching degree greater than a matching degree threshold.

[0127] A re-identification module is configured to, if the first judgment result is yes, determine a multi-modal feature vector of a pedestrian in a video sequence according to any video sequence and a corresponding matching probe request sequence to implement re-identification of the pedestrian.

[0128] An update iteration module is configured to, if the first judgment result is no, determine two to-be-updated matching probe request sequences as one to-be-updated sequence group, update all probe request sequences based on all to-be-updated sequence groups, and return to “calculate a matching degree between any video sequence and each probe request sequence” until the stop condition is met.

[0129] Embodiment 3

[0130] An electronic device includes:

[0131] One or more processors.

[0132] A storage device having one or more programs stored thereon.

[0133] When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-modal pedestrian re-identification method as in embodiment 1.

[0134] Embodiment 4

[0135] A storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the multi-modal pedestrian re-identification method in embodiment 1.

[0136] The various embodiments are described in a progressive manner in the specification, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be mutually referred to. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0137] The principles and implementation manners of the present application are described by using specific examples in the specification. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges can be changed according to the idea of the present application. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A multi-modal pedestrian re-identification method, characterized in that, The method includes: Acquire the video to be re-identified and multiple Wi-Fi 802.11 detection request frames within a preset time period; The video to be re-identified is input into a multi-object detection model to obtain multiple video sequences; each video frame in the video sequence contains pedestrians and bounding boxes; the multi-object detection model is obtained by training the ByteTrack network using multiple videos and corresponding video sequences. Using a probe request association algorithm, multiple probe request sequences are determined based on each probe request frame; Calculate the matching degree between any video sequence and each detection request sequence; The probe request sequence corresponding to the maximum matching degree among all matching degrees that match any video sequence is determined as the matching probe request sequence; Determine whether the stopping condition is met to obtain a first determination result; the stopping condition is whether the number of matching probe request sequences to be updated is less than 2 or reaches a preset number of updates; the matching probe request sequences to be updated are matching probe request sequences with a matching degree greater than the matching degree threshold. If the first judgment result is yes, then the multimodal feature vector of the pedestrian in the video sequence is determined according to any video sequence and the corresponding matching detection request sequence, so as to realize the re-identification of the pedestrian; If the first judgment result is negative, then the two matching detection request sequences to be updated are determined as a sequence group to be updated; all detection request sequences are updated based on all sequence groups to be updated, and "calculate the matching degree between any video sequence and each detection request sequence respectively" is returned until the stopping condition is met; Using a probe request association algorithm, multiple probe request sequences are determined based on each probe request frame, including: Divide all probe request frames within a preset time period into multiple batches of probe request frames; The probe request frames for each batch are grouped according to their MAC addresses to obtain multiple groups of probe request frames; Determine the similarity probability between any two groups of probe request frames in any current batch as the first similarity probability; Determine whether any first similarity probability is greater than the similarity probability threshold to obtain the second determination result; If the second judgment result is yes, then the two groups of detection request frames with a first similarity probability greater than the similarity probability threshold are merged into one group of detection request frames to obtain multiple updated groups of detection request frames. Then, the multiple groups of detection request frames are replaced with the updated groups of detection request frames, and the function "calculate the similarity probability between any two groups of detection request frames in all groups of detection request frames" is returned until all first similarity probabilities are less than or equal to the similarity probability threshold. The multiple updated detection request frames in the current batch are then used as the target detection request frames for the current batch. The updated detection request frames in the current batch include one or more groups of detection request frames. If the second judgment result is negative, then all group detection request frames in the current batch will be determined as the target detection request frames of the current batch. The similarity probability between any two batches of target detection request frames is determined as the second similarity probability; Determine whether any second similarity probability is greater than the similarity probability threshold to obtain the third judgment result; If the third judgment result is yes, then the two batches of target detection request frames with a second similarity probability greater than the similarity probability threshold are merged into one batch of target detection request frames to obtain multiple updated batches of target detection request frames. The multiple batches of detection request frames are replaced with the updated multiple batches of target detection request frames, and the function "determine the similarity probability between any two batches of target detection request frames as the second similarity probability" is returned until all second similarity probabilities are less than or equal to the similarity probability threshold, and multiple updated batches of target detection request frames are obtained as the target batch of target detection request frames. If the third judgment result is negative, then all batches of target detection request frames are identified as target batches of target detection request frames; All target batches of multiple target detection request frames are sorted according to timestamps to obtain multiple detection request sequences.

2. The multi-modal pedestrian re-identification method of claim 1, wherein, Calculate the matching degree between any video sequence and each probe request sequence, including: Each of the video sequences and each of the detection request sequences is defined as a sequence group, resulting in multiple sequence groups; Calculate the similarity between the video sequences and the probe request sequences in each of the sequence groups; the similarity includes distance similarity and visual similarity; For any sequence group, the matching degree between the video sequence and the probe request sequence in the sequence group is determined based on the similarity of the sequence group.

3. The multi-modal pedestrian re-identification method of claim 2, wherein, The process of calculating the distance similarity between video sequences and probe request sequences in any current sequence group includes: Calculate the near-end overlap between video sequences and probe request sequences in the current sequence group; The degree of far-end overlap between video sequences and probe request sequences in the current sequence group; The distance similarity is calculated based on the proximal overlap and the distal overlap.

4. The multimodal pedestrian re-identification method according to claim 3, characterized in that, The process of calculating the visual similarity between video sequences and probe request sequences in any current sequence group includes: The video sequence is input into the feature vector determination model to obtain the video feature vector; the feature vector determination model is obtained by training an initial network using multiple training video sequences and corresponding video feature vectors, and the initial network is AP3D, PSTA or SINet; Determine the probe request feature vector based on all distance similarities to the probe request sequences in the current sequence group; The visual similarity is obtained by calculating the cosine distance between the video feature vector and the probe request feature vector.

5. A multimodal pedestrian re-identification system, characterized in that, The system includes: The raw data acquisition module is used to acquire the video to be re-identified and multiple Wi-Fi 802.11 detection request frames within a preset time period; The video sequence determination module is used to input the video to be re-identified into a multi-object detection model to obtain multiple video sequences; each video frame in the video sequence contains pedestrians and bounding boxes; the multi-object detection model is obtained by training the ByteTrack network using multiple videos and corresponding multiple video sequences. The probe request sequence determination module is used to determine multiple probe request sequences based on each probe request frame using a probe request association algorithm. The matching degree determination module is used to calculate the matching degree between any video sequence and each detection request sequence. The matching probe request sequence determination module is used to determine the probe request sequence corresponding to the maximum matching degree among all matching degrees that match any video sequence as the matching probe request sequence; The judgment module is used to determine whether the stopping condition is met and obtain a first judgment result; the stopping condition is whether the number of matching probe request sequences to be updated is less than 2 or reaches a preset number of updates; the matching probe request sequences to be updated are matching probe request sequences whose corresponding matching degree is greater than the matching degree threshold. The re-identification module is used to determine the multimodal feature vector of the pedestrian in the video sequence based on any video sequence and the corresponding matching detection request sequence if the first judgment result is yes, so as to realize the re-identification of the pedestrian; The update iteration module is used to determine two matching detection request sequences to be updated as a sequence group to be updated if the first judgment result is negative; update all detection request sequences based on all sequence groups to be updated, and return "calculate the matching degree between any video sequence and each detection request sequence respectively" until the stopping condition is met; Using a probe request association algorithm, multiple probe request sequences are determined based on each probe request frame, including: Divide all probe request frames within a preset time period into multiple batches of probe request frames; The probe request frames for each batch are grouped according to their MAC addresses to obtain multiple groups of probe request frames; Determine the similarity probability between any two groups of probe request frames in any current batch as the first similarity probability; Determine whether any first similarity probability is greater than the similarity probability threshold to obtain the second determination result; If the second judgment result is yes, then the two groups of detection request frames with a first similarity probability greater than the similarity probability threshold are merged into one group of detection request frames to obtain multiple updated groups of detection request frames. Then, the multiple groups of detection request frames are replaced with the updated groups of detection request frames, and the function "calculate the similarity probability between any two groups of detection request frames in all groups of detection request frames" is returned until all first similarity probabilities are less than or equal to the similarity probability threshold. The multiple updated detection request frames in the current batch are then used as the target detection request frames for the current batch. The updated detection request frames in the current batch include one or more groups of detection request frames. If the second judgment result is negative, then all group detection request frames in the current batch will be determined as the target detection request frames of the current batch. The similarity probability between any two batches of target detection request frames is determined as the second similarity probability; Determine whether any second similarity probability is greater than the similarity probability threshold to obtain the third judgment result; If the third judgment result is yes, then the two batches of target detection request frames with a second similarity probability greater than the similarity probability threshold are merged into one batch of target detection request frames to obtain multiple updated batches of target detection request frames. The multiple batches of detection request frames are replaced with the updated multiple batches of target detection request frames, and the function "determine the similarity probability between any two batches of target detection request frames as the second similarity probability" is returned until all second similarity probabilities are less than or equal to the similarity probability threshold, and multiple updated batches of target detection request frames are obtained as the target batch of target detection request frames. If the third judgment result is negative, then all batches of target detection request frames are identified as target batches of target detection request frames; All target batches of multiple target detection request frames are sorted according to timestamps to obtain multiple detection request sequences.

6. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal pedestrian re-identification method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the multimodal pedestrian re-identification method as described in any one of claims 1 to 4.

Images