A cross-border tracking method based on multimodality
Through the multimodal cross-border tracking method, combined with visible light and infrared images, the yolov5 target detection and pedestrian re-identification model is used to solve the problem of low recognition accuracy and low computing efficiency of cross-modular pedestrian re-identification at night or when there is insufficient light. , achieving higher recognition accuracy and efficiency.
Patent Information
- Application Number
- CN202111025677.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-02
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-09-02
AI Technical Summary
The existing cross-modal pedestrian re-identification technology has low recognition accuracy at night or when there is insufficient light, low computing efficiency, and complex network structure, making it difficult to achieve good results in practical application scenarios.
By using multimodal cross-border tracking method, by obtaining visible light and infrared images, using the yolov5 target detection algorithm to detect pedestrians, construct a pedestrian re-identification model, set a similarity threshold, perform feature extraction and similarity measurement, and combine modal weight sorting, Improve recognition accuracy.
Improve the accuracy and calculation efficiency of cross-modal pedestrian re-identification, reduce the missed detection rate of the same suspect, enhance the diversity of features and discrimination, optimize the output of network results, and improve the accuracy and recall of Top-n Rate.
Smart Images

Figure CN113837024B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a multimodal cross-border tracking method. Background Art
[0002] Traditional RGB-RGB single-mode pedestrian re-identification technology can only solve the recognition task under sufficient light, while visible light cameras are almost useless at night or in dim light, and criminals usually move around at night. Most of today's surveillance cameras have infrared and visible light functions configured together. When there is sufficient light, clear RGB images will be collected, and when there is insufficient light, the infrared function will be turned on to collect infrared images, which provides favorable conditions for the research of cross-modal pedestrian re-identification.
[0003] Pedestrian re-identification is a hot research topic in the field of computer vision. It mainly solves the problem of pedestrian recognition and retrieval across cameras and scenes. As a supplement to face recognition technology, it continuously tracks pedestrians whose faces cannot be clearly seen across cameras. It is widely used in security and intelligent monitoring. It has difficulties such as large intra-class differences (the appearance of the same person may be very different) and small inter-class differences (the appearance of different people may be very similar). This is mainly due to factors such as camera shooting angle, lighting differences, pedestrian posture changes, and occlusion. Night scenes are also a key area in monitoring, security and other fields.
[0004] Most of the existing cross-modal pedestrian re-identification methods based on visible light and infrared images are dedicated to solving the differences between the two modalities. The solutions include: 1) using parameter-sharing convolutional neural networks to learn the shared features between the two modalities; 2) using generative adversarial networks to learn the association between the modalities by training the generator and the discriminator. However, most of the existing technologies are based on innovations in network structure, and often do not consider more specific problems and challenges of cross-modal pedestrian re-identification in actual application scenarios, which increases the complexity of the network structure and the training time cost, and it is difficult to achieve good recognition accuracy and effect in actual application scenarios.
[0005] A multimodal cross-border tracking method is needed to solve the above problems. Summary of the invention
[0006] The present invention aims to solve the problem that the current cross-modal pedestrian re-identification technology in the prior art aims to eliminate the difference between the two modalities, but it will lead to complex network structure, low computational efficiency and poor recognition effect. A cross-border tracking method based on multimodality is provided, which solves the problems of low accuracy, poor effect and low computational efficiency in cross-modal pedestrian re-identification in actual application scenarios by allowing the difference between the two modalities of visible light and infrared images to be retained.
[0007] The present invention provides a cross-border tracking method based on multimodality, comprising the following steps:
[0008] S1, obtain visible light image and infrared image of the same target;
[0009] S2. Use the yolov5 target detection algorithm to detect pedestrians in videos taken by different cameras, and extract them to generate an image set pedestrian library;
[0010] S3, build a pedestrian re-identification model;
[0011] S4, setting a similarity threshold a;
[0012] S5, extracting features from the visible light image of the target to be queried, the infrared image of the target to be queried, and the feature vectors in the pedestrian database of the image set;
[0013] S6, performing similarity measurement on the visible light image of the target to be queried, the infrared image of the target to be queried and the feature vector in the pedestrian database of the image set;
[0014] S7, obtaining the comprehensive similarity of the visible light modality according to the similarity of the results returned by the visible light modality and the preset weight of the visible light modality; obtaining the comprehensive similarity of the infrared modality according to the similarity of the infrared modality and the preset weight of the infrared;
[0015] S8. Intersect the results of the two modalities and compare the IOU, and sort them in descending order of comprehensive similarity;
[0016] S9. Remove the same pictures according to the camera ID and picture name to obtain the final result.
[0017] In the cross-border tracking method based on multimodality described in the present invention, as a preferred embodiment, the similarity measurement between the visible light image of the target to be queried and the feature vector in the pedestrian library of the image set in step S6 is specifically:
[0018] For the visible light image of the target to be queried, the pedestrian library of the image set is used as the search space, and the feature vector of the visible light image of the target to be queried is measured for similarity with the feature vector in the pedestrian library of the image set, as follows:
[0019] d s (q RGB ,g j ) = d (k) (q RGB ,g j )
[0020] Among them, d s (q RGB ,g j) represents the K similarity distance metrics between the target visible light modality image and the pedestrian database in the image set; q RGB g represents the RGB modality image of the target query; j Represents the pedestrian gallery feature library;
[0021] Returns results that are greater than the set threshold a, as follows:
[0022]
[0023] Among them, ψ RGB (q RGB ,k q ,d s ,th RGB ) indicates the top_k values greater than the set threshold q Results; q RGB represents the query image in the target RGB mode; k q Indicates the top_k in the result q Results;d s Indicates distance similarity; RGB Indicates the threshold set in RGB mode.
[0024] In the cross-border tracking method based on multimodality described in the present invention, as a preferred embodiment, the similarity measurement between the infrared image of the target to be queried and the feature vector in the pedestrian library of the image set in step S6 is specifically:
[0025] For the infrared image of the target to be queried, the pedestrian library of the image set is used as the search space, and the feature vector of the infrared image of the target to be queried is measured for similarity with the feature vector in the pedestrian library of the image set, as follows:
[0026] d s (q IR ,g j ) = d (k) (q IR ,g j ),
[0027] Among them, d s (q IR ,g j ) represents the distance similarity between the target infrared IR modal image and the Kth pedestrian in the gallery pedestrian library; q IR g represents the IR modality image of the target query; j Represents the pedestrian gallery feature library;
[0028] Returns results that are greater than the set threshold a, as follows:
[0029]
[0030] Among them, ψ IR (q IR ,k q ,d s ,th IR ) indicates the top_k values greater than the set threshold q Results; q IR represents the query image in the target IR infrared mode; k q Indicates the top_k in the result q Results;d s Indicates distance similarity; IR Indicates the threshold set in IR mode.
[0031] In the cross-border tracking method based on multimodality described in the present invention, as a preferred embodiment, the similarity of the visible light modality return result and the preset weight of the visible light modality in step S7 to obtain the comprehensive similarity of the visible light modality is specifically:
[0032]
[0033] Among them, α represents the preset weight under RGB mode; Ψ IR (q IR ,k q ,d s ,th IR ) represents the weighted comprehensive similarity in IR mode.
[0034] The cross-border tracking method based on multimodality described in the present invention is, as a preferred embodiment, the infrared modality similarity and the infrared preset weight in step S7 are used to obtain the infrared modality comprehensive similarity:
[0035]
[0036] Where, β represents the preset weight in IR mode; Ψ RGB (q RGB ,k q ,d s ,th RGB ) represents the weighted comprehensive similarity under RGB mode.
[0037] The multimodal cross-border tracking method described in the present invention is, as a preferred method, a pedestrian re-identification model is used to perform feature extraction and feature discrimination on both visible light and infrared image modalities.
[0038] The cross-border tracking method based on multimodality described in the present invention is, as a preferred method, in step S8, the results of the two modes are intersected and compared with IOU, and sorted in descending order of comprehensive similarity. The specific formula is as follows:
[0039]
[0040] Among them, D s It represents the distance similarity of a set of results returned after the intersection and comparison of the results of the two modes and then sorting them in descending order of comprehensive similarity.
[0041] The beneficial effects of the present invention are as follows:
[0042] (1) This technical solution obtains two modalities, namely, visible light image and infrared image of the query target image as input data sources, and can obtain corresponding modal data by converting the modalities, thereby increasing the diversity and discriminability of the input data sources;
[0043] (2) It allows the differences between different modal images in the gallery to be preserved;
[0044] (3) Design a ReID model with strong feature extraction and discrimination capabilities for both visible light and infrared modalities, and extract feature vectors from multimodal data;
[0045] (4) The two modes of the query target are searched in the gallery pedestrian database respectively, and the results with similarity close to the own mode are returned, which reduces the missed detection rate of the same suspect target;
[0046] (5) The returned results of the two modalities are sorted according to the preset weights and comprehensive similarities, and the final results are returned, which improves the recognition accuracy of cross-modal pedestrian re-identification;
[0047] (6) This technical solution improves the accuracy of pedestrian re-identification without increasing the complexity of the network structure and additional computing efficiency;
[0048] (7) This technical solution avoids the complexity of the network structure and the difference between modalities caused by forcibly mapping different modalities to the same feature space. By retrieving different modalities of the image to be retrieved separately, the diversity and discriminability of the features are increased, thereby improving the performance of cross-modal pedestrian re-identification;
[0049] (8) This technical solution optimizes the output of network results by using preset weights and comprehensive similarity sorting, which can effectively improve the accuracy and recall rate of Top-n. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 Schematic diagram of a cross-border tracking method based on multimodality. DETAILED DESCRIPTION
[0051] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0052] Example 1
[0053] like Figure 1 As shown, a multimodal cross-border tracking method includes the following steps:
[0054] S1. Obtain two modal data of visible light image and infrared image of the query under the same target ID. For example, the two modal images of the suspect target can be obtained by browsing the summary results through the video summary module in the cross-border tracking system; if only one modal image can be obtained, the modal conversion method can be used to obtain the image of the other modal corresponding to the target;
[0055] S2. Use the yolov5 target detection algorithm to detect pedestrians in videos taken by different cameras, and extract them to generate an image set pedestrian library;
[0056] S3. Build a pedestrian re-identification (ReID) model. The model has been trained to have strong feature extraction and discrimination capabilities. It extracts features from the query data and the gallery pedestrian database. The query is The feature vector of gallery is n*2048 feature vector;
[0057] S4, setting a similarity threshold a;
[0058] S5, extracting features from the visible light image of the target to be queried, the infrared image of the target to be queried, and the feature vectors in the pedestrian database of the image set;
[0059] S6, performing similarity measurement on the visible light image of the target to be queried, the infrared image of the target to be queried and the feature vector in the pedestrian database of the image set;
[0060] S7, obtaining the comprehensive similarity of the visible light modality according to the similarity of the results returned by the visible light modality and the preset weight of the visible light modality; obtaining the comprehensive similarity of the infrared modality according to the similarity of the infrared modality and the preset weight of the infrared;
[0061] S8. Intersect the results of the two modalities and compare the IOU, and sort them in descending order of comprehensive similarity;
[0062] S9. Because the visible light RGB mode and the infrared IR mode may retrieve and return the same image at the same time, the same images are removed according to the camera ID and image name to obtain the final result.
[0063] In step S6, the similarity measurement between the visible light image of the target to be queried and the feature vector in the pedestrian database of the image set is specifically:
[0064] For the visible light image of the target to be queried, the pedestrian library of the image set is used as the search space, and the feature vector of the visible light image of the target to be queried is measured for similarity with the feature vector in the pedestrian library of the image set, as follows:
[0065] d s (q RGB ,g j ) = d (k) (q RGB ,g j )
[0066] Among them, d s (q RGB ,g j ) represents the K similarity distance metrics between the target visible light modality image and the pedestrian database in the image set; q RGB g represents the RGB modality image of the target query; j Represents the pedestrian gallery feature library;
[0067] Returns results that are greater than the set threshold a, as follows:
[0068]
[0069] Among them, ψ RGB (q RGB ,k q ,d s ,th RGB ) indicates the top_k values greater than the set threshold q Results; q RGB represents the query image in the target RGB mode; k q Indicates the top_k in the result q Results;d s Indicates distance similarity; RGB Indicates the threshold set in RGB mode
[0070] In step S6, the similarity measurement between the infrared image of the target to be queried and the feature vector in the pedestrian library of the image set is specifically as follows:
[0071] For the infrared image of the target to be queried, the pedestrian library of the image set is used as the search space, and the feature vector of the infrared image of the target to be queried is measured for similarity with the feature vector in the pedestrian library of the image set, as follows:
[0072] d s (q IR ,g j ) = d (k) (q IR ,g j )
[0073] Among them, d s (q IR ,g j ) represents the distance similarity between the target infrared IR modal image and the Kth pedestrian in the gallery pedestrian library; q IR g represents the IR modality image of the target query; j Represents the pedestrian gallery feature library;
[0074] Returns results that are greater than the set threshold a, as follows:
[0075]
[0076] Among them, ψ IR (q IR ,k q ,d s ,th IR ) indicates the top_k values greater than the set threshold q Results; q IR represents the query image in the target IR infrared mode; k q Indicates the top_k in the result q Results; d s Indicates distance similarity; IR Indicates the threshold set in IR mode.
[0077] The similarity of the visible light modality return result in step S7 and the preset weight of the visible light modality to obtain the comprehensive similarity of the visible light modality is specifically:
[0078]
[0079] Wherein, α represents the preset weight under RGB mode; IR (q IR ,k q ,d s ,th IR ) represents the weighted comprehensive similarity in IR mode.
[0080] In step S7, the infrared modal similarity and the infrared preset weight are combined to obtain the infrared modal comprehensive similarity:
[0081]
[0082] Where, β represents the preset weight in IR mode; Ψ RGB (q RGB ,k q ,d s ,th RGB ) represents the weighted comprehensive similarity under RGB mode.
[0083] The pedestrian re-identification model is used to perform feature extraction and feature discrimination on both visible light and infrared image modalities.
[0084] Step S8: The results of the two modes are compared by IOU and sorted in descending order of comprehensive similarity. The specific formula is as follows:
[0085]
[0086] Among them, D s It represents the distance similarity of a set of results returned after the intersection and comparison of the results of the two modes and then sorting them in descending order of comprehensive similarity.
[0087] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A multimodal cross-border tracking method, characterized by: The following steps are involved: S1, obtain visible light image and infrared image of the same target; S2. Use the yolov5 target detection algorithm to detect pedestrians in videos taken by different cameras, and extract them to generate an image set pedestrian library; S3, build a pedestrian re-identification model; S4, setting a similarity threshold a; S5, extracting features from the visible light image of the target to be queried, the infrared image of the target to be queried, and the feature vectors in the pedestrian database of the image set; S6, performing similarity measurement on the visible light image of the target to be queried, the infrared image of the target to be queried, and the feature vectors in the pedestrian database of the image set; S7, obtaining the comprehensive similarity of the visible light modality according to the similarity of the results returned by the visible light modality and the preset weight of the visible light modality; obtaining the comprehensive similarity of the infrared modality according to the similarity of the infrared modality and the preset weight of the infrared; S8. Intersect the results of the two modalities and compare the IOU, and sort them in descending order of comprehensive similarity; S9. Remove the same pictures according to the camera ID and picture name to obtain the final result.
2. The multimodal cross-border tracking method according to claim 1, characterized in that: The similarity measurement between the visible light image of the target to be queried and the feature vector in the pedestrian database of the image set in step S6 is specifically as follows: For the visible light image of the target to be queried, the pedestrian library of the image set is used as the search space, and the feature vector of the visible light image of the target to be queried is measured for similarity with the feature vector in the pedestrian library of the image set, as follows: d s (q RGB ,g j )=d (k) (q RGB ,g j ) Among them, the d s (q RGB ,g j ) represents the K similarity distance metrics between the target visible light modality image and the pedestrian database of the image set; the q RGB Represents the RGB modality image of the target query; the g j Represents the pedestrian gallery feature library; Returns results that are greater than the set threshold a, as follows: Among them, the ψ RGB (q RGB ,k q ,d s ,th RGB ) indicates the top_k values greater than the set threshold q Results; the q RGB represents the query image in the target RGB mode; the k q Indicates the top_k in the result q Results; the s Represents distance similarity; RGB Indicates the threshold set in RGB mode.
3. The multimodal cross-border tracking method according to claim 1, characterized in that: The similarity measurement between the infrared image of the target to be queried and the feature vector in the pedestrian database of the image set in step S6 is specifically as follows: For the infrared image of the target to be queried, the pedestrian library of the image set is used as the search space, and the feature vector of the infrared image of the target to be queried is measured for similarity with the feature vector in the pedestrian library of the image set, as follows: d s (q IR ,g j )=d (k) (q IR ,g j ) Among them, the d s (q IR ,g j ) represents the distance similarity between the target infrared IR modal image and the Kth pedestrian in the gallery pedestrian library; the q IR represents the IR infrared modality image of the target query; the g j Represents the pedestrian gallery feature library; Returns results that are greater than the set threshold a, as follows: Among them, the ψ IR (q IR ,k q ,d s ,th IR ) indicates the top_k values greater than the set threshold q Results; the q IR represents the query image to be queried in the target IR infrared mode; the k q Indicates the top_k in the result q Results; the s Represents distance similarity; IR Indicates the threshold set in IR mode.
4. The multimodal cross-border tracking method according to claim 2, characterized in that: The similarity of the visible light modality returned in step S7 and the preset weight of the visible light modality to obtain the comprehensive similarity of the visible light modality is specifically: Wherein, α represents the preset weight under RGB mode; RGB (q RGB ,k q ,d s ,th RGB ) represents the weighted comprehensive similarity under RGB mode.
5. The multimodal cross-border tracking method according to claim 4, characterized in that: In step S7, the infrared modal similarity and the infrared preset weight are used to obtain the infrared modal comprehensive similarity, which is specifically: Wherein, β represents the preset weight in IR infrared mode; IR (q IR ,k q ,d s ,th IR ) represents the weighted comprehensive similarity in IR mode.
6. The multimodal cross-border tracking method according to claim 1, characterized in that: The pedestrian re-identification model is used to perform feature extraction and feature discrimination on both visible light and infrared image modalities.
7. The multimodal cross-border tracking method according to claim 5, characterized in that: In step S8, the results of the two modes are compared by IOU and sorted in descending order of comprehensive similarity. The specific formula is as follows: Wherein D s It represents the distance similarity of a set of results returned after the intersection and comparison of the results of the two modes and then sorting them in descending order of comprehensive similarity.
