Cross-camera pedestrian re-identification method, system, equipment and medium
Through the complementary multi-scale view fusion perceived feature extraction network, the problem of poor recognition effect and low efficiency of pedestrian re-identification in cross-camera environment is solved, and more accurate pedestrian recognition is achieved, especially in different view angles and appearance differences are large. Down.
Patent Information
- Application Number
- CN202411279322.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-27
AI Technical Summary
The existing pedestrian re-identification technology is affected by factors such as facial occlusion, shooting angle, and brightness, resulting in poor recognition effect and low recognition efficiency. Especially in cross-camera environments, the appearance of the same pedestrian is significantly different, and the complex monitoring scenes lead to background noise and occlusion problems. serious.
A feature extraction network with complementary multi-scale view fusion perception is adopted to generate pedestrian feature vectors through human pose estimation, image segmentation, and multi-scale feature extraction and fusion, and match them with the pre-constructed pedestrian feature library to improve the accuracy of pedestrian re-identification and stability.
It significantly improves the accuracy and stability of the pedestrian re-identification system, and can more accurately distinguish visually similar but actually different pedestrians, as well as the same pedestrians from different perspectives, improving overall performance.
Smart Images
Figure CN119206784A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of pedestrian re-identification, and in particular to a method, system, device and medium for cross-camera pedestrian re-identification. Background Art
[0002] Current pedestrian re-identification technology is subject to poor recognition results and low recognition efficiency due to factors such as facial occlusion, shooting angle, and brightness. Influencing factors may include: 1) Due to the influence of camera parameters such as viewing angle, resolution, and shooting distance, the resolution of pedestrians in surveillance images is generally low, and the scale and posture of the same pedestrian under different devices vary greatly; 2) The appearance of the same pedestrian captured by different cameras varies significantly (such as changing clothes); 3) Different pedestrians may have visually similar shapes such as clothes, which may lead to recognition errors.
[0003] In addition, due to the wide monitoring field of view of the camera, a large amount of background noise and irrelevant information is often introduced into the monitoring image; most importantly, the monitoring scene is complex, and pedestrians can easily block each other or key information can be covered. Summary of the invention
[0004] The purpose of this application is to provide a cross-camera pedestrian re-identification method, system, device and medium to alleviate the above-mentioned technical problems existing in the prior art.
[0005] In a first aspect, the present invention provides a method for cross-camera pedestrian re-identification, the method comprising:
[0006] In response to the pedestrian image to be queried provided by the user, human body posture estimation processing is performed on the pedestrian image to obtain pedestrian key points;
[0007] Perform image segmentation on the query pedestrian image based on the pedestrian key points to obtain multiple segmented area images;
[0008] The feature extraction network of complementary multi-scale view fusion perception is used to extract features from multiple regional images to obtain regional feature vectors, and multi-scale pedestrian feature fusion is performed based on the regional feature vectors to generate a query pedestrian feature vector; wherein the feature extraction network of complementary multi-scale view fusion perception includes a pedestrian upper body feature extraction network, a pedestrian lower body feature extraction network, a pedestrian full body feature extraction network and a pedestrian feature fusion network;
[0009] The query pedestrian feature vector is matched with the pedestrian features in the pre-built pedestrian feature library to determine the target pedestrian that matches the query pedestrian image.
[0010] In an optional implementation, feature extraction is performed on multiple regional images through a complementary multi-scale view fusion perception feature extraction network to obtain a regional feature vector, including:
[0011] The pedestrian full-body feature extraction network uses a graph convolutional network as the network backbone. The regional image is input into the network backbone of the pedestrian full-body feature extraction network, and the feature information output by the network backbone is input into a global maximum pooling layer and a global average pooling layer to extract the pedestrian full-body features.
[0012] The pedestrian orientation feature extraction network includes a convolution structure of a fully connected layer, a random forest, and a preset convolution kernel. The regional image is input into the pedestrian orientation feature extraction network to obtain pedestrian orientation information.
[0013] FCOS is used as the network backbone in the pedestrian upper body feature extraction network and the pedestrian lower body feature extraction network, and a KAN network is added to each residual module in the FCOS network to extract local key point information from the pedestrian upper body features and lower body features respectively, and map the local key point information to the key point coordinate space through the pooling layer.
[0014] In an optional implementation, multi-scale pedestrian feature fusion is performed based on the regional feature vector to generate a query pedestrian feature vector, including:
[0015] The high-level features output from the pedestrian orientation feature extraction network, the upper body feature extraction network, the pedestrian lower body feature extraction network and the pedestrian full body feature extraction network are complementary fused to generate a query pedestrian feature vector.
[0016] In an optional implementation, high-level features output from the pedestrian orientation feature extraction network, the upper body feature extraction network, the pedestrian lower body feature extraction network, and the pedestrian full body feature extraction network are complementary feature fused to generate a query pedestrian feature vector, including:
[0017] Complementary features are fused using the following formula:
[0018]
[0019] Where W×H represents the size of the feature map, C1, C2, C3, and C4 represent the number of channels of high-level features output from the pedestrian orientation feature extraction network, the pedestrian upper body feature extraction network, the pedestrian lower body feature extraction network, and the pedestrian full body feature extraction network, respectively; Concat(·) represents the concatenation operation; and l is the weight coefficient.
[0020] In an optional implementation, the construction of the pre-built pedestrian feature library includes:
[0021] A pedestrian image video stream is obtained through multiple cameras, and then the pedestrian image video stream is decoded and framed, and filtering, noise reduction and cropping preprocessing operations are performed to obtain a preprocessed pedestrian image;
[0022] The pedestrian region in the preprocessed pedestrian image is extracted, and features are extracted through a complementary multi-scale view fusion-aware feature extraction network to obtain a pedestrian feature vector;
[0023] The pedestrian feature vector is stored in a pre-built pedestrian feature library.
[0024] In an optional implementation, matching the query pedestrian feature vector with pedestrian features in a pre-built pedestrian feature library to determine a target pedestrian that matches the pedestrian image to be queried includes:
[0025] The query pedestrian feature vector is matched with the pedestrian features in the pre-built pedestrian feature library to find the matching feature vector with the highest similarity, and the matching feature vector corresponding to the matching feature vector is determined as the target pedestrian that matches the query pedestrian image.
[0026] In an optional embodiment, the method further comprises:
[0027] After obtaining the pedestrian key points, the visibility of the pedestrian key points in the current viewing angle is determined based on the pedestrian's orientation and posture, so as to perform image segmentation when the visibility indicates that all the pedestrian key points are visible.
[0028] In a second aspect, the present invention provides a cross-camera pedestrian re-identification system, the system comprising:
[0029] A human posture estimation module is used to perform human posture estimation processing on the pedestrian image to be queried in response to the pedestrian image to be queried provided by the user, and obtain the pedestrian key points;
[0030] An image segmentation module is used to perform image segmentation on the query pedestrian image based on the pedestrian key points to obtain multiple segmented area images;
[0031] A feature extraction and fusion module is used to extract features from multiple regional images through a feature extraction network of complementary multi-scale view fusion perception to obtain regional feature vectors, and to perform multi-scale pedestrian feature fusion based on the regional feature vectors to generate a query pedestrian feature vector; wherein the feature extraction network of complementary multi-scale view fusion perception includes a pedestrian upper body feature extraction network, a pedestrian lower body feature extraction network, a pedestrian full body feature extraction network and a pedestrian feature fusion network;
[0032] The feature matching module is used to match the query pedestrian feature vector with the pedestrian features in the pre-built pedestrian feature library to determine the target pedestrian that matches the pedestrian image to be queried.
[0033] In a third aspect, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the cross-camera pedestrian re-identification method of any one of the aforementioned implementations.
[0034] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the cross-camera pedestrian re-identification method of any one of the aforementioned implementations.
[0035] The cross-camera pedestrian re-identification method, system, device and medium provided in this application have the following beneficial effects:
[0036] By constructing a complementary multi-scale view fusion-aware feature extraction network (the network includes pedestrian orientation feature extraction, pedestrian upper body feature extraction, pedestrian lower body feature extraction, pedestrian full body feature extraction and pedestrian feature fusion network), the feature extraction network is used to extract pedestrian features from multiple angles. By using information from different scales and perspectives, through multi-level feature extraction and fusion, the system can more accurately distinguish between visually similar but actually different pedestrians, as well as the features of the same pedestrian at different perspectives, thereby significantly improving the accuracy and stability of the pedestrian re-identification system. It can more accurately distinguish between visually similar but actually different pedestrians, as well as the same pedestrian at different perspectives, thereby significantly improving the overall performance of pedestrian re-identification. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0038] Figure 1 A flowchart of a cross-camera pedestrian re-identification method provided in an embodiment of the present application;
[0039] Figure 2 A flowchart of a specific cross-camera pedestrian re-identification method provided in an embodiment of the present application;
[0040] Figure 3 A schematic diagram of pedestrian feature extraction and fusion provided in an embodiment of the present application;
[0041] Figure 4 A structural diagram of a cross-camera pedestrian re-identification system provided in an embodiment of the present application;
[0042] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0044] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0045] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0046] In recent years, with the in-depth promotion of the concept of "safe city", more and more attention has been paid to the development of the security field. As a key means of security work, the construction of video surveillance network is an important tool for protecting people's lives and property, maintaining social order and stability, and preventing terrorist activities. In the massive amount of unstructured video data, there is a large amount of pedestrian identity, behavior and trajectory information to be mined. As the planner and manipulator of abnormal events / activities, pedestrian targets are key objects for investigation and evidence collection by government departments. Therefore, in the actual evidence collection process, locking the identity information of the target person based on the surveillance video of the crime scene is an important auxiliary means for accurately judging the case.
[0047] However, traditional evidence collection methods mainly rely on manual offline browsing and checking of surveillance videos. Although they have achieved certain results, there are still many problems: 1) Evidence collection is difficult and the effect is poor. On the one hand, with the intelligence and concealment of criminal methods, target persons often hide their appearance characteristics through disguise (such as wearing masks); on the other hand, due to differences in camera models and climatic conditions, the resolution of surveillance images is often low, and criminal suspects are often obscured or their movements are changeable. These factors make it extremely difficult to identify suspected targets. 2) Evidence collection costs are high and efficiency is low. Therefore, how to use image processing technology to "re-identify" specific pedestrian targets in surveillance images has become an important issue that needs to be urgently solved in the field of public security.
[0048] Based on the above problems, when face recognition technology fails, pedestrian re-identification technology comes into being and becomes an effective alternative. However, unlike face acquisition, the camera has a lot of randomness when acquiring human images, and the quality of the pedestrian images obtained is limited and difficult to use directly. It is mainly manifested in the following two aspects: 1. Affected by camera parameters such as viewing angle, resolution and shooting distance, the resolution of pedestrians in monitoring images is generally low, and the scale and posture of the same pedestrian under different devices vary greatly; 2. The appearance of the same pedestrian captured by different cameras is significantly different (such as changing clothes); in addition, due to the wide monitoring field of the camera, a large amount of background noise and irrelevant information are often introduced into the monitoring image; most importantly, the monitoring scene is complex, and pedestrians are easy to block each other or key information is covered.
[0049] Based on this, the embodiments of the present application provide a cross-camera pedestrian re-identification method, system, device and medium, which can more accurately distinguish between visually similar but actually different pedestrians, as well as the same pedestrian under different viewing angles, thereby significantly improving the overall performance of pedestrian re-identification. In the images of cross-camera devices, when the images of pedestrians change dramatically in scale, posture, resolution, etc., or when there is occlusion in the pedestrian images, the accuracy and robustness of the pedestrian re-identification algorithm are improved.
[0050] This application embodiment provides a method for cross-camera pedestrian re-identification. Figure 1 As shown, the method mainly includes the following steps:
[0051] Step S110 , in response to the pedestrian image to be queried provided by the user, human body posture estimation processing is performed on the pedestrian image to be queried to obtain pedestrian key points.
[0052] In one embodiment, the pedestrian key points may include eight key points, namely, the top of the head, shoulders, hands, legs, and waist, for marking the key points of the pedestrian. The above pedestrian key points may be marked by marking coordinates so as to accurately identify the key points by coordinates.
[0053] In an optional implementation, after obtaining the pedestrian key points, the visibility of the pedestrian key points at the current viewing angle is determined based on the pedestrian's orientation and posture, so as to perform image segmentation when the visibility indicates that all the pedestrian key points are visible.
[0054] Step S120 , performing image segmentation on the query pedestrian image based on the pedestrian key points to obtain a plurality of segmented region images.
[0055] The queried pedestrian image is segmented based on pedestrian key points, and the obtained multiple segmented region images may include an image of the upper body (from the top of the head to the waist) and an image of the lower body (from the waist to the feet) obtained by segmenting the pedestrian image.
[0056] Step S130 , extracting features from the multiple regional images through a complementary multi-scale view fusion perception feature extraction network to obtain regional feature vectors, and performing multi-scale pedestrian feature fusion based on the regional feature vectors to generate a query pedestrian feature vector.
[0057] In one embodiment, the complementary multi-scale view fusion perception feature extraction network includes a pedestrian upper body feature extraction network, a pedestrian lower body feature extraction network, a pedestrian full body feature extraction network and a pedestrian feature fusion network.
[0058] When performing feature extraction, the image of the upper body (from the top of the head to the waist), the image of the lower body (from the waist to the feet), and the overall unsegmented pedestrian image can be put together and input into the above-mentioned complementary multi-scale view fusion perception feature extraction network.
[0059] The feature extraction network through complementary multi-scale view fusion perception extracts features from multiple regional images to obtain regional feature vectors, which can include:
[0060] 1. The pedestrian full-body feature extraction network uses a graph convolutional network as the network backbone, inputs the regional image into the network backbone of the pedestrian full-body feature extraction network, and inputs the feature information output by the network backbone into a global maximum pooling layer and a global average pooling layer to extract the full-body features of the pedestrian;
[0061] 2. The pedestrian orientation feature extraction network includes a convolution structure of a fully connected layer, a random forest and a preset convolution kernel. The regional image is input into the pedestrian orientation feature extraction network to obtain pedestrian orientation information. The preset convolution kernel can be set according to actual conditions. In one embodiment, the convolution kernel size includes a 1*1 convolution structure.
[0062] 3. FCOS is used as the network backbone in the pedestrian upper body feature extraction network and the pedestrian lower body feature extraction network, and a KAN network is added to each residual module in the FCOS network to extract local key point information from the pedestrian upper body features and lower body features respectively, and map the local key point information to the key point coordinate space through the pooling layer.
[0063] Furthermore, the above-mentioned multi-scale pedestrian feature fusion based on the regional feature vector generates a query pedestrian feature vector, and the high-level features output from the pedestrian orientation feature extraction network, the upper body feature extraction network, the pedestrian lower body feature extraction network, and the pedestrian full body feature extraction network can be complementary feature fused to generate a query pedestrian feature vector. Specifically, the high-level features output from the pedestrian orientation feature extraction network, the upper body feature extraction network, the pedestrian lower body feature extraction network, and the pedestrian full body feature extraction network are complementary feature fused to generate a query pedestrian feature vector. The complementary feature fusion can be performed by the following formula:
[0064]
[0065] Where W×H represents the size of the feature map, C1, C2, C3, and C4 represent the number of channels of high-level features output from the pedestrian orientation feature extraction network, the pedestrian upper body feature extraction network, the pedestrian lower body feature extraction network, and the pedestrian full body feature extraction network, respectively; Concat(·) represents the concatenation operation; and l is the weight coefficient.
[0066] Step S140 , matching the query pedestrian feature vector with pedestrian features in a pre-built pedestrian feature library to determine a target pedestrian that matches the pedestrian image to be queried.
[0067] In an optional embodiment, the pre-constructed pedestrian feature library is constructed by acquiring a pedestrian image video stream through multiple cameras, then decoding the pedestrian image video stream to extract frames, and performing filtering, noise reduction and cropping preprocessing operations to obtain a pre-processed pedestrian image; extracting the pedestrian area in the pre-processed pedestrian image, and performing feature extraction through a feature extraction network of complementary multi-scale view fusion perception to obtain a pedestrian feature vector; and storing the pedestrian feature vector in the pre-constructed pedestrian feature library.
[0068] The feature extraction used in constructing the pedestrian feature library, that is, feature extraction through a feature extraction network of complementary multi-scale view fusion perception to obtain a pedestrian feature vector, is the same as the network structure and extraction method in the aforementioned step S130, and will not be repeated here.
[0069] In one embodiment, the query pedestrian feature vector is matched with the pedestrian features in a pre-built pedestrian feature library to determine the target pedestrian that matches the pedestrian image to be queried. The query pedestrian feature vector can be matched with the pedestrian features in the pre-built pedestrian feature library to find the matching feature vector with the highest similarity, and the matching feature vector corresponding to the matching feature vector is determined as the target pedestrian that matches the pedestrian image to be queried.
[0070] Figure 2A flowchart of a specific cross-camera pedestrian re-identification method is shown, which mainly includes the following steps S1 to S6:
[0071] Step S1, using multiple cameras (camera 1, camera 2, ..., camera n) to capture pedestrian images, then decoding the video stream to extract frames, and then performing pre-processing operations such as filtering, noise reduction, and cropping, and transmitting it to the pedestrian detection module;
[0072] Step S2, in the pedestrian detection module, a pedestrian detection algorithm is used to perform pedestrian detection on the image obtained from the image preprocessing module and the detected pedestrians are cut out and then transmitted to the pedestrian feature extraction and fusion module;
[0073] Step S3, in the pedestrian feature extraction and fusion module, the pedestrian feature extraction algorithm proposed in the present invention is used to extract pedestrian features and the extracted pedestrian features are stored in a pedestrian feature library as a reference for subsequent query and matching.
[0074] Step S4: The user provides a pedestrian image to be queried, and features are extracted from the query image to generate a query feature vector.
[0075] Step S5, matching the query feature vector with the features in the pedestrian feature library to find the feature vector with the highest similarity.
[0076] Step S6: output the query result based on the matching result, and display the pedestrian that is most similar to the query image.
[0077] The core part of the embodiment of the present application is mainly in the pedestrian feature extraction and fusion module part, see Figure 3 As shown, the specific process is as follows:
[0078] 1. Key point detection
[0079] In the pedestrian key point detection module, the present invention first detects the eight key points of the pedestrian's head, shoulders, hands, legs, and waist designed by the present invention through a human posture estimation algorithm, and then uses information such as the pedestrian's orientation and posture to determine whether the above key points are visible under the current viewing angle, and uses the serial numbers of the visible key points to find the corresponding coordinates.
[0080] 2. Pedestrian Image Segmentation
[0081] In the pedestrian image segmentation module, the present invention segments the pedestrian image based on the coordinates of the visible key points, and then puts the image of the upper body (from the top of the head to the waist), the image of the lower body (from the waist to the feet), and the overall unsegmented pedestrian image together and inputs them into the multi-scale pedestrian feature extraction and fusion module.
[0082] 3. Multi-scale pedestrian feature extraction
[0083] In the multi-scale pedestrian feature extraction and fusion module, the present invention designs four branches for pedestrian feature extraction, which mainly include:
[0084] The pedestrian orientation feature extraction network and the pedestrian upper body feature extraction network are used to extract upper body features, mainly used to extract the pedestrian's orientation and high-level feature information of the upper body; the pedestrian lower body feature extraction network is used to extract high-level feature information of the lower body, focusing on richer detail information; the pedestrian full body feature extraction network is used to extract the global features of the entire image and the global relationship between the two parts of the image after comprehensive segmentation.
[0085] (1) The present invention adopts a graph convolutional network as the backbone of the pedestrian full-body feature extraction network, and inputs the feature information output by the backbone into a global maximum pooling layer and a global average pooling layer to extract the most obvious features of the pedestrian's whole body. The network improves the utilization rate of the pedestrian's full-body high-level feature information, enhances the scale invariance of the feature map, alleviates the separation of contextual information in the feature map, and effectively improves the accuracy of pedestrian re-identification under blurry and occluded conditions.
[0086] (2) Pedestrian Orientation Feature Extraction Network The present invention designs a fully connected layer + random forest + 1X1 convolution structure to map the pedestrian's orientation information to four directions: front, back, left, and right.
[0087] (3) FCOS is used as the network backbone in the pedestrian upper body feature extraction network and the pedestrian lower body feature extraction network, and a KAN network is added to each residual module in the FCOS network. Local key point information can be extracted from the pedestrian upper body features and lower body features, and mapped to the key point coordinate space through the pooling layer, which effectively improves the accuracy of re-identification when the pedestrian image changes drastically in terms of scale, posture, resolution, etc. The calculation method is:
[0088] F * =fc keypoints (pool(ReLU(F)))
[0089] 4. Multi-scale pedestrian feature fusion
[0090] In the multi-scale pedestrian feature extraction and fusion module, the present invention performs complementary feature fusion on the high-level features output from the pedestrian orientation feature extraction network, the upper body feature extraction network, the pedestrian lower body feature extraction network, and the pedestrian full body feature extraction network. The fusion feature obtained by the complementary feature fusion can be expressed as:
[0091]
[0092] Where W×H represents the size of the feature map, C1, C2, C3, and C4 represent the number of channels of high-level features output from the pedestrian orientation feature extraction network, the pedestrian upper body feature extraction network, the pedestrian lower body feature extraction network, and the pedestrian full body feature extraction network, respectively. Concat(·) represents the concatenation operation; l is the weight coefficient.
[0093] In summary, the embodiments of the present application are mainly used to solve the problem of pedestrian re-identification in cases where pedestrians are wearing similar clothes, have large differences in posture angles, and have significant changes in pedestrian features under different viewing angles. By constructing a feature extraction network with complementary multi-scale view fusion perception (the network includes pedestrian orientation feature extraction, pedestrian upper body feature extraction, pedestrian lower body feature extraction, pedestrian full body feature extraction and pedestrian feature fusion network), pedestrian features are extracted from multiple angles through the feature extraction network, and information from different scales and viewing angles is used. Through multi-level feature extraction and fusion, visually similar but actually different pedestrians, as well as the features of the same pedestrian under different viewing angles, are more accurately distinguished, thereby significantly improving the accuracy and stability of the pedestrian re-identification system.
[0094] Based on the above method embodiment, the present application embodiment also provides a cross-camera pedestrian re-identification system, see Figure 4 As shown, the system mainly includes the following parts:
[0095] A human posture estimation module 410 is used to perform human posture estimation processing on the pedestrian image to be queried provided by the user to obtain pedestrian key points;
[0096] An image segmentation module 420 is used to perform image segmentation on the query pedestrian image based on pedestrian key points to obtain a plurality of segmented region images;
[0097] The feature extraction and fusion module 430 is used to extract features from multiple regional images through a feature extraction network of complementary multi-scale view fusion perception to obtain regional feature vectors, and perform multi-scale pedestrian feature fusion based on the regional feature vectors to generate a query pedestrian feature vector; wherein the feature extraction network of complementary multi-scale view fusion perception includes a pedestrian upper body feature extraction network, a pedestrian lower body feature extraction network, a pedestrian full body feature extraction network and a pedestrian feature fusion network;
[0098] The feature matching module 440 is used to match the query pedestrian feature vector with the pedestrian features in the pre-built pedestrian feature library to determine the target pedestrian that matches the pedestrian image to be queried.
[0099] In a feasible implementation manner, the feature extraction and fusion module 430 is used to:
[0100] The pedestrian full-body feature extraction network uses a graph convolutional network as the network backbone. The regional image is input into the network backbone of the pedestrian full-body feature extraction network, and the feature information output by the network backbone is input into a global maximum pooling layer and a global average pooling layer to extract the pedestrian full-body features.
[0101] The pedestrian orientation feature extraction network includes a convolution structure of a fully connected layer, a random forest, and a preset convolution kernel. The regional image is input into the pedestrian orientation feature extraction network to obtain pedestrian orientation information.
[0102] FCOS is used as the network backbone in the pedestrian upper body feature extraction network and the pedestrian lower body feature extraction network, and a KAN network is added to each residual module in the FCOS network to extract local key point information from the pedestrian upper body features and lower body features respectively, and map the local key point information to the key point coordinate space through the pooling layer.
[0103] In a feasible implementation, the feature extraction and fusion module 430 is used to perform complementary feature fusion on the high-level features output from the pedestrian orientation feature extraction network, the upper body feature extraction network, the pedestrian lower body feature extraction network, and the pedestrian full body feature extraction network to generate a query pedestrian feature vector.
[0104] In a feasible implementation manner, the feature extraction and fusion module 430 is used to:
[0105] Complementary features are fused using the following formula:
[0106]
[0107] Where W×H represents the size of the feature map, C1, C2, C3, and C4 represent the number of channels of high-level features output from the pedestrian orientation feature extraction network, the pedestrian upper body feature extraction network, the pedestrian lower body feature extraction network, and the pedestrian full body feature extraction network, respectively; Concat(·) represents the concatenation operation; and l is the weight coefficient.
[0108] In a feasible implementation manner, the above system further includes a pedestrian feature library construction module, which is used to:
[0109] A pedestrian image video stream is obtained through multiple cameras, and then the pedestrian image video stream is decoded and framed, and filtering, noise reduction and cropping preprocessing operations are performed to obtain a preprocessed pedestrian image;
[0110] The pedestrian region in the preprocessed pedestrian image is extracted, and features are extracted through a complementary multi-scale view fusion-aware feature extraction network to obtain a pedestrian feature vector;
[0111] The pedestrian feature vector is stored in a pre-built pedestrian feature library.
[0112] In a feasible implementation manner, the feature matching module 440 is used to:
[0113] The query pedestrian feature vector is matched with the pedestrian features in the pre-built pedestrian feature library to find the matching feature vector with the highest similarity, and the matching feature vector corresponding to the matching feature vector is determined as the target pedestrian that matches the query pedestrian image.
[0114] In a feasible implementation manner, the system further includes: a key point feasibility determination module, which is used to:
[0115] After obtaining the pedestrian key points, the visibility of the pedestrian key points in the current viewing angle is determined based on the pedestrian's orientation and posture, so as to perform image segmentation when the visibility indicates that all the pedestrian key points are visible.
[0116] The cross-camera pedestrian re-identification system provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for parts not mentioned in the embodiment of the cross-camera pedestrian re-identification system, reference may be made to the corresponding contents in the aforementioned cross-camera pedestrian re-identification method embodiment.
[0117] The present application also provides an electronic device, such as Figure 5 As shown, it is a schematic diagram of the structure of the electronic device, wherein the electronic device 100 includes a processor 51 and a memory 50, the memory 50 stores computer executable instructions that can be executed by the processor 51, and the processor 51 executes the computer executable instructions to implement any of the above-mentioned cross-camera pedestrian re-identification methods.
[0118] exist Figure 5 In the illustrated embodiment, the electronic device further includes a bus 52 and a communication interface 53 , wherein the processor 51 , the communication interface 53 and the memory 50 are connected via the bus 52 .
[0119] Among them, the memory 50 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 53 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 52 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 52 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0120] The processor 51 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 51. The above processor 51 can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor 51 reads the information in the memory and completes the steps of the cross-camera pedestrian re-identification method of the aforementioned embodiment in combination with its hardware.
[0121] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned cross-camera pedestrian re-identification method. The specific implementation can be found in the aforementioned method embodiment, which will not be repeated here.
[0122] The computer program products of the cross-camera pedestrian re-identification method, system, device and medium provided in the embodiments of the present application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.
[0123] Unless otherwise specifically stated, the relative steps, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present application.
[0124] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.
[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A cross-camera pedestrian re-identification method, characterized in that: The method comprises: In response to a pedestrian image to be queried provided by a user, human body posture estimation is performed on the pedestrian image to be queried to obtain pedestrian key points; Performing image segmentation on the pedestrian image to be queried based on the pedestrian key points to obtain a plurality of segmented region images; Extract features from multiple regional images through a feature extraction network based on complementary multi-scale view fusion perception to obtain regional feature vectors, and perform multi-scale pedestrian feature fusion based on the regional feature vectors to generate a query pedestrian feature vector; wherein the feature extraction network based on complementary multi-scale view fusion perception includes a pedestrian upper body feature extraction network, a pedestrian lower body feature extraction network, a pedestrian full body feature extraction network and a pedestrian feature fusion network; The query pedestrian feature vector is matched with pedestrian features in a pre-built pedestrian feature library to determine a target pedestrian that matches the pedestrian image to be queried.
2. The cross-camera pedestrian re-identification method according to claim 1, characterized in that: The feature extraction network of complementary multi-scale view fusion perception is used to extract features from multiple regional images to obtain regional feature vectors, including: The pedestrian full body feature extraction network adopts a graph convolutional network as a network backbone, inputs the regional image into the network backbone of the pedestrian full body feature extraction network, and inputs the feature information output by the network backbone into a global maximum pooling layer and a global average pooling layer to extract the pedestrian full body features; The pedestrian orientation feature extraction network includes a convolution structure of a fully connected layer, a random forest and a preset convolution kernel, and the regional image is input into the pedestrian orientation feature extraction network to obtain pedestrian orientation information; FCOS is used as the network backbone in the pedestrian upper body feature extraction network and the pedestrian lower body feature extraction network, and a KAN network is added to each residual module in the FCOS network to extract local key point information from the pedestrian upper body features and lower body features, respectively, and map the local key point information to the key point coordinate space through a pooling layer.
3. The cross-camera pedestrian re-identification method according to claim 1, characterized in that: Performing multi-scale pedestrian feature fusion based on the regional feature vector to generate a query pedestrian feature vector includes: Complementary feature fusion is performed on high-level features output from the pedestrian orientation feature extraction network, the upper body feature extraction network, the pedestrian lower body feature extraction network and the pedestrian full body feature extraction network to generate a query pedestrian feature vector.
4. The cross-camera pedestrian re-identification method according to claim 3, characterized in that: The high-level features outputted from the pedestrian orientation feature extraction network, the upper body feature extraction network, the pedestrian lower body feature extraction network and the pedestrian full body feature extraction network are complementaryly fused to generate a query pedestrian feature vector, including: Complementary features are fused using the following formula: Where W×H represents the size of the feature map, C1, C2, C3, and C4 represent the number of channels of high-level features output from the pedestrian orientation feature extraction network, the pedestrian upper body feature extraction network, the pedestrian lower body feature extraction network, and the pedestrian full body feature extraction network, respectively; Concat(·) represents the concatenation operation; and l is the weight coefficient.
5. The cross-camera pedestrian re-identification method according to claim 1, characterized in that: The construction of the pre-built pedestrian feature library includes: Acquire a pedestrian image video stream through multiple cameras, then decode and extract frames of the pedestrian image video stream, and perform filtering, noise reduction and cropping preprocessing operations to obtain a preprocessed pedestrian image; Extracting the pedestrian region in the preprocessed pedestrian image, and extracting features through a complementary multi-scale view fusion perception feature extraction network to obtain a pedestrian feature vector; The pedestrian feature vector is stored in a pre-built pedestrian feature library.
6. The cross-camera pedestrian re-identification method according to claim 1, characterized in that: Matching the query pedestrian feature vector with pedestrian features in a pre-built pedestrian feature library to determine a target pedestrian that matches the to-be-queried pedestrian image includes: The query pedestrian feature vector is matched with pedestrian features in a pre-built pedestrian feature library to find a matching feature vector with the highest similarity, and the matching feature vector corresponding to the matching feature vector is determined as the target pedestrian that matches the pedestrian image to be queried.
7. The cross-camera pedestrian re-identification method according to claim 1, characterized in that: The method further comprises: After obtaining the pedestrian key points, the visibility of the pedestrian key points at the current viewing angle is determined based on the pedestrian orientation and posture, so as to perform image segmentation when the visibility indicates that all the pedestrian key points are visible.
8. A cross-camera pedestrian re-identification system, characterized in that: The system comprises: A human posture estimation module is used to perform human posture estimation processing on the pedestrian image to be queried in response to the pedestrian image to be queried provided by the user, so as to obtain pedestrian key points; An image segmentation module is used to perform image segmentation on the pedestrian image to be queried based on the pedestrian key points to obtain a plurality of segmented area images; A feature extraction and fusion module is used to extract features from multiple regional images through a feature extraction network of complementary multi-scale view fusion perception to obtain regional feature vectors, and to perform multi-scale pedestrian feature fusion based on the regional feature vectors to generate a query pedestrian feature vector; wherein the feature extraction network of complementary multi-scale view fusion perception includes a pedestrian upper body feature extraction network, a pedestrian lower body feature extraction network, a pedestrian full body feature extraction network and a pedestrian feature fusion network; The feature matching module is used to match the query pedestrian feature vector with pedestrian features in a pre-built pedestrian feature library to determine a target pedestrian that matches the pedestrian image to be queried.
9. An electronic device, characterized in that: It includes a processor and a memory, the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the cross-camera pedestrian re-identification method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor, the computer-executable instructions prompt the processor to implement the cross-camera pedestrian re-identification method according to any one of claims 1 to 7.