A long-term cross-camera target association method and system based on appearance and morphology decoupling

Through dense analysis of human bodies and edge estimation, separation of pedestrian appearance and morphological information, combined with high-dimensional semantic learning, the interpretability and reliability problems of target associations under changes in pedestrian clothing in the existing technology are solved, and accurate pedestrians are achieved over a long period of time Feature matching and completeness of intelligent monitoring systems.

CN114092873BActive Publication Date: 2024-11-05PEKING UNIV SHENZHEN GRADUATE SCHOOL
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202111271643.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-29
Publication Date
2024-11-05
Estimated Expiration
2041-10-29

AI Technical Summary

Technical Problem

The existing long-term cross-camera target association method lacks interpretability when dealing with changes in pedestrian clothing, relies on complex models or unreliable face and gait information, and is not suitable for color image scenarios, making it difficult to accurately match target pedestrians over a long period of time. .

Method used

Pedestrian outline images are obtained through dense analysis of human bodies and edge estimation, residuals are calculated to separate appearance and morphological information, and target correlation learning is performed in high-dimensional semantic space, and feature extraction and optimization are used for independent morphological and appearance representation models to avoid hidden limitations of formal and explicit feature learning.

Benefits of technology

It realizes reliable extraction and matching of pedestrian identity characteristics in long-term cross-camera scenarios, which is suitable for target associations in real scenarios, and improves the performance of the intelligent monitoring system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114092873B_ABST
    Figure CN114092873B_ABST
Patent Text Reader

Abstract

The present invention relates to a long-term cross-camera target association method and system based on appearance and morphology decoupling. The method extracts a human body contour image from an input training image through dense human body analysis estimation and edge estimation, obtains the residual between the training image and the human body contour image, and obtains a human body appearance image; uses a morphological representation model to extract high-level semantic features from the human body contour image, and uses an appearance representation model to extract high-level semantic features from the human body appearance image; uses a triplet loss function and a recognition loss function to train the appearance representation model and the morphological representation model, uses the learned two representation models to extract morphological features and appearance features from the query pedestrian image and the candidate pedestrian image, and obtains the associated targets in the cross-camera scene by calculating the similarity score. The present invention can extract the morphological information of pedestrians without using a complex adversarial learning model, and can avoid mining face information involving privacy and unreliable pedestrian gait information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of robot vision technology and intelligent monitoring, and specifically relates to a method and system for long-term cross-camera target association based on decoupling of appearance and morphology; the body contour of pedestrians is estimated by dense human body analysis method, and the residual between the original pedestrian appearance image and the body contour is obtained to obtain the pedestrian appearance information and morphology information respectively. Furthermore, the two types of information are mapped to a high-dimensional semantic space for target association learning to achieve long-term cross-camera target association. Background Art

[0002] The cross-camera target association task aims to solve how to match the identity of pedestrian targets in cross-camera scenarios to achieve continuous tracking and trajectory association. In particular, existing cross-camera target association tasks are often targeted at short-term scenarios, that is, it is assumed that the target pedestrian will not change his clothes in cross-camera scenarios. In order to break through this limitation, the need to solve the long-term cross-camera target association task becomes more urgent without restricting whether the target pedestrian's clothes change. For the long-term cross-camera target association task, it is critical to mine reliable pedestrian identity features. Therefore, according to the different ways of mining identity information, the existing long-term cross-camera target association methods and systems can be roughly divided into two categories, namely implicit identity feature learning and explicit identity feature learning.

[0003] Long-term cross-camera object association methods and systems based on implicit identity feature learning are often dedicated to implicitly decoupling identity-independent clothing features and identity-related pedestrian morphological features from pedestrian images. In particular, a common solution (Li YJ, Weng X, Kitani K M. Learning shape representations for person re-identification under clothing change. Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision (WACV). 2021: 2432-2441.) is to use a generative adversarial network to generate a large number of pedestrian images with clothing changes and introduce these images into the identity representation learning process. Using the original pedestrian images and the generated clothing-changed pedestrian images, some clothing-independent pedestrian re-identification models learn identity-related pedestrian features by improving the compactness of intra-class pedestrian features and the sparsity of inter-class pedestrian features. In contrast, implicit identity feature learning is not as interpretable as explicit identity feature learning, so a more complex model is needed to construct the relationship between the original pedestrian samples and the pedestrian samples with clothing changes in the feature space.

[0004] The long-term cross-camera target association method and system based on explicit identity feature learning aims to find key body regions that can contain fine-grained pedestrian identity information. In the literature (Yang Q, Wu A, Zheng W S. Person re-identification by contour sketch under moderate clothing change. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019.), a learnable polarity transformation is used to extract discriminative curve patterns from pedestrian body contours. In the literature (Jin X, He T, Zheng K, et al. Cloth-changing person re-identification from a single image with gait prediction and regularization. ArXiv preprint arXiv:2103.15537, 2021.), gait recognition is taken as an additional task, by utilizing the unique gait information of pedestrians to guide the target association model to learn clothing-independent pedestrian identity representation. With the continuous breakthroughs in face recognition tasks, the literature (Wan F, Wu Y, Qian X, et al. When person re-identification meets changing clothes. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2020: 830-831.) jointly extracts pedestrian identity information from the face area and body area, and fuses the two as a unified pedestrian identity representation.

[0005] In contrast, although these methods have promoted the development of long-term cross-camera target association tasks, they still have many limitations. For example, the long-term cross-camera target association method based on face information relies on the face detection method, but is limited by the fact that the face detection method cannot detect non-frontal faces well; the method based on implicit morphological representation lacks interpretability and requires a complex model to construct the relationship between the clothing-changing pedestrian sample and the original sample in the feature space; gait information needs to rely on video data to be accurately obtained, which is not suitable for the long-term cross-camera target association task based on color images. Summary of the invention

[0006] In view of the technical problems existing in the prior art, the purpose of the present invention is to provide a method and system for long-term cross-camera target association based on decoupling of appearance and morphology. By estimating the outline of pedestrian targets and retaining the appearance information in a residual learning manner, the present invention can extract the morphological information of pedestrians without adopting the complex adversarial learning model in implicit identity feature learning, and can avoid mining the privacy-related facial information and unreliable pedestrian gait information in explicit identity feature learning.

[0007] The technical solution adopted by the present invention is as follows:

[0008] A long-term cross-camera target association method based on appearance and morphology decoupling includes the following steps:

[0009] 1) Input training images from a long-term cross-camera object association dataset;

[0010] 2) Perform dense parsing estimation of human body on the input training image;

[0011] 3) using edge estimation method to process the obtained human body dense analytical estimation result to extract the human body contour image;

[0012] 4) obtaining the residual between the training image in step 1) and the human body contour image in step 3) to obtain a human body appearance image;

[0013] 5) extracting high-level semantic features from the human body contour image in step 3) using the morphological representation model;

[0014] 6) extracting high-level semantic features from the human appearance image in step 4) using the appearance representation model;

[0015] 7) Using the triplet loss function and the recognition loss function to perform supervised learning on the features extracted in step 5) and step 6), and realizing parameter optimization of the appearance representation model and the morphology representation model;

[0016] 8) In the actual deployment stage, the human body contour image and human body appearance image can be extracted from the test image in the manner of steps 1) to 4). The morphological representation model and appearance representation model learned in step 7) are used to extract morphological features and appearance features from the human body contour image and human body appearance image respectively. The morphological features and appearance features extracted from the same pedestrian image are connected in series and combined into the final pedestrian features.

[0017] 9) Using the method of extracting pedestrian image features mentioned in step 8), extract features from the query pedestrian image and the candidate pedestrian image respectively, and calculate the cosine distance between the features as the similarity score. The candidate pedestrian with the highest score is selected as the associated target in the cross-camera scene.

[0018] Among them, step 1)-step 4) belong to the image preprocessing stage, including the generation of human body contour image and the extraction of appearance image; step 5)-step 7) is the training part of the morphological representation model and the appearance representation model, and the parameters are optimized to facilitate the extraction of features in the deployment stage; step 8)-step 9) belongs to the testing stage of the long-term cross-camera target association method based on the decoupling of appearance and morphology, which is used for matching and associating pedestrians to be queried in cross-camera scenarios during actual deployment.

[0019] Furthermore, the training images, the pedestrian images to be queried and the candidate pedestrian images mentioned in step 1) and step 9) are all results that can be obtained by directly detecting using existing pedestrian detection algorithms, rather than raw data directly obtained from surveillance cameras.

[0020] Furthermore, step 2) uses a human body dense analytical estimation method to transform the original pedestrian image in order to further extract the human body contour image; step 3) uses an edge estimation method and a human body dense analytical estimation method to extract the human body contour image, which is further used for morphological feature extraction of pedestrians; step 4) uses a residual calculation method to achieve explicit decoupling of appearance information and morphological information in pedestrian images. This method is used to solve problems such as pedestrians may change clothes in long-term scenes; step 5) and step 6) respectively extract the morphological representation and appearance representation of pedestrians in the high-level space.

[0021] Furthermore, the morphological representation model and the appearance representation model mentioned in step 5) and step 6) are the same network structure, and the existing commonly used image classification model (such as ResNet-50) can be used. However, the parameters of the morphological representation model and the appearance representation model are trained independently during the training phase, and there is no sharing relationship between the two. Since the same network is used, the pedestrian morphological features and appearance features extracted using the morphological representation model and the appearance representation model also have the same dimension.

[0022] A long-term cross-camera object association system based on appearance and morphology decoupling, comprising:

[0023] The image preprocessing module is responsible for performing dense analytical estimation of the human body on the input training image, and processing the obtained dense analytical estimation result of the human body using the edge estimation method to extract the human body contour image, and then calculating the residual between the training image and the human body contour image to obtain the human body appearance image;

[0024] The long-term cross-camera target association model training module is responsible for parameter optimization learning of the pedestrian morphology representation model and appearance representation model in the long-term cross-camera target association model; the parameters of the two representation models are optimized under the supervision of the triple loss function and the recognition loss function;

[0025] The long-term cross-camera target association module is responsible for inputting the pedestrian image to be queried and the candidate pedestrian image into the image preprocessing module respectively, and extracting the corresponding human body contour image and human body appearance image; then, the obtained human body contour image and human body appearance image are respectively sent to the morphological representation model and appearance representation model trained by the long-term cross-camera target association model training module to extract the morphological features and appearance features of the corresponding images; then, by concatenating the morphological features and appearance features, the final pedestrian features of the pedestrian image to be queried and the candidate pedestrian image are obtained; finally, by comparing the similarity between the two, it is determined whether the candidate pedestrian image is the target, thereby realizing long-term cross-camera target association.

[0026] The beneficial effects of the present invention are as follows:

[0027] The present invention can solve the problem of difficulty in matching target pedestrians in long-term cross-camera scenarios by decoupling the appearance information and morphological information of pedestrian images. Explicitly separating the appearance information and morphological information of pedestrians and representing them separately in high-level feature spaces helps to extract more semantic pedestrian identity features. In addition, the appearance features and morphological features of pedestrians are connected in series to obtain more reliable pedestrian identity features. The present invention attempts to mine the morphological information of pedestrians, which helps to solve the problem of pedestrians changing clothes in long-term scenarios and is more suitable for cross-camera target association in real scenarios. The invention can be introduced into intelligent monitoring systems, target tracking systems, and target association systems to achieve a more complete intelligent monitoring technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 . Schematic diagram of long-term cross-camera object association model training based on decoupling of appearance and morphology.

[0029] Figure 2 .Flowchart of long-term cross-camera target association test based on decoupling of appearance and morphology. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described below through specific embodiments and drawings.

[0031] like Figure 1 The long-term cross-camera target association model diagram based on appearance and morphology decoupling of the present invention includes the following steps:

[0032] Step 1: Input the training set images and their corresponding data labels.

[0033] Assume the original input image is I i ,In the actual training process, the input image data is often a batch of ,data to ensure that the model parameters can be optimized more stably during the ,training optimization process.

[0034] Step 2: Use the existing method DensePose (Alp Güler R, Neverova N, Kokkinos I. Densepose: Dense human pose estimation in the wild. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2018: 7297-7306.) to perform dense human pose estimation. The estimation process can be expressed as:

[0035] c * (x,y)=argmax c P(c|I i (x,y)

[0036] Where (x, y) represents the image I i The Cartesian coordinates of the pixels in i (x,y)) represents pixel I i The probability that (x,y) belongs to the cth body part, and c * (x,y) represents pixel I i (x,y) Predicted body part.

[0037] Step 3: Use any edge detection algorithm to extract the pedestrian contour image of the estimation result in step 2. The pedestrian contour image is expressed as

[0038] Step 4: Using the original image I in step 1 i and the pedestrian contour image obtained in step 3 Extracting pedestrian appearance images The extraction method is to calculate the residual, that is,

[0039]

[0040] Step 5: and The features are respectively sent to the morphological characterization model and the appearance characterization model to extract features.

[0041] Among them, the morphological representation model and the appearance representation model can select the currently popular classic convolutional neural network structure, such as VGG Net, Inception Net, ResNet-50 and ResNet-101. The purpose of the two models is to map the corresponding images into high-level semantic spaces respectively. Using the morphological representation model, we can get the contour image from the contour image. Extract the pedestrian morphological representation f i s , which can be expressed as:

[0042]

[0043] Using the appearance representation model, we can Extract the pedestrian appearance representation f i a , which can be expressed as:

[0044]

[0045] Among them, W shape and W app They represent the morphological representation model and the appearance representation model respectively. The two models use the same network structure but do not share parameters.

[0046] Step 6: Use the triplet loss function (Schroff F, Kalenichenko D, Philbin J. Facenet: Aunified embedding for face recognition and clustering. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2015: 815-823) and the recognition loss function to identify the feature f i s and f i a Supervised learning is performed (using triplet loss function and recognition loss function to simultaneously supervise the learning of morphological features and appearance features) to achieve parameter optimization of the morphological representation model and the appearance representation model.

[0047] Among them, the triple loss function is defined as:

[0048]

[0049] Feature f represents the feature of any sample in the training batch. is a set of positive samples with the same pedestrian identity as feature f, is a set of negative samples with the same pedestrian identity as feature f. and Respectively represent f a The positive and negative samples of . m is the threshold. d represents the distance between features.

[0050] The recognition loss function is defined as:

[0051]

[0052] Among them, C is the number of pedestrians in the training set, w is the trained classifier, T is the transposition operation, and f j is the j-th class feature. It is worth noting that whether it is the f mentioned in the triple loss function or the f mentioned in the recognition loss, in this method, f is used respectively. i s and f i a Make substitutions to calculate the loss value.

[0053] Step 7: Use a commonly used optimization algorithm to perform back propagation to update the parameters in the morphological representation model and the appearance representation model.

[0054] like Figure 2 The flowchart of long-term cross-camera target association test based on decoupling of appearance and morphology includes the following steps:

[0055] Step 1: Input the pedestrian image to be queried and the candidate pedestrian images in the cross-camera scene.

[0056] Step 2: Perform dense human body analysis estimation on the query pedestrian image and candidate pedestrian images respectively.

[0057] Step 3: Use the edge estimation algorithm used in the training phase to perform edge estimation on the result estimated in step 2) to extract the contour image of the pedestrian.

[0058] Step 4: Calculate the residual between the original image and the contour image to respectively extract the pedestrian appearance image corresponding to the pedestrian image to be queried and the candidate pedestrian image in the cross-camera scene.

[0059] Step 5: Use the morphological representation model and appearance representation model trained in the training phase to perform morphological feature extraction and appearance feature extraction on the contour image obtained in step 3) and the appearance image obtained in step 4) respectively.

[0060] Step 6: Concatenate the morphological features and appearance features of the pedestrian image to be queried as the pedestrian feature to be queried. Concatenate the morphological features and appearance features of the candidate pedestrian image as the candidate pedestrian feature.

[0061] Step 7: Calculate the cosine distance between the pedestrian feature to be queried and the candidate pedestrian feature.

[0062] Step 8: Repeat steps 1 to 7 for all candidate images. Sort the cosine distances between the features of the queried pedestrian and the features of different candidate pedestrians, and select the candidate pedestrian image with the largest cosine distance as the target pedestrian image to achieve long-term cross-camera target matching. This step uses cosine distance as a similarity evaluation method to compare the similarity between the queried pedestrian image and the candidate pedestrian images.

[0063] Based on the same inventive concept, another embodiment of the present invention provides a long-term cross-camera target association system based on appearance and morphology decoupling, which includes:

[0064] The image preprocessing module is responsible for performing dense analytical estimation of the human body on the input training image, and processing the obtained dense analytical estimation result of the human body using the edge estimation method to extract the human body contour image, and then calculating the residual between the training image and the human body contour image to obtain the human body appearance image;

[0065] The long-term cross-camera target association model training module is responsible for parameter optimization learning of the pedestrian morphology representation model and appearance representation model in the long-term cross-camera target association model; the parameters of the two representation models are optimized under the supervision of the triple loss function and the recognition loss function;

[0066] The long-term cross-camera target association module is responsible for inputting the pedestrian image to be queried and the candidate pedestrian image into the image preprocessing module respectively, and extracting the corresponding human body contour image and human body appearance image; then, the obtained human body contour image and human body appearance image are respectively sent to the morphological representation model and appearance representation model trained by the long-term cross-camera target association model training module to extract the morphological features and appearance features of the corresponding images; then, by concatenating the morphological features and appearance features, the final pedestrian features of the pedestrian image to be queried and the candidate pedestrian image are obtained; finally, by comparing the similarity between the two, it is determined whether the candidate pedestrian image is the target, thereby realizing long-term cross-camera target association.

[0067] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0068] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, CD), which stores a computer program. When the computer program is executed by a computer, it implements the various steps of the method of the present invention.

[0069] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A long-term cross-camera target association method based on appearance and morphology decoupling, comprising the following steps: Input training images from the Long Term Cross-Camera Object Association dataset; Perform dense parsing estimation of human body on the input training image; The obtained human body dense analytical estimation result is processed by edge estimation method to extract the human body contour image; Obtain the residual between the training image and the human body contour image to obtain the human body appearance image; Use the morphological representation model to extract high-level semantic features from human body contour images; Use appearance representation model to extract high-level semantic features from human appearance images; The extracted features are supervised by using triple loss function and recognition loss function to optimize the parameters of appearance representation model and morphology representation model. The learned morphological representation model and appearance representation model are used to extract morphological features and appearance features from the query pedestrian image and the candidate pedestrian images. The distance between the features is calculated as the similarity score, and the candidate pedestrian with the highest similarity score is selected as the associated target in the cross-camera scene.

2. The method according to claim 1, characterized in that The human body dense analytical estimation is used to transform the original pedestrian image so as to further extract the human body contour image.

3. The method according to claim 1, characterized in that By calculating the residual, explicit decoupling of appearance information and morphological information in pedestrian images is achieved.

4. The method according to claim 1, characterized in that The morphology characterization model and the appearance characterization model are the same model, but their parameters are not shared.

5. The method according to claim 1, characterized in that The triplet loss function and the recognition loss function are used to simultaneously supervise the learning of morphological features and appearance features.

6. The method according to claim 1, characterized in that The method of extracting morphological features and appearance features from the query pedestrian image and the candidate pedestrian image using the learned morphological representation model and appearance representation model includes: Performing human body contour image extraction and human body appearance image extraction on the query pedestrian image and the candidate pedestrian image; The learned morphological representation model and appearance representation model are used to extract morphological features and appearance features from the human body contour image and human body appearance image respectively; The morphological features and appearance features extracted from the same pedestrian image are concatenated and combined into the final pedestrian features.

7. The method according to claim 1, characterized in that The similarity score is obtained by using the cosine distance to compare the similarity between the pedestrian image to be queried and the candidate pedestrian images.

8. A long-term cross-camera target association system based on appearance and morphology decoupling, characterized in that: include: The image preprocessing module is responsible for performing dense analytical estimation of the human body on the input training image, and processing the obtained dense analytical estimation result of the human body using the edge estimation method to extract the human body contour image, and then calculating the residual between the training image and the human body contour image to obtain the human body appearance image; The long-term cross-camera target association model training module is responsible for parameter optimization learning of the pedestrian morphology representation model and appearance representation model in the long-term cross-camera target association model; Under the supervision of the triplet loss function and the recognition loss function, the parameters of the two representation models are optimized; The long-term cross-camera target association module is responsible for inputting the pedestrian image to be queried and the candidate pedestrian image into the image preprocessing module respectively, and extracting the corresponding human contour image and human appearance image; Then, the obtained human contour image and human appearance image are respectively sent to the morphological representation model and appearance representation model trained by the long-term cross-camera target association model training module to extract the morphological features and appearance features of the corresponding images; then, by concatenating the morphological features and appearance features, the final pedestrian features of the query pedestrian image and the candidate pedestrian image are obtained; finally, by comparing the similarity between the two, it is determined whether the candidate pedestrian image is the target, thereby realizing long-term cross-camera target association.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Personnel identity recognition method, device and equipment and readable storage medium

    CN109815874A

  • Cross-domain pedestrian re-identification method based on unsupervised joint multi-loss model

    CN111126360A

  • Gait recognition system based on deep learning

    CN111626242A

  • Depth discrimination network model method for pedestrian re-recognition in image or video

    WO2019010950A1