Content adaptive pedestrian re-identification dataset generation method and system

Abstract

The application discloses a method and system for generating an adaptive pedestrian re-identification dataset, which comprises the following steps: acquiring a monitoring video and extracting key frames from the monitoring video; performing pedestrian target trajectory tracking on the monitoring video and acquiring a pedestrian frame image and corresponding identity information of each key frame; screening out full-body pedestrian frame images without occlusion and motion blur from the acquired pedestrian frame images, and ensuring that the screened-out full-body pedestrian frame images have no similar pictures in adjacent frames; performing quality evaluation on the screened-out full-body pedestrian frame images to obtain quality scores of the screened-out pedestrian frame images; performing intra-domain data enhancement on the pedestrian frame images with quality scores higher than a set score threshold to obtain frame images after data enhancement; performing inter-domain data generation according to the frame images after data enhancement to acquire cross-camera feature maps of pedestrians; and constructing a pedestrian re-identification dataset through the frame images after data enhancement and the cross-camera feature maps of the pedestrians. A high-quality and diverse pedestrian re-identification dataset is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pedestrian re-identification, and particularly relates to a content adaptive pedestrian re-identification dataset generation method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Pedestrian re-identification is an important research topic in intelligent monitoring systems, and is suitable for technical fields such as security and public place searching. The main purpose is to identify the same pedestrian under the non-overlapping camera view of the cross-region, that is, to determine whether a given pedestrian in the camera appears under other cameras. It is an automatic target recognition technology that can quickly locate the pedestrian target of interest in the monitoring network, and is an important step in intelligent video monitoring.

[0004] At present, a discriminative model is obtained by using a convolutional neural network combined with supervised learning for pedestrian re-identification. The supervised learning method requires a large number of training datasets, that is, the current training data is used to estimate and simulate future data.

[0005] However, the number of samples in the current dataset for pedestrian re-identification is limited, and the samples have problems such as poor image quality, blur and occlusion, half-body people, and high similarity. When the existing dataset is used to train the pedestrian re-identification model, the accuracy of the trained pedestrian re-identification model is low, and overfitting phenomenon is prone to occur.

[0006] To solve the problem of limited number of samples and poor quality of samples in the dataset, artificial methods are often used to make and expand the dataset, which requires a large amount of human intervention and is costly.

[0007] In view of the above problems of dataset making and expansion, there are three existing methods, one is to obtain more video or picture data, and to expand the dataset by combining manual and machine labeling. Manual labeling requires a large amount of human intervention and is costly, and the reliability of machine labeling is also not high; the second is to add non-labeled data through unsupervised learning to intelligently improve the recognition performance. The data introduced by this method usually does not meet the independent and identically distributed criterion of data, resulting in errors in the data, so that the improvement effect is limited; the third is to perform linear transformation on the existing data to obtain more data. This method cannot obtain diverse data, and has poor robustness and high computational complexity.

[0008] Therefore, the present application believes that the number of samples in the existing dataset for pedestrian re-identification is limited, the quality of the samples is poor, and the dataset cannot be effectively expanded. SUMMARY

[0009] In order to solve the above problems, the application provides a content adaptive pedestrian re-identification dataset generation method and system, which filters the pedestrian frame image without occlusion, motion blur, similar image and high quality score, and performs corresponding intra-domain data enhancement and inter-domain data expansion according to the characteristics of different domains, so as to obtain a high-quality and diverse pedestrian re-identification dataset.

[0010] In order to achieve the above object, the application adopts the following technical scheme:

[0011] In the first aspect, a content adaptive pedestrian re-identification dataset generation method is provided, which comprises:

[0012] Obtaining a monitoring video and extracting key frames from the monitoring video;

[0013] Tracking the pedestrian target trajectory of the monitoring video and obtaining the pedestrian frame image and the corresponding identity information of each key frame;

[0014] From the obtained pedestrian frame image, the full-body pedestrian frame image without occlusion and motion blur is screened out, and the screened full-body pedestrian frame image has no similar image in adjacent frames;

[0015] Quality evaluation is performed on each screened full-body pedestrian frame image to obtain the quality score of each screened pedestrian frame image;

[0016] The pedestrian frame image with a quality score higher than a set score threshold is subjected to intra-domain data enhancement to obtain a data-enhanced frame image;

[0017] According to the data-enhanced frame image, inter-domain data is generated to obtain the cross-camera feature image of each pedestrian;

[0018] The pedestrian re-identification dataset is constructed by the data-enhanced frame image and the cross-camera feature image of each pedestrian.

[0019] In the second aspect, a content adaptive pedestrian re-identification dataset generation system is provided, which comprises:

[0020] A key frame acquisition module is configured to obtain a monitoring video and extract key frames from the monitoring video;

[0021] A pedestrian frame image acquisition module is configured to track the pedestrian target trajectory of the monitoring video and obtain the pedestrian frame image and the corresponding identity information of each key frame;

[0022] A pedestrian frame image screening module is configured to screen the full-body pedestrian frame image without occlusion and motion blur from the obtained pedestrian frame image, and the screened full-body pedestrian frame image has no similar image in adjacent frames;

[0023] A quality evaluation module is configured to evaluate the quality of each filtered whole-body pedestrian frame diagram, and obtain a quality score of each filtered pedestrian frame diagram;

[0024] An intra-domain data enhancement module is configured to perform intra-domain data enhancement on the pedestrian frame diagram with a quality score higher than a set score threshold, and obtain a data-enhanced frame diagram;

[0025] An inter-domain data generation module is configured to perform inter-domain data generation according to the data-enhanced frame diagram, and obtain a cross-camera feature map of each pedestrian;

[0026] A data set construction module is configured to construct a pedestrian re-identification data set by using the data-enhanced frame diagram and the cross-camera feature map of each pedestrian.

[0027] In a third aspect, an electronic device is provided, which includes a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the content-adaptive pedestrian re-identification data set generation method are completed.

[0028] In a fourth aspect, a computer readable storage medium is provided, which is used to store computer instructions, when the computer instructions are executed by a processor, the steps of the content-adaptive pedestrian re-identification data set generation method are completed.

[0029] Compared with the prior art, the beneficial effects of the present application are:

[0030] 1. The present application filters pedestrian frame diagrams without occlusion, motion blur, similar images and high quality scores, and performs corresponding intra-domain data enhancement and inter-domain data expansion according to the characteristics of different domains, thereby obtaining a high-quality and diverse pedestrian re-identification data set.

[0031] 2. The intra-domain data enhancement strategy of the present application screens the optimal data enhancement strategy that minimizes the pedestrian re-identification loss from different data enhancement strategies, and effectively enhances the intra-domain data using the optimal data enhancement strategy.

[0032] The advantages of the additional aspects of the present application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0033] The drawings accompanying the specification of this application form a part thereof, serve to provide further understanding of the application, and together with the description of the exemplary embodiments of the application, serve to explain the application, and do not constitute an improper limitation on the application.

[0034] Figure 1 The whole process schematic diagram of the method disclosed in Example 1 is shown in the figure;

[0035] Figure 2 Figure illustrating the image quality evaluation network disclosed in Example 1;

[0036] Figure 3 Figure illustrating the intra-domain data augmentation process disclosed in Example 1;

[0037] Figure 4 Figure illustrating the inter-domain data generation network disclosed in Example 1. DETAILED DESCRIPTION

[0038] The application will be further described below in conjunction with the accompanying drawings and examples.

[0039] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0040] Terminology:

[0041] 1. Pedestrian re-identification: Given a surveillance pedestrian image, retrieve the pedestrian image across devices to make up for the visual limitations of fixed cameras.

[0042] 2. Intra-domain and inter-domain: Intra-domain refers to the same scene at different times; inter-domain refers to different scenes.

[0043] 3. Intra-class and inter-class: Intra-class refers to the same pedestrian at different times or in different scenes; inter-class refers to different pedestrians.

[0044] 4. Global average pooling: Sliding a convolution window over the entire feature map and taking the average of all values in the window.

[0045] 5. Domain generalization: Let the model trained on source domain data generalize to target domain with arbitrary data distribution.

[0046] 6. Data augmentation: Increase the amount of data and improve the performance and accuracy of machine learning models at low cost by making minor changes to existing data or creating new synthetic data from existing data.

[0047] 7. Hyperparameter optimization algorithm: Based on the past evaluation results of the objective function, an alternative function is established to find the value that minimizes the objective function.

[0048] Example 1

[0049] In this example, a content adaptive pedestrian re-identification dataset generation method is disclosed, as shown in Figure 1 , comprising:

[0050] S1: Obtain surveillance video and extract key frames from the surveillance video.

[0051] The monitoring video for pedestrian re-identification is acquired, start frame position, end frame position and frame interval are set according to actual requirements, a plurality of key frames are extracted from the monitoring video according to the frame interval, and video information is simplified on the basis of retaining most video features.

[0052] S2: track the pedestrian target trajectory of the monitoring video, and acquire the pedestrian frame image and corresponding identity information of each key frame.

[0053] Specifically, a multi-target tracking algorithm is used to track the multiple target pedestrians in the monitoring video, and the pedestrian trajectory and identity information are retained. In the trajectory tracking process, the high-score frame and the low-score frame are divided, the high-score frame is matched with the existing trajectory first, when the matching is unsuccessful, the low-score frame is matched with the existing trajectory again, and the identity exchange problem in target tracking is optimized. Then, the key frame images are input into the target tracking detection frame calibration part for target detection and identity calibration, and the pedestrian frame boundary is judged, and the pedestrian frame image of each key frame and the corresponding identity information of each pedestrian frame image are output.

[0054] In this embodiment, the Kalman filter is used to predict the position of the tracking trajectory of the current frame in the next frame during target tracking, and then the intersection-over-union matching and the Hungarian algorithm optimization matching are used to track the trajectories of each pedestrian in the complete video and acquire the corresponding identity information. The identity information is used as the identity label of the subsequent pedestrian frame.

[0055] The prediction and correction formulas of Kalman filter are as follows:

[0056] Prediction formula:

[0057]

[0058]

[0059] Correction formula:

[0060]

[0061]

[0062]

[0063] wherein, and respectively represent the posterior state estimation value at k-1 and k time, i.e. the updated result; is the prior state estimation value at k time, which is the result of predicting k time; P k-1 and P k respectively represent the posterior estimation covariance at k-1 and k time, and represent the uncertainty of the state; is the prior estimation covariance at time k (Hk is the covariance of the measurement zk k is the measurement value; K k is the filter gain matrix, Kalman gain; A is the state transition matrix; Q is the process excitation noise covariance, which can represent the error between the state transition matrix and the actual process; R is the measurement noise covariance; B is the matrix for converting the input matrix to the state.

[0064] Based on the key frames obtained in S1, each key frame image is input into a target tracking detection frame calibration part for target detection and identity calibration, and each pedestrian frame image and the corresponding identity information in the tracking video are output. At the same time, the coordinate range of each pedestrian frame is judged, that is, whether it exceeds the image boundary of the video frame is judged, and the part exceeding the boundary is filled with zero to prevent unknown background noise interference. Thus, the pedestrian re-identification initial data set of each type of pedestrian can be obtained.

[0065] S3: From the obtained pedestrian frame images, full-body pedestrian frame images without occlusion and motion blur are screened out, and the screened full-body pedestrian frame images have no similar pictures in adjacent frames, and the process is as follows:

[0066] S31: From the pedestrian frame images of each key frame, full-body pedestrian frame images without occlusion and motion blur are screened out.

[0067] The process of screening the pedestrian frame images without occlusion is as follows:

[0068] For each key frame, when there are multiple pedestrian frame images, for each pedestrian frame image, the intersection over union with the remaining pedestrian frame images is calculated, and from all the pedestrian frame images, the pedestrian frame image whose intersection over union with the remaining pedestrian frame images is less than or equal to a set threshold is screened out as the pedestrian frame image without occlusion; when there is only a single pedestrian frame image, the pedestrian frame image is directly taken as the pedestrian frame image without occlusion.

[0069] From the pedestrian frame images without occlusion, the full-body pedestrian frame images in which the key parts of the pedestrians are all visible, the height-width ratio of the pedestrians is between the first set value and the second set value, and there is no motion blur are screened out as the final screened full-body pedestrian frame images without occlusion and motion blur.

[0070] Occlusion and motion blur are important factors affecting the robustness of the pedestrian re-identification model. In order to obtain a high-quality pedestrian data set, the intersection over union anti-overlapping algorithm and the pedestrian key point detection algorithm are used in this embodiment to judge whether the pedestrian frame image is occluded and whether it has motion blur, and then the initial data set obtained in S2 is preliminarily screened.

[0071] In this embodiment, according to the pedestrian frame image data existing in each key frame, the corresponding algorithm is selectively enabled.

[0072] When there are multiple pedestrian frame images in the key frame, it indicates that there are multiple target pedestrians in the frame, and the IOU anti-overlapping algorithm in target tracking is enabled to determine whether each pedestrian frame image is occluded, and the steps are as follows: set a pedestrian frame image IOU setting threshold a; calculate the IOU of each pedestrian frame image and other pedestrian frame images in turn, compare the IOUs and store the maximum value, and if the maximum value of the IOU of the pedestrian frame image is greater than the setting threshold a, discard the pedestrian frame image, which can avoid multiple pedestrian overlapping.

[0073] When there is only a single pedestrian frame image in the key frame, it indicates that the pedestrian is not occluded in the key frame.

[0074] The embodiment performs secondary screening on the screened non-occluded pedestrian frame image, and screens out a full-body pedestrian frame image in which the key parts of the pedestrian are visible, the height-width ratio of the pedestrian is between the first and second setting values, the detection confidence is greater than the confidence threshold, and the boundary box width is greater than the set pixel.

[0075] When the key parts of the pedestrian in the pedestrian frame image are visible, and the height-width ratio of the pedestrian is between the first and second setting values, the pedestrian frame image is determined to be a full-body pedestrian frame image.

[0076] When the detection confidence of the full-body pedestrian frame image is greater than the confidence threshold, and the boundary box width is greater than the set pixel, the full-body pedestrian frame image is determined to be a full-body pedestrian frame image without motion blur.

[0077] In specific implementation, the human body key points in the screened non-occluded pedestrian frame image are identified by a pedestrian key point detection network to obtain a heat map of each human body key point, and the position with the highest score of each key point is obtained from the heat map, and the position information with the highest score of each key point is mapped to the pedestrian frame image,

[0078] Preferably, the first setting value is 1.4, the second setting value is 5, the human body key parts include the head, the upper body and the lower body, the confidence threshold is 0.72, and the set pixel is 48 pixels. The following specifications are set to screen the final full-body pedestrian frame image without occlusion and motion blur: 1) the head and the upper body of the pedestrian are visible; 2) the hips or knees of the pedestrian exist, i.e. the lower body is visible; 3) the height-width ratio of the pedestrian is between 1.4 and 5; 4) the detection confidence must be greater than 0.72; 5) the boundary box width must be greater than 48 pixels. That is, if the above conditions are met, the pedestrian frame image is considered to be a full-body pedestrian frame image without occlusion and motion blur, and the pedestrian images seriously occluded by the environment or objects or the pedestrian images with a large degree of motion blur are excluded.

[0079] The human key point detection network takes the pedestrian frame graph as input, takes the heat map of the human key point in the pedestrian frame graph as output, adopts the mean square error loss function, and finally the total loss is summed up for each key point by relying on the weight. Among them, in the key point detection, OKS is used to measure the similarity between the predicted key point and the true key point, the value is between 0 and 1, the larger the value is, the more similar it is. The loss function and the OKS formula are specifically expressed as follows:

[0080] Mean square error loss function:

[0081]

[0082] Wherein, MSE(y, y') is the mean square error between the predicted value and the true value, n represents the sample number, y i represents the true value, y i ' represents the predicted value.

[0083] OKS expression:

[0084]

[0085] Wherein, i represents the ith key point; v i represents the visibility of the ith key point; δ(x) is 1 when x is True, and 0 when x is False; d i is the Euclidean distance between the ith predicted key point and the corresponding GT; s is the square root of the target area; k i is a decay constant used to control the key point category.

[0086] The embodiment preliminarily screens the unoccluded pedestrian frame graph for preprocessing, so that the proportion of the preprocessed pedestrian frame graph is the same as that of the human key point detection network. The boundary part of the preprocessed image is filled accordingly, and then input into the human key point detection network.

[0087] The target original proportion is scaled to 4:3, which is the same as the training network proportion, and the missing image boundary part is filled accordingly, and then input into the human key point detection network, and the heat map of each human key point is output.

[0088] S32: For the screened full-body pedestrian frame graph without occlusion and motion blur, similar pictures of adjacent key frames are removed to obtain the screened full-body pedestrian frame graph.

[0089] To ensure the difference and diversity between the same pedestrian re-identification data sets, similar pictures with repeated information must be removed, specifically:

[0090] According to the time sequence of the pedestrian bounding box diagram, the screened full-body pedestrian bounding box diagrams without occlusion and motion blur are segmented, and in each segment, multiple frames of full-body pedestrian bounding box diagrams without occlusion and motion blur are included;

[0091] For each pedestrian target in each segment, the pedestrian bounding box diagrams in two frames are sequentially differentiated to calculate the inter-frame difference value between the time-adjacent pedestrian bounding box diagrams, and the average inter-frame difference value between the adjacent pedestrian bounding box diagrams is taken to further calculate the average inter-frame difference strength; and the pedestrian bounding box diagram with the average inter-frame difference strength as a local maximum value is screened out as the full-body pedestrian bounding box diagram of the pedestrian target screened from the segment, which can effectively eliminate intra-class similar pictures and avoid information redundancy.

[0092] All the pedestrian bounding box diagrams screened for each pedestrian target from different segments are combined to form the screened full-body pedestrian bounding box diagram.

[0093] S4: Quality evaluation is performed on the screened full-body pedestrian bounding box diagrams to obtain the quality score of each pedestrian bounding box diagram.

[0094] In this embodiment, the image quality evaluation network is used to perform quality evaluation on the screened full-body pedestrian bounding box diagrams to obtain the quality score of each pedestrian bounding box diagram.

[0095] As shown in FIG. 1, the image quality evaluation network includes a semantic feature extraction backbone network, a local distortion perception model, a learning perception rule super network, and a quality prediction target network. Figure 2

[0096] The semantic feature extraction backbone network adopts a residual neural network, and the last two layers, i.e., the average pooling layer and the full connection layer, of the original network are deleted. The residual neural network extracts multi-scale feature streams S ms (x) and global semantic features S(x) from the full-body pedestrian bounding box diagram input into the image quality evaluation network, wherein S ms (x) are extracted by the middle three convolution layers and can be used to capture local distortion, and S(x) is extracted by the last convolution layer and represents the overall image content. The multi-scale features S ms (x) are input into the local distortion perception model to output multi-scale content vectors v x The local distortion perception model is a patch extractor based on attention, which includes dividing the multi-scale features S ms (x) into non-overlapping patches, stacking the patches along the channel dimension, performing 1x1 convolution, and globally averaging and merging them into a vector to output the multi-scale content vector v x (x) and the multi-scale content vector v x ​The input quality prediction target network. The global semantic feature S(x) is input into the learning perception rule super network, which is composed of three 1x1 convolutional layers and four weight generation branches, each of which is composed of a convolutional layer, a shaping layer, a pooling layer and a fully connected layer. The specific process is that the learning perception rule super network obtains a feature matrix after convolution by three layers of convolution on the global semantic feature S(x); the feature matrix after convolution is input into four weight generation branches respectively, and the specific operation of each weight branch is: the feature matrix after convolution is input into a 3x3 convolutional layer to obtain a feature, the feature is shaped by a shaping layer to obtain a fully connected layer weight for perceiving the rule of image quality, guiding the target network to predict the image quality; at the same time, the feature matrix after convolution is input into a pooling layer, and the output of the pooling layer is input into a fully connected layer, and the fully connected layer outputs a fully connected layer bias value. The generated fully connected layer weights and fully connected layer bias values are input into the quality prediction target network, which includes four fully connected layers connected in turn. The quality prediction target network receives the multi-scale content feature vector v x as input, and the four fully connected layers determined by the generated fully connected layer weights are propagated layer by layer to obtain the final quality score. The pedestrian frame graph with a quality score higher than the set score threshold is selected from the selected whole body pedestrian frame graph, and the pedestrian frame graph with large blur, distortion and noise is excluded, so as to obtain a high-quality pedestrian re-identification data set.

[0097] Image quality evaluation network loss function:

[0098]

[0099] Wherein, N is the total number of patches of each training image, is a network model, v is a content perception vector, H is a super network mapping function, S() is an image semantic feature, γ is a super network parameter, p i is the i-th training patch score, Q i is the i-th real score.

[0100] S5: performing intra-domain data enhancement on the pedestrian frame graph with a quality score higher than the set score threshold to obtain the frame graph after data enhancement.

[0101] The same scene exists intra-domain difference at different time periods, and if it is costly to make a large number of training sets of the same scene at each time period, content adaptive intra-domain data enhancement can be performed according to the scene characteristics, such as Figure 3 As shown in the figure, the process of intra-domain data enhancement is as follows:

[0102] For each scene, obtain the pedestrian frame graph with a quality score higher than the set threshold in the same scene at different times to form a small amount of high-quality data set;

[0103] dividing the high-quality dataset into a training set and a validation set;

[0104] training the pedestrian re-identification model using the training set to obtain a trained pedestrian re-identification model;

[0105] performing data augmentation on the data in the validation set by a data augmentation strategy, and optimizing the enhancement amplitude of the data augmentation strategy by a hyperparameter optimization algorithm, in the optimization process, the validation set is expanded by using different enhancement amplitudes to obtain different data-augmented validation sets;

[0106] performing pedestrian re-identification on the different data-augmented validation sets by the trained pedestrian re-identification model to obtain different pedestrian re-identification losses;

[0107] selecting the enhancement amplitude with the smallest pedestrian re-identification loss as the optimal enhancement amplitude;

[0108] performing in-domain data augmentation on the small amount of high-quality dataset by the data augmentation strategy and the optimal enhancement amplitude to obtain a data-augmented bounding box.

[0109] For example, for the domain G, the process of adaptive data augmentation is as follows: the pedestrian bounding boxes with quality scores higher than a set threshold at different times are formed into a small amount of high-quality dataset D, the dataset D is randomly divided into a training set D T and a validation set D V , a pedestrian re-identification model θ is trained using the training set to obtain a trained pedestrian re-identification model θ; then n data augmentation sub-strategies are set, the data augmentation sub-strategies can be rotation, automatic contrast adjustment, inverse color, equalization, exposure enhancement, brightness, color, sharpness, etc., each data augmentation sub-strategy is combined to form a data augmentation strategy, and an initial enhancement amplitude {λ1,...,λ n} is set for each data augmentation sub-strategy; finally, the data augmentation strategy and the initial enhancement amplitude {λ1,...,λ n}The extended verification set is used to calculate the pedestrian re-identification loss on the extended verification set by using the trained pedestrian re-identification model θ. A surrogate model is established by using a hyperparameter optimization algorithm, that is, different enhancement amplitude vectors are sampled in a certain range, the pedestrian re-identification loss under different enhancement amplitude vectors is calculated, and each enhancement amplitude vector is divided into two categories of good and poor according to the loss value, the optimal point is collected, the probability density under the optimal point distribution is estimated, the probability density of the optimal point is sampled, and the corresponding expected value is calculated, that is, the actual output loss reduction value compared with the set loss threshold is calculated, the larger expected value is selected, the enhancement amplitude vector is updated according to the expected value, and the above optimization operation is repeated. When the loss does not decrease for many times in succession, the algorithm stops searching, and the optimal enhancement amplitude of the enhancement strategy can be obtained at this time. When the amplitude of a certain enhancement sub-strategy in the enhancement amplitude vector is zero, the data enhancement sub-strategy is not used. The data enhancement strategy and the enhancement amplitude vector can perform intra-domain data enhancement on the domain G, increase the diversity of intra-domain samples, and improve the generalization ability of the model.

[0110] S6: Inter-domain data generation is performed according to the frame diagram after data enhancement, and the cross-camera feature maps of each pedestrian are obtained.

[0111] In order to reduce the cost of collecting cross-camera samples for pedestrian re-identification model training, cross-domain feature generation is used for inter-domain data expansion, as shown in FIG. 8, and stable feature generation under isolated camera conditions can be realized. Figure 4 The process of obtaining the cross-camera feature maps of each pedestrian is as follows:

[0112] For each scene, the pedestrian frame diagram of the same pedestrian is obtained from the frame diagram after data enhancement, and the target pedestrians in the pedestrian frame diagrams obtained in different scenes are different;

[0113] The cross-camera feature maps of each pedestrian are obtained by using the inter-domain data generation network and the pedestrian frame diagrams of different scenes.

[0114] In specific implementation, the inter-domain data generation network is used to perform inter-domain data generation according to the frame diagram after data enhancement, and the cross-camera feature maps of each pedestrian are obtained.

[0115] The inter-domain data generation network takes the pedestrian frame diagrams of different scenes as input and takes the cross-camera feature maps of each pedestrian as output, and includes an encoder E and a generator G. The backbone network of the encoder E is a convolutional neural network, and the generator G is composed of a camera-adjusted variational autoencoder. The encoder E takes the frame diagram after data enhancement as input and outputs the pedestrian image features of each pedestrian frame diagram. The output of the encoder E is input into the generator G, the generator G outputs the cross-camera samples generated by the pedestrian image features in the feature space, and the cross-camera feature maps of each pedestrian are obtained.

[0116] As for pedestrians a, b, c, domains A, B, C, respectively, obtain the box plot data of pedestrian a in domain A, the box plot data of pedestrian b in domain B, and the box plot data of pedestrian c in domain C, generate cross-camera samples in the feature space through the inter-domain data generation network, generate sample data of pedestrian a in domains B and C, sample data of pedestrian b in domains A and C, and sample data of pedestrian c in domains A and B, then only through the data of pedestrian a in a specific domain A, the box plot data of pedestrian b in domain B, and the box plot data of pedestrian c in domain C, the data of pedestrian a in other domains can be generated, and the missing camera view data of pedestrian a is compensated.

[0117] The inter-domain data generation network is trained using the box plot data of each pedestrian in a specific domain for training, and a trained inter-domain data generation network is obtained after training.

[0118] The box plot of pedestrians in different scenes is input into the trained inter-domain data generation network, and the cross-camera feature map of each pedestrian is obtained.

[0119] When training the inter-domain data generation network, the box plot data of each pedestrian in a specific domain is input into the encoder E to extract pedestrian image features, and the output of the encoder E is used as the input of the feature generator G for training the generator G. The training set of the generator G is Each training sample is a triple, i.e., pedestrian image x n , identity label y n ∈{p1,...,p M}, camera label C n ∈{v1,...,v c}, and M and C represent the total number of different identities and camera views used for training, respectively. Among them, the cross-camera samples of the same pedestrian do not exist in the training set at the same time, i.e. If c i ≠c j , then y i ≠y j , the loss of the feature generator G is L G|E . G generates features of different cameras {v1,…,v c} for the training of the encoder E, and the loss of the encoder E is L E|G . The encoder E and the generator G are jointly optimized end-to-end, so that the total loss L (G,E) reaches a minimum, the generator G can generate reliable features in other camera scenes, and the features of people in the encoder E are more discriminative between cameras. Therefore, cross-domain pedestrian features can be obtained, and inter-domain data generation is realized.

[0120] The loss function in the network is as follows:

[0121] L G|E = αLEST + (1 - a)L GSNN

[0122] L E|G = λ1L CCFA + λ2L ID + λ3L MCNL

[0123] L (G,E) = L G|E + L E|G

[0124] wherein L G|E is the generator G training loss function, L EST is the maximum conditional log-likelihood function, L GSNN is the Gaussian random neural network loss, a is a balance hyperparameter, L E|G is the encoder E training loss function, L CCFA is the cross-camera feature alignment loss, L ID is the cross-entropy loss, L MCNL is the pedestrian image feature similarity loss.

[0125] S7: Construct a pedestrian re-identification data set by the data enhanced bounding box and the cross-camera feature map of each pedestrian.

[0126] In this embodiment, the data enhanced bounding box obtained by S5 and the cross-camera feature map of each pedestrian obtained by S6 are used to construct a pedestrian re-identification data set.

[0127] The pedestrian re-identification model is trained by using the pedestrian re-identification data set, and the trained pedestrian re-identification model is obtained after the training is completed. When the trained pedestrian re-identification model is used for pedestrian re-identification, the accuracy of pedestrian re-identification can be ensured.

[0128] Embodiment 2

[0129] In this embodiment, a content adaptive pedestrian re-identification data set generation system is disclosed, comprising:

[0130] A key frame acquisition module is configured to acquire a monitoring video and extract key frames from the monitoring video.

[0131] A pedestrian bounding box acquisition module is configured to track pedestrian target trajectories in the monitoring video and acquire pedestrian bounding box images and corresponding identity information of each key frame.

[0132] A pedestrian bounding box screening module is configured to screen out full-body pedestrian bounding box images without occlusion and motion blur from the acquired pedestrian bounding box images, and the screened full-body pedestrian bounding box images have no similar images in adjacent frames.

[0133] The quality evaluation module is configured to evaluate the quality of each of the screened whole-body pedestrian frame diagram, and obtain a quality score of each of the screened pedestrian frame diagram;

[0134] The intra-domain data enhancement module is configured to perform intra-domain data enhancement on the pedestrian frame diagram with the quality score higher than the set score threshold, and obtain a frame diagram after data enhancement;

[0135] The inter-domain data generation module is configured to perform inter-domain data generation according to the frame diagram after data enhancement, and obtain a cross-camera feature map of each pedestrian;

[0136] The data set construction module is configured to construct a pedestrian re-identification data set by using the frame diagram after data enhancement and the cross-camera feature map of each pedestrian.

[0137] Embodiment 3

[0138] In this embodiment, an electronic device is disclosed, which comprises a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the adaptive pedestrian re-identification data set generation method disclosed in embodiment 1 are completed.

[0139] Embodiment 4

[0140] In this embodiment, a computer readable storage medium is disclosed, which is configured to store computer instructions, when the computer instructions are executed by a processor, the steps of the adaptive pedestrian re-identification data set generation method disclosed in embodiment 1 are completed.

[0141] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application but not to limit it, although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that: the specific embodiments of the present application can be modified or replaced by the same, without departing from the spirit and scope of the present application, any modification or equivalent replacement, which should be covered in the protection scope of the claims of the present application.

Claims

1. A method for generating a content-adaptive pedestrian re-identification dataset, characterized in that, The method comprises the following steps: acquiring a monitoring video and extracting key frames from the monitoring video; tracking a pedestrian target trajectory of the monitoring video and acquiring a pedestrian bounding box image and corresponding identity information of each key frame; screening a full-body pedestrian bounding box image without occlusion and motion blur from the acquired pedestrian bounding box image, and the screened full-body pedestrian bounding box image has no similar image in adjacent frames; performing quality evaluation on each screened full-body pedestrian bounding box image to obtain a quality score of each screened pedestrian bounding box image; performing intra-domain data enhancement on a pedestrian bounding box image with a quality score higher than a set score threshold to obtain a data-enhanced bounding box image; generating inter-domain data according to the data-enhanced bounding box image to acquire a cross-camera feature map of each pedestrian; intra-domain refers to the same scene in different time periods; inter-domain refers to different scenes; constructing a pedestrian re-identification data set through the data-enhanced bounding box image and the cross-camera feature map of each pedestrian. 2.The method of claim 1, wherein, The process of acquiring the screened full-body pedestrian bounding box image comprises the following steps: screening a full-body pedestrian bounding box image without occlusion and motion blur from the pedestrian bounding box image of each key frame; and removing similar images of adjacent key frames from the screened full-body pedestrian bounding box image without occlusion and motion blur to obtain the screened full-body pedestrian bounding box image.

3. The method of claim 2, wherein, For each key frame, when there are multiple pedestrian bounding box images, for each pedestrian bounding box image, the intersection over union with the remaining pedestrian bounding box images is calculated, and from all the pedestrian bounding box images, the pedestrian bounding box image with an intersection over union with the remaining pedestrian bounding box images less than or equal to a set threshold is screened out as a full-body pedestrian bounding box image without occlusion; when there is only a single pedestrian bounding box image, the pedestrian bounding box image is directly taken as a full-body pedestrian bounding box image without occlusion. From the full-body pedestrian bounding box image without occlusion, a full-body pedestrian bounding box image is screened out, in which the key parts of the pedestrian are visible, the height-width ratio of the pedestrian is between a first set value and a second set value, the detection confidence is greater than a confidence threshold, and the bounding box width is greater than a set pixel, as the final screened full-body pedestrian bounding box image without occlusion and motion blur.

4. The method of claim 2, wherein, According to the time sequence of the pedestrian bounding box image, the screened full-body pedestrian bounding box image without occlusion and motion blur is divided into segments, and in each segment, there are multiple full-body pedestrian bounding box images without occlusion and motion blur. For each pedestrian target in each segment, the difference value between adjacent pedestrian bounding box images is calculated. According to the inter-frame difference value between adjacent pedestrian bounding box images, the average inter-frame difference intensity is calculated. And from it, the pedestrian bounding box image with the maximum average difference intensity is screened out as the full-body pedestrian bounding box image of the pedestrian target screened out from the segment. All the pedestrian bounding box images screened out from different segments for each pedestrian target form the screened full-body pedestrian bounding box image.

5. The method of claim 1, wherein, The quality of each screened full-body pedestrian bounding box image is evaluated by an image quality evaluation network to obtain the quality score of each screened pedestrian bounding box image.

6. The method of claim 1, wherein, The process of performing intra-domain data enhancement comprises the following steps: For each scene, acquire a pedestrian bounding box image with a quality score higher than a set threshold in the same scene at different times to form a small amount of high-quality data set; divide the high-quality data set into a training set and a validation set; train a pedestrian re-identification model using the training set to obtain a trained pedestrian re-identification model; The data in the verification set is data-augmented through a data augmentation strategy, and a hyperparameter optimization algorithm is used to optimize the augmentation amplitude of the data augmentation strategy for optimization. In the optimization process, the verification set is expanded using different augmentation amplitudes to obtain different data-augmented verification sets. The pedestrian re-identification model is trained to perform pedestrian re-identification on different data-augmented verification sets to obtain different pedestrian re-identification losses. The enhancement amplitude with the smallest pedestrian re-identification loss is selected as the optimal enhancement amplitude. The intra-domain data augmentation is performed on the small amount of high-quality data set through the data augmentation strategy and the optimal enhancement amplitude to obtain the data-augmented frame diagram.

7. The method of claim 1, wherein, The process of obtaining the cross-camera feature map of each pedestrian is as follows: For each scene, the pedestrian frame diagram of the same pedestrian is obtained from the data-augmented frame diagram, and the target pedestrian in the pedestrian frame diagram obtained from different scenes is different. The cross-camera feature map of each pedestrian is obtained by using the inter-domain data generation network and the pedestrian frame diagram of different scenes.

8. A system for generating a content-adaptive pedestrian re-identification dataset, the system comprising: It includes: A key frame acquisition module for acquiring a monitoring video and extracting key frames from the monitoring video; A pedestrian frame diagram acquisition module for tracking a pedestrian target trajectory of the monitoring video and acquiring a pedestrian frame image and corresponding identity information of each key frame; A pedestrian frame diagram screening module for screening full-body pedestrian frame diagrams without occlusion and motion blur from the acquired pedestrian frame images, and the screened full-body pedestrian frame diagrams have no similar images in adjacent frames; A quality evaluation module for evaluating the quality of each full-body pedestrian frame diagram to obtain a quality score of each pedestrian frame diagram; An intra-domain data augmentation module for performing intra-domain data augmentation on the pedestrian frame diagram with a quality score higher than a set score threshold to obtain a data-augmented frame diagram; An inter-domain data generation module for generating inter-domain data based on the data-augmented frame diagram to obtain a cross-camera feature map of each pedestrian; Intra-domain refers to the same scene in different time periods; Inter-domain refers to different scenes; A data set construction module for constructing a pedestrian re-identification data set by using the data-augmented frame diagram and the cross-camera feature map of each pedestrian.

9. An electronic device, comprising: It includes a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the content adaptive pedestrian re-identification data set generation method in any one of claims 1-7 are completed.

10. A computer-readable storage medium, characterized in that, For storing computer instructions, when the computer instructions are executed by the processor, the steps of the content adaptive pedestrian re-identification data set generation method in any one of claims 1-7 are completed.

Images