A method for constructing domain generalization target re-identification model based on dynamic normalization technology

Abstract

The present invention discloses a method for constructing a domain generalization target re-identification model based on dynamic normalization technology. First, a dynamic normalization DN module is constructed, and then the dynamic normalization DN module is inserted into a residual convolution block to obtain a DN convolution block. Then the DN convolution block is applied to a ResNet-50 model, which is named a domain generalization target re-identification model based on dynamic normalization technology. The model accesses data from multiple source data domains during the training phase, regards each source domain as an independent task, and provides a shared feature extractor for these tasks. Then, based on the cross entropy loss and triplet loss, the overall goal is to minimize the average value of the loss in each data domain. The model of the present invention can strike a balance between eliminating irrelevant specific data domain features and adapting to a single data domain or instance.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition, and in particular relates to a method for constructing a domain generalization target re-identification model. Background Art

[0002] Object Re-ID aims to match the same object in different cameras. Many supervised object Re-ID methods achieve good results when trained and evaluated on images from the same environment. However, the performance of these methods often drops significantly when tested on images from environments that the model has not seen. It is widely believed that capturing changes in the environment, such as changes in lighting, viewpoint, and season, leads to data domain shift, and existing methods are not robust to these changes. In response, domain generalization object Re-ID aims to build an object Re-ID model that is more robust to data domain changes and can work in unseen environments.

[0003] In past works, instance normalization has been widely used to produce data domain representation by removing statistical contrasts across feature channels. However, the removed statistics not only encode irrelevant domain-specific patterns, but also contain discriminative information that can be used to identify specific domains or instances. To address this issue, it is necessary to build a module that is flexible enough to learn a mapping function to combine normalized features processed by instance normalization and non-normalized features. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a method for constructing a domain generalization target re-identification model based on dynamic normalization technology. First, a dynamic normalization DN module is constructed, and then the dynamic normalization DN module is inserted into the residual convolution block to obtain a DN convolution block. Then the DN convolution block is applied to the ResNet-50 model, which is named a domain generalization target re-identification model based on dynamic normalization technology. During the training phase, the model accesses data from multiple source data domains, regards each source domain as an independent task, and provides a shared feature extractor for these tasks. Then, based on the cross entropy loss and triplet loss, the overall goal is to minimize the average loss in each data domain. The model of the present invention can strike a balance between eliminating irrelevant specific data domain features and adapting to a single data domain or instance.

[0005] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0006] Step 1: Construct a dynamic normalized DN module;

[0007] First, a convolution kernel of size 1×1 is used to adjust the channel dimension of the input feature, and then instance normalization is performed on the adjusted feature. Then, dynamic convolution is used to transform the normalized features extracted by instance normalization. The parameters of the dynamic convolution are generated according to the features before instance normalization. The above operation can be written as:

[0008]

[0009] Among them, F and F DTIN are the input features and output features of the dynamic normalization DN module respectively; CC represents the convolution layer with a convolution kernel size of 1×1 for channel compression, which is used to compress the channel dimension of the feature from C to K before performing the instance normalization operation; φ w (F′) indicates that the adaptive parameter predictor generates adaptive parameters including weights and biases according to F′, and F′ is the feature used for convolution parameter prediction; avgPool indicates the spatial average pooling operation, and global spatial pooling is performed here; θ1 and θ2 indicate the parameters of the fully connected layer FC;

[0010] Step 2: Apply the dynamic normalization DN module to the ResNet-50 model. The ResNet-50 model consists of 4 parts, each of which contains a different number of residual convolution blocks. Use the DN module to replace the 3×3 convolution layer in the residual convolution block to obtain the DN convolution block, and then use the DN convolution block to replace the last residual convolution block in the 2nd to 4th parts of the ResNet-50 model; the model after applying the dynamic normalization DN module is named the domain generalization target re-identification model based on dynamic normalization technology; in addition, insert an instance normalization layer after the first part for convergence;

[0011] Step 3: Multi-task training strategy;

[0012] The domain generalization target re-identification model based on dynamic normalization technology accesses data from multiple source data domains during the training phase. The source data domain is referred to as the source domain; the source data domain is represented as Where S is the number of available source domains, represents the kth source domain, The i-th labeled sample in the k-th source domain, For this sample; is the label information of the sample, that is, the target ID corresponding to the sample; N k represents the number of samples in the kth source domain;

[0013] Treat each source domain as an independent task and provide a shared feature extractor for these tasks; create a target ID classifier for each source domain To perform target ID classification, the kth classifier The label space is all target IDs in the k-th source domain; the cross entropy loss is applied to the result of each sample after the target ID classifier and its label information; the triple loss is applied to the samples randomly sampled from the k-th source domain; the overall goal of the multi-task training strategy is to minimize the average loss in each data domain, so the overall loss function is expressed as follows:

[0014]

[0015] Among them, N k It represents the number of samples in the k-th source domain, and They are randomly captured positive and negative samples; ψ represents the domain generalization target re-identification model based on dynamic normalization technology; L c.e. and L tri. They represent cross entropy loss and triplet loss respectively.

[0016] Preferably, K=64.

[0017] The beneficial effects of the present invention are as follows:

[0018] The present invention can construct a feature extraction model with dynamic characteristics. When extracting features from images in unseen data domains, the model can dynamically extract features based on the inherent information of the current input image and information that is easy to distinguish, thereby achieving a balance between eliminating irrelevant specific data domain features and adapting to a single data domain or instance, and helping to better perform domain generalization target re-identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a model framework diagram of the present invention. DETAILED DESCRIPTION

[0020] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0021] In order to solve the negative impact of data domain changes on model performance in target re-identification tasks and better extract data domain representation, this paper proposes a domain generalization target re-identification model based on dynamic normalization technology. The model can strike a balance between eliminating irrelevant specific data domain features and adapting to a single data domain or instance. Its structure is as follows Figure 1 As shown, the main work of the present invention includes two parts: a dynamic normalization DN module and a multi-task training strategy.

[0022] 1. Dynamic Normalization DN Module and Its Deployment

[0023] The present invention integrates dynamic convolution with instance normalization to overcome the shortcomings of instance normalization, and adaptively recalibrates the normalized features through dynamic convolution to obtain a new representation that can adapt to various domains and instances. This is the dynamically normalized DN module (Dynamically Normalization Module) proposed in the present invention.

[0024] like Figure 1 As shown in (c), in the dynamic normalization DN module, the present invention first uses a convolution kernel of size 1×1 to adjust the channel dimension of the input feature, then performs instance normalization on the adjusted feature, and then uses dynamic convolution to convert the normalized features extracted by instance normalization, wherein the parameters of the dynamic convolution are generated according to the features before the instance normalization operation. Under such a design idea, the instance normalization operation can remove the data domain inherent information encoded in the mean and standard deviation of the feature channel in the input feature, and try to eliminate the influence of different probability distributions on feature extraction. The subsequent dynamic convolution operation can adaptively adjust the parameters according to the input data domain-level features and instance-level features to capture sensitive content, compensate for the representation information lost in the instance normalization operation, and effectively capture the representation information that is beneficial to the target re-identification task. The above operation can be written as:

[0025]

[0026] Among them, F and F DTIN are the input features and output features of the dynamic normalization DN module respectively; CC represents the convolution layer with a convolution kernel size of 1×1 for channel compression, which is used to compress the channel dimension of the feature from C to K (K=64) before performing the instance normalization operation; φ w (F′) indicates that the adaptive parameter predictor generates adaptive parameters including weights and biases according to F′. F′ is the feature used for convolution parameter prediction, and its composition will be explained in detail later; avgPool indicates the spatial average pooling operation, and global spatial pooling is performed here; θ1 and θ2 indicate the parameters of the fully connected layer FC. It is worth mentioning that after the instance normalization operation, the normalized features do not need to be normalized. An additional static affine transformation is applied because the subsequent dynamic convolution operation can be more flexible to calibrate instead of the static affine transformation.

[0027] It should be noted that the present invention does not simply connect the instance normalization module and the dynamic convolution module in a form similar to series or parallel connection, but integrates the two into one module in a more compact and more relevant way, which is mainly reflected in the work carried out on adaptive parameter generation. In order to provide more adaptive parameters for dynamic convolution, on the one hand, the present invention retains the input features before the instance normalization operation, that is, the non-normalized features, to guide the generation of adaptive parameters. On the other hand, the present invention designs a feature transfer path, which integrates low-level features into semantic supplements of high-level retained features, and passes them into the dynamic normalization DN module, sums them with the features, and obtains fused features for adaptive parameter generation, so as to ensure that the prediction parameters contain sufficient semantic information, so as to perform more appropriate extraction and calibration operations for each input feature.

[0028] The present invention inserts the above-mentioned dynamic normalization DN module into the residual convolution block to obtain the DN convolution block, and then applies it to the widely used ResNet-50 model, which is named the domain generalization target re-identification model based on dynamic normalization technology. ResNet-50 consists of four main parts, each of which contains a different number of residual blocks, such as Figure 1 As shown in (a) and (b), in order to avoid introducing too much computational consumption, the present invention only uses the DN module to replace the 3×3 convolution layer in the last residual convolution block of parts 2-4, and obtains a DN convolution block to refine the features extracted in these parts. In addition, the present invention experimentally inserts a common instance normalization layer after the first residual part to promote convergence.

[0029] 2. Multi-task training strategy

[0030] As for the common data domain generalization problem, the model can access data from multiple source data domains (referred to as source domains) during the training phase. The present invention also adopts the same setting in the target re-identification task based on the idea of ​​data domain generalization. These accessible source data domains are represented as Where S is the number of available source domains, represents the kth source domain, where The i-th labeled sample in the k-th source domain, For this sample, is the label information of the sample, that is, the target ID corresponding to the sample, N k It represents the number of samples in the kth source domain.

[0031] The present invention treats each source domain as an independent task and provides a shared feature extractor for these tasks. In other words, this is a multi-task training process. Based on this, the present invention creates a target ID classifier for each domain. To perform target ID classification. Where k is consistent with the previous description, indicating the data domain sequence number, and the kth classifier The label space is all target IDs in the kth domain. The present invention applies cross-entropy loss to the result of each sample after the target ID classifier and its label information. In addition, the present invention also applies triplet loss to the samples randomly sampled from the kth domain. Based on the above two losses, the overall goal of the multi-task training strategy of the present invention is to minimize the average loss in each data domain. Therefore, the overall loss function can be expressed as the following formula:

[0032]

[0033] Among them, N k represents the number of samples in the kth source domain, and and They are randomly captured positive and negative samples; in addition, ψ represents the feature extraction model, which is the domain generalization target re-identification model based on dynamic normalization technology proposed in this invention, and L c.e. and L tri. They represent cross entropy loss and triplet loss respectively. Specific embodiment:

[0035] The following is a specific example of using the present invention to perform target re-identification. The present invention provides a domain generalization target re-identification model based on dynamic normalization technology, and the specific implementation process is as follows.

[0036] 1. Dataset selection

[0037] In terms of data set setting, the present invention uses a re-identification data set to evaluate the recognition ability of the method of the present invention in the domain generalization target re-identification task, and compares the results with other target re-identification algorithms that are also based on the idea of ​​data domain generalization under the same data set setting to analyze the generalization ability of the domain generalization target re-identification model based on dynamic normalization technology proposed by the present invention. Specifically, the present invention uses five data sets such as Market-1501, DukeMTMC-ReID, Cuhk02, Cuhk03, and Cuhk-SYSU to construct training sets, because the image data in these data sets are all taken by fixed cameras or ground handheld cameras. After completing the model training on the data sets constructed by the above five data sets, the present invention tests the trained model on four small re-identification data sets such as VIPeR, PRID, GRID, and QMUL i-LIDS.

[0038] 2. Implementation details setting

[0039] As for training, before formal training, we first pre-trained the domain generalization object re-identification model based on dynamic normalization technology on the ImageNet dataset. The pre-training operation is based on two considerations. First, ImageNet is a training dataset with massive labeled data suitable for image classification tasks. Its huge amount of data will provide great help for training more effective parameters for the model. Second, ImageNet includes thousands of categories, which is general image data not limited to object re-identification. It is not highly correlated with the data domain, has good versatility, and is suitable for various tasks.

[0040] As for data enhancement, in both the training and testing phases, the present invention pre-adjusts each input image to a size of 256×128, and assigns each image a horizontal flip probability of 0.5 during the training phase, thereby enhancing the diversity of training data to achieve data enhancement. In addition, the present invention also applies image transformation strategies such as random erasing and padding for data enhancement.

[0041] As far as the hyperparameter setting is concerned, since the model has been pre-trained on ImageNet, only fine-tuning is required in the formal training phase to complete the model training. Therefore, the present invention selects a smaller learning rate. The present invention sets the initial learning rate to 3.5×10-4, and reduces it by one order of magnitude in the 40th and 90th rounds, that is, divided by 10, to avoid the failure of the model to converge due to excessive learning rate. Based on experience, the present invention sets the total number of Epochs of training to 120 rounds to ensure that the model fully converges. In addition, considering the need to balance server performance and model convergence effect, the present invention sets the batch size of the number of samples to 64.

[0042] 3. Implementation environment

[0043] The present invention uses Sugon-W580-G20 server and Linux Ubuntu 16.04.4LTS operating system to implement the model proposed in the present invention, uses two NVIDIA GeForce GTX 1080Ti graphics processors for training, and uses NVIDIA CUDA 10.2 platform to accelerate training. In terms of software configuration, the present invention uses Python 3.8.13 (GCC 7.5.0) and PyTorch 1.6.0, as well as Numpy 1.19.2, Fast-Reid 1.3, Pillow 9.0.1, Torchvision0.7.0, cv2 4.5.5, CuDNN 7.6.5 and other dependent libraries.

[0044] 4. Model Application

[0045] At this stage, the present invention does not perform data augmentation on the input data, but only samples the input data to an image size of 256*128. At the same time, the present invention fixes the model parameters and no longer updates the network parameters through the stochastic gradient descent algorithm, but only uses the network as an extractor of image features. In the actual reasoning process, the present invention uses the ResNet-50 model with a dynamic normalization module deployed, that is, the output feature F of the domain generalization target re-identification model based on dynamic normalization technology. For the query sample, the feature F obtained after model feature reasoning q The present invention extracts the features of all images in the image library and stores them as feature sequences {F1,…,F m}, and then calculate F q The Euclidean distance of all image features in the image library:

[0046] d=||F q -F i ||2,i=1,2,3,…,m (5)

[0047] Then the present invention obtains the distance sequence D=[d1, d2, ..., d m ], and then the present invention reorders D by distance, and takes the L images closest to the query sample. If there is an image with the same ID as the query sample among these images, it is considered that the query is successful.

Claims

1. A method for constructing a domain generalization target re-identification model based on dynamic normalization technology, characterized in that: The steps include: Step 1: Construct a dynamic normalized DN module; First, a convolution kernel of size 1×1 is used to adjust the channel dimension of the input feature, and then instance normalization is performed on the adjusted feature. Then, dynamic convolution is used to transform the normalized features extracted by instance normalization. The parameters of the dynamic convolution are generated according to the features before instance normalization. The above operation can be written as: Among them, F and F DTIN are the input features and output features of the dynamic normalization DN module respectively; CC represents the convolution layer with a convolution kernel size of 1×1 for channel compression, which is used to compress the channel dimension of the feature from C to K before performing the instance normalization operation; φ w (F′) indicates that the adaptive parameter predictor generates adaptive parameters including weights and biases according to F′, and F′ is the feature used for convolution parameter prediction; avgPool indicates the spatial average pooling operation, and global spatial pooling is performed here; θ1 and θ2 indicate the parameters of the fully connected layer FC; Step 2: Apply the dynamic normalization DN module to the ResNet-50 model. The ResNet-50 model consists of 4 parts, each of which contains a different number of residual convolution blocks. Use the DN module to replace the 3×3 convolution layer in the residual convolution block to obtain the DN convolution block, and then use the DN convolution block to replace the last residual convolution block in the 2nd to 4th parts of the ResNet-50 model; the model after applying the dynamic normalization DN module is named the domain generalization target re-identification model based on dynamic normalization technology; in addition, insert an instance normalization layer after the first part for convergence; Step 3: Multi-task training strategy; The domain generalization target re-identification model based on dynamic normalization technology accesses data from multiple source data domains during the training phase. The source data domain is referred to as the source domain; the source data domain is represented as Where S is the number of available source domains, represents the kth source domain, The i-th labeled sample in the k-th source domain, For this sample; is the label information of the sample, that is, the target ID corresponding to the sample; N k represents the number of samples in the kth source domain; Treat each source domain as an independent task and provide a shared feature extractor for these tasks; create a target ID classifier for each source domain To perform target ID classification, the kth classifier The label space is all target IDs in the k-th source domain; the cross entropy loss is applied to the result of each sample after the target ID classifier and its label information; the triple loss is applied to the samples randomly sampled from the k-th source domain; the overall goal of the multi-task training strategy is to minimize the average loss in each data domain, so the overall loss function is expressed as follows: Among them, N k It represents the number of samples in the k-th source domain, and They are randomly captured positive and negative samples; ψ represents the domain generalization target re-identification model based on dynamic normalization technology; L c.e. and L tri. They represent cross entropy loss and triplet loss respectively.

2. According to the method for constructing a domain generalization target re-identification model based on dynamic normalization technology according to claim 1, it is characterized in that: Said K=64.

Images