A domain-adaptive person re-identification method based on 3D generative adversarial network

Abstract

The present invention discloses a domain-adaptive pedestrian re-identification method based on a 3D generative adversarial network. The method mainly comprises the following steps: detecting and tracking pedestrians in video data to obtain a sequence of pedestrian images; inputting the pedestrian sequence images into a 3D-GAN model; obtaining a feature sequence after the image sequence passes through an encoder, adding random Gaussian noise and a random direction sequence into the feature sequence, and changing the distribution of the feature sequence; using the changed feature sequence as the input of a decoder, and inputting the original image sequence and the generated image sequence into a discriminator together, corresponding to two outputs; setting a training mechanism of the 3D-GAN model so that the discriminator and the generator confront each other until a balance is reached; and realizing the generation of cross-domain data by using 3D-GAN to obtain rich feature information of the target domain, thereby improving the domain adaptation capability of the model.

Description

Technical Field

[0001] The present invention relates to the field of image recognition research, and in particular to a domain adaptation pedestrian re-identification method based on a 3D generative adversarial network. Background Art

[0002] As more and more intelligent methods continue to make innovative breakthroughs and are applied to various fields of urban construction, the safety of the public domain is particularly important because it is closely related to the general public. With the popularization of public cameras, it is urgent to ensure public safety at all times through image-based intelligent means. As one of the hot research directions of artificial intelligence, pedestrian re-identification has always attracted much attention. Pedestrian re-identification technology can assist in finding lost persons through video images, etc., which can not only reduce the burden on state agencies, but also solve problems well, and plays an important role in the security field. At present, there are also some difficulties in the actual use of pedestrian re-identification, such as large differences in imaging styles across cameras, partial occlusion of pedestrians, and large differences in pedestrian postures. Although there are related methods to initially solve the above problems, the depth and breadth of pedestrian feature mining are very limited, and the technical effect needs to be further improved. Summary of the invention

[0003] In order to overcome the shortcomings of the prior art, the present invention provides a domain adaptation person re-identification method based on 3D generative adversarial network, using 3D-GAN to generate cross-domain data, thereby obtaining rich feature information of the target domain and improving the model domain adaptation capability. The technical solution is as follows:

[0004] The present invention provides a domain adaptation pedestrian re-identification method based on 3D generative adversarial network, which mainly includes the following steps:

[0005] Step 1: Detect and track pedestrians in the video data to obtain a sequence of pedestrian images I real , whose dimension is H×W×S, where H and W are the height and width of the pedestrian image, respectively, and S is the number of sequences.

[0006] Step 2: Pedestrian sequence image I real The input is into the 3D-GAN model, which consists of a generator G and a discriminator D. The generator G contains an encoder and a decoder. The encoder consists of 4 3D convolutional layers and 3 3D downsampling layers, which are used to extract the common features of pedestrians from the sequence pictures and prepare for the decoder to generate cross-domain data.

[0007] Step 3: After the image sequence passes through the encoder, a feature sequence is obtained, and random Gaussian noise z is added to the feature sequence. rand and the random direction sequence d rand, changing the distribution of feature sequences to form conditions for domain changes; random direction sequence d rand The change of feature sequence is affected by adding randomly generated spatial coordinates to each sequence channel.

[0008] Step 4: The changed feature sequence is used as the input of the decoder. The decoder consists of 4 3D deconvolution layers and 3 3D upsampling layers to restore the feature sequence to an image sequence. A jump connection is used between the encoder and the decoder to generate the image sequence I gen For: I gen =G(I real ,z rand ,d rand ).

[0009] Step 5: Convert the original image sequence I real And the generated image sequence I gen The discriminator consists of 5 3D convolutional layers and 4 downsampling layers. The discriminator has two outputs. gen When , the feature sequence output by the discriminator is f gen , the direction sequence is d gen ; When the original image sequence I is input real When , the feature sequence output by the discriminator is f real .

[0010] Step 6: The training mechanism of the 3D-GAN model is that the discriminator should identify the generated image sequence as accurately as possible as it is fake, while the generator should make the generated image sequence deceive the discriminator as much as possible. The two compete until a balance is reached.

[0011] In step 7, in addition to being connected to the discriminator, the generator also has a branch connected to the domain information interaction network. The domain information interaction module consists of a domain space interaction module, a domain sequence interaction module, a feature fusion layer, and a classification network. The domain space interaction module spatially intersects the original image sequence and the generated image sequence, and then connects a 1×1×1 3D convolution layer for preliminary feature fusion. The domain sequence interaction module intersects the original image sequence and the generated image sequence on the sequence channel, and then connects a 1×1×1 3D convolution layer for preliminary feature fusion.

[0012] In step 8, the outputs of the domain spatial interaction module and the domain sequence interaction module are cascaded on the sequence channel and input into the feature fusion layer. The cascaded features first pass through 3D convolutional layers with convolution kernels of 3×3×3, 5×5×5, and 7×7×7, respectively. The outputs of these three convolutional layers are cascaded on the sequence channel and then connected to a 1×1×1 3D convolutional layer to achieve multi-scale feature fusion.

[0013] Step 9: Use the output of the feature fusion layer as the input of the classification network to further refine the fused features and mine deeper common feature information.

[0014] Preferably, the loss function used in step 6 is expressed as:

[0015]

[0016] Where γ is an adjustable weight factor.

[0017] Preferably, step 9 is specifically that the classification network consists of 5 3D convolutional layers, 4 3D downsampling layers and one fully connected layer. The fused features are further refined through the convolutional layers and the downsampling layers to mine deeper common feature information. The refined 3D features are then flattened into 2D features and input into the fully connected layer, and the predicted value p is obtained through softmax.

[0018] Furthermore, the loss function in step 9 adopts multi-category cross entropy, and the formula can be expressed as:

[0019]

[0020] Where N is the total number of image sequences, y ic represents the category label of the i-th image sequence, p ic Represents the prediction result corresponding to the i-th image sequence;

[0021] The total loss function is

[0022] Compared with the prior art, one of the above technical solutions has the following beneficial effects: this method takes a segment of image sequence as a whole as the input of the system to pave the way for the learning of common features; uses 3D-GAN to generate cross-domain data, thereby obtaining rich feature information of the target domain and improving the model's domain adaptability; and further adopts a domain information interaction network, which can not only effectively fuse the internal feature information of the sequence, but also enable the features between domains to fully interact, so that the model can mine deeper common features from cross-domain sequence images. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of a domain-adaptive person re-identification method based on a 3D generative adversarial network is provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to illustrate the technical solution and working principle of the present invention, the following will further describe the embodiments of the present invention in detail with reference to the accompanying drawings. All the above optional technical solutions can be combined in any way to form optional embodiments of the present invention, which will not be described one by one here.

[0025] The terms "step 1", "step 2", "step 3" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those described herein.

[0026] The present disclosure provides a domain adaptation pedestrian re-identification method based on a 3D generative adversarial network: Figure 1 A flowchart of a domain adaptation pedestrian re-identification method based on a 3D generative adversarial network is provided in an embodiment of the present disclosure. Combined with the figure, the method mainly includes the following steps:

[0027] Step 1: Detect and track pedestrians in the video data to obtain a sequence of pedestrian images I real , whose dimension is H×W×S, where H and W are the height and width of the pedestrian image, respectively, and S is the number of sequences.

[0028] Step 2: Pedestrian sequence image I real The input is into the 3D-GAN model, which consists of a generator G and a discriminator D. The generator G contains an encoder and a decoder. The encoder consists of 4 3D convolutional layers and 3 3D downsampling layers, which are used to extract the common features of pedestrians from the sequence pictures and prepare for the decoder to generate cross-domain data.

[0029] Step 3: After the image sequence passes through the encoder, a feature sequence is obtained, and random Gaussian noise z is added to the feature sequence. rand and the random direction sequence d rand , changing the distribution of feature sequences to form conditions for domain changes; random direction sequence d rand The change of feature sequence is affected by adding randomly generated spatial coordinates to each sequence channel.

[0030] Step 4: The changed feature sequence is used as the input of the decoder. The decoder consists of 4 3D deconvolution layers and 3 3D upsampling layers, which are used to restore the feature sequence to an image sequence. In order to make the restored image sequence retain more pedestrian information, a jump connection is used between the encoder and the decoder, so that the generated image sequence not only changes in the domain mode but also retains the characteristic information of the pedestrian as much as possible. Generated image sequence I gen For: I gen =G(I real ,z rand ,d rand ).

[0031] Step 5: Convert the original image sequence Ireal And the generated image sequence I gen The discriminator consists of five 3D convolutional layers and four downsampling layers. The discriminator has two outputs, the feature sequence f and the predicted direction sequence d. gen When , the feature sequence output by the discriminator is f gen , the direction sequence is d gen ; When the original image sequence I is input real When , the feature sequence output by the discriminator is f real .

[0032] Step 6: The training mechanism of the 3D-GAN model is that the discriminator should identify the generated image sequence as accurately as possible as it is fake, while the generator should make the generated image sequence deceive the discriminator as much as possible. The two compete until a balance is reached.

[0033] Preferably, the loss function in step 6 is expressed as:

[0034]

[0035] Where γ is an adjustable weight factor.

[0036] In step 7, in addition to being connected to the discriminator, the generator also has a branch connected to the domain information interaction network. The domain information interaction module consists of a domain space interaction module, a domain sequence interaction module, a feature fusion layer, and a classification network. The domain space interaction module spatially intersects the original image sequence and the generated image sequence, and then connects a 1×1×1 3D convolution layer for preliminary feature fusion. The domain sequence interaction module intersects the original image sequence and the generated image sequence on the sequence channel, and then connects a 1×1×1 3D convolution layer for preliminary feature fusion.

[0037] In step 8, the outputs of the domain spatial interaction module and the domain sequence interaction module are cascaded on the sequence channel and input into the feature fusion layer. The cascaded features first pass through 3D convolutional layers with convolution kernels of 3×3×3, 5×5×5, and 7×7×7, respectively. The outputs of these three convolutional layers are cascaded on the sequence channel and then connected to a 1×1×1 3D convolutional layer to achieve multi-scale feature fusion.

[0038] Step 9: Use the output of the feature fusion layer as the input of the classification network to further refine the fused features and mine deeper common feature information.

[0039] Preferably, step 9 is specifically that the classification network consists of 5 3D convolutional layers, 4 3D downsampling layers and one fully connected layer. The fused features are further refined through the convolutional layers and the downsampling layers to mine deeper common feature information. The refined 3D features are then flattened into 2D features and input into the fully connected layer, and the predicted value p is obtained through softmax.

[0040] Furthermore, the loss function in step 9 adopts multi-category cross entropy, and the formula can be expressed as:

[0041]

[0042] Where N is the total number of image sequences, y ic represents the category label of the i-th image sequence, p ic Represents the prediction result corresponding to the i-th image sequence, and the total loss function is

[0043] The present invention is described above by way of example in conjunction with the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-mentioned method. All non-substantial improvements made by adopting the method concept and technical solution of the present invention; or directly applying the above-mentioned concept and technical solution of the present invention to other occasions without improvement or equivalent replacement, are within the protection scope of the present invention.

Claims

1. A domain-adaptive person re-identification method based on 3D generative adversarial network, It is characterized in that The method mainly includes the following steps: Step 1: Detect and track pedestrians in the video data to obtain a sequence of pedestrian images I real , whose dimensions are H×W×S, where H and W are the height and width of the pedestrian image, respectively, and S is the number of sequences; Step 2: Pedestrian sequence image I real The input is sent to the 3D-GAN model, which consists of a generator G and a discriminator D. The generator G contains an encoder and a decoder. The encoder consists of 4 3D convolutional layers and 3 3D downsampling layers, which are used to extract the common features of pedestrians from the sequence pictures, so as to prepare for the decoder to generate cross-domain data. Step 3: After the image sequence passes through the encoder, a feature sequence is obtained, and random Gaussian noise z is added to the feature sequence. rand and the random direction sequence d rand , changing the distribution of feature sequences to form conditions for domain changes; random direction sequence d rand The change of feature sequence is affected by adding randomly generated spatial coordinates to each sequence channel; Step 4: The changed feature sequence is used as the input of the decoder. The decoder consists of 4 3D deconvolution layers and 3 3D upsampling layers to restore the feature sequence to an image sequence. A jump connection is used between the encoder and the decoder to generate the image sequence I gen For: I gen =G(I real ,z rand ,d rand ); Step 5: Convert the original image sequence I real And the generated image sequence I gen The discriminator consists of 5 3D convolutional layers and 4 downsampling layers. The discriminator has two outputs. When the input generates the image sequence I gen When , the feature sequence output by the discriminator is f gen , the direction sequence is d gen ; When the original image sequence I is input real When , the feature sequence output by the discriminator is f real ; Step 6: The training mechanism of the 3D-GAN model is that the discriminator should identify the generated image sequence as accurately as possible as it is fake, while the generator should make the generated image sequence deceive the discriminator as much as possible. The two compete until a balance is reached.

2. According to the domain adaptation pedestrian re-identification method based on 3D generative adversarial network according to claim 1, It is characterized in that The loss function used in step 6 is expressed as: Where γ is an adjustable weight factor.

3. According to the domain adaptation pedestrian re-identification method based on 3D generative adversarial network according to claim 2, It is characterized in that The method further comprises steps 7-9: Step 7: In addition to being connected to the discriminator, the generator also leads to a branch connected to the domain information interaction network; the domain information interaction module consists of a domain space interaction module, a domain sequence interaction module, a feature fusion layer, and a classification network; the domain space interaction module spatially crosses the original image sequence and the generated image sequence, and then connects a 1×1×1 3D convolution layer for preliminary feature fusion; The domain sequence interaction module crosses the original image sequence and the generated image sequence on the sequence channel, and then connects a 1×1×1 3D convolution layer for preliminary feature fusion; Step 8: The outputs of the domain spatial interaction module and the domain sequence interaction module are cascaded on the sequence channel and then input into the feature fusion layer. The cascaded features first pass through 3D convolutional layers with convolution kernels of 3×3×3, 5×5×5, and 7×7×7, respectively. The outputs of these three convolutional layers are cascaded on the sequence channel and then connected to a 1×1×1 3D convolutional layer to achieve multi-scale feature fusion. Step 9: Use the output of the feature fusion layer as the input of the classification network to further refine the fused features and mine deeper common feature information.

4. According to claim 3, a domain adaptation pedestrian re-identification method based on 3D generative adversarial network, It is characterized in that Step 9 specifically consists of a classification network consisting of 5 3D convolutional layers, 4 3D downsampling layers and a fully connected layer. The fused features are further refined through the convolutional layers and downsampling layers to dig out deeper common feature information. The refined 3D features are then flattened into 2D features and input into the fully connected layer, and the predicted value p is obtained through softmax.

5. According to the domain adaptation pedestrian re-identification method based on 3D generative adversarial network according to claim 4, It is characterized in that The loss function in step 9 uses multi-category cross entropy, and the formula can be expressed as: Where N is the total number of image sequences, y ic represents the category label of the i-th image sequence, p ic Represents the prediction result corresponding to the i-th image sequence; The total loss function is

Images