A person re-identification method and system based on view-decoupled self-attention network

Abstract

The present invention discloses a pedestrian re-identification method and system based on a view-decoupled self-attention network, the method comprising: constructing a data set; constructing a view-decoupling model based on a Vision Transformer Base network; training the view-decoupling model based on the data set to obtain a recognition model; processing the test data based on the recognition model to obtain a recognition result. The system comprises: a data construction module, a model construction module, a model training module and a model application module. By using the present invention, the interference of different viewpoint differences in an air-ground camera network on the model's extraction of discriminative identity features can be alleviated, thereby achieving pedestrian re-identification in a camera network with heterogeneous viewpoints. The present invention can be widely applied to the field of image processing.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a pedestrian re-identification method and system based on a view-decoupled self-attention network. Background Art

[0002] Person re-identification (PDR) has received widespread attention and development as a key technology application of computer vision in the field of social security. Existing PDR mainly focuses on associating the identity of the target pedestrian in camera networks with homogeneous perspectives (e.g., camera networks deployed under purely terrestrial perspectives). However, air-ground PDR in camera networks with heterogeneous perspectives (e.g., drone-ground camera networks deployed under aerial and terrestrial perspectives), as a more widespread scenario and demand, has received little attention. We believe that the difference in imaging perspective between aerial and terrestrial cameras is the biggest challenge facing air-ground PDR, and one of the problems that existing methods based on homogeneous camera networks cannot effectively solve. Summary of the Invention

[0003] In view of this, in order to solve the technical problem that existing pedestrian re-identification methods cannot effectively eliminate the interference of different perspective differences in air-ground camera networks on the model's extraction of discriminative identity features, which in turn makes it difficult to achieve pedestrian re-identification with heterogeneous perspectives, in the first aspect, the present invention proposes a pedestrian re-identification method based on a perspective-decoupled self-attention network, which includes the following steps:

[0004] Build a dataset;

[0005] Build a perspective decoupling model based on the Vision Transformer Base network;

[0006] Training the perspective decoupling model based on the data set to obtain a recognition model;

[0007] Processing the test data based on the recognition model to obtain a recognition result;

[0008] The training process is as follows:

[0009] inputting the data set into the perspective decoupling model;

[0010] Segmenting the images in the data set and constructing feature vectors to obtain image passwords, global passwords, and viewpoint passwords;

[0011] Assigning position features to the image password, the global password, and the viewing angle password to obtain a final input image matrix;

[0012] Propagate the final input image matrix forward block by block in the view decoupling model to participate in calculation, update the global password and the view password, and obtain an updated password;

[0013] Separate the view command from the updated command, and use the separated global command to replace the global one in the subsequent self-attention block operation;

[0014] Repeat the steps of password updating and password separation until all self-attention blocks are calculated to obtain identity features and view features;

[0015] A loss function is calculated based on the identity feature and the view feature, and the view decoupling model is updated to obtain a recognition model.

[0016] In some embodiments, the step of constructing a dataset specifically includes:

[0017] Collect real data;

[0018] Creating a synthetic pedestrian model and an open-ground camera group scene, placing the synthetic pedestrian model in the open-ground camera group scene for movement to obtain virtual data;

[0019] Combine real data and virtual data to get the data set.

[0020] Through this preferred step, the scale of training data can be expanded at low cost.

[0021] In some embodiments, the loss function includes a cross entropy loss function, a hard triplet loss function, and a feature orthogonal loss function.

[0022] In some embodiments, the step of calculating the loss function based on the identity feature and the view feature, and updating the view decoupling model to obtain the recognition model specifically includes:

[0023] Calculating a first loss value based on the identity feature and a cross entropy loss function;

[0024] Calculating a second loss value based on the identity feature and the difficulty triplet loss function;

[0025] Calculating a third loss value based on the perspective feature and the cross entropy loss function;

[0026] Calculating a fourth loss value based on the identity feature, the view feature, and a feature orthogonal loss function;

[0027] The parameters of the perspective decoupling model are updated in combination with the first loss value, the second loss value, the third loss value, and the fourth loss value to obtain a recognition model.

[0028] In some embodiments, the fourth loss value is calculated as follows:

[0029]

[0030] Among them, the value of |B| is the number of training samples in a batch, ||·||2 represents the L2 norm of the feature, Indicates identity characteristics, represents the viewing angle feature, |<·,·>| represents the absolute value of the inner product of the two features, N+1 represents the output of the last layer of the network, x i represents the i-th input image.

[0031] In some embodiments, it further includes:

[0032] Dividing the data set into a training set and a test set;

[0033] Performing data enhancement on the training set to obtain a data-enhanced training set;

[0034] The training set after data enhancement is used for training.

[0035] Among them, data enhancement includes image expansion, image cropping and random erasure processing.

[0036] The present invention also proposes a person re-identification system based on a view-decoupled self-attention network, the system comprising:

[0037] Data construction module, used to construct datasets;

[0038] Model building module, building a perspective decoupling model based on the Vision Transformer Base network

[0039] A model training module trains the view decoupling model based on the data set to obtain a recognition model; the training process is as follows: inputting the data set into the view decoupling model; segmenting the images in the data set and constructing feature vectors to obtain image passwords, global passwords and view passwords; assigning position features to the image passwords, the global passwords and the view passwords to obtain a final input image matrix; propagating the final input image matrix forward block by block in the view decoupling model to participate in calculations, updating the global password and the view password to obtain updated passwords; separating the view password from the updated password, and replacing the global password with the separated global password to participate in the calculation of subsequent self-attention blocks; looping the steps of password updating and password separation until the calculations of all self-attention blocks are completed to obtain identity features and view features; calculating the loss function based on the identity features and view features, and updating the view decoupling model to obtain a recognition model;

[0040] The model application module processes the test data based on the recognition model to obtain a recognition result.

[0041] The present invention also proposes a person re-identification device based on a view-decoupled self-attention network, comprising:

[0042] at least one processor;

[0043] at least one memory for storing at least one program;

[0044] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned pedestrian re-identification method based on the view-decoupled self-attention network.

[0045] Based on the above scheme, the present invention provides a pedestrian re-identification method and system based on a view-decoupled self-attention network, and builds a self-attention network framework based on progressive view decoupling. The framework designs two main components to decouple view-related and view-independent features, namely, feature layer-by-layer separation and feature orthogonal constraint. The former separates the view-related features and the view-independent features from the total features of the image layer by layer in the self-attention model, and the latter constrains the two separated features to be orthogonal and independent of each other in the form of a loss function, thereby improving the model's perception of pedestrian features and the final overall retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of the steps of a pedestrian re-identification method based on a view-decoupled self-attention network of the present invention;

[0047] Figure 2 This is a data flow diagram of a specific embodiment of the present invention;

[0048] Figure 3 This is a structural block diagram of a pedestrian re-identification system based on a view-decoupled self-attention network in the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] It should be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0051] It should be understood that the terms "system," "device," "unit," and / or "module" used in this application are a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other terms can achieve the same purpose, the terms may be replaced by other expressions.

[0052] Unless the context clearly indicates an exception, the terms "a," "an," "an," and / or "the" are not intended to refer to the singular and may include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list; a method or apparatus may also include other steps or elements. The phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, product, or apparatus that includes the elements.

[0053] In the description of the embodiments of this application, "plurality" refers to two or more than two. The terms "first" and "second" below are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0054] In addition, flow charts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps may be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0055] Reference Figure 1 and Figure 2 The present invention proposes a person re-identification method based on a view-decoupled self-attention network. The method can be applied to a computer device. The person re-identification method proposed in this embodiment may include but is not limited to the following steps:

[0056] Step S1, constructing a data set;

[0057] Step S2: construct a view decoupling model based on the Vision Transformer Base network;

[0058] Step S3: training the view decoupling model based on the data set to obtain a recognition model;

[0059] The training process is as follows:

[0060] S3.1. Inputting the data set into the perspective decoupling model;

[0061] The overall network structure of the present invention is a self-attention model based on view decoupling, which is composed of view decoupling self-attention blocks to extract the identity and view features of the input data. This embodiment uses VisionTransformer Base (ViT-B) as the basic network structure of the present invention, which contains N = 12 encoder modules as the basic structure of the view decoupling self-attention block of the present invention, and uses its weights pre-trained on the ImageNet dataset to initialize the network model;

[0062] S3.2. Segment the images in the dataset and construct feature vectors to obtain image passwords, global passwords, and viewpoint passwords;

[0063] S3.3. Assign position features to the image password, the global password, and the view password to obtain a final input image matrix;

[0064] S3.4. Propagate the final input image matrix forward block by block in the view decoupling model to participate in the calculation, update the global password and the view password, and obtain an updated password;

[0065] S3.5. Separate the viewpoint command from the updated command, and use the separated global command to replace the global command in the subsequent self-attention block operation;

[0066] S3.6. Repeat the steps of password updating and password separation until all self-attention blocks are calculated and the identity feature and view feature are obtained.

[0067] S3.7. Calculate the loss function based on the identity feature and the view feature, and update the view decoupling model to obtain a recognition model.

[0068] Step S4: processing the test data based on the recognition model to obtain a recognition result;

[0069] The recognition result includes the recognized viewing angle features and identity features.

[0070] In some feasible embodiments, step S1 specifically includes:

[0071] S1.1. Collect real data;

[0072] S1.2. Create a synthetic pedestrian model and an air-ground camera cluster scene, place the synthetic pedestrian model in the air-ground camera cluster scene and move it to obtain virtual data;

[0073] MakeHuman and Unity3D tools are used to create synthetic pedestrian models and air-ground camera group scenes. The synthetic human models are placed in the air-ground camera group for random movement, thereby generating a large number of high-quality synthetic data images combined with real images.

[0074] S1.3. Combine real data and virtual data to obtain a data set.

[0075] Through this embodiment, a large number of high-quality synthetic data images are generated and combined with real images to expand the scale of training data at a low cost.

[0076] In some feasible embodiments, step S3.2 specifically includes:

[0077] S3.2.1, image serialization, constructing image password;

[0078] For every input image x i , it is divided into M = 128 16 × 16 image blocks without overlap. On this basis, for each image block, it is straightened into a d = 768-dimensional feature vector, that is, the image password t p,i More specifically, the above serialization operation can be implemented by a convolutional layer (where the convolution kernel size is 16×16, the stride is 16, and the number of convolution kernels is 768), which ultimately maps the input image of size 256×128×3 to an M×d input image matrix.

[0079] S3.2.2. Add global password and view password.

[0080] The 128 image passwords t obtained in S3.2.1 p,i On the basis of , add a global password t of the same dimension (d = 768) m and a perspective command t v It is worth noting that t m and t v are learnable variables that are randomly initialized and appended to the image password. There is no explicit constraint on the row positions of the two passwords appended to the input image matrix. Without loss of generality, this embodiment will respectively m and t v Append to the first and last rows to get the input image matrix with dimension (M+2)×d.

[0081] In some feasible embodiments, the step S3.3 specifically includes:

[0082] For image password t p,i In this embodiment, the corresponding one-dimensional index number is assigned according to its original position in the image; for the global password t m and perspective command t v , and give it two additional position features. Specifically, the position features come from a position matrix E with a dimension of (M+2)×d pos, each row of the matrix corresponds to the position feature of the input image matrix. It is worth noting that E pos is a learnable variable, which is randomly initialized and added to the image matrix. i The modification can be formally defined as follows:

[0083]

[0084] At this point, the data preparation for the input network structure is completed.

[0085] In some feasible embodiments, step S3.4 specifically includes:

[0086] The input matrix obtained from S3.3 In the view-decoupled self-attention model, forward propagation is performed block by block to participate in the calculation. In each view-decoupled self-attention block, After the passwords are interacted with each other, the updated password features are calculated through the encoding layer. The feature interaction method in this encoding layer is the same as the encoding layer feature calculation strategy in ViT-B. The dimension of the password features does not change before and after passing through the encoding layer, so the above operation can be written as:

[0087]

[0088] in, represents the j-th view-decoupled self-attention block in the proposed model, Represents the global password and view password output by the j-th view-decoupled self-attention block.

[0089] In some feasible embodiments, the S3.5 specifically includes:

[0090] The view feature is separated from the global password, and the separated global feature is used to replace the original global feature to participate in the subsequent self-attention block operation, which can be recorded as:

[0091]

[0092] In some feasible embodiments, the S3.6 specifically includes:

[0093] Repeat S3.4 and S3.5 in each view-decoupled self-attention block until the operation is completed in the last block to obtain the global password feature and perspective command features Since feature subtraction is performed in each self-attention block (separating the view password feature from the global password feature), the global password feature obtained here is Can be recorded as perspective-independent identity features, perspective password features It can be recorded as the viewing angle feature. So far, the feature extraction of the viewing angle related and the viewing angle independent of the input image has been completed.

[0094] In some feasible embodiments, step S3.7 specifically includes:

[0095] S3.7.1. Calculate a first loss value based on the identity feature and the cross entropy loss function;

[0096] This loss is used to determine whether the target identity of the input image is correctly expressed. The specific form is:

[0097]

[0098] Among them, |B| represents the number of training samples in a batch, y i and Represents the true identity label for which loss needs to be calculated and the identity label predicted by the view-decoupled self-attention model.

[0099] S3.7.2. Calculate a second loss value based on the identity feature and the difficult triplet loss function;

[0100] This loss enables the view-independent features to achieve better feature cohesion based on the target identity. The specific form is:

[0101]

[0102] Among them, |B| represents the number of training samples in a batch, and is x i In the hard positive and hard negative samples in the current batch, ‖·‖2 represents the L2 norm of the feature, α is a hyperparameter that balances the distance between positive and negative sample pairs, [·] + Represents the function max(·, 0).

[0103] S3.7.3. Calculate a third loss value based on the view feature and the cross entropy loss function;

[0104] This loss is used to determine whether the view label of the input image (from a ground camera or a drone camera) is correctly expressed. The specific form is:

[0105]

[0106] Among them, |B| represents the number of training samples in a batch, v i and Represents the true view label for which loss needs to be calculated and the view label predicted by the view-decoupled self-attention model.

[0107] S3.7.4. Calculate a fourth loss value based on the identity feature, the view feature, and the feature orthogonal loss function;

[0108] This loss reduces the redundancy between the two features, and each focuses on different semantic features of the input image. The specific form is:

[0109]

[0110] Among them, |<·,·>| represents the absolute value of the inner product of the two features, and N represents the total number of layers of the Vision Transformer Base network.

[0111] S3.7.5. Update the parameters of the perspective decoupling model by combining the first loss value, the second loss value, the third loss value, and the fourth loss value to obtain a recognition model.

[0112] Finally, the four sub-loss functions mentioned above are added together to obtain the final optimization goal of the network:

[0113]

[0114] Here, λ is a hyperparameter that balances identity learning and perspective learning.

[0115] like Figure 3 As shown in FIG, a person re-identification system based on a view-decoupled self-attention network includes:

[0116] Data construction module, used to construct data sets;

[0117] Model building module, building a perspective decoupling model based on the Vision Transformer Base network

[0118] A model training module trains the view decoupling model based on the data set to obtain a recognition model; the training process is as follows: inputting the data set into the view decoupling model; segmenting the images in the data set and constructing feature vectors to obtain image passwords, global passwords and view passwords; assigning position features to the image passwords, the global passwords and the view passwords to obtain a final input image matrix; propagating the final input image matrix forward block by block in the view decoupling model to participate in calculations, updating the global password and the view password to obtain updated passwords; separating the view password from the updated password, and replacing the global password with the separated global password to participate in the calculation of subsequent self-attention blocks; looping the steps of password updating and password separation until the calculations of all self-attention blocks are completed to obtain identity features and view features; calculating the loss function based on the identity features and view features, and updating the view decoupling model to obtain a recognition model;

[0119] The model application module processes the test data based on the recognition model to obtain a recognition result.

[0120] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0121] A person re-identification device based on view-decoupled self-attention network:

[0122] at least one processor;

[0123] at least one memory for storing at least one program;

[0124] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned pedestrian re-identification method based on the view-decoupled self-attention network.

[0125] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0126] A storage medium stores processor-executable instructions, which, when executed by a processor, are used to implement a person re-identification method based on a view-decoupled self-attention network as described above.

[0127] The contents of the above method embodiments are all applicable to the present storage medium embodiment. The functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0128] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A person re-identification method based on view-decoupled self-attention network, characterized by: The following steps are involved: Build a dataset; Build a perspective decoupling model based on the Vision Transformer Base network; Training the perspective decoupling model based on the data set to obtain a recognition model; Processing the test data based on the recognition model to obtain a recognition result; The training process is as follows: inputting the data set into the perspective decoupling model; Segmenting the images in the data set and constructing feature vectors to obtain image passwords, global passwords, and viewpoint passwords; Assigning position features to the image password, the global password, and the viewing angle password to obtain a final input image matrix; Propagate the final input image matrix forward block by block in the view decoupling model to participate in calculation, update the global password and the view password, and obtain an updated password; Separate the view command from the updated command, and use the separated global command to replace the global one in the subsequent self-attention block operation; Repeat the steps of password updating and password separation until all self-attention blocks are calculated to obtain identity features and view features; A loss function is calculated based on the identity feature and the view feature, and the view decoupling model is updated to obtain a recognition model.

2. The person re-identification method based on view-decoupled self-attention network according to claim 1, characterized in that: The step of constructing a data set specifically includes: Collect real data; Creating a synthetic pedestrian model and an open-ground camera group scene, placing the synthetic pedestrian model in the open-ground camera group scene for movement to obtain virtual data; Combine real data and virtual data to get the data set.

3. The person re-identification method based on view-decoupled self-attention network according to claim 1, characterized in that: The loss functions include cross entropy loss function, difficult triple loss function and feature orthogonal loss function.

4. The person re-identification method based on view-decoupled self-attention network according to claim 3, characterized in that: The step of calculating the loss function based on the identity feature and the view feature, and updating the view decoupling model to obtain the recognition model specifically includes: Calculating a first loss value based on the identity feature and a cross entropy loss function; Calculating a second loss value based on the identity feature and the difficulty triplet loss function; Calculating a third loss value based on the perspective feature and the cross entropy loss function; Calculating a fourth loss value based on the identity feature, the view feature, and a feature orthogonal loss function; The parameters of the perspective decoupling model are updated in combination with the first loss value, the second loss value, the third loss value, and the fourth loss value to obtain a recognition model.

5. The person re-identification method based on view-decoupled self-attention network according to claim 4, characterized in that: The calculation formula of the fourth loss value is as follows: Among them, the value of |B| is the number of training samples in a batch, ||·||2 represents the L2 norm of the feature, Indicates identity characteristics, represents the viewing angle feature, |<·,·>| represents the absolute value of the inner product of the two features, N+1 represents the output of the last layer of the network, x i represents the i-th input image.

6. The person re-identification method based on view-decoupled self-attention network according to claim 1, characterized in that: Also includes: Dividing the data set into a training set and a test set; Performing data enhancement on the training set to obtain a data-enhanced training set; The training set after data enhancement is used for training.

7. A person re-identification system based on view-decoupled self-attention network, characterized by: include: Data construction module, used to construct data sets; Model building module, building a perspective decoupling model based on the Vision Transformer Base network A model training module trains the view decoupling model based on the data set to obtain a recognition model; the training process is as follows: inputting the data set into the view decoupling model; segmenting the images in the data set and constructing feature vectors to obtain image passwords, global passwords, and view passwords; Assigning position features to the image password, the global password, and the view password to obtain a final input image matrix; forward-propagating the final input image matrix block by block in the view decoupling model to participate in calculation, updating the global password and the view password to obtain an updated password; Separating the view password from the updated password and using the separated global password to replace the global password in the calculation of subsequent self-attention blocks; looping the password updating and password separation steps until all self-attention blocks are calculated to obtain identity features and view features; calculating the loss function based on the identity features and view features, and updating the view decoupling model to obtain a recognition model; The model application module processes the test data based on the recognition model to obtain a recognition result.

8. A person re-identification device based on view-decoupled self-attention network, characterized by: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a pedestrian re-identification method based on a view-decoupled self-attention network as described in any one of claims 1-6.

Images