Attribute and image cross-modal pedestrian re-identification method and device based on double-branch transformer network

Abstract

The application discloses a kind of attribute and image cross-modal pedestrian re-identification method and device based on double-branch Transformer network, obtain pedestrian attribute and pedestrian image, pedestrian image comes from pedestrian registration image set;Double-branch Transformer network is constructed, and double-branch Transformer network is trained, and attribute and image cross-modal pedestrian re-identification model is obtained, double-branch Transformer network includes the attribute Transformer branch and image Transformer branch of same type;Pedestrian attribute and pedestrian image are input into attribute and image cross-modal pedestrian re-identification model, and attribute feature and image feature are extracted by attribute Transformer branch and image Transformer branch respectively;Attribute feature and image feature are compared with similarity, and the re-identification result corresponding to pedestrian attribute is obtained.Attribute Transformer branch and image Transformer branch all belong to the same type Transformer structure, which is conducive to controlling the modal heterogeneity problem of text attribute and pedestrian image in feature space, thereby improving attribute-image cross-modal pedestrian re-identification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, in particular to an attribute and image cross-modal pedestrian re-identification method and device based on a double-branch Transformer network. BACKGROUND

[0002] In practice, some cases have only witnesses but no image evidence, and only the witness's attribute description of the suspect, such as gender, hairstyle, clothing, etc., can be used as a retrieval condition to search for matching pedestrian images from a pedestrian image set. Attribute-image cross-modal pedestrian re-identification aims to use pedestrian attributes as queries to find matching pedestrian images from a massive pedestrian image database, which has reuse applications for searching the activity track of a criminal suspect in the absence of images. Since pedestrian attributes and pedestrian images belong to two different data modalities, existing methods usually use different feature learning structures to learn features separately, such as a fully connected neural network for pedestrian attributes and a convolutional neural network for pedestrian images. However, the differences in feature learning structures further exacerbate the modality heterogeneity problem of text attributes and pedestrian images in the feature space, which restricts the accuracy of attribute-image cross-modal pedestrian re-identification. SUMMARY

[0003] To solve the above technical problems mentioned in the background section, the embodiments of the present application propose an attribute and image cross-modal pedestrian re-identification method and device based on a double-branch Transformer network.

[0004] In a first aspect, the present application provides an attribute and image cross-modal pedestrian re-identification method based on a double-branch Transformer network, comprising the following steps:

[0005] S1, obtaining pedestrian attributes and pedestrian images, the pedestrian images coming from a pedestrian registration image set;

[0006] S2, constructing a double-branch Transformer network and training the double-branch Transformer network to obtain an attribute and image cross-modal pedestrian re-identification model, the double-branch Transformer network including attribute Transformer branches and image Transformer branches of the same type;

[0007] S3, inputting the pedestrian attributes and pedestrian images into the attribute and image cross-modal pedestrian re-identification model to extract attribute features and image features through the attribute Transformer branches and the image Transformer branches, respectively;

[0008] S4, comparing the attribute features and the image features for similarity to obtain a re-identification result corresponding to the pedestrian attributes.

[0009] As preferred, the attribute Transformer branch comprises an attribute embedding module, a first encoder and an attribute feature fusion layer, the attribute embedding module comprises a grouping encoding layer, a first full connection layer and a first position encoding layer, in the grouping encoding layer, the pedestrian attribute is divided into several equal-length attribute groups by a windowing strategy, and the several attribute groups are input into the first full connection layer to extract features to obtain a set of feature embeddings of all attribute groups, and the first position encoding layer provides a set of first position encodings for the feature embeddings of all attribute groups.

[0010] As preferred, the calculation process of the grouping encoding layer is as follows:

[0011] C = [C1, C2,..., Cn] ; n ];

[0012] Wherein, s i ∈ [1, n-m+1] ;

[0013] Wherein, A is the pedestrian attribute, the s i+1 th pedestrian attribute of A; C represents a set of attribute groups after grouping, C i represents the i i th attribute group, i = 1, 2,..., n, and m represents the vector length of C

[0014] The calculation process of the first full connection layer is as follows:

[0015]

[0016] Wherein, FC represents a full connection structure, θ is a learnable parameter of FC; f is a set of feature embeddings of all attribute groups, f i represents the feature embedding learned by FC for the i i th attribute group C i , and d is the vector length of f.

[0017] The calculation process of the first position encoding layer is as follows:

[0018]

[0019]

[0020] Wherein, attpos i is the i iis the first position encoding of the i-th attribute group, which is a d-dimensional feature embedding; i represents the attribute group serial number, i = 1, 2, 3,..., n; k is the dimension serial number of the feature embedding of each attribute group, k = 1, 2, 3,..., d.

[0021] As preferred, the attribute Transformer branch further comprises a first encoder and an attribute feature fusion layer, and the set of feature embeddings of all attribute groups and the set of first position encodings are input into the first encoder for global encoding to obtain global encoding features corresponding to all attribute groups, and the calculation formula is as shown below:

[0022] g = Encode (f + att pos) ;

[0023] Wherein, Encode represents a stacked encoder structure, and g is the global encoding feature corresponding to all attribute groups.

[0024] The global encoding features corresponding to all attribute groups are input into the attribute feature fusion layer for feature fusion to obtain attribute features, and the calculation formula is as shown below:

[0025] a = FC (g, ω) ;

[0026] Wherein, FC represents a full connection structure, ω is a learnable parameter of FC, and a represents the attribute feature learned by FC from the feature g of all attribute groups.

[0027] As preferred, the image Transformer branch comprises an image embedding module, a second encoder and an image feature fusion layer, the first encoder and the second encoder adopt the same encoding structure, and the attribute feature fusion layer and the image feature fusion layer both adopt a full connection structure.

[0028] As preferred, the image embedding module comprises a block encoding layer, a second full connection layer, a flag module and a second position encoding layer; the pedestrian image is input into the block encoding layer, and the pedestrian image is divided into N image blocks Wherein, H x W represents the resolution of the pedestrian image, C represents the channel number, h x w represents the resolution of each image block, i is the image block serial number, represents the number of image blocks.

[0029] A plurality of image blocks are input into the second full connection layer, and the feature embedding set of all image blocks is obtained through feature extraction by the second full connection layer, and the calculation formula is as shown below:

[0030]

[0031] Wherein, FC represents a full connection structure, η is a learnable parameter of FC, h is the feature embedding set of all image blocks, and hi represents the i-th image patch x i learned feature embedding by FC; d is the vector dimension of its feature embedding;

[0032] In the logo module, the logo parameters are obtained, and the calculation formula is as follows:

[0033]

[0034] where t is the initialized logo parameter, and d is the vector dimension thereof;

[0035] A second position encoding layer is used to provide a set of second position encodings for the feature embeddings of all image patches, and the calculation formula is as follows:

[0036]

[0037]

[0038] where imgpos i is the feature embedding h i of the image patch, and the corresponding second position encoding is a d-dimensional feature embedding; k is the dimension sequence number of the feature embedding of each image patch, k = 1, 2, 3,..., d; when i = N + 1, the corresponding imgpos N+1 is the second position encoding of the logo parameter t;

[0039] The logo parameter t is expanded and then concatenated with the set of feature embeddings h of all attribute groups, and the set of second position encodings is input into a second encoder for global encoding to obtain the global encoding features corresponding to all image patches, and the calculation formula is as follows:

[0040] l = Encoder (ExtCat (h, t) + imgpos);

[0041] where Encoder represents a number of stacked encoder structures, ExtCat represents the concatenation operation of the expanded logo parameter t and the set of feature embeddings h of all attribute groups, and l is the global encoding feature corresponding to all image patches;

[0042] The global encoding feature corresponding to the logo parameter is input into an image feature fusion layer for feature fusion to obtain the image feature, and the calculation formula is as follows:

[0043]

[0044] where FC represents a fully connected structure, is the learnable parameter of FC; l N+1 represents the global encoding feature corresponding to the logo parameter; o represents the lN+1 image features learned via FC.

[0045] As preferred, a binary cross-entropy loss function is used in the training process of the dual-branch Transformer network, and the calculation formula is as follows:

[0046]

[0047] wherein, is the image Transformer branch, and ξ is the learnable parameter thereof; is the attribute Transformer branch, and ζ is the learnable parameter thereof;

[0048] is the learnable parameter of the binary cross-entropy loss function, d represents the feature dimension extracted by the attribute Transformer branch and the image Transformer branch, n is the number of attributes; img and att are randomly extracted image sample set and attribute sample set, and the sample number of both is z; and respectively represent the attribute feature of the i-th attribute sample att i and the image feature of the i-th image sample img i .

[0049] In a second aspect, the application provides an attribute and image cross-modal pedestrian re-identification device based on a dual-branch Transformer network, comprising:

[0050] a data acquisition module configured to acquire pedestrian attributes and pedestrian images, the pedestrian images being from a pedestrian registration image set;

[0051] a model construction module configured to construct a dual-branch Transformer network, and train the dual-branch Transformer network to obtain an attribute and image cross-modal pedestrian re-identification model, the dual-branch Transformer network comprising an attribute Transformer branch and an image Transformer branch of the same type;

[0052] a feature extraction module configured to input the pedestrian attributes and the pedestrian images into the attribute and image cross-modal pedestrian re-identification model, and extract attribute features and image features through the attribute Transformer branch and the image Transformer branch respectively;

[0053] a comparison module configured to compare the attribute features and the image features in similarity, and obtain a re-identification result corresponding to the pedestrian attributes.

[0054] In a third aspect, the present application provides an electronic device, comprising one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementation manners of the first aspect.

[0055] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, when the computer program is executed by a processor, the method described in any of the implementation manners of the first aspect is implemented.

[0056] Compared with the prior art, the present application has the following beneficial effects:

[0057] (1) The attribute and image cross-modal pedestrian re-identification method based on the dual-branch Transformer network proposed in the present application designs a dual-branch Transformer structure of the same type for pedestrian attribute and pedestrian image feature learning, controls the modal heterogeneity of the text attribute and the pedestrian image in the feature space at the level of the feature learning structure, and thus improves the attribute-image cross-modal pedestrian re-identification accuracy.

[0058] (2) In the attribute and image cross-modal pedestrian re-identification method based on the dual-branch Transformer network proposed in the present application, the attribute Transformer branch and the image Transformer branch learn features from the pedestrian attribute and the pedestrian image respectively, and can complete the text attribute feature learning and image feature learning tasks respectively. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0060] Figure 1 is an example device architecture diagram to which an embodiment of the present application can be applied;

[0061] Figure 2 is a flowchart of the attribute and image cross-modal pedestrian re-identification method based on the dual-branch Transformer network of the embodiment of the present application;

[0062] Figure 3 is a schematic diagram of the attribute and image cross-modal pedestrian re-identification model based on the dual-branch Transformer network of the embodiment of the present application;

[0063] Figure 4This is a schematic diagram of a cross-modal pedestrian re-identification device based on a dual-branch Transformer network and images, representing an embodiment of this application.

[0064] Figure 5 This is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the present application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0066] Figure 1 An exemplary device architecture 100 is shown, which can be applied to the attribute and image cross-modal pedestrian re-identification method or the attribute and image cross-modal pedestrian re-identification device based on the dual-branch Transformer network of the present application embodiments.

[0067] like Figure 1 As shown, the device architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0068] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications, such as data processing applications and file processing applications, can be installed on terminal devices 101, 102, and 103.

[0069] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0070] The server 105 can be a server providing various services, such as a background data processing server processing files or data uploaded by the terminal devices 101, 102, and 103. The background data processing server can process the obtained files or data to generate a processing result.

[0071] It should be noted that the attribute and image cross-modal pedestrian re-identification method based on the dual-branch Transformer network provided in the embodiments of the present application can be executed by the server 105, or can be executed by the terminal devices 101, 102, and 103. Correspondingly, the attribute and image cross-modal pedestrian re-identification device based on the dual-branch Transformer network can be arranged in the server 105, or can be arranged in the terminal devices 101, 102, and 103.

[0072] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above-mentioned device architecture is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks, and servers. In the case where the data to be processed does not need to be obtained remotely, the above-mentioned device architecture can not include a network, but only a server or a terminal device.

[0073] Figure 2 An attribute and image cross-modal pedestrian re-identification method based on a dual-branch Transformer network provided by an embodiment of the present application is shown, which includes the following steps:

[0074] S1, obtaining a pedestrian attribute and a pedestrian image, the pedestrian image being from a pedestrian registration image set.

[0075] Specifically, the pedestrian attribute q is used as a query parameter, and the attribute and image cross-modal pedestrian re-identification method based on the dual-branch Transformer network provided by the embodiments of the present application is used to find a pedestrian image matching the pedestrian attribute in the pedestrian registration image set G.

[0076] S2, constructing a dual-branch Transformer network and training the dual-branch Transformer network to obtain an attribute and image cross-modal pedestrian re-identification model, the dual-branch Transformer network including a same-type attribute Transformer branch and an image Transformer branch.

[0077] In specific embodiments, referring to Figure 3 The attribute Transformer branch includes an attribute embedding module, a first encoder, and an attribute feature fusion layer. The attribute embedding module includes a grouping encoding layer, a first full connection layer, and a first position encoding layer. The pedestrian attribute is input into the grouping encoding layer, and is divided into a plurality of equal-length attribute groups by a windowing strategy. The calculation process is as follows:

[0078] C = [C1, C2,..., Cn] ; n ];

[0079] where s i ∈ [1, n-m+1] ;

[0080] where A is pedestrian attribute, represents the s i+1 th pedestrian attribute of A; C represents the attribute group set after grouping, C i represents the i-th attribute group, i = 1, 2,..., n, m represents the vector length of C i , n is the number of groups. The correlation between attribute groups C i is calculated by the Transformer attention mechanism, focusing on key attribute groups.

[0081] A number of attribute groups are input into the first fully connected layer, and feature extraction is performed through the first fully connected layer to obtain a set of feature embeddings of all attribute groups. The calculation process is as shown in the following formula:

[0082]

[0083] where FC represents the fully connected structure, θ is the learnable parameter of FC; f is the set of feature embeddings of all attribute groups, f i represents the feature embedding of the i-th attribute group C i learned by FC, and d is the vector length of f i .

[0084] The first position encoding layer provides the first position encoding for the feature embedding of each attribute group, and the first position encoding layer provides a set of first position encodings for the feature embedding of all attribute groups. The calculation process is as shown in the following formula:

[0085]

[0086]

[0087] where attpos i is the first position encoding of the i-th attribute group C i , which is a d-dimensional feature vector; i represents the attribute group number, i = 1, 2, 3,..., n; k is the dimension number of the feature embedding of each attribute group, k = 1, 2, 3,..., d, and the first position encoding layer provides position information for each f i .

[0088] In specific embodiments, the attribute Transformer branch further comprises a first encoder and an attribute feature fusion layer, the set of feature embeddings of all attribute groupings and the set of first positional encodings are input into the first encoder for global encoding to obtain global encoding features corresponding to all attribute groupings, and the calculation formula is as shown below:

[0089] g = Encode (f + attpos) ;

[0090] wherein, Encode represents a number of stacked encoder structures, and g is the global encoding features corresponding to all attribute groupings. The global encoding of f + attpos is obtained by using the encoder structure, and the main operations of the first encoding layer include multi-head attention calculation, normalization, full connection, etc., and a plurality of encoder structures can be stacked in practice to increase the feature learning ability

[0091] The global encoding features corresponding to all attribute groupings are input into the attribute feature fusion layer for feature fusion to obtain attribute features, and the calculation formula is as shown below:

[0092] a = FC (g, ω) ;

[0093] wherein, FC represents a full connection structure, ω is a learnable parameter of FC, and a represents attribute features learned by FC from the features g corresponding to all attribute groupings.

[0094] In specific embodiments, the image Transformer branch comprises an image embedding module, a second encoder and an image feature fusion layer, the first encoder and the second encoder adopt the same encoding structure, and the attribute feature fusion layer and the image feature fusion layer both adopt a full connection structure.

[0095] In specific embodiments, the image embedding module comprises a block encoding layer, a second full connection layer, a flag module and a second position encoding layer; the pedestrian image is input into the block encoding layer, and the pedestrian image is divided into N image blocks wherein, H x W represents the resolution of the pedestrian image, C represents the number of channels, h x w represents the resolution of each image block, i is the image block number, and N represents the number of image blocks.

[0096] A plurality of image blocks are input into the second full connection layer, and feature extraction is performed through the second full connection layer to obtain a set of feature embeddings of all image blocks, and the calculation formula is as shown below:

[0097]

[0098] wherein, FC represents a full connection structure, η is a learnable parameter of FC, h is the set of feature embeddings of all image blocks, and h ixi i the feature embedding learned by FC; d is the vector dimension of its feature embedding.

[0099] In the logo module, the logo parameters are obtained, and the calculation formula is as follows:

[0100]

[0101] where t is the initialized logo parameter, and d is the vector dimension thereof.

[0102] The second position encoding layer provides a set of second position encodings for the feature embeddings of all image patches, and the calculation formula is as follows:

[0103]

[0104]

[0105] where imgpos i is the feature embedding h i of the image patch; k is the dimension number of the feature embedding of each image patch, k = 1, 2, 3,..., d; when i = N + 1, the corresponding imgpos N+1 is the second position encoding of the logo parameter t.

[0106] The logo parameter t is expanded and then concatenated with the set of feature embeddings h of all attribute groups, and the set of second position encodings is input into the second encoder for global encoding to obtain the global encoding features corresponding to all image patches, and the calculation formula is as follows:

[0107] l = Encoder (ExtCat (h, t) + imgpos);

[0108] where Encoder represents a structure of several stacked encoders, ExtCat represents the concatenation operation of the expanded logo parameter t and the set of feature embeddings h of all attribute groups, which will obtain a vector with the same dimension as imgpos; l is the global encoding feature corresponding to all image patches, the Encoder here adopts the same structure as the attribute Transformer branch encoder to learn the global encoding feature l, and the Encoder here also includes multi-head attention calculation, normalization, full connection, etc. In practice, multiple encoder structures can also be stacked to increase the feature learning ability.

[0109] The global encoding feature corresponding to the logo parameter is input into the image feature fusion layer for feature fusion to obtain the image feature, and the calculation formula is as follows:

[0110]

[0111] wherein FC denotes a full connection structure, is a learnable parameter of FC; l N+1 denotes a global encoding feature corresponding to the landmark parameter; o denotes l N+1 image features learned via FC.

[0112] In specific embodiments, a binary cross-entropy loss function is used in the training process of the dual-branch Transformer network, and the calculation formula is as follows:

[0113]

[0114] wherein, is an image Transformer branch, and ξ is a learnable parameter thereof; is an attribute Transformer branch, and ζ is a learnable parameter thereof; is a learnable parameter of the binary cross-entropy loss function, d denotes the feature dimension extracted by the attribute Transformer branch and the image Transformer branch, n is the number of attributes; img and att are randomly extracted image sample sets and attribute sample sets, and the sample number of both is z; and denote the attribute feature of the i-th attribute sample att i and the image feature of the i-th image sample img i .

[0115] Specifically, in the present embodiment, the training process of the dual-branch Transformer network includes the following steps:

[0116] (1) Construct a pedestrian attribute data set and a pedestrian image data set, and perform data augmentation on the images in the pedestrian image data set, including random cropping, horizontal flipping, random occlusion, etc., to obtain an image sample set and an attribute sample set.

[0117] (2) Construct a dual-branch Transformer network. As shown in Figure 3 , the dual-branch Transformer network is composed of an image Transformer branch and an attribute Transformer branch.

[0118] (3) Randomly extract samples from the image sample set and the attribute sample set, and train the dual-branch Transformer network using a binary cross-entropy loss function. The training process is optimized using a stochastic gradient descent method until convergence, thereby obtaining an attribute and image cross-modal pedestrian re-identification model based on the dual-branch Transformer network.

[0119] S3, input the pedestrian attribute and the pedestrian image input attribute and image cross-modal pedestrian re-identification model, respectively through attribute Transformer branch and image Transformer branch to extract attribute features and image features.

[0120] S4, compare the attribute features and the image features for similarity, and obtain the re-identification result corresponding to the pedestrian attribute.

[0121] Specifically, the test process of the embodiment includes the following steps:

[0122] (1) The attribute and image cross-modal pedestrian re-identification model is used to extract the features of the pedestrian attribute q to be queried and the pedestrian registration image set G respectively, and the attribute features and image features are obtained. As shown in the following formula (1), the attribute features and image features refer to the outputs of the attribute feature fusion layer and the image feature fusion layer in the attribute Transformer branch and the image Transformer branch respectively. Figure 3

[0123] (2) Based on the extracted attribute features and image features, the Euclidean distances between the attribute features of the pedestrian attribute q to be queried and all image features in the pedestrian registration image set G are calculated, and sorted, and the pedestrian registration images with close distances are selected as the re-identification results of the pedestrian attribute q to be queried.

[0124] Further referring to Figure 4 , as an implementation of the method shown in the above figures, the application provides an embodiment of an attribute and image cross-modal pedestrian re-identification device based on a double-branch Transformer network. The device embodiment corresponds to the method embodiment shown in Figure 2 , and the device can be applied to various electronic devices.

[0125] The application embodiment provides an attribute and image cross-modal pedestrian re-identification device based on a double-branch Transformer network, which includes:

[0126] The data acquisition module 1 is configured to acquire pedestrian attributes and pedestrian images, and the pedestrian images come from a pedestrian registration image set.

[0127] The model construction module 2 is configured to construct a double-branch Transformer network, train the double-branch Transformer network, obtain an attribute and image cross-modal pedestrian re-identification model, and the double-branch Transformer network includes an attribute Transformer branch and an image Transformer branch of the same type.

[0128] ​The feature extraction module 3 is configured to input the pedestrian attribute and the pedestrian image input attribute into the image cross-modal pedestrian re-identification model through an attribute Transformer branch and an image Transformer branch respectively to extract attribute features and image features;

[0129] The comparison module 4 is configured to compare the attribute features and the image features in similarity to obtain a re-identification result corresponding to the pedestrian attribute.

[0130] Reference will be made to the following description and drawings Figure 5 which shows a structural schematic diagram of a computer device 500 of an electronic device (for example Figure 1 a server or a terminal device) suitable for being used to implement the embodiments of the present application. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0131] As shown in Figure 5 , the computer device 500 includes a central processing unit (CPU) 501 and a graphics processor (GPU) 502, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 503 or programs loaded from a storage portion 509 into a random access memory (RAM) 504. In the RAM 504, various programs and data required for the operation of the device 500 are also stored. The CPU 501, the GPU 502, the ROM 503, and the RAM 504 are connected to each other through a bus 505. An input / output (I / O) interface 506 is also connected to the bus 505.

[0132] The following components are connected to the I / O interface 506: an input portion 507 including a keyboard, a mouse, and the like; an output portion 508 including a display such as a cathode ray tube (CRT) display, a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage portion 509 including a hard disk, and the like; and a communication portion 510 including a network interface card such as a LAN card, a modem, and the like. The communication portion 510 performs communication processing via a network such as the Internet. A drive 511 can also be connected to the I / O interface 506 as needed. A removable recording medium 512 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 511 as needed, so that a computer program read therefrom is installed in the storage portion 509 as needed.

[0133] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 510, and / or installed from the removable medium 512. When the computer program is executed by the central processing unit (CPU) 501 and the graphics processor (GPU) 502, the above-described functions defined in the methods of the present application are executed.

[0134] It should be noted that the computer readable medium described in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor device, device or means, or any combination of the above. More specific examples of computer readable medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, device or means. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable medium that can send, propagate or transmit the program for use by or in conjunction with an instruction execution device, device or means. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0135] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0136] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0137] The modules described in the embodiments of the present application can be implemented by software, or by hardware. The described modules can also be arranged in a processor.

[0138] As another aspect, the present application also provides a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: obtain a pedestrian attribute and a pedestrian image, the pedestrian image being from a pedestrian registration image set; construct a double-branch Transformer network and train the double-branch Transformer network to obtain an attribute and image cross-modal pedestrian re-identification model, the double-branch Transformer network including an attribute Transformer branch and an image Transformer branch of the same type; input the pedestrian attribute and the pedestrian image into the attribute and image cross-modal pedestrian re-identification model, and extract an attribute feature and an image feature through the attribute Transformer branch and the image Transformer branch, respectively; and compare the attribute feature and the image feature in similarity, to obtain a re-identification result corresponding to the pedestrian attribute.

[0139] The above description is merely the preferred embodiments of the present application and the explanation of the technical principles used. It should be understood by those skilled in the art that the inventive scope of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features can be replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form the technical solutions.

Claims

1. A method for cross-modal pedestrian re-identification based on attributes and images using a dual-branch Transformer network, characterized in that, Includes the following steps: S1, Obtain pedestrian attributes and pedestrian images, wherein the pedestrian images are from the pedestrian registered image set; S2. Construct a two-branch Transformer network and train it to obtain a cross-modal pedestrian re-identification model based on attributes and images. The two-branch Transformer network includes a homogeneous attribute Transformer branch and an image Transformer branch. The attribute Transformer branch includes an attribute embedding module, a first encoder, and an attribute feature fusion layer. The attribute embedding module includes a group coding layer, a first fully connected layer, and a first position coding layer. The pedestrian attributes are input into the group coding layer and divided into several attribute groups of equal length using a windowing strategy. These attribute groups are then input into the first fully connected layer for feature extraction, resulting in a set of feature embeddings for all attribute groups. The first position coding layer provides a set of first position codes for the feature embeddings of all attribute groups. The set of feature embeddings and the set of first position codes for all attribute groups are then input into the first encoder for global encoding, resulting in globally encoded features corresponding to all attribute groups. The calculation formula is shown below: ； Where Encode represents several stacked encoder structures, and g is the global encoded feature corresponding to all attribute groups; The global encoded features corresponding to all attribute groups are input into the attribute feature fusion layer for feature fusion to obtain the attribute features. The calculation formula is as follows: ； Where FC represents a fully connected structure. Here are the learnable parameters for FC; 'a' represents the features corresponding to all attribute groups. Attributes and characteristics learned through FC; S3, the pedestrian attributes and pedestrian image input attributes are combined with the image cross-modal pedestrian re-identification model, and attribute features and image features are extracted respectively through the attribute Transformer branch and the image Transformer branch; S4, compare the similarity between the attribute features and the image features to obtain the re-identification result corresponding to the pedestrian attributes.

2. The method for cross-modal pedestrian re-identification based on attributes and images using a dual-branch Transformer network according to claim 1, characterized in that, The calculation process of the block coding layer is shown in the following formula: ； ,in, ; Where A represents the pedestrian attribute. Describes the sth term of A i+1 Individual pedestrian attributes; C represents the set of attribute groups after grouping, C i This indicates that it is a group based on the i-th attribute. m represents C i The length of the vector, where n is the number of groups; The calculation process of the first fully connected layer is shown in the following formula: ； Where FC represents a fully connected structure. These are the learnable parameters for FC (Full Functional). A set of feature embeddings grouped by all attributes. Indicates the i-th attribute group The feature embedding learned by FC, d is The length of the vector; The calculation process of the first positional coding layer is shown in the following formula: ； ； in, Group C for the i-th attribute i The first position code is A dimensional feature vector; i represents the attribute grouping index. k is the dimension number of the feature embedding for each group attribute. .

3. The method for cross-modal pedestrian re-identification based on attributes and images using a dual-branch Transformer network according to claim 1, characterized in that, The image Transformer branch includes an image embedding module, a second encoder, and an image feature fusion layer. The first encoder and the second encoder adopt the same encoding structure, and both the attribute feature fusion layer and the image feature fusion layer adopt a fully connected structure.

4. The method for cross-modal pedestrian re-identification based on dual-branch Transformer networks according to claim 3, characterized in that, The image embedding module includes a block coding layer, a second fully connected layer, a marker module, and a second position coding layer; the pedestrian image is input into the block coding layer, and the pedestrian image is... Divide the image into N blocks. ,in, C represents the resolution of the pedestrian image, and C represents the number of channels. This represents the resolution of each image block, where i is the image block number, and N = Indicates the number of image blocks; The image blocks are input into the second fully connected layer, and feature extraction is performed through the second fully connected layer to obtain a set of feature embeddings for all image blocks. The calculation formula is as follows: ； Where FC represents a fully connected structure. represents the learnable parameters of the fully connected (FC) image; h is the set of feature embeddings for all image blocks. Represents the i-th image block The feature embeddings learned by FC; d is the vector dimension of its feature embeddings; The flag parameters are obtained in the flag module, and the calculation formula is as follows: ； Where t is the initial flag parameter, and d is its vector dimension; The second positional coding layer provides a set of second positional codes for the feature embeddings of all image blocks, and the calculation formula is as follows: ， ； ； in, embedding h for image blocks i The corresponding second position code is The feature embedding is dimensional; k is the dimension number of the feature embedding for each image block. When i = N+1, the corresponding It is the second position code of the flag parameter t; The flag parameter t is expanded and then concatenated with the set h of feature embeddings for all attribute groups. This concatenation is then combined with the set of second position codes and input into the second encoder for global encoding to obtain the global encoded features corresponding to all image blocks. The calculation formula is as follows: ； Where Encoder represents several stacked encoder structures, ExtCat represents the operation of expanding the flag parameter t and then concatenating it with the set h of feature embeddings grouped by all attributes, and l is the global encoded feature corresponding to all image blocks. The globally encoded features corresponding to the flag parameters are input into the image feature fusion layer for feature fusion to obtain the image features. The calculation formula is as follows: ； Where FC represents a fully connected structure. These are the learnable parameters for FC (Full Functional). This represents the global encoding feature corresponding to the flag parameter; o indicates... Image features learned through FC.

5. The method for cross-modal pedestrian re-identification based on attributes and images using a dual-branch Transformer network according to claim 1, characterized in that, The training process of the dual-branch Transformer network uses the binary cross-entropy loss function, and the calculation formula is as follows: ； in, For the image Transformer branch, Its learnable parameters; For the Transformer property branch, Its learnable parameters; These are the learnable parameters of the binary cross-entropy loss function. This represents the feature dimension extracted by the attribute Transformer branch and the image Transformer branch, where n is the number of attributes; img and att are randomly selected image sample sets and attribute sample sets, respectively, and the number of samples for both is z. and They represent the i-th attribute sample respectively. The attribute features and the i-th image sample Image features.

6. A cross-modal pedestrian re-identification device based on attributes and images using a dual-branch Transformer network, characterized in that, include: The data acquisition module is configured to acquire pedestrian attributes and pedestrian images, wherein the pedestrian images are from a registered pedestrian image set; The model building module is configured to construct a two-branch Transformer network and train it to obtain a cross-modal pedestrian re-identification model based on attributes and images. The two-branch Transformer network includes a homogeneous attribute Transformer branch and an image Transformer branch. The attribute Transformer branch includes an attribute embedding module, a first encoder, and an attribute feature fusion layer. The attribute embedding module includes a group coding layer, a first fully connected layer, and a first positional coding layer. Pedestrian attributes are input into the group coding layer and divided into several equal-length attribute groups using a windowing strategy. These attribute groups are then input into the first fully connected layer for feature extraction, resulting in a set of feature embeddings for all attribute groups. The first positional coding layer provides a set of first positional codes for the feature embeddings of all attribute groups. The set of feature embeddings and the set of first positional codes are then input into the first encoder for global encoding to obtain the globally encoded features corresponding to all attribute groups. The calculation formula is shown below: ； Where Encode represents several stacked encoder structures, and g is the global encoded feature corresponding to all attribute groups; The global encoded features corresponding to all attribute groups are input into the attribute feature fusion layer for feature fusion to obtain the attribute features. The calculation formula is as follows: ； Where FC represents a fully connected structure. Here are the learnable parameters for FC; 'a' represents the features corresponding to all attribute groups. Attributes and characteristics learned through FC; The feature extraction module is configured to extract attribute features and image features by combining the pedestrian attributes and pedestrian image input attributes with the image cross-modal pedestrian re-identification model through the attribute Transformer branch and the image Transformer branch, respectively. The comparison module is configured to perform a similarity comparison between the attribute features and image features to obtain the re-identification result corresponding to the pedestrian attributes.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images