Person re-identification method, person re-identification system, and computer-readable recording medium recording person re-identification program

Abstract

The present disclosure provides a technology capable of improving the precision of person re-identification, and relates to a person re-identification method, a person re-identification system, and a computer-readable recording medium recording a person re-identification program. According to the person re-identification method of the present disclosure, the posture of a target person is estimated in an image (10) of a person who is a re-identification target. Furthermore, according to the person re-identification method of the present disclosure, based on the estimated posture of the target person, a predetermined number of blocks (14) are cropped from the image (10) along the body of the target person, and in addition, position information (16) of each of the predetermined number of blocks (14) is generated. According to the person re-identification method of the present disclosure, the cropped predetermined number of blocks are input to a visual transformer encoder (132) together with the position information. The output of the visual transformer encoder (132) is input to a neural network (142) including an MLP, and the output of the neural network is obtained as a re-identification result of the target person.

Description

Technical Field

[0001] This disclosure relates to a person re-identification method, a person re-identification system, and a computer-readable recording medium containing a person re-identification program. Background Technology

[0002] Patent document 1 discloses a technique for re-identification using image data. According to this prior art, a learned model is used to detect faces, and the face position is detected based on image data. Furthermore, local image data extracted from the detected face position is generated. This local image data of the face is processed by an application processor for personal identification, facial authentication, and individual image collection, etc.

[0003] In addition to Patent Document 1, Patent Document 2 and Patent Document 3 may also be cited as documents indicating the level of technology in the technical field related to this disclosure.

[0004] Existing technical documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-025261

[0006] Patent Document 2: Japanese Patent Application Publication No. 2021-012707

[0007] Patent Document 1: Japanese Patent No. 6788929 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] The aforementioned existing technologies pertain to face re-identification, while research on person re-identification based on overall image data of people, such as pedestrian image data, is also underway. However, there is room for improvement in the accuracy of currently proposed person re-identification methods.

[0010] This disclosure is made in view of the problems mentioned above, and aims to provide a technique that can improve the accuracy of person re-identification.

[0011] Technical solutions for solving the problem

[0012] This disclosure provides a person re-identification technique for achieving the aforementioned objectives. In this person re-identification technique, a Vision Transformer (ViT) is applied for person re-identification. Compared to Convolutional Neural Networks (CNNs), a conventional image processing technique, the Vision Transformer is superior in terms of computational efficiency and scalability. Furthermore, the input to the encoder used in the Vision Transformer, i.e., the Vision Transformer encoder, is further improved in this person re-identification technique.

[0013] The person re-identification technology disclosed herein includes a person re-identification method, a person re-identification system, and a person re-identification program.

[0014] The person re-identification method disclosed herein includes the following steps: Step 1: Estimating the pose of the person to be re-identified (hereinafter referred to as the target person) from an image. Step 2: Based on the estimated pose of the target person, cropping (cutting out) a predetermined number of patches from the image along the body of the target person. Step 3: Generating positional information for each of the predetermined number of cropped patches. Step 4: Inputting the predetermined number of cropped patches along with their respective positional information into a visual transformer encoder. Step 5: Inputting the output of the visual transformer encoder into a neural network. Step 6: Obtaining the output of the neural network as the re-identification result of the target person. However, some of the above steps can be appropriately integrated.

[0015] The person re-recognition system disclosed herein includes one or more processors and a program memory, which, in conjunction with the one or more processors, stores a plurality of executable instructions. These executable instructions are configured to cause the one or more processors to perform the following processes: A first process is to estimate the pose of the subject person in an image. A second process is to crop a predetermined number of blocks from the image along the subject person's body based on the estimated pose. A third process is to generate position information for each of the predetermined number of cropped blocks. A fourth process is to input the predetermined number of cropped blocks along with their respective position information into a visual transformer encoder. A fifth process is to input the output of the visual transformer encoder into a neural network. A sixth process is to obtain the output of the neural network as the person re-recognition result. However, some of the above processes can be appropriately integrated.

[0016] The person re-identification program disclosed herein is configured to cause a computer to perform the following processes: A first process is to estimate the pose of the subject person from an image of the subject person. A second process is to crop a predetermined number of blocks from the image along the subject person's body based on the estimated pose. A third process is to generate position information for each of the predetermined number of cropped blocks. A fourth process is to input the predetermined number of cropped blocks along with their respective position information into a visual transformer encoder. A fifth process is to input the output of the visual transformer encoder into a neural network. A sixth process is to obtain the output of the neural network as the re-identification result of the subject person. However, some of the above processes can be appropriately integrated. Furthermore, the person re-identification program disclosed herein can also be recorded on a computer-readable recording medium.

[0017] According to the person re-identification technique disclosed herein, blocks are cropped from an image along the subject's body based on the subject's pose. Therefore, unwanted background portions of the image are cropped around the subject's body and excluded from the input of the visual transformer encoder. Furthermore, the size, number, and order of the blocks input to the visual transformer encoder are determined. By standardizing the data input to the visual transformer encoder in this way, the variance of the input data is reduced. This improves the performance of neural network-based recognition and increases the accuracy of person re-identification.

[0018] In the person re-identification technology disclosed herein, the estimated pose may also include the estimated joint positions of the person being identified. In this case, cropping a predetermined number of blocks may also include cropping the same number of blocks as the joints, centered on their positions, and generating position information may also include generating joint position information. By cropping blocks centered on their joint positions, the entire human body can be cut out evenly from the blocks. Furthermore, the positions of the blocks input to the vision transducer encoder can be made consistent (uniform).

[0019] Furthermore, in the person re-identification technology disclosed herein, the predetermined number of blocks may also include at least one pair of blocks that partially overlap with each other. By allowing the blocks to partially overlap, the portion of the human body not cut out according to the blocks can be reduced.

[0020] In the person re-identification technology disclosed herein, the aforementioned predetermined number can also be less than the number of segments required when dividing the image into blocks. Accordingly, the computational load can be reduced compared to the size of the blocks into which the image is divided.

[0021] Invention Effects

[0022] As described above, the person re-identification technology disclosed herein can improve the recognition performance based on neural networks and improve the accuracy of person re-identification. Attached Figure Description

[0023] Figure 1 This diagram illustrates the configuration of a system for implementing the person re-identification method according to embodiments of the present disclosure.

[0024] Figure 2 This is a diagram illustrating the features of the person re-identification method according to the embodiments of this disclosure.

[0025] Figure 3 This is a diagram illustrating an example of the hardware configuration of a person re-identification system according to an embodiment of this disclosure.

[0026] Label Explanation

[0027] 10 Image; 12 Joint; 14 Block; 16 Position Information; 100 Person Re-identification System; 110 Pose Estimation Unit; 120 Block Cropping Unit; 130 Feature Extraction Unit; 132 Visual Transformer Encoder; 134 Linear Embedding; 136 Position Embedding; 138 CLS Token; 140 Recognition Unit; 142 MLP (Multi-layer Perceptron); 200 Computer; 202 Processor; 204 Program Memory; 206 Instructions; 208 Data Storage; 220 Display Device; 240 Input Device. Detailed Implementation

[0028] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, when the number, quantity, quantity, range, etc., of each element are mentioned in the embodiments described below, the technical concept involved in the present disclosure is not limited to the mentioned number, unless specifically stated or clearly determined in principle. In addition, the structures, etc., described in the embodiments described below are not necessarily necessary for the technical concept involved in the present disclosure, unless specifically stated or clearly determined in principle.

[0029] Figure 1 This diagram illustrates the structure of a system for implementing the person re-identification method according to embodiments of the present disclosure, i.e., a person re-identification system. The person re-identification system 100 according to this embodiment includes a pose estimation unit 110, a block clipping unit 120, a feature extraction unit 130, and a recognition unit 140.

[0030] First, the description will begin with the feature extraction unit 130. CNNs are commonly used methods for extracting features from images. However, in the person re-identification system 100 of this embodiment, a Visual Transformer (ViT) is used instead of a CNN as the method for extracting features from the image 10 of the person to be re-identified. That is, the feature extraction unit 130 is configured as a ViT. ViT is an image processing model that does not utilize CNNs, proposed in the paper “Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An image is worth 16×16 words: Transformers for image recognition at scale. arXiv 2020, arXiv:2010.11929.”

[0031] The input to ViT requires one-dimensional sequence data. Therefore, the image 10 of the object, a two-dimensional data person, will not be the input to ViT. In the feature extraction unit 130, the individual blocks of the multiple blocks 14 cropped from the image 10 are planarized, i.e., transformed into one-dimensional sequence data, through the linear embedding function 134. Based on the linear embedding function 134, the one-dimensional sequence data transformed from the multiple blocks 14 is then linearly projected using a learned filter. The final embedded patch sequence is obtained through linear projection.

[0032] In the feature extraction unit 130, the embedded block sequence is embedded with position information 16 based on the position embedding function 136. The position information 16 is information used to identify where each of the multiple blocks 14 is located in the image 10. In addition, in order to enable image classification, a [class] token is added to the beginning of the embedded block sequence.

[0033] The feature extraction unit 130 includes a visual transformer encoder (hereinafter referred to as the ViT encoder) 132. A sequence of embedding blocks with added positional embeddings is input to the ViT encoder 132. The architecture of the ViT encoder 132 is as disclosed in the aforementioned paper on ViT, therefore details thereof are omitted here. The ViT encoder 132 outputs a feature map.

[0034] The recognition unit 140 receives input of a feature map from the ViT encoder 132. The recognition unit 140 incorporates a neural network. A multilayer perceptron (MLP) 142 is one example. The MLP 142 performs image classification on the feature map. The output of the MLP 142 is obtained as the re-identification result of the object / person. Furthermore, the neural network constituting the recognition unit 140 is not limited to the MLP 142. For example, a CNN can also be used in the recognition unit 140.

[0035] Next, the block cropping unit 120 will be described. Multiple blocks 14 input to the ViT encoder 132 in the feature extraction unit 130 are cropped from the image 10 by the block cropping unit 120. However, the cropping method of the block cropping unit 120 for cropping blocks 14 differs from the method in existing ViT methods.

[0036] In the existing ViT, the original image is segmented into square blocks. Furthermore, an embedding block sequence is generated from these blocks. Additionally, with the original image resolution set to (H, W) and the block size set to (P, P), the number N of blocks input to the encoder in the existing ViT is represented by N = (H × W) / (P × P). That is, the number of blocks depends on the resolution of the original image and the block size. Furthermore, in the embedding block sequence, the sequence number of each block is embedded as positional information for that block.

[0037] In contrast, in the block cropping unit 120, a predetermined number of blocks 14 are cropped from the image 10 along the human body of the subject. Cropping blocks 14 along the human body of the subject means that, based on blocks 14, the portion of the subject reflected in the image 10 is cut out from the image 10. Thus, the portion not reflecting the subject is left intact. In other words, there is no block 14 that only cuts out the portion not reflecting the subject. Furthermore, unlike conventional methods where the number of blocks is determined by the image resolution and the block size, according to the block cropping unit 120, the number of blocks 14 cropped from the image 10 is constant and independent of the resolution of the image 10.

[0038] More specifically, the cropping of block 14 from image 10 is centered on the human body's joint 12. Figure 1 In the example shown, 15 blocks 14 are cut out centered on the positions of 15 joints 12, including the joints of both wrists, both elbows, both shoulders, both ankles, both knees, both thighs, the waist, and the neck, as well as the top of the head, which is considered a joint. That is, the number of blocks 14 cut by the block cutting unit 120 is the same as the number of predefined joints 12. Moreover, the position of each joint 12 becomes the position of the block 14 to be cut. In addition, each joint 12 is assigned a number. The number assigned to the joint 12 becomes the sequence number when the block 14 is input into the ViT encoder 132.

[0039] The positions of the joints 12 of the subject are estimated by the pose estimation unit 110. The pose estimation unit 110 acquires an image 10 of the subject and estimates the pose of the subject using a known pose estimation method. Estimating the pose of the subject includes estimating the positions of the joints 12 of the subject. As a pose estimation method for the pose estimation unit 110, for example, the method disclosed in the paper "Gregory Rogez, Philippe Weinzaepfel, Cordelia Schmid: LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images. IEEE Trans. Pattern Anal. Mach. Intell. 42(5):1146-1161(2020)" can be used.

[0040] As described above, the person re-identification method involved in this embodiment does not utilize the existing ViT as is, but rather further improves the input to the ViT encoder 132. Figure 2 This diagram illustrates the features of the person re-identification method described in this embodiment compared to existing methods.

[0041] Figure 2 (A) represents a new method for inputting the person re-identification method to the ViT encoder based on the person re-identification method involved in this embodiment. Figure 2 (B) represents the existing method. Figure 2 In the existing method shown in (B), the image is segmented according to blocks. Therefore, in the existing method, blocks contain many unwanted background portions, and there are also blocks formed solely by background portions. Furthermore, in the existing method, sub-images obtained by equally dividing the image into multiple squares are called blocks, so the blocks do not overlap. Moreover, although in... Figure 2 (B) shows joints, but in the existing method, the pose of the object figure containing the joints is not presumed.

[0042] On the other hand, Figure 2 In the new method shown in (A), unwanted background portions of the image are cropped around the subject's body because blocks are cut from the image centered on predefined joint positions. Furthermore, instead of simply cropping a portion of the subject's body, blocks of predetermined sizes are created centered on the positions of each joint. That is, the new method ensures consistency in the order, position, and size of the blocks. This reduces the variance of the input data to the ViT encoder, accelerating learning and allowing the focus to concentrate on learning only the interrelated aspects. Consequently, the neural network-based recognition performance in later stages is improved, increasing the accuracy of person re-identification.

[0043] Furthermore, the new method crops fewer blocks from the image than existing methods that segment the image by block size. This reduces the computational load on the ViT encoder compared to the block size of image segmentation. Allowing partial overlap between blocks is another feature of the new method, distinguishing it from existing approaches.

[0044] Finally, refer to Figure 3 An example of the hardware configuration of the person re-identification system 100 according to this embodiment will be described.

[0045] The person re-identification system 100 includes a computer 200, a display device 220, and an input device 240. The computer 200 includes a processor 202, a program memory 204, and a data storage device 208. The processor 202 is integrated with the program memory 204 and the data storage device 208.

[0046] Program memory 204 is a non-transitory memory that stores multiple executable instructions 206. Data memory 208, such as flash memory, SSD, and / or HDD, stores image 10 and data required to execute instructions 206. Instructions 206 constitute a person re-recognition program. By executing some or all of the instructions 206 by processor 202, the functions of pose estimation unit 110, block cropping unit 120, feature extraction unit 130, and recognition unit 140 are implemented in computer 200.

[0047] Display device 220 displays the calculation results obtained by computer 200. Input device 240, such as a keyboard or mouse, handles operations on computer 200. Furthermore, person re-identification system 100 can be composed of multiple computers connected via a network or a server on the Internet.

Claims

1. A person re-identification method, characterized in that, include: In the image of the person who becomes the object of re-identification, the position of the predefined joints of the person is estimated; Based on the position of the joint, cut out the same number of blocks of a predetermined size from the image along the body of the person, centered on the position of the joint; Generate the position information of each of the joints; The same number of blocks as the joint, along with the position information of the joint, are input into the vision transformer encoder in a predetermined joint order. The output of the visual transducer encoder is input into the neural network; and The output of the neural network is obtained as the re-identification result of the person.

2. The person re-identification method according to claim 1, characterized in that, The same number of blocks as the joint includes at least one pair of blocks that partially overlap each other.

3. The person re-identification method according to claim 1 or 2, characterized in that, The number of blocks is less than the number of segments when the image is divided according to the size of the blocks.

4. A person re-identification system, characterized in that, have: One or more processors; and A program memory, which is combined with the one or more processors, stores a plurality of executable instructions. The plurality of executable instructions are configured to cause the one or more processors to execute: In the image of the person who becomes the object of re-identification, the position of the predefined joints of the person is estimated; Based on the position of the joint, cut out the same number of blocks of a predetermined size from the image along the body of the person, centered on the position of the joint; Generate the position information of each of the joints; The same number of blocks as the joint, along with the position information of the joint, are input into the vision transformer encoder in a predetermined joint order. The output of the visual transducer encoder is input into the neural network; and The output of the neural network is obtained as the re-identification result of the person.

5. A computer-readable recording medium, characterized in that, The record contains a person re-identification program, which is configured to be executed by a computer: In the image of the person who becomes the object of re-identification, the position of the predefined joints of the person is estimated; Based on the position of the joint, cut out the same number of blocks of a predetermined size from the image along the body of the person, centered on the position of the joint; Generate the position information of each of the joints; The same number of blocks as the joint, along with the position information of the joint, are input into the vision transformer encoder in a predetermined joint order. The output of the visual transducer encoder is input into the neural network; and The output of the neural network is obtained as the re-identification result of the person.

Images