PERSON RE-IDENTIFICATION METHOD, PERSON RE-IDENTIFICATION SYSTEM, AND PERSON RE-IDENTIFICATION PROGRAM

Abstract

To provide a technique to improve person re-identification accuracy.SOLUTION: According to a person re-identification method of the present disclosure, a pose of a target person is estimated in an image 10 of a person to be a re-identification target. Then, according to the person re-identification method of the present disclosure, a predetermined number of patches 14 are clipped from the acquired image 10 along a body of the target person based on the estimated pose of the target person. Further, positional information 16 of each of the predetermined number of patches 14 is generated. According to the person re-identification method of the present disclosure, the predetermined number of clipped patches 14 are input to a vision transformer encoder 132 together with the positional information 16. An output of the vision transformer encoder 132 is input to a neural network 142 including an MLP, and an output of the neural network 142 is acquired as a re-identification result of the target person.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a person re-identification method, a person re-identification system, and a person re-identification program. [Background technology]

[0002] Patent Document 1 discloses a technology related to re-identification using image data. According to this conventional technology, face detection is performed using a trained learning model, and a face position is detected from image data. Then, partial image data extracted from the detected face position is generated. The partial image data of the face is processed by an application processor for personal identification, face authentication, image collection for each individual, etc.

[0003] In addition to Patent Document 1, Patent Documents 2 and 3 can be mentioned as examples of documents showing the state of the art in the technical field related to the present disclosure. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-025261 A [Patent Document 2] Patent Publication No. 2021-012707 [Patent Document 3] Patent No. 6788929 Summary of the Invention [Problem to be solved by the invention]

[0005] The above conventional techniques are related to face re-identification, but research is also being conducted on person re-identification, which involves re-identifying a person from image data of the entire person, such as image data of a pedestrian. However, there is room for improvement in the accuracy of currently proposed person re-identification techniques.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a technology that can improve the accuracy of person re-identification. [Means for solving the problem]

[0007] The present disclosure provides a person re-identification technology for achieving the above object. In the person re-identification technology of the present disclosure, a vision transformer is applied to person re-identification. The vision transformer is a technology that is superior in terms of computational efficiency and scalability compared to a convolutional neural network (CNN), which is a conventional image processing technology. In the person re-identification technology of the present disclosure, further improvements are made to the encoder used in the vision transformer, i.e., the input to the vision transformer encoder.

[0008] The person re-identification technology of the present disclosure includes a person re-identification method, a person re-identification system, and a person re-identification program.

[0009] The person re-identification method of the present disclosure includes the following steps. The first step is to estimate the posture of a person to be re-identified (hereinafter, referred to as the target person) in an image of the target person. The second step is to cut out a predetermined number of patches from the image along the body of the target person based on the estimated posture of the target person. The third step is to generate position information of each of the cut-out predetermined number of patches. The fourth step is to input the cut-out predetermined number of patches together with their respective position information to a vision transform encoder. The fifth step is to input the output of the vision transform encoder to a neural network. And the sixth step is to obtain the output of the neural network as a re-identification result of the target person. However, some of the above steps can be appropriately integrated.

[0010] The person re-identification system of the present disclosure includes one or more processors and a program memory coupled to the one or more processors and storing a plurality of executable instructions. The plurality of executable instructions are configured to cause the one or more processors to execute the following processes. The first process is to estimate a posture of a target person in an image of the target person. The second process is to cut out a predetermined number of patches from the image along the target person's body based on the estimated posture of the target person. The third process is to generate position information of each of the cut-out predetermined number of patches. The fourth process is to input the cut-out predetermined number of patches together with their respective position information to a vision transformer encoder. The fifth process is to input the output of the vision transformer encoder to a neural network. And the sixth process is to obtain the output of the neural network as a re-identification result of the target person. However, some of the above processes can be appropriately integrated.

[0011] The person re-identification program of the present disclosure is configured to cause a computer to execute the following processes. The first process is to estimate the posture of the target person in the image of the target person. The second process is to cut out a predetermined number of patches from the image along the target person's body based on the estimated posture of the target person. The third process is to generate position information for each of the cut-out predetermined number of patches. The fourth process is to input the cut-out predetermined number of patches together with their respective position information to a vision transformer encoder. The fifth process is to input the output of the vision transformer encoder to a neural network. And the sixth process is to obtain the output of the neural network as a re-identification result of the target person. However, some of the above processes can be appropriately integrated.

[0012] According to the person re-identification technique of the present disclosure, patches are cut out from an image along the target person's body based on the target person's posture, so that unnecessary background parts of the image are cut out around the target person's body and excluded from the input to the vision transform encoder. Furthermore, the size, number, and order of the patches input to the vision transform encoder are fixed. In this way, the data input to the vision transform encoder is normalized, thereby reducing the variance of the data for each input. This improves the identification performance of the neural network and increases the accuracy of person re-identification.

[0013] In the person re-identification technique of the present disclosure, estimating the posture may include estimating the positions of the joints of the target person. In this case, cutting out the predetermined number of patches may include cutting out the same number of patches as the number of joints, centered on the positions of the joints, and generating the position information may include generating the position information of the joints. By cutting out the patches centered on the positions of the joints, the entire human body can be cut out evenly by the patches. In addition, the positions of the patches input to the vision transformer encoder can be made consistent.

[0014] In the person re-identification technique of the present disclosure, the predetermined number of patches may include at least a pair of patches that partially overlap each other. By allowing the patches to partially overlap each other, it is possible to reduce the portion of the human body that is not cut out by the patches.

[0015] In the person re-identification technique of the present disclosure, the predetermined number may be smaller than the number of divisions when an image is divided into the patch sizes, which can reduce the calculation load compared to dividing an image into the patch sizes. Effect of the Invention

[0016] As described above, according to the person re-identification technology of the present disclosure, it is possible to improve the identification performance by a neural network and increase the accuracy of person re-identification. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a system for implementing a person re-identification method according to an embodiment of the present disclosure. [Diagram 2] 1A to 1C are diagrams illustrating features of a person re-identification method according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 is a diagram illustrating an example of a hardware configuration of a person re-identification system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. However, when the numbers, quantities, amounts, ranges, etc. of each element are mentioned in the embodiments shown below, the technical idea of ​​the present disclosure is not limited to the mentioned numbers unless otherwise specified or clearly specified in principle. Furthermore, the structures etc. described in the embodiments shown below are not necessarily essential to the technical idea of ​​the present disclosure unless otherwise specified or clearly specified in principle.

[0019] 1 is a diagram showing the configuration of a system for realizing a person re-identification method according to an embodiment of the present disclosure, i.e., a person re-identification system 100 according to the present embodiment. The person re-identification system 100 includes a pose estimation unit 110, a patch extraction unit 120, a feature extraction unit 130, and a recognition unit 140.

[0020] First, the feature extraction unit 130 will be described. CNN is a common means for extracting features from an image. However, in the person re-identification system 100 according to this embodiment, a vision transformer (ViT) is used instead of a CNN as a means for extracting features from an image 10 of a target person to be re-identified. In other words, the feature extraction unit 130 is configured as a ViT. ViT is a model of image processing that does not use a CNN, proposed in the paper "Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An image is worth 16 × 16 words: Transformers for image recognition at scale. arXiv 2020, arXiv:2010.11929."

[0021] The input of ViT must be one-dimensional sequence data. Therefore, the image 10 of the target person, which is two-dimensional data, is not the input of ViT. In the feature extraction unit 130, the linear embedding function 134 flattens each of the multiple patches 14 cut out from the image 10, that is, converts them into one-dimensional sequence data. The linear embedding function 134 further performs linear projection using a trained filter on the one-dimensional sequence data converted from the multiple patches 14. The linear projection results in a final embedded patch sequence.

[0022] In the feature extraction unit 130, the position embedding function 136 embeds position information 16 into the embedded patch sequence. The position information 16 is information for identifying where each of the multiple patches 14 is located in the image 10. In addition, a [class] token 138 is added to the beginning of the embedded patch sequence to enable image classification.

[0023] The feature extraction unit 130 includes a vision transformer encoder (hereinafter referred to as ViT encoder) 132. The embedded patch sequence with added position embedding is input to the ViT encoder 132. The architecture of the ViT encoder 132 is as disclosed in the above paper on ViT, so a detailed description of the architecture is omitted here. The ViT encoder 132 outputs a feature map.

[0024] The recognition unit 140 receives an input of a feature map from the ViT encoder 132. The recognition unit 140 includes a neural network. An example of the neural network is a multi-layer perceptron (MLP) 142. The MLP 142 performs image classification on the feature map. The output of the MLP 142 is obtained as a re-identification result of the target person. Note that the neural network constituting the recognition unit 140 is not limited to the MLP 142. For example, a CNN can also be used for the recognition unit 140.

[0025] Next, a description will be given of the patch extraction unit 120. The multiple patches 14 input to the ViT encoder 132 in the feature extraction unit 130 are extracted from the image 10 by the patch extraction unit 120. However, the method of extracting the patches 14 by the patch extraction unit 120 differs from the method used in conventional ViT.

[0026] In conventional ViT, the original image is divided into square patches. An embedded patch sequence is then generated from the patches obtained by dividing the original image. If the resolution of the original image is (H, W) and the size of the patch is (P, P), the number of patches N input to the encoder in conventional Vit is expressed as N = (H × W) / (P × P). In other words, the number of patches depends on the resolution of the original image and the size of the patch. In addition, the sequence number of each patch is embedded in the embedded patch sequence as the position information of each patch.

[0027] In contrast, in the patch extraction unit 120, a predetermined number of patches 14 are extracted from the image 10 along the body of the target person. Extracting the patches 14 along the body of the target person means that the parts of the image 10 in which the target person is shown are extracted by the patches 14. Therefore, the parts in which the target person is not shown remain. In other words, there are no patches 14 in which only the parts in which the target person is not shown are extracted. Also, in the conventional method, the number of patches is determined by the resolution of the image and the size of the patches, whereas in the patch extraction unit 120, the number of patches 14 extracted from the image 10 is constant regardless of the resolution of the image 10.

[0028] More specifically, the patches 14 are extracted from the image 10 by setting the joints 12 of the human body as the centers of the patches 14. In the example shown in FIG. 1, 15 patches 14 are extracted centered on the positions of 15 joints 12, including the joints of both wrists, both elbows, both shoulders, both ankles, both knees, both thighs, waist, and neck, and the top of the head, which is considered as a joint. That is, the number of patches 14 extracted by the patch extraction unit 120 is the same as the number of joints 12 defined in advance. The position of each joint 12 becomes the position of the patch 14 to be extracted. Also, each joint 12 is assigned a number. The number assigned to the joint 12 becomes a sequence number when the patch 14 is input to the ViT encoder 132.

[0029] The positions of the joints 12 of the target person are estimated by the pose estimation unit 110. The pose estimation unit 110 acquires an image 10 of the target person and estimates the pose of the target person by a known pose estimation method. Estimating the pose of the target person includes estimating the positions of the joints 12 of the target person. As a pose estimation method by the pose estimation unit 110, for example, the method disclosed in the paper "Gregory Rogez, Philippe Weinzaepfel, Cordelia Schmid: LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images. IEEE Trans. Pattern Anal. Mach. Intell. 42(5): 1146-1161 (2020)" can be used.

[0030] As described above, the person re-identification method according to this embodiment does not directly use the conventional ViT, but is a method in which further improvements have been made to the input to the ViT encoder 132. Fig. 2 is a diagram for explaining the features of the person re-identification method according to this embodiment in comparison with the conventional method.

[0031] FIG. 2(A) shows a new approach to input to the ViT encoder using the person re-identification method according to the present embodiment, and FIG. 2(B) shows a conventional approach. In the conventional approach shown in FIG. 2(B), an image is divided into patches. Therefore, in the conventional approach, the patches contain many unnecessary background parts, and some patches consist only of background parts. In addition, in the conventional approach, the patches are obtained by equally dividing the image into multiple squares, so the patches do not overlap with each other. Note that although joints are shown in FIG. 2(B), the posture of the target person, including the positions of the joints, is not estimated in the conventional approach.

[0032] In contrast, in the new approach shown in Figure 2(A), patches are cut out from the image around predefined joint positions, so that unnecessary background parts of the image are cut out around the target person's body. Also, instead of simply cutting out parts of the target person's body, patches of a given size are always created around the position of each joint. In other words, the new approach can ensure consistency in the order of patches, patch positions, and patch sizes. This reduces the variance of data for each input to the ViT encoder, accelerating learning and allowing only parts with mutual correlation to be learned in a concentrated manner. As a result, the classification performance of the subsequent neural network can be improved, and the accuracy of person re-identification can be improved.

[0033] In addition, the new approach cuts out fewer patches from an image than the conventional approach, which divides an image into patches. This reduces the computational load of the ViT encoder compared to dividing an image into patches. Another feature of the new approach that differs from the conventional approach is that patches are allowed to overlap partially.

[0034] Finally, an example of the hardware configuration of the person re-identification system 100 according to this embodiment will be described with reference to FIG.

[0035] The person re-identification system 100 includes a computer 200, a display device 220, and an input device 240. The computer 200 comprises a processor 202, a program memory 204, and a data storage 208. The processor 202 is coupled to the program memory 204 and the data storage 208.

[0036] The program memory 204 is a non-transitory memory that stores a number of executable instructions 206. The data storage 208 is, for example, a flash memory, SSD, or HDD, and stores the images 10 and data required for the execution of the instructions 206. The instructions 206 constitute a person re-identification program. When some or all of the instructions 206 are executed by the processor 202, the computer 200 is implemented with the functions of the pose estimation unit 110, the patch extraction unit 120, the feature extraction unit 130, and the recognition unit 140.

[0037] The display device 220 displays the results of calculation by the computer 200. The input device 240 is, for example, a keyboard or a mouse, and accepts operations for the computer 200. Note that the person re-identification system 100 may be configured by a plurality of computers connected via a network, or may be configured by a server on the Internet. [Explanation of symbols]

[0038] 10 Images 12 Joints 14 Patches 16 Location information 100 Person Re-Identification System 110 Pose Estimation Unit 120 Patch Extraction Unit 130 Feature Extraction Unit 132 Vision Transformer Encoder 134 Linear Embedding 136 Position Embedding 138 CLS Tokens 140 Recognition Units 142 MLP 200 Computers 202 Processor 204 Program Memory 206 Instructions 208 Data Storage 220 Display device 240 Input Device

Claims

1. estimating positions of pre-defined joints of a person to be re-identified in an image of the person; extracting patches of a predetermined size, the same number as the number of joints, from the image along the body of the person based on the positions of the joints, with the joints at the center; generating position information for each of said joints; inputting the patches, the number of which is equal to the number of joints, into a vision transformer encoder in a predetermined joint order together with the position information of the joints; inputting the output of the vision transformer encoder into a neural network; and obtaining an output of the neural network as a re-identification result for the person. A person re-identification method comprising:

2. The person re-identification method according to claim 1, The predetermined number of patches includes at least one pair of patches that partially overlap each other. A person re-identification method comprising:

3. The person re-identification method according to claim 1 or 2, The predetermined number is smaller than the number of divisions when the image is divided by the size of the patch. A person re-identification method comprising:

4. one or more processors; a program memory coupled to the one or more processors and storing a plurality of executable instructions; The executable instructions may be configured to cause the one or more processors to: estimating positions of pre-defined joints of a person to be re-identified in an image of the person; extracting patches of a predetermined size, the same number as the number of joints, from the image along the body of the person based on the positions of the joints, with the joints at the center; generating position information for each of said joints; inputting the patches, the number of which is equal to the number of joints, into a vision transformer encoder in a predetermined joint order together with the position information of the joints; inputting the output of the vision transformer encoder into a neural network; and obtaining an output of the neural network as a re-identification result for the person. A person re-identification system comprising:

5. estimating positions of pre-defined joints of a person to be re-identified in an image of the person; extracting patches of a predetermined size, the same number as the number of joints, from the image along the body of the person based on the positions of the joints, with the joints at the center; generating position information for each of said joints; inputting the patches, the number of which is equal to the number of joints, into a vision transformer encoder in a predetermined joint order together with the position information of the joints; inputting the output of the vision transformer encoder into a neural network; and obtaining an output of the neural network as a re-identification result for the person. A person re-identification program.

Images