Re-identification system

Abstract

To improve the processing speed of re-identification of a moving body.SOLUTION: The re-identification system performs re-identification of a moving object among a plurality of cameras. The plurality of cameras includes a first camera and a second camera having partially overlapping fields of view. A first overlap area in a first image shot by the first camera and a second overlap area in a second image shot by the second camera correspond to partially overlapping fields of view. When the re-identification system performs re-identification on the target moving object appearing in the second overlap area in the second image, the re-identification system performs the re-identification by focusing on the first overlap area in the first image without focusing on an area other than the first overlap area in the first image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a technique for re-identifying a moving object. [Background technology]

[0002] Person re-identification is a technology that identifies the same person from multiple images. Specifically, a person is detected from an image, the overall features of the detected person are extracted, and person re-identification is performed based on the extracted features. Feature extraction is performed using a person re-identification model based on machine learning. This type of person re-identification makes it possible to track the same person.

[0003] Patent Document 1 discloses a keypoint generation program for generating keypoints for a target image. The keypoints are used for image matching.

[0004] Non-Patent Document 1 discloses dense feature matching. Dense feature matching extracts features from each of two or more images and performs feature matching, thereby making it possible to estimate with high accuracy pixel pairs that correspond to the same 3D point in two or more images. In other words, dense feature matching makes it possible to estimate with high accuracy the overlapping area in two or more images on a pixel-by-pixel basis. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-170905 [Non-patent literature]

[0006] [Non-Patent Document 1] Johan Edstedt et al., "RoMa: Robust Dense Feature Matching," arXiv:2305.15404v2 [cs.CV], 11 Dec 2023. Summary of the Invention [Problem to be solved by the invention]

[0007] An object of the present disclosure is to provide a technology that can improve the processing speed of re-identifying a moving object. [Means for solving the problem]

[0008] One aspect of the present disclosure relates to a re-identification system. The re-identification system comprises one or more processors configured to perform re-identification of a moving object among a plurality of cameras. The multiple cameras include a first camera and a second camera having partially overlapping fields of view. A first overlap area in a first image captured by a first camera and a second overlap area in a second image captured by a second camera correspond to overlapping fields of view. The one or more processors are configured to, when re-identifying a target moving object that appears in the second overlap area in the second image, perform the re-identification by focusing on the first overlap area in the first image, without focusing on areas other than the first overlap area in the first image. [Effects of the Invention]

[0009] According to the present disclosure, a first overlap area in a first image captured by a first camera and a second overlap area in a second image captured by a second camera are taken into consideration. In re-identifying a moving object, non-overlapping areas other than the overlap area are excluded, thereby improving the processing speed of the re-identification. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a conceptual diagram for explaining an overview of a re-identification system. [Figure 2] FIG. 1 is a conceptual diagram illustrating an overlap camera. [Figure 3] FIG. 10 is a conceptual diagram for explaining a re-identification process for an overlapping camera. [Figure 4] 10 is a flowchart showing the processing performed by the re-identification system. [Figure 5] FIG. 10 is a conceptual diagram illustrating a first example of a technique for identifying an overlap area. [Figure 6] FIG. 1 is a conceptual diagram for explaining projective transformation. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0012] 1. Overview of the re-identification system Moving object re-identification is a technology for identifying the same moving object from multiple images. The concept of "image" encompasses video. Examples of moving objects include people (pedestrians), robots, vehicles, etc. In the following explanation, we will consider, as an example, the case where the moving object is a person (person re-identification). When generalizing, "person" in the following explanation will be read as "moving object." Also, in the following explanation, re-identification may be referred to as "ReID."

[0013] FIG. 1 is a conceptual diagram for explaining an overview of a re-identification system (ReID system) 100 according to this embodiment. The re-identification system 100 re-identifies a person in a predetermined area. Examples of the predetermined area include a town, a building, a facility, a store, etc. The re-identification system 100 acquires multiple images IMG taken by one or multiple cameras CAM installed in the predetermined area. The re-identification system 100 then identifies the same person from among the multiple images IMG.

[0014] More specifically, the re-identification system 100 performs person detection in each image IMG. Furthermore, the re-identification system 100 extracts features of the detected person (hereinafter referred to as "ReID features") based on a partial image corresponding to the person. Typically, feature extraction is performed using a ReID model based on machine learning. The ReID model may be a Transformer-based model. The re-identification system 100 then calculates the similarity between the first person and the second person based on the first ReID feature of the first person and the second ReID feature of the second person. If the similarity is equal to or greater than a threshold, the re-identification system 100 determines that the first person and the second person are the same person. Unique personal identification information is assigned to the same person. This re-identification process makes it possible to track the same person.

[0015] For example, multiple images IMG are captured by multiple different cameras CAM. In this case, it can be said that the re-identification system 100 performs person re-identification and tracking across the multiple cameras CAM. This type of system is also called MTMC (Multi-Target Multi-Camera Tracking). In MTMC, re-identification and tracking are performed for multiple people in parallel.

[0016] As another example, the multiple images IMG may be taken at different times by the same camera CAM.

[0017] The re-identification system 100 may be configured with a single node or multiple nodes. More specifically, the re-identification system 100 includes one or more processors 110 (hereinafter simply referred to as "processors 110"), one or more storage devices 120 (hereinafter simply referred to as "storage devices 120"), and an interface 130. The processor 110 executes various processes. Examples of the processor 110 include a general-purpose processor, a specific-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc. The processor 110 can also be referred to as processing circuitry. The storage device 120 stores various information 200 required for processing. Examples of the storage device 120 include a volatile memory, a non-volatile memory, a hard disk drive (HDD), a solid-state drive (SSD), etc. The interface 130 receives various data from the outside and outputs various data to the outside.

[0018] The processor 110 acquires a plurality of images IMG via the interface 130. The plurality of images IMG are captured by one or more cameras CAM installed in a predetermined area. The processor 110 performs a re-identification process based on the plurality of images IMG. The processor 110 then outputs the results of the re-identification process via the interface 130.

[0019] The processor 110 may execute a computer program. In this case, the functions of the re-identification system 100 are realized by cooperation between the processor 110 executing the computer program and the storage device 120. The computer program is stored in the storage device 120. The computer program may be recorded on a computer-readable recording medium.

[0020] 2. Re-identification process for overlapping cameras In the following description, we consider a case where multiple images IMG are taken by multiple different cameras CAM. In other words, the re-identification system 100 performs person re-identification across multiple camera CAMs. The multiple camera CAMs may include cameras with partially overlapping fields of view. Such two cameras with partially overlapping fields of view will be referred to as "overlapping cameras" hereinafter.

[0021] Figure 2 is a conceptual diagram illustrating overlapping cameras. The fields of view of the first camera CAM-1 and the second camera CAM-2 partially overlap. The first image IMG-1 is the image IMG captured by the first camera CAM-1. The second image IMG-2 is the image IMG captured by the second camera CAM-2.

[0022] The first image IMG-1 captured by the first camera CAM-1 includes a first overlap area OA-1 and a first non-overlap area NOA-1. The first overlap area OA-1 corresponds to the area where the field of view of the first camera CAM-1 and the field of view of the second camera CAM-2 overlap. The first non-overlap area NOA-1 is the area in the first image IMG-1 other than the first overlap area OA-1.

[0023] The second image IMG-2 captured by the second camera CAM-2 includes a second overlap area OA-2 and a second non-overlap area NOA-2. The second overlap area OA-2 corresponds to the area where the field of view of the first camera CAM-1 and the field of view of the second camera CAM-2 overlap. The second non-overlap area NOA-2 is the area in the second image IMG-2 other than the second overlap area OA-2.

[0024] An example of a method for identifying the first overlap area OA-1 and the second overlap area OA-2 is described in Section 3 below.

[0025] FIG. 3 is a conceptual diagram for explaining the re-identification process for overlapping cameras. Person re-identification is performed between the first camera CAM-1 and the second camera CAM-2. In other words, person re-identification is performed between the first image IMG-1 and the second image IMG-2. Assume that the first image IMG-1 and the second image IMG-2 are acquired at approximately the same time. In this case, by focusing on the first overlap area OA-1 and the second overlap area OA-2, person re-identification can be performed efficiently and quickly.

[0026] For example, suppose a person PT moves from the field of view of the first camera CAM-1 to the field of view of the second camera CAM-2. At a certain point, the person PT appears in the second overlap area OA-2 in the second image IMG-2. Consider the re-identification process for the person PT captured in the second overlap area OA-2 in the second image IMG-2. The person PT that is the target of the re-identification process is called the target person PT. The target person PT captured in the second overlap area OA-2 in the second image IMG-2 should also be captured in the first overlap area OA-1 in the first image IMG-1. Conversely, the target person PT captured in the second overlap area OA-2 in the second image IMG-2 is not captured in the first non-overlap area NOA-1 in the first image IMG-1. Therefore, when performing re-identification processing on the target person PT appearing in the second overlap area OA-2, it is not necessary to focus on the first non-overlap area NOA-1 in the first image IMG-1; it is sufficient to focus only on the first overlap area OA-1 in the first image IMG-1. Person PA appears in the first overlap area OA-1 in the first image IMG-1. Therefore, it is sufficient to determine whether the target person PT and person PA are the same based on the ReID feature amount of the target person PT and the ReID feature amount of person PA. Although person PB appears in the first non-overlap area NOA-1 in the first image IMG-1, in terms of the re-identification processing on the target person PT, person PB may be excluded from the comparison target.

[0027] From the above perspective, the re-identification system 100 recognizes the first overlap area OA-1 in the first image IMG-1 and the second overlap area OA-2 in the second image IMG-2. When performing re-identification processing on the target person PT appearing in the second overlap area OA-2 in the second image IMG-2, the re-identification system 100 performs the re-identification processing by focusing on the first overlap area OA-1 in the first image IMG-1, without focusing on the first non-overlap area NOA-1 in the first image IMG-1. In other words, when performing re-identification processing between the first camera CAM-1 and the second camera CAM-2, the re-identification system 100 limits the search range for feature matching to the overlap areas OA-1 and OA-2. Since the non-overlap areas NOA-1 and NOA-2 are excluded from the search range and only the overlap areas OA-1 and OA-2 are used in the re-identification processing, the re-identification processing can be performed efficiently and quickly. Furthermore, since people in the non-overlapping areas NOA-1 and NOA-2 are excluded from comparison, the error rate of the re-identification process is also reduced. In other words, according to this embodiment, it is possible to improve the processing speed and accuracy of the re-identification process. This is preferable from the perspective of real-time re-identification processing.

[0028] The overlap area information 210 is information for recognizing the first overlap area OA-1 and the second overlap area OA-2. By identifying the first overlap area OA-1 and the second overlap area OA-2 in advance, such overlap area information 210 can be prepared in advance. An example of a method for identifying the first overlap area OA-1 and the second overlap area OA-2 will be described later in Section 3. The overlap area information 210 is stored in advance in the storage device 120. When performing re-identification processing between the first camera CAM-1 and the second camera CAM-2, the re-identification system 100 can recognize the first overlap area OA-1 and the second overlap area OA-2 based on the overlap area information 210 stored in advance in the storage device 120. Using the overlap area information 210 generated in advance further improves processing speed. This is also preferable from the perspective of real-time re-identification processing.

[0029] The overlap area information 210 may include mask information 220. As shown in FIG. 3, the first mask MSK-1 masks (hides) the first non-overlap area NOA-1 in the first image IMG-1. The second mask MSK-2 masks (hides) the second non-overlap area NOA-2 in the second image IMG-2. The mask information 220 indicates the arrangement of the first mask MSK-1 in the first image IMG-1 and the arrangement of the second mask MSK-2 in the second image IMG-2. When performing re-identification processing between the first camera CAM-1 and the second camera CAM-2, the re-identification system 100 may perform the re-identification processing by masking the first non-overlap area NOA-1 in the first image IMG-1 and the second non-overlap area NOA-2 in the second image IMG-2 based on the mask information 220.

[0030] 4 is a flowchart showing the processing by the re-identification system 100 (processor 110). The camera pair that constitutes the overlapping camera is identified in advance. In step S100, the re-identification system 100 determines whether the target camera pair that is the target of the current re-identification processing is an overlapping camera.

[0031] If the target camera pair is an overlapping camera pair (step S100; Yes), the process proceeds to steps S110 and S120. In step S110, the re-identification system 100 acquires overlap area information 210 related to the target camera pair from the storage device 120. Then, in step S120, the re-identification system 100 refers to the overlap area information 210 to recognize the overlap area OA and non-overlap area NOA related to the target camera pair. Then, the re-identification system 100 performs re-identification processing on the target camera pair by focusing on the overlap area OA, without focusing on the non-overlap area NOA.

[0032] On the other hand, if the target camera pair is not an overlapping camera (step S100; No), the process proceeds to step S130. In step S130, the re-identification system 100 performs a normal re-identification process on the target camera pair.

[0033] As described above, according to this embodiment, the non-overlapping area NOA is excluded from the re-identification process, thereby improving the processing speed and accuracy of the re-identification process. Furthermore, by using overlap area information 210 generated in advance, the processing speed can be further improved. These are desirable from the perspective of real-time re-identification processing.

[0034] 3. Example of how to identify overlapping areas An example of a method for identifying the overlap area OA will be described below.

[0035] 3-1. First example FIG. 5 is a conceptual diagram illustrating a first example of a technique for identifying overlap areas OA. In the first example, "dense feature matching" is used (see Non-Patent Document 1). Dense feature matching extracts features from each of two or more images and performs feature matching, thereby making it possible to estimate with high accuracy pixel pairs that correspond to the same 3D points in two or more images. In other words, dense feature matching makes it possible to estimate with high accuracy the overlapping areas in two or more images on a pixel-by-pixel basis.

[0036] The re-identification system 100 estimates pixel pairs that correspond to the same 3D points in the first image IMG-1 and the second image IMG-2 by applying dense feature matching to the first image IMG-1 and the second image IMG-2. In the example shown in Figure 5, the first pixel group PC-1 in the first image IMG-1 and the second pixel group PC-2 in the second image IMG-2 are a set of pixel pairs that correspond to the same 3D point group. By applying dense feature matching, a large number of pixel pairs can be extracted with high accuracy.

[0037] Next, the re-identification system 100 extracts N pixel pairs as "keypoints" from the large number of pixel pairs corresponding to the same 3D point cloud. N is an integer equal to or greater than 4. The first keypoint KP-1 is a keypoint in the first image IMG-1, and the second keypoint KP-2 is a keypoint in the second image IMG-2. The first keypoint KP-1 and the second keypoint KP-2 correspond to the same 3D point. N pairs of the first keypoint KP-1 and the second keypoint KP-2 are extracted. Note that, as described above, because a large number of pixel pairs are extracted by dense feature matching, N may be set to a number greater than 4. For example, N may be set to several hundred.

[0038] Next, the re-identification system 100 calculates a homography matrix H between the first image IMG-1 and the second image IMG-2 based on N pairs of the first keypoint KP-1 and the second keypoint KP-2. FIG. 6 is a conceptual diagram for explaining homography. The intra-image position of the first keypoint KP-1 in the first image IMG-1 is (x, y), and the intra-image position of the second keypoint KP-2 in the second image IMG-2 is (x', y'). The homography matrix H between the first image IMG-1 and the second image IMG-2 can be calculated based on the intra-image positions of the N pairs of the first keypoint KP-1 and the second keypoint KP-2. Note that the camera parameters of the first camera CAM-1 and the second camera CAM-2 are not required here.

[0039] Referring again to FIG. 5, the re-identification system 100 generates a first converted image CIMG-1 by applying a projective transformation matrix H to the first image IMG-1. The first converted image CIMG-1 corresponds to an image obtained when the 3D point cloud captured in the first image IMG-1 is photographed by the second camera CAM-2. Therefore, the re-identification system 100 can identify a second overlap area OA-2 in the second image IMG-2 by overlaying the first converted image CIMG-1 on the second image IMG-2. The area of ​​the first converted image CIMG-1 overlaid on the second image IMG-2 is the second overlap area OA-2.

[0040] Similarly, the re-identification system 100 generates a second transformed image CIMG-2 by applying a projective transformation matrix H to the second image IMG-2. The second transformed image CIMG-2 corresponds to an image obtained when the 3D point cloud captured in the second image IMG-2 is photographed by the first camera CAM-1. Therefore, the re-identification system 100 can identify a first overlap area OA-1 in the first image IMG-1 by overlaying the second transformed image CIMG-2 on the first image IMG-1. The area of ​​the second transformed image CIMG-2 overlaid on the first image IMG-1 is the first overlap area OA-1.

[0041] As described above, according to the first example, it is possible to identify the overlap area OA with high accuracy by using dense feature matching. Furthermore, when dense feature matching is used, the camera parameters of the first camera CAM-1 and the second camera CAM-2 are not required. Therefore, the first example is also applicable to uncalibrated cameras.

[0042] 3-2. Second example In the second example, consider calibrated cameras. That is, the camera parameters of the first camera CAM-1 and the second camera CAM-2 are known. The camera parameters include intrinsic parameters and extrinsic parameters. The intrinsic parameters include distortion parameters, focal length, etc. The extrinsic parameters include the position and rotation of the camera in the world coordinate system.

[0043] Also, the depth map of the first camera CAM-1 and the depth map of the second camera CAM-2 are assumed to be known information.

[0044] As can be seen from FIG. 6, by performing an inverse perspective transformation using the camera parameters and depth map of the first camera CAM-1, points in the first image coordinate system of the first image IMG-1 can be transformed (projected) into 3D points in the world coordinate system. Furthermore, by performing a perspective transformation using the camera parameters and depth map of the second camera CAM-2, world coordinate points in the world coordinate system can be transformed (projected) into points in the second image coordinate system of the second image IMG-2. The image consisting of the points in the second image obtained in this way is the first transformed image CIMG-1. The first transformed image CIMG-1 corresponds to the image obtained when the 3D point cloud captured in the first image IMG-1 is captured by the second camera CAM-2. In this way, the first image IMG-1 captured by the first camera CAM-1 can be transformed into the first transformed image CIMG-1 as seen from the second camera CAM-2.

[0045] Similarly, the second image IMG-2 captured by the second camera CAM-2 can be converted into a second converted image CIMG-2 as seen from the first camera CAM-1.

[0046] In the second example, the overlap area OA is identified based on the above viewpoints. More specifically, transformation information including the camera parameters and depth maps of the first camera CAM-1 and the second camera CAM-2 is prepared in advance. This transformation information is stored in the storage device 120 in advance. The re-identification system 100 (processor 110) acquires the transformation information from the storage device 120.

[0047] The re-identification system 100 converts the first image IMG-1 into a first converted image CIMG-1 as seen from the second camera CAM-2 by perspective projection transformation using the transformation information. The first converted image CIMG-1 corresponds to the image obtained when the 3D point cloud captured in the first image IMG-1 is photographed by the second camera CAM-2. Therefore, the re-identification system 100 can identify a second overlap area OA-2 in the second image IMG-2 by overlaying the first converted image CIMG-1 on the second image IMG-2. The area of ​​the first converted image CIMG-1 overlaid on the second image IMG-2 is the second overlap area OA-2.

[0048] Similarly, the re-identification system 100 converts the second image IMG-2 into a second converted image CIMG-2 as seen from the first camera CAM-1 by perspective projection transformation using the transformation information. The second converted image CIMG-2 corresponds to the image obtained when the 3D point cloud captured in the second image IMG-2 is photographed by the first camera CAM-1. Therefore, the re-identification system 100 can identify the first overlap area OA-1 in the first image IMG-1 by overlaying the second converted image CIMG-2 on the first image IMG-1. The area of ​​the second converted image CIMG-2 overlaid on the first image IMG-1 is the first overlap area OA-1.

[0049] As described above, according to the second example, it is possible to identify the overlap area OA by using the conversion information. Even in a situation where dense feature matching does not work properly due to occlusion or the like, the overlap area OA can be identified by the method according to the second example. [Explanation of symbols]

[0050] 100 Re-identification System (ReID System) 110 processors 120 Storage device 130 Interface 210 Overlap Area Information 220 Mask Information CAM camera IMG image OA overlap area NOA Non-Overlapping Area

Claims

1. one or more processors configured to perform re-identification of moving objects between a plurality of cameras; the plurality of cameras includes a first camera and a second camera having overlapping fields of view; a first overlapping area in a first image captured by the first camera and a second overlapping area in a second image captured by the second camera correspond to the overlapping fields of view; When performing the re-identification on a target moving object captured in the second overlap area in the second image, the one or more processors are configured to perform the re-identification by focusing on the first overlap area in the first image without focusing on areas other than the first overlap area in the first image. Re-identification system.

2. 10. The re-identification system of claim 1, The one or more processors further extracting a first keypoint in the first image and a second keypoint in the second image that correspond to the same three-dimensional point by applying dense feature matching to the first image and the second image; Calculating a projective transformation matrix between the first image and the second image based on the intra-image positions of the first keypoints in the first image and the intra-image positions of the second keypoints in the second image; generating a first transformed image by applying the projective transformation matrix to the first image, and identifying the second overlap area in the second image by overlaying the first transformed image on the second image; generating a second transformed image by applying the projective transformation matrix to the second image, and identifying the first overlap area in the first image by overlaying the second transformed image on the first image; It was configured as Re-identification system.

3. 10. The re-identification system of claim 1, The one or more processors further acquiring camera parameters and a depth map of each of the first camera and the second camera as transformation information; transforming the first image into a first transformed image viewed from the second camera by a perspective projection transformation using the transformation information, and identifying the second overlap area in the second image by overlaying the first transformed image on the second image; The second image is transformed into a second transformed image viewed from the first camera by a perspective projection transformation using the transformation information, and the first overlap area in the first image is identified by overlaying the second transformed image on the first image. It was configured as Re-identification system.

4. A re-identification system according to any one of claims 1 to 3, one or more storage devices configured to store overlap area information for identifying the first overlap area and the second overlap area; The one or more processors are further configured to recognize the first overlap area and the second overlap area based on the overlap area information when performing the re-identification. Re-identification system.

5. 5. The re-identification system of claim 4, a first non-overlapping area is an area in the first image other than the first overlapping area; a second non-overlapping area is an area in the second image other than the second overlapping area; the overlap area information includes mask information indicating the arrangement of a first mask for masking the first non-overlap area and a second mask for masking the second non-overlap area; The one or more processors are further configured to perform the re-identification by masking the first non-overlapping area in the first image and the second non-overlapping area in the second image based on the mask information. Re-identification system.

Images