Identification method, identification program, and information processing device
The information processing device achieves robust markerless person re-identification by generating correspondence relationships between image elements, addressing the challenge of identifying objects without markers and improving accuracy and efficiency in surveillance systems.
Patent Information
- Application Number
- PCT/JP2023/020563
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-06-01
- Publication Date
- 2024-12-05
AI Technical Summary
Conventional surveillance camera systems face difficulties in identifying objects without markers in overlapping areas, leading to challenges in accurate person re-identification and increased processing load due to the need for coordinate transformation and marker installation.
An information processing device that performs person re-identification by generating correspondence relationships between image elements based on detected positions across multiple cameras, eliminating the need for markers and reducing processing load by using the object as a clue for position matching.
Enables robust person re-identification without markers, reducing installation efforts and minimizing errors from coordinate transformation, while maintaining accuracy across varying viewpoints and illumination conditions.
Smart Images

Figure JP2023020563_05122024_PF_FP_ABST
Abstract
Description
Identification method, identification program, and information processing device
[0001] The present invention relates to an identification method, an identification program, and an information processing device.
[0002] Tracking technology using camera footage is being used in a variety of scenarios, including crime prevention, marketing analysis, and customer behavior analysis. As the range of uses for tracking technology expands, the task of Re-ID (Re-IDentification), which identifies the identity of objects captured by different cameras, is becoming increasingly important.
[0003] As one technology related to Re-ID, the following surveillance camera terminal has been proposed (see, for example, Patent Document 1). For example, the position of an object is detected from frame images captured by each of the multiple surveillance camera terminals using four markers marked on the floor surface within an overlapping area monitored by the multiple surveillance camera terminals when the surveillance camera terminals are installed as references. Furthermore, the position of the object detected for each frame image from each surveillance camera terminal is converted to a position in a common coordinate system according to coordinate transformation parameters calculated using the four markers. Objects located in the overlapping area are identified based on the distance between the objects converted to positions in the common coordinate system in this way.
[0004] International Publication No. 2011 / 010490
[0005] However, in the conventional technology represented by the above-mentioned surveillance camera terminal, it is difficult to identify an object without a marker placed within the overlapping area.
[0006] In one aspect, an object is to provide an identification method, an identification program, and an information processing device that can realize markerless object identification.
[0007] In one embodiment of the identification method, a computer executes a process in which a first person is detected from a first image taken using a first camera, and when a second person is detected from a second image taken using a second camera, relationship information is generated that associates the position in the first image where the first person is detected with the position in the second image where the second person is detected, and the first person and the second person are identified based on characteristic information of the first person and characteristic information of the second person.
[0008] According to one embodiment, object identification can be achieved without markers.
[0009] FIG. 1 is a block diagram showing an example of the functional configuration of an information processing device. FIG. 2 is a schematic diagram showing an example of person re-identification. FIG. 3 is a diagram showing an example of camera arrangement. FIG. 4 is a diagram showing an example of person detection. FIG. 5 is a diagram showing an example of generation of positional correspondence between image elements. FIG. 6 is a diagram showing an example of camera arrangement. FIG. 7 is a diagram showing an example of generation of positional correspondence between image elements. FIG. 8 is a diagram showing an example of camera arrangement. FIG. 9 is a schematic diagram showing an example of person re-identification. FIG. 10 is a diagram showing an example of correspondence data configuration. FIG. 11 is a flowchart showing the procedure of the overall processing of the information processing device. FIG. 12 is a flowchart showing the procedure of second person re-identification processing. FIG. 13 is a diagram showing an example of person detection. FIG. 14 is a diagram showing an example of generation of positional correspondence between image elements. FIG. 15 is a diagram explaining interpolation between blocks. FIG. 16 is a diagram showing an example of correspondence data configuration. FIG. 17 is a flowchart showing the procedure of second person re-identification processing. FIG. 18 is a diagram explaining an example of check-in according to an application example. FIG. 19 is a flowchart showing the procedure of person tracking processing according to an application example. FIG. 20 illustrates an example of a hardware configuration.
[0010] Hereinafter, embodiments of the identification method, identification program, and information processing device according to the present application will be described with reference to the accompanying drawings. Each embodiment merely illustrates one example or aspect, and does not limit the range of values, functions, or usage scenarios. Furthermore, each embodiment can be appropriately combined within the scope of not causing any contradiction in the processing content.
[0011] <First Embodiment> <Overall Configuration> Fig. 1 is a block diagram showing an example of the functional configuration of an information processing device 10. The information processing device 10 shown in Fig. 1 provides a multi-camera tracking function for tracking an object using cameras 30A to 30N.
[0012] The information processing device 10 is an example of a computer that provides the multi-camera tracking function. For example, the information processing device 10 may be realized as a server that provides the multi-camera tracking function on-premise. Alternatively, the information processing device 10 may be realized as a PaaS (Platform as a Service) or SaaS (Software as a Service) application, thereby providing the multi-camera tracking function as a cloud service.
[0013] 1, cameras 30A to 30N are connected to an information processing device 10 via a network NW. For example, the network NW may be realized by any type of communication network, such as the Internet or a local area network (LAN), whether wired or wireless.
[0014] The cameras 30A to 30N are imaging devices that capture images. Hereinafter, when there is no need to distinguish between the individual cameras 30A to 30N, the cameras 30A to 30N will be referred to as "camera 30."
[0015] These cameras 30A to 30N may be arranged so that the entire area supported by the multi-camera tracking function is covered by the camera's 30A to 30N imaging range. In this case, the cameras 30 may be installed so that their imaging ranges partially overlap. For example, the imaging range of one camera 30 may overlap with the imaging range of one or more other cameras 30.
[0016] Furthermore, the operations of the cameras 30A to 30N to capture images may be synchronized among the cameras 30A to 30N. In this manner, the images captured by the cameras 30A to 30N can be transmitted to the information processing device 10 on a frame-by-frame basis.
[0017] <Multi-camera tracking> In the following, a "person" will be exemplified as an object to be tracked by the multi-camera tracking function. However, the object is not limited to this and can be any object, such as a moving body. For example, the object may be a living thing other than a human, such as a horse or a dog, or an inanimate object, such as a two-wheeled or four-wheeled vehicle, or an air vehicle, such as a drone.
[0018] For example, use cases for the multi-camera tracking function include monitoring, marketing analysis, and customer behavior analysis targeting public facilities such as stations, commercial facilities such as shopping malls, or complex facilities.
[0019] In such a use case, the person being tracked may move within the ranges of the cameras 30A to 30N. In this case, in order to prevent the movement line of the person obtained by tracking from being interrupted, if a person captured on one camera is captured on another camera, the people can be linked as the same person, and the viewpoint of the camera used for tracking can be switched.
[0020] <Person Re-Identification> For this reason, the information processing device 10 performs person re-identification, so-called Person Re-ID, to identify the identity of a person photographed by the multiple cameras 30A to 30N, as part of the multi-camera tracking function described above.
[0021] FIG. 2 is a schematic diagram showing an example of person re-identification. For ease of explanation, FIG. 2 shows a schematic top view of a facility photographed by cameras 30A and 30B, and the angles of view of cameras 30A and 30B are also shown in the top view. Furthermore, FIG. 2 shows the object detection results for image 20A captured by camera 30A, for example, the bounding box of an object whose class is "Person." Hereinafter, the bounding box may be referred to as "Bbox." Furthermore, FIG. 2 shows the object detection results for image 20B captured by camera 30B, for example, the Bbox of an object whose class is "Person."
[0022] As shown in FIG. 2, Person Re-ID is a task for linking the same person photographed by a plurality of different cameras 30, that is, camera 30A and camera 30B, that is, people shown with the same hatching.
[0023] As just one example, person re-identification can be achieved by comparing feature information extracted from an image corresponding to a Bbox region. Hereinafter, an image corresponding to a Bbox region may be referred to as a "Bbox image." For example, an example of feature information is a feature amount (feature vector) obtained by embedding a Bbox image into a feature space. Person re-identification can be achieved by evaluating the similarity or distance between pairs of such feature information.
[0024] For example, in the example shown in Figure 2, B box 21 detected from image 20A and B box 24 detected from image 20B are identified as belonging to the same person. In this case, the same ID "1" is assigned to B box 21 and B box 24. Furthermore, B box 22 detected from image 20A and B box 23 detected from image 20B are identified as belonging to the same person. In this case, the same ID "2" is assigned to B box 22 and B box 23.
[0025] <One aspect of the problem> However, as explained in the background art section above, person re-identification, as represented by the above-mentioned surveillance camera terminal, has the aspect that it is difficult to identify an object without a marker marked within the overlapping area.
[0026] <One Aspect of the Problem-Solving Approach> Therefore, the information processing device 10 according to the present embodiment generates correspondences between elements in the images of each camera based on the detected positions of people detected in the images of each camera, while performing person re-identification by comparing feature information of people in images of multiple cameras in the background. Note that the "elements" referred to here are elements of an image, and may be pixels or blocks that are collections of pixels.
[0027] Fig. 3 is a diagram showing an example of the arrangement of cameras 30. For ease of explanation, Fig. 3 shows a schematic top view of the facility that is the subject of photography by cameras 30A and 30B, and also shows the angles of view of cameras 30A and 30B in the top view.
[0028] 3 shows an example in which a person moves from a position in frame t (solid black) to a position in frame t+1 (hatched dots) using such an arrangement of cameras 30. In this case, in each of frames t and t+1, person detection results shown in FIG. 4 are obtained for each of cameras 30A and 30B, and positional correspondences between image elements shown in FIG. 5 are generated.
[0029] Fig. 4 is a diagram showing an example of person detection. Fig. 5 is a diagram showing an example of generating positional correspondences between image elements. Fig. 4 shows the detected positions of people in each frame, frame t and frame t+1, for camera 30A and camera 30B. Furthermore, Fig. 4 plots the midpoints of the bases of the four sides included in box B with white circles as the detected positions of people.
[0030] 4, the image captured by camera 30 is divided into a total of 24 blocks arranged in 4 rows and 6 columns, as an example only. These blocks are used as elements of the image of camera 30, and the positional correspondence between the elements is recorded frame by frame between camera 30A and camera 30B.
[0031] For example, in image 20A captured by camera 30A in frame t, the detected position of a person is in block number "15," while in image 20B captured by camera 30B in frame t, the detected position of the person is in block number "14." In this case, as shown in Fig. 5, the correspondence between block number "15," which is an element of image 20A captured by camera 30A, and block number "14," which is an element of image 20B captured by camera 30B, is recorded.
[0032] Furthermore, in image 21A captured by camera 30A in frame t+1, the detected position of the person is in block number "21," while in image 21B captured by camera 30B in frame t+1, the detected position of the person is in block number "13." In this case, as shown in Figure 5, the correspondence between block number "21," which is an element of image 21A captured by camera 30A, and block number "13," which is an element of image 21B captured by camera 30B, is recorded.
[0033] As such, in the example shown in Figure 4, the detected positions of people in frames t and t+1 are illustrated merely as an example, but it is obvious that as more time passes from frame t+2 onwards, it will be possible to accumulate positional correspondences between more elements.
[0034] In addition, in the example shown in FIG. 3, two cameras 30, camera 30A and camera 30B, are illustrated, but the number of cameras 30 is not limited to two, and the number of cameras 30 may be three or more.
[0035] Fig. 6 is a diagram showing an example of the arrangement of cameras 30. For ease of explanation, Fig. 6 shows a schematic top view of the facility that is the subject of photography by three cameras 30A to 30C, and also shows the angles of view of cameras 30A to 30C in the top view.
[0036] Fig. 7 is a diagram showing an example of generating correspondence relationships between images from multiple cameras. Fig. 7 shows the detected positions of people in frame t for each of cameras 30A to 30C. Furthermore, Fig. 7 plots the midpoints of the bases of the four sides included in box B as white circles to represent the detected positions of people.
[0037] 7, the image captured by camera 30 is divided into a total of 24 blocks arranged in 4 rows and 6 columns, just as an example. These blocks are used as elements of the image of camera 30, and the positional correspondence between the elements is recorded frame by frame between cameras 30A to 30C.
[0038] For example, in image 20A captured by camera 30A in frame t, the detected position of the person is in block number "15." In image 20B captured by camera 30B in frame t, the detected position of the person is in block number "14." In image 20C captured by camera 30C in frame t, the detected position of the person is in block number "7."
[0039] In this case, the correspondence between block number "15", which is an element of image 20A from camera 30A, block number "14", which is an element of image 20B from camera 30B, and block number "7", which is an element of image 20C from camera 30C, is recorded.
[0040] Although FIG. 6 illustrates an example of the detected position of a person in frame t, by continuing to process data from frame t+1 onward, it is possible to accumulate position correspondence data 13B between more elements.
[0041] In this way, in the background of person re-identification, which involves matching feature information of people in images taken by multiple cameras, it is possible to generate a correspondence relationship between the positions of elements in the images taken by each camera 30. This correspondence relationship makes it possible to realize person re-identification using logic that is independent of and different from person re-identification based on matching feature information.
[0042] Fig. 8 is a diagram showing an example of the arrangement of cameras 30. For ease of explanation, Fig. 8 shows a schematic top view of the facility that is the subject of photography by three cameras 30A to 30C, and also shows the angles of view of cameras 30A to 30C in the top view.
[0043] 8 shows an example in which two people, person A and person B, are present within the shooting ranges of the three cameras 30A to 30C under such an arrangement of cameras 30A to 30C. In this case, person re-identification can be performed by comparing the combinations of person detection positions obtained for each of images 20A to 20C captured by the three cameras 30A to 30C with the correspondence data 13B shown in FIG.
[0044] Fig. 9 is a schematic diagram showing an example of person re-identification. Fig. 9 shows images 20A to 20C captured by cameras 30A to 30C, respectively, and the person detection results. Furthermore, Fig. 9 plots the midpoints of the bases of the four sides included in the Bbox of an object whose class is "Person" as white circles to indicate the detected positions of the people.
[0045] 9, in image 20A captured by camera 30A, block number "7" is detected as person detection position d11, and block number "15" is detected as person detection position d12. In image 20B captured by camera 30B, block number "11" is detected as person detection position d21, and block number "14" is obtained as person detection position d22. In image 20C captured by camera 30C, block number "7" is detected as person detection position d31, and block number "20" is detected as person detection position d32.
[0046] In this case, the combination of the person detection position d12 of camera 30A, the person detection position d21 of camera 30B, and the person detection position d31 of camera 30C matches the data entry in the first row of correspondence data 13B. This enables person A, who appears in block number "15" of camera 30A, block number "14" of camera 30B, and block number "7" of camera 30C, to be identified as the same person.
[0047] Furthermore, the combination of the detected position d11 of the person from camera 30A, the detected position d22 of the person from camera 30B, and the detected position d32 of the person from camera 30C matches the data entry in the second row of correspondence data 13B. This enables person B, who appears in block number "7" of camera 30A, block number "11" of camera 30B, and block number "20" of camera 30C, to be identified as the same person.
[0048] As described above, the information processing device 10 of this embodiment generates a correspondence between the positions of elements in the images of each camera 30 using the object to be tracked as a clue, rather than a marker such as tape that needs to be installed in advance.
[0049] Therefore, the information processing device 10 according to this embodiment can achieve person re-identification without using markers. This eliminates the need to install markers such as tape in the space to be photographed in advance. Furthermore, when person re-identification is performed using the positional correspondence between elements in the images of each camera 30, coordinate conversion to a common coordinate system, as in the person re-identification performed by the above-described surveillance camera terminal, is not required, thereby reducing the processing load. In addition, in the person re-identification performed by the above-described surveillance camera terminal, errors in coordinate conversion increase as the position of the subject person moves away from the four markers, but the occurrence and expansion of such errors can be suppressed.
[0050] Furthermore, the information processing device 10 according to this embodiment can achieve more robust person re-identification than person re-identification based on matching of feature information. That is, when person re-identification is performed by matching of feature information, the accuracy of person re-identification decreases more significantly as the change in viewpoint or lighting conditions between the cameras 30 increases, and the accuracy of person re-identification also decreases more significantly as the resolution of the cameras 30 decreases. On the other hand, when person re-identification is performed using the positional correspondence between elements between images from each camera 30, as in the information processing device 10 according to this embodiment, the effect of changes in viewpoint, lighting conditions, camera resolution, etc. is small, and therefore the decrease in accuracy of person re-identification can be suppressed.
[0051] <Configuration of Information Processing Device 10> Fig. 1 shows a schematic diagram of blocks related to the multi-camera tracking function of the information processing device 10. As shown in Fig. 1, the information processing device 10 has a communication control unit 11, a storage unit 13, and a control unit 15. Note that Fig. 1 only shows a selection of functional units related to the above-described multi-camera tracking function, and the information processing device 10 may also be provided with functional units other than those shown.
[0052] The communication control unit 11 is a functional unit that controls communication with other devices such as the cameras 30A to 30N. As just one example, the communication control unit 11 can be realized by a network interface card such as a LAN card. In one aspect, the communication control unit 11 can receive images from the camera 30 on a frame-by-frame basis, or receive video for a certain period of time. In another aspect, the communication control unit 11 can output the results of multi-camera tracking to any external device.
[0053] The storage unit 13 is a functional unit that stores various types of data. As just one example, the storage unit 13 is realized by internal, external, or auxiliary storage of the information processing device 10. For example, the storage unit 13 stores ID information 13A and correspondence data 13B. The ID information 13A and the correspondence data 13B will be described together with a scene in which the ID information 13A and the correspondence data 13B are referenced, generated, or registered.
[0054] The control unit 15 is a functional unit that performs overall control of the information processing device 10. For example, the control unit 15 can be realized by a hardware processor. Alternatively, the control unit 15 can be realized by hardwired logic. As shown in FIG. 1 , the control unit 15 includes an acquisition unit 15A, a person detection unit 15B, a feature extraction unit 15C, a tracking unit 15D, a first re-identification unit 15E, a correspondence generation unit 15F, and a second re-identification unit 15G.
[0055] The acquisition unit 15A is a processing unit that acquires images. As just one example, the acquisition unit 15A can acquire images captured by the camera 30 via the network NW. In this case, the acquisition unit 15A can acquire images of each frame in real time, or can acquire a video including images for an arbitrary period of time.
[0056] The person detection unit 15B is a processing unit that detects a person from each image captured by the camera 30. As just one example, the person detection unit 15B can be realized by a machine learning model that receives an image as input and outputs a B-box of an object. Such a machine learning model may be realized by YOLO (You Only Look Once) or SSD (Single Shot MultiBox Detector), as well as a model that combines a CNN (Convolutional Neural Network) with a Transformer.
[0057] The feature extraction unit 15C is a processing unit that extracts feature amounts of each person detected from an image captured by the camera 30. As just one example, the feature extraction unit 15C can be realized by a machine learning model that receives an image, i.e., a B-box image, as input and outputs feature amounts (feature vectors) that represent visual features, or so-called appearance features. Such a machine learning model may be realized by a CNN that embeds an input image into a feature space.
[0058] The tracking unit 15D is a processing unit that tracks a person detected from an image captured by the camera 30. As just one example, the tracking unit 15D can be realized by a machine learning model that links objects between frames, such as MOT (Multiple-Object Tracking), SORT (Simple Online and Real Time Tracking), or DeepSORT.
[0059] More specifically, the tracking unit 15D can execute the following processes in parallel for each camera 30. For example, the tracking unit 15D assigns a person ID stored in the ID information 13A to a person B box detected in a frame whose image is acquired by the acquisition unit 15A. Here, the ID information 13A may be data in which the person B box detected in the image of each frame is linked to the person ID. At this time, the tracking unit 15D calculates the similarity and distance between the feature vector of the person B box extracted by the feature extraction unit 15C and the feature vector of the person B box detected in the previous frame. Then, the person B box extracted by the feature extraction unit 15C is assigned the person ID linked to the person B box with the highest similarity or the smallest distance among the people detected in the previous frame. At this time, if the highest similarity is equal to or less than a threshold or the smallest distance is equal to or greater than a threshold, a new person ID can be generated and assigned to the person B box extracted by the feature extraction unit 15C. Thereafter, the result of the assignment of the person ID to the person B box by the tracking unit 15D is registered as the result of the assignment of the person ID of the current frame in the ID information 13A.
[0060] Although only the previous frame is used for the calculation here, any number of past frames can also be used for calculating the similarity and distance. Also, although an example in which tracking is performed based on the evaluation of the similarity and distance of appearance features has been given here, tracking may also be performed based on the degree of overlap between B-boxes between frames.
[0061] The first re-identification unit 15E is a processing unit that performs person re-identification, so-called Person Re-ID, by comparing feature vectors of people in images captured by the multiple cameras 30A to 30N.
[0062] More specifically, for each pair of two cameras 30, the first re-identification unit 15E compares the feature vector of the B box image detected from the image of one camera 30 with the feature vector of the B box image detected from the image of the other camera 30. For example, the first re-identification unit 15E assigns the same person ID to a pair of B boxes having the highest similarity between their feature vectors or the smallest distance between their feature vectors. Here, as a mere example, the person ID assigned first by the tracking unit 15D, i.e., the oldest person ID, is assigned to both pairs of person IDs. In this case, if the highest similarity is equal to or less than a threshold or the smallest distance is equal to or greater than a threshold, the same person ID may not be assigned, and each of the person IDs of the pair assigned by the tracking unit 15D may be maintained. Thereafter, the result of the assignment of the person ID to the person B box by the first re-identification unit 15E is updated to the result of the assignment of the person ID for the current frame of the ID information 13A.
[0063] The correspondence generation unit 15F is a processing unit that generates positional correspondences between elements between images of each camera 30. As just one example, the correspondence generation unit 15F starts processing when the correspondence data 13B is incomplete. Here, "incomplete" refers to a state in which positional correspondences have not been accumulated in the correspondence data 13B to an extent that they can be used for person re-identification by the second re-identification unit 15G. Whether such correspondence data 13B is complete or incomplete can be determined based on the following criteria.
[0064] In one aspect, the correspondence generation unit 15F can determine that the correspondence data 13B is incomplete if the elapsed time from the start time when the generation of the correspondence data 13B began is less than a threshold value, and can determine that the correspondence data 13B is complete if the elapsed time is greater than or equal to the threshold value.
[0065] In another aspect, the correspondence generation unit 15F can determine that the correspondence data 13B is incomplete if the total number of data entries registered in the correspondence data 13B is less than a threshold value, and can determine that the correspondence data 13B is complete if the total number of data entries is equal to or greater than the threshold value.
[0066] As a further aspect, the correspondence generation unit 15F calculates the coverage rate of blocks from the data entries registered in the correspondence data 13B. The "coverage rate" here refers to the proportion of blocks for which positional correspondences are covered among all blocks, and is, for example, the value obtained by dividing the number of types of blocks for which positional correspondences are registered in the correspondence data 13B by the total number of blocks included in the image. If the coverage rate of such blocks is below a threshold, the correspondence data 13B can be determined to be incomplete, and if the coverage rate of the blocks is equal to or greater than the threshold, the correspondence data 13B can be determined to be complete.
[0067] If the correspondence data 13B is incomplete, the correspondence generation unit 15F determines whether the number of people present in the space being photographed is less than a threshold value, for example, 2. Such a determination of the number of people can be realized by determining whether the number of people detected from each of all images captured by the cameras 30A to 30N, i.e., objects with a class name of "Person," is less than a threshold value, for example, 2.
[0068] The above-described determination of the number of people is not limited to being realized by image processing, and may be realized by any other means. For example, a user input of a time or period when the number of people is less than a threshold, such as a frame number, may be accepted via a user interface (not shown). Furthermore, whether the number of people present in the space to be photographed is less than a threshold can be determined based on the number of terminals receiving beacons from user terminals by beacon receivers placed in the space to be photographed.
[0069] Here, if the number of people present in the space to be photographed is less than a threshold value, the correspondence generation unit 15F executes the following process. That is, the correspondence generation unit 15F adds to the correspondence data 13B a data entry including a combination of block numbers corresponding to the detected positions of people detected for each camera 30 by the person detection unit 15B in the frame in which the image of each camera 30 is acquired by the acquisition unit 15A. Note that the generation of positional correspondence can be executed as already described using FIGS. 3 to 7.
[0070] An example of how such correspondence data 13B is stored is as follows. For example, the correspondence data 13B may be stored in list format. In this case, as described with reference to FIGS. 5, 7, and 9, the correspondence data 13B stores, as data entries, combinations of block numbers corresponding to the positions at which people are detected in the images from each camera 30.
[0071] Alternatively, the correspondence data 13B may be stored in a matrix format. Fig. 10 is a diagram showing an example of the configuration of the correspondence data 13B. Fig. 10 shows correspondence data 13B1 in which the correspondence between the positions of elements in images captured by two cameras 30, camera 30A and camera 30B, is stored in a list format.
[0072] 10 , the list-format correspondence data 13B1 can be converted into matrix-format correspondence data 13B2. In the matrix-format correspondence data 13B2, the columns indicate the block numbers of camera 30A, and the rows indicate the block numbers of camera 30B. Each element of the matrix-format correspondence data 13B2 holds two values, "0" or "1," indicating whether or not there is a correspondence. For example, if there is a correspondence, a value of "1" is stored, and if there is no correspondence, a value of "0" is stored.
[0073] For example, the data entry in the first row of the list-format correspondence data 13B1, i.e., the data entry hatched with diagonal lines, indicates that there is a correspondence between block number "1" of camera 30A and block number "0" of camera 30B. Recording of an equivalent correspondence is realized by storing "1" in the element in the first row and second column of the matrix-format correspondence data 13B2.
[0074] Furthermore, the data entry in the fourth row of the list-format correspondence data 13B1, i.e., the data entry displayed in reverse video, indicates that there is a correspondence between block number "21" of camera 30A and block number "23" of camera 30B. Recording of a correspondence equivalent to this is realized by storing "1" in the element in the 23rd row and 21st column of the matrix-format correspondence data 13B2.
[0075] 10 illustrates two cameras 30, camera 30A and camera 30B, but positional correspondences for three or more cameras 30 can be stored in the same manner. That is, regardless of the number of cameras 30, positional correspondences can be stored for each pair. For example, if there are N cameras 30, N×(N−1) pieces of matrix-formatted correspondence data 13B2 shown in FIG. 10 can be generated.
[0076] Returning to the explanation of Figure 1, the second re-identification unit 15G is a processing unit that performs person re-identification by comparing the combination of person detection positions in the images of each camera 30 with the combination of image elements between multiple cameras registered in the correspondence data 13B.
[0077] As merely an example, the second re-identification unit 15G can be constantly activated for each frame, or can be activated when a specific condition is met, for example, when a person is detected in an image from any of the cameras 30A to 30N. In this way, the constantly activated or conditionally activated mode can be selected, as merely an example, depending on the performance of the processor and memory installed in the information processing device 10, or the usage scene, such as the time of day.
[0078] After starting up, second re-identification unit 15G executes the following process for a total of K pairs of two cameras 30 selected from N cameras 30A to 30N. Hereinafter, one camera 30 included in the k-th pair is identified as camera i, and the other camera 30 is identified as camera j.
[0079] For example, the second re-identification unit 15G searches the correspondence data 13B for combinations of the number L of B boxes detected from the image of the camera i of the kth pair and the number M of B boxes detected from the image of the camera j of the kth pair.
[0080] More specifically, the second re-identification unit 15G searches the correspondence data 13B for a combination of the block number kil corresponding to the detection position of the lth Bbox of the camera i and the block number kjm corresponding to the detection position of the mth Bbox of the camera j.
[0081] In this case, if the combination of block number kil and block number kjm is found, it can be identified that the person in the lth Bbox of camera i and the person in the mth Bbox of camera j are the same person.
[0082] In this case, the second re-identification unit 15G determines whether or not there is a mismatch between the person re-identification result by the second re-identification unit 15G and the person re-identification result by the first re-identification unit 15E. Hereinafter, the person re-identification result by the first re-identification unit 15E may be referred to as the "first person re-identification result," and the person re-identification result by the second re-identification unit 15G may be referred to as the "second person re-identification result."
[0083] For example, if the first re-identification unit 15E assigns different person IDs to the lth Bbox of camera i and the mth Bbox of camera j, it is determined that the first person re-identification result and the second person re-identification result do not match.
[0084] In this case, the second re-identification unit 15G assigns the same person ID to the lth B box of camera i and the mth B box of camera j. For example, if the same person ID is assigned to a pair of B boxes including either the lth B box of camera i or the mth B box of camera j in another pair of cameras 30, the second re-identification unit 15G assigns the person ID preferentially. In addition, the second re-identification unit 15G preferentially assigns person IDs that were assigned earlier by the tracking unit 15, i.e., older person IDs, or person IDs for which tracking has continued for a large number of frames. As a result, the result of assigning person IDs to person B boxes by the second re-identification unit 15G is updated to the result of assigning person IDs for the current frame of the ID information 13A.
[0085] On the other hand, if the combination of block number kil and block number kjm does not match, it can be determined that the person in the lth Bbox of camera i and the person in the mth Bbox of camera j are not the same person.
[0086] In this case, the second re-identification unit 15G determines whether or not the person re-identification result by the second re-identification unit 15G and the person re-identification result by the first re-identification unit 15E are inconsistent.
[0087] For example, if the first re-identification unit 15E assigns the same person ID to the lth Bbox of camera i and the mth Bbox of camera j, it is determined that the first person re-identification result and the second person re-identification result do not match.
[0088] In this case, the second re-identification unit 15G assigns different person IDs to the lth B box of camera i and the mth B box of camera j. For example, the second re-identification unit 15G returns the person IDs to the person IDs that were assigned to the lth B box of camera i and the mth B box of camera j by the tracking unit 15D before the same person IDs were assigned by the first re-identification unit 15E. As a result, the result of assigning a person ID to a person B box by the second re-identification unit 15G is updated to the result of assigning a person ID for the current frame of the ID information 13A.
[0089] The ID information 13A thus obtained can be output, by way of example only, to software that executes processes such as monitoring, marketing analysis, and customer behavior analysis, or to a backend that provides services.
[0090] <Processing Flow> Next, a processing flow of the information processing device 10 according to this embodiment will be described. Here, (1) Overall processing executed by the information processing device 10 will be described, and then (2) Second person re-identification processing will be described.
[0091] (1) Overall Processing Fig. 11 is a flowchart showing the procedure of the overall processing of the information processing device 10. The processing shown in Fig. 11 is merely an example and can be executed on a frame-by-frame basis. As shown in Fig. 11, the acquisition unit 15A acquires images captured by each camera 30 via the network NW (step S101).
[0092] Next, the person detection unit 15B detects a person from each image captured by each camera 30 (step S102). Then, the feature extraction unit 15C extracts a feature amount of each person detected from the image captured by the camera 30 (step S103).
[0093] Then, the tracking unit 15D tracks the people detected from the images captured by the camera 30 between frames, and assigns the person ID stored in the ID information 13A to each person detected from the images captured by the camera 30 (step S104).
[0094] Next, the first re-identification unit 15E performs person re-identification by comparing feature amounts of people in images from the multiple cameras 30A to 30N, and assigns the same person ID to the same person (step S105).
[0095] If the correspondence data 13B is incomplete (Yes in step S106), the correspondence generation unit 15F determines whether the number of people present in the space to be photographed is less than a threshold value, for example, "2" (step S107).
[0096] At this time, if the number of people present in the space to be photographed is less than the threshold value (Yes in step S107), the correspondence generation unit 15F executes the following process: The correspondence generation unit 15F adds a data entry including a combination of block numbers corresponding to the detection positions of people detected for each camera 30 in the current frame to the correspondence data 13B (step S108).
[0097] On the other hand, if the correspondence data 13B is complete (No in step S106), the second re-identification unit 15G executes the following process: The second re-identification unit 15G executes person re-identification by comparing a combination of the detected positions of the person in the image of each camera 30 with a combination of image elements between the multiple cameras registered in the correspondence data 13B (step S109).
[0098] Thereafter, the results of the assignment of the person ID assigned to each person by the tracking unit 15D, the first re-identification unit 15E, and the second re-identification unit 15G are output to an arbitrary output destination (step S110), and then the processing is terminated.
[0099] (2) Second Person Re-Identification Process FIG. 12 is a flowchart showing the procedure of the second person re-identification process. The process shown in FIG. 12 corresponds to the process of step S109 shown in FIG. 11. As shown in FIG. 12, the second re-identification unit 15G executes loop process 1, which repeats the processes from step S301 to step S309 a total number of times, K, for combining pairs of two cameras 30 from the N cameras 30, cameras 30A to 30N. Note that while FIG. 12 shows an example in which the processes from step S301 to step S309 are repeated, this process can also be executed in parallel.
[0100] Furthermore, the second re-identification unit 15G executes loop process 2 in which the processes from step S301 to step S308 are repeated for the number L of B boxes detected from the image of the camera i of the kth pair. Note that although an example in which the processes from step S301 to step S308 are repeated is shown in FIG. 12, this process can also be executed in parallel.
[0101] In addition, the second re-identification unit 15G executes loop process 2 in which the processes from step S301 to step S307 are repeated for the number M of B boxes detected from the image of the camera j of the kth pair. Note that although an example in which the processes from step S301 to step S307 are repeated is shown in FIG. 12, this process can also be executed in parallel.
[0102] That is, the second re-identification unit 15G searches the correspondence data 13B for a combination of the block number kil corresponding to the detection position of the lth Bbox of the camera i and the block number kjm corresponding to the detection position of the mth Bbox of the camera j (step S301).
[0103] At this time, if the combination of block number kil and block number kjm is found (Yes in step S302), the person in the lth Bbox of camera i and the person in the mth Bbox of camera j can be identified as the same person.
[0104] In this case, the second re-identification unit 15G determines whether the second person re-identification result obtained by branching to Yes in step S302 is inconsistent with the first person re-identification result obtained in step S105 (step S303).
[0105] If the first person re-identification result and the second person re-identification result do not match (Yes in step S303), the second re-identification unit 15G assigns the same person ID to the lth Bbox of camera i and the mth Bbox of camera j (step S304). Note that if the first person re-identification result and the second person re-identification result match (No in step S303), the processing of step S304 is skipped.
[0106] On the other hand, if the combination of block number kil and block number kjm does not match (No in step S302), it can be determined that the person in the lth Bbox of camera i and the person in the mth Bbox of camera j are not the same person.
[0107] In this case, the second re-identification unit 15G determines whether the second person re-identification result obtained by branching to No in step S302 is inconsistent with the first person re-identification result obtained in step S105 (step S305).
[0108] Then, if the first person re-identification result and the second person re-identification result do not match (Yes in step S305), the second re-identification unit 15G assigns different person IDs to the lth Bbox of camera i and the mth Bbox of camera j (step S306).
[0109] Thereafter, a loop counter m that counts M Bboxes detected from the image of the camera j of the kth pair is incremented, and loop process 3 is repeated until the loop counter m exceeds M.
[0110] By repeating this loop process 3, the M B boxes detected from the image of the j camera of the kth pair are matched with the l-th B box out of the L B boxes detected from the image of the i camera of the kth pair.
[0111] Thereafter, a loop counter l that counts L Bboxes detected from the image of the camera i of the kth pair is incremented, and loop process 2 is repeated until the loop counter l exceeds L.
[0112] By repeating this loop process 2, all combinations of L B boxes detected from the image of camera i of the kth pair and M B boxes detected from the image of camera j of the kth pair are matched.
[0113] Thereafter, a loop counter k that counts the total number of combinations of pairs of two cameras 30 from the N cameras 30A to 30N is incremented, and loop process 1 is repeated until loop counter k exceeds K.
[0114] By repeating this loop process 1, the second person re-identification process is completed for all combinations of pairs of two cameras 30 selected from the N cameras 30A to 30N.
[0115] <One aspect of the effect> As described above, the information processing device 10 according to this embodiment generates correspondences between elements in the images of each camera based on the detection positions of people detected in the images of each camera, in the background of performing person re-identification by comparing feature information of people in images between multiple cameras.
[0116] Therefore, the information processing device 10 of this embodiment can generate positional correspondences between elements in the images of each camera 30 using the object to be tracked as a clue, rather than a marker such as tape that needs to be installed in advance.
[0117] Therefore, the information processing device 10 according to this embodiment can achieve person re-identification without using markers. This eliminates the need to install markers such as tape in the space to be photographed in advance. Furthermore, when person re-identification is performed using the positional correspondence between elements in the images of each camera 30, coordinate conversion to a common coordinate system, as in the person re-identification performed by the above-described surveillance camera terminal, is not required, thereby reducing the processing load. In addition, in the person re-identification performed by the above-described surveillance camera terminal, errors in coordinate conversion increase as the position of the subject person moves away from the four markers, but the occurrence and expansion of such errors can be suppressed.
[0118] Furthermore, the information processing device 10 according to this embodiment can achieve more robust person re-identification than person re-identification based on matching of feature information. That is, when person re-identification is performed by matching of feature information, the accuracy of person re-identification decreases more significantly as the change in viewpoint or lighting conditions between the cameras 30 increases, and the accuracy of person re-identification also decreases more significantly as the resolution of the cameras 30 decreases. On the other hand, when person re-identification is performed using the positional correspondence between elements between images from each camera 30, as in the information processing device 10 according to this embodiment, the effect of changes in viewpoint, lighting conditions, camera resolution, etc. is small, and therefore the decrease in accuracy of person re-identification can be suppressed.
[0119] Although the embodiments of the disclosed device have been described above, the present invention may be embodied in various different forms other than the above-described embodiments. Therefore, other embodiments included in the present invention will be described below.
[0120] <Orientation of Person> In the above-described first embodiment, an example has been given in which the correspondence between the positions of blocks between multiple cameras is registered in the correspondence data 13B, but other data can also be registered. As just one example, the orientation of a person, for example, a moving direction, can also be registered in the correspondence data 13B for each camera 30.
[0121] FIG. 13 is a diagram showing an example of person detection. FIG. 14 is a diagram showing an example of generating positional correspondences between image elements. FIG. 13 shows the detected positions of people detected in frames t and t+1 for each of cameras 30A and 30B, based on the camera arrangement and person movement shown in FIG. 3. Furthermore, FIG. 13 plots the midpoints of the bases of the four sides included in box B with white circles as the detected positions of people. Furthermore, FIG. 13 plots the movement direction calculated from the difference between the detected position of the person in the previous frame and the detected position of the person in the current frame with an arrow as the orientation of the person.
[0122] 13, the image captured by camera 30 is divided into a total of 24 blocks arranged in 4 rows and 6 columns, just as an example. These blocks are used as elements of the image of camera 30, and the positional correspondence between the elements is recorded frame by frame between camera 30A and camera 30B.
[0123] For example, in image 20A captured by camera 30A in frame t, the detected position of a person is in the block with block number "15," while in image 20B captured by camera 30B in frame t, the detected position of the person is in the block with block number "14." In this case, as shown in FIG. 14 , a correspondence is recorded between block number "15," which is an element of image 20A captured by camera 30A, and block number "14," which is an element of image 20B captured by camera 30B. Also, as shown in FIG. 13 , in image 20A captured by camera 30A in frame t, the movement direction, i.e., an arrow pointing downward and to the left, can be calculated from the difference between the detected position of the person in frame t−1 and the detected position of the person in frame t. Therefore, as shown in FIG. 14 , the direction "arrow pointing downward and to the left" is recorded in association with camera 30A in the data entry for frame t. 13, in image 20B captured by camera 30B in frame t, the movement direction, i.e., the leftward arrow, can be calculated from the difference between the detected position of the person in frame t-1 and the detected position of the person in frame t. Therefore, as shown in FIG. 14, the direction "leftward arrow" is recorded in association with camera 30B in the data entry for frame t.
[0124] Furthermore, as shown in FIG. 13 , in image 21A captured by camera 30A in frame t+1, the detected position of the person is in the block with block number "21," while in image 21B captured by camera 30B in frame t+1, the detected position of the person is in the block with block number "13." In this case, as shown in FIG. 14 , a correspondence is recorded between block number "21," which is an element of image 21A captured by camera 30A, and block number "13," which is an element of image 21B captured by camera 30B. Also, as shown in FIG. 13 , in image 21A captured by camera 30A in frame t+1, the movement direction, i.e., an arrow pointing downward and to the left, can be calculated from the difference between the detected position of the person in frame t and the detected position of the person in frame t+1. Therefore, as shown in FIG. 14 , the direction "arrow pointing downward and to the left" is recorded in association with camera 30A in the data entry for frame t+1. Furthermore, as shown in Fig. 13, in image 21B captured by camera 30B in frame t+1, the movement direction, i.e., the leftward arrow, can be calculated from the difference between the detected position of the person in frame t and the detected position of the person in frame t+1. Therefore, as shown in Fig. 14, the direction "leftward arrow" is recorded in association with camera 30B in the data entry for frame t+1.
[0125] As such, in the example shown in Figure 13, the detected positions of people in frames t and t+1 are illustrated as just one example, but it is obvious that as more time passes from frame t+2 onwards, it will be possible to accumulate positional correspondences between more elements.
[0126] For example, as shown in Fig. 14, by registering the orientation of a person in each camera 30 for each frame in the correspondence data 13B, if multiple people exist in the same block, they can be distinguished by their orientations if they are facing different directions. In other words, people can be distinguished by comparing their orientations between frames. In this way, when the second re-identification unit 15G uses the orientation of a person, the correspondence data 13B can be continuously updated even after the correspondence data 13B is completed.
[0127] <Person's Speed> As described above, an example has been given in which the orientation of a person in each camera 30 is registered in the correspondence data 13B for each frame. However, other data can also be registered. For example, the correspondence data 13B can also register the person's speed or speed ratio, such as the moving speed, for each camera 30. Such a moving speed can be calculated by converting the moving distance determined from the detected position of the person in the previous frame and the detected position of the person in the current frame into a unit time value. In this case, when multiple people exist in the same block, they can be distinguished based on their differences in speed. In other words, people can be distinguished by comparing the speeds between frames. Even when the second re-identification unit 15G uses the person's speed in this way, the correspondence data 13B can continue to be updated even after the correspondence data 13B is completed.
[0128] <Interpolation between blocks> For example, when the combinations of block numbers of each camera 30 are registered in chronological order in the correspondence data 13B, if the arrangement of block numbers corresponding to the detection positions of a person detected in chronological order for each camera 30 is an arrangement in which the person moves in any of eight directions (up and down, left and right, upper left, lower left, upper right, and lower right) by skipping a specific number of blocks, for example, one block, the correspondence generation unit 15F additionally registers the combination of block numbers sandwiched between the arrangement of block numbers in the correspondence data 13B.
[0129] FIG. 15 is a diagram illustrating interpolation between blocks. FIG. 15 shows an example in which images 20A and 20B captured by cameras 30A and 30B are divided into a total of 24 blocks arranged in 4 rows and 6 columns. Furthermore, FIG. 15 indicates the presence or absence of a person detected in each block of image 20A and image 20B using a binary value of "0" or "1." As shown in FIG. 15 , an example is given in which the block numbers corresponding to the positions of people detected in time series by camera 30A are block numbers "9" and "19," and the block numbers corresponding to the positions of people detected in time series by camera 30B are block numbers "16" and "14." In this case, the block numbers for camera 30A are arranged so that every other block is alternated to the lower left. Furthermore, the block numbers for camera 30B are arranged so that every other block is alternated to the right. In this case, the combination of block numbers sandwiched between the arrays of block numbers of camera 30A and camera 30B, that is, the combination of block number "14" and block number "15", is additionally registered in correspondence data 13B.
[0130] <Frequency> In the first embodiment described above, the presence or absence of a positional correspondence is registered in the correspondence data 13B. However, if the same block number combination is detected when the correspondence data 13B is generated, the frequency of the positional correspondence may be maintained by updating the number of positional correspondences. FIG. 16 is a diagram illustrating an example of the configuration of correspondence data. In the matrix-format correspondence data 13B2 shown in FIG. 16, the columns indicate the block numbers of camera 30A, and the rows indicate the block numbers of camera 30B. Each element of the matrix-format correspondence data 13B2 stores the frequency of the correspondence. For example, the element in the first row and second column of the matrix-format correspondence data 13B2 indicates that the frequency of the combination of block number "1" of camera 30A and block number "0" of camera 30B is "5." This frequency may be an actual measurement of the number of times the block number combination is detected, or a normalized value. For example, the maximum number of detections may be determined, and the frequency may be normalized according to the following equation (1). By storing the frequency of positional correspondence in correspondence data 13B in this manner, if the first person re-identification result and the second person re-identification result do not match, a correction can be performed to match the first person re-identification result to the second person re-identification result by narrowing it down to cases where the frequency of the block number combination exceeds a threshold value.
[0131] Frequency = Number of detections / Maximum number of detections (1)
[0132] 17 is a flowchart showing the procedure of the second person re-identification process. In FIG. 17, different step numbers are assigned to steps in which different procedures are performed compared to the flowchart shown in FIG.
[0133] 17, the process executed in the branch of Yes in step S302 is different from the flowchart shown in Fig. 12. That is, when the combination of block number kil and block number kjm is found (Yes in step S302), the second re-identification unit 15G determines whether the frequency of the combination of block number kil and block number kjm exceeds a threshold (step S501).
[0134] If the frequency of the combination of block numbers kil and kjm exceeds the threshold (Yes in step S501), the process proceeds to step S303. On the other hand, if the frequency of the combination of block numbers kil and kjm does not exceed the threshold (No in step S501), the process proceeds to step S305.
[0135] By adding the processing of step S501, correction can be performed only in cases where the correction from the first person re-identification result to the second person re-identification result occurs frequently, thereby improving the accuracy of Re-ID.
[0136] <Distribution and Integration> Furthermore, the components of each device shown in the figure do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of the devices can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. For example, the acquisition unit 15A, the person detection unit 15B, the feature extraction unit 15C, the tracking unit 15D, the first re-identification unit 15E, the correspondence generation unit 15F, or the second re-identification unit 15G may be connected via a network as an external device to the information processing device 10. Furthermore, the acquisition unit 15A, the person detection unit 15B, the feature extraction unit 15C, the tracking unit 15D, the first re-identification unit 15E, the correspondence generation unit 15F, or the second re-identification unit 15G may each be included in a separate device and connected to a network to cooperate with each other to realize the functions of the information processing device 10.
[0137] <Application Example> Next, an application example will be described with reference to Fig. 18. The information processing device 10 can analyze the behavior of a person who has checked in using images captured by the cameras 30A to 30N. The facility 1 is a railway facility, an airport, a store, etc. Furthermore, the gate G1 located at the facility 1 is located at the entrance to a store, a railway facility, a boarding gate at an airport, etc.
[0138] First, an example will be described in which the check-in target is a railway facility or an airport. In the case of a railway facility or an airport, gate G1 is located at a ticket gate at the railway facility, a counter at the airport, or an inspection area. In this case, if the person's biometric information has been pre-registered as a train or airplane passenger, the information processing device 10 determines that authentication using the person's biometric information has been successful.
[0139] Next, an example will be described in which the check-in target is a store. When the check-in target is a store, gate G1 is located at the entrance of the store. In this case, when the biometric information of the person who checked in is registered as a member of the store, the information processing device 10 determines that authentication using the biometric information of the person has been successful.
[0140] Here, the details of check-in will be described. Authentication is performed by acquiring a vein image or the like from the biometric sensor 31, for example, a vein sensor, from the biometric sensor 31. This identifies the ID, name, etc. of the person checking in.
[0141] At that time, the information processing device 10 uses the cameras 30A to 30N to acquire images of the person checking in. Next, the information processing device 10 detects the person from the image. The tracking unit 15D of the information processing device 10 tracks the person detected from the image captured by the camera 30 between frames. The information processing device 10 associates the ID and name of the person checking in with the person being tracked.
[0142] An application example will now be described with reference to Fig. 19 , assuming that the facility is a store. When checking in, the information processing device 10 acquires biometric information of a person passing through gate G1 located at a predetermined position within the store (step S601). Specifically, the information processing device 10 acquires, from biometric sensor 31 mounted on gate G1 located at the entrance to the store, for example, a vein image acquired by a vein sensor, and performs authentication. At this time, the information processing device 10 identifies the user's ID, name, etc. from the biometric information.
[0143] The biometric sensor 31 is mounted on a gate G1 located at a predetermined position in the facility and detects biometric information of people passing through the gate G1. The cameras 30A to 30N are installed on the ceiling of the store.
[0144] In addition, instead of using the biometric sensor 31, the information processing device 10 may perform authentication by acquiring biometric information from a face image captured by a camera mounted on a gate G1 located at the entrance to the store.
[0145] Next, the information processing device 10 determines whether authentication using the person's biometric information has been successful (step S602). If authentication has been successful (step S602: Yes), the information processing device 10 proceeds to step S603. On the other hand, if authentication has not been successful (step S602: No), the information processing device 10 proceeds to step S601.
[0146] Then, the information processing device 10 analyzes an image including a person passing through gate G1 to identify the person included in the image as a person who has checked in to facility 1 (step S603). The information processing device 10 stores in a storage unit the identification information of the person specified from the biometric information and the identified person in association with each other. Specifically, the information processing device 10 stores the ID and name of the person checking in and the identified person in association with each other.
[0147] Thereafter, the information processing device 10 analyzes the images captured by the cameras 30A to 30N and tracks the identified person using the results of identifying the first person and the second person (step S604). That is, the information processing device 10 identifies the identity of the person captured by the multiple cameras 30A to 30N. Then, the information processing device 10 identifies the route along which the identified person was tracked, thereby identifying the trajectory of the identified person within the facility 1.
[0148] This allows for the analysis of a person's purchasing behavior by determining whether the person who checked in acquired a product placed in the store after the person checked in. A person's purchasing behavior will now be described. The information processing device 10 generates skeletal information of the person by analyzing an image including the tracked person. The information processing device 10 then uses the generated skeletal information to identify the behavior of the tracked person in acquiring a product. In other words, after the person checks in to a store, the information processing device 10 determines whether the person acquired any of a plurality of products placed in the store from the time the person entered the store until the time the person left the store. The information processing device 10 then stores the result of whether or not the product was acquired in association with the ID and name of the person who checked in.
[0149] Specifically, the information processing device 10 uses existing object detection technology to identify customers staying in a store and products placed in the store from images captured by cameras 30A-30N. Furthermore, the information processing device 10 uses existing skeletal detection technology to generate skeletal information of the identified person from images captured by cameras 30A-30N and estimate the position and posture of each joint of the person. Then, based on the positional relationship between the skeletal information and the product, the information processing device 10 detects actions such as grasping a product or placing a product in a basket or cart. For example, the information processing device 10 determines that a product is being grasped when the skeletal information located at the position of the person's arm overlaps with the area of the product.
[0150] The existing object detection algorithm may be, for example, an object detection algorithm using deep learning such as Faster R-CNN (Convolutional Neural Network). It may also be an object detection algorithm such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector). The existing skeleton estimation algorithm may be, for example, a skeleton estimation algorithm using deep learning such as HumanPose Estimation such as DeepPose or OpenPose.
[0151] Although a vein image is used as an example of biometric information here, the biometric information may be a face image, a fingerprint image, an iris image, or the like. Furthermore, in the above-described first embodiment, an example was given in which the feature quantity of an image captured by the camera 30 embedded in a feature space by the feature extraction unit 15C is used for the first person re-identification by the first re-identification unit 15E, but this is not limiting. For example, biometric information detected by the cameras 30A to 30N or feature quantities extracted from the biometric information may also be used for the first person re-identification.
[0152] <Hardware Configuration> The various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer, a workstation, etc. Therefore, an example of a computer that executes an identification program having the same functions as those in the first and second embodiments will be described below with reference to FIG.
[0153] Fig. 20 is a diagram showing an example of a hardware configuration. As shown in Fig. 20, a computer 100 has an operation unit 110a, a speaker 110b, a camera 110c, a display 120, and a communication unit 130. The computer 100 also has a CPU 150, a ROM 160, an HDD 170, and a RAM 180. These units 110 to 180 are connected via a bus 140.
[0154] As shown in Fig. 20, the HDD 170 stores an identification program 170a that performs functions similar to those of the acquisition unit 15A, person detection unit 15B, feature extraction unit 15C, tracking unit 15D, first re-identification unit 15E, correspondence generation unit 15F, and second re-identification unit 15G shown in Fig. 1. This identification program 170a may be integrated or separated, similar to the components of the acquisition unit 15A, person detection unit 15B, feature extraction unit 15C, tracking unit 15D, first re-identification unit 15E, correspondence generation unit 15F, and second re-identification unit 15G shown in Fig. 1. In other words, it is not necessary for all of the data shown in the first embodiment to be stored in the HDD 170; it is sufficient that the data used for processing is stored in the HDD 170.
[0155] Under such an environment, the CPU 150 reads the identification program 170a from the HDD 170 and loads it into the RAM 180. As a result, the identification program 170a functions as an identification process 180a, as shown in FIG. 20 . The identification process 180a loads various data read from the HDD 170 into an area of the storage area of the RAM 180 allocated to the identification process 180a, and executes various processes using the loaded data. For example, examples of processes executed by the identification process 180a include the processes shown in FIGS. 11 , 12 , and 17 . Note that the CPU 150 does not necessarily need to operate all of the processing units shown in the first embodiment; it is sufficient that the processing units corresponding to the processes to be executed are virtually implemented.
[0156] It should be noted that the above-described identification program 170a does not necessarily have to be stored in the HDD 170 or the ROM 160 from the beginning. For example, each program may be stored on a "portable physical medium" such as a flexible disk, a so-called FD, a CD-ROM, a DVD disk, a magneto-optical disk, or an IC card that is inserted into the computer 100. The computer 100 may then acquire and execute each program from such a portable physical medium. Alternatively, each program may be stored in another computer or server device connected to the computer 100 via a public line, the Internet, a LAN, a WAN, or the like, and the computer 100 may acquire and execute each program from such a computer or server device.
[0157] REFERENCE SIGNS LIST 10 Information processing device 11 Communication control unit 13 Storage unit 13A ID information 13B Correspondence data 15 Control unit 15A Acquisition unit 15B Person detection unit 15C Feature extraction unit 15D Tracking unit 15E First re-identification unit 15F Correspondence generation unit 15G Second re-identification unit 30A, 30B, ..., 30N Camera
Claims
1. An identification method characterized by being executed by a computer to perform the following processes: when a first person is detected from a first image taken with a first camera and a second person is detected from a second image taken with a second camera, generate relationship information that associates the position in the first image where the first person is detected with the position in the second image where the second person is detected; and identify the first person and the second person based on characteristic information of the first person and characteristic information of the second person.
2. The identification method described in claim 1, characterized in that the generating process includes a process of generating the relationship information when the number of people detected from the first image and the number of people detected from the second image are less than a threshold value.
3. The identification method of claim 1, further comprising the computer further executing a process of identifying the first person and the second person based on whether or not a combination of a position in the first image where the first person is detected and a position in the second image where the second person is detected is registered in the relationship information, and correcting the first identification result to the second identification result if there is a mismatch between a first identification result obtained using the feature information and a second identification result obtained using the relationship information.
4. The identification method according to claim 3, characterized in that the process of identification using the feature information includes a process of identifying the first person and the second person when the elapsed time since the generation of the relationship information began is equal to or greater than a threshold.
5. The identification method described in claim 3, characterized in that the process of identification using the feature information includes a process of identifying the first person and the second person when the number of associations between positions in the first image and positions in the second image in the relationship information is equal to or greater than a threshold value.
6. The identification method described in claim 3, characterized in that the process of identification using the feature information includes a process of identifying the first person and the second person when the coverage rate of the combination of positions in the first image and positions in the second image in the relationship information is equal to or greater than a threshold.
7. The identification method of claim 1, characterized in that the generating process further includes a process of correlating an orientation of the first person detected between frames of the first image and an orientation of the second person detected between frames of the second image.
8. The identification method of claim 1, wherein the generating process further includes a process of correlating a velocity of the first person detected between frames of the first image and a velocity of the second person detected between frames of the second image.
9. The method of identification described in claim 1, further comprising the steps of: mounting a sensor or camera on a gate placed at a predetermined position in a facility; acquiring biometric information of a person passing through the gate based on detection of the biometric information of the person by the sensor or camera; when authentication using the acquired biometric information of the person is successful, analyzing an image including the person passing through the gate to identify the person included in the image as a person who has checked in to the facility; storing identification information of the person specified from the biometric information in association with the identified person; and tracking the identified person using the results of identifying the first person and the second person.
10. The identification method described in claim 9, characterized in that the facility is a store, the gate is located at the entrance of the store, and when the acquired biometric information of the person is registered as a member of the store, it is determined that authentication using the biometric information of the person has been successful, and the purchasing behavior of the person from entering the store to leaving the store is identified by tracking the person as they move through the store.
11. The identification method described in claim 10, further comprising: generating skeletal information of the person by analyzing an image including the tracked person; and using the generated skeletal information to identify whether the tracked person has performed an action related to the purchase in the store, such as acquiring a product placed in the store.
12. The identification method described in claim 9, characterized in that the facility is either a railway facility or an airport, the gate is located at a ticket gate at the railway facility, or a counter or inspection area at an airport, and when the acquired biometric information of the person has been pre-registered as a railway or airplane passenger, it is determined that authentication using the biometric information of the person has been successful.
13. An identification program that causes a computer to execute a process of: detecting a first person from a first image taken with a first camera, and when detecting a second person from a second image taken with a second camera, generating relationship information that associates the position in the first image where the first person is detected with the position in the second image where the second person is detected; and identifying the first person and the second person based on characteristic information of the first person and characteristic information of the second person.
14. The identification program described in claim 13, characterized in that a sensor or camera is mounted on a gate placed at a predetermined position in a facility, and the biometric information of a person passing through the gate is acquired based on the detection of the biometric information of the person by the sensor or camera, and when authentication using the acquired biometric information of the person is successful, an image including the person passing through the gate is analyzed to identify the person included in the image as a person who has checked in to the facility, identification information of the person specified from the biometric information is stored in correspondence with the identified person, and the identified person is tracked using the results of identifying the first person and the second person.
15. The identification program described in claim 14, characterized in that the facility is a store, the gate is located at the entrance of the store, and when the acquired biometric information of the person is registered as a member of the store, it is determined that authentication using the biometric information of the person has been successful, and the purchasing behavior of the person from entering the store to leaving the store is identified by tracking the person as they move through the store.
16. The identification program described in claim 15, further comprising: generating skeletal information of the tracked person by analyzing an image including the tracked person; and using the generated skeletal information to identify whether the tracked person has engaged in an action related to the purchase, such as acquiring a product placed in the store, in the store.
17. The identification program described in claim 14, characterized in that the facility is either a railway facility or an airport, the gate is located at a boarding gate of the railway facility or airport, and when the acquired biometric information of the person has been pre-registered as a railway or airplane passenger, it is determined that authentication using the biometric information of the person has been successful.
18. An information processing device having a control unit that executes a process in which, when a first person is detected from a first image taken using a first camera and a second person is detected from a second image taken using a second camera, relationship information is generated that associates the position in the first image where the first person is detected with the position in the second image where the second person is detected, and the first person and the second person are identified based on characteristic information of the first person and characteristic information of the second person.
Citation Information
Patent Citations
Surveillance camera terminal
WO2011010490A1
Moving body tracking system using a plurality of cameras and its method
JP2004072628A
Monitoring system with autonomous mobile body, monitoring device, autonomous mobile body, monitoring method and monitoring program
JP2012078950A
Position identification device, position identification program, route specification device, and route specification program
JP2017182761A
Store management device and store management method
WO2019181499A1
