Assessment method, assessment program, and information processing device

By synchronizing image processing across cameras with different frame rates and determining a unified execution interval, the method improves person tracking and determination accuracy in biometric authentication systems, enabling continuous authentication across multiple cameras.

WO2024236682A1PCT designated stage expired Publication Date: 2024-11-21FUJITSU LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/018060
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-15
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

In biometric authentication systems using multiple cameras with different shooting intervals, images taken at intervals shorter than the set interval are not utilized, leading to a deterioration in person tracking and determination accuracy, especially when multiple people are present with different movement directions.

Method used

A method that calculates the trajectory of individuals from images acquired by cameras with different frame rates, determines a unified execution interval for image processing, and synchronizes image processing to maintain high tracking accuracy by selecting images based on the lowest frame rate, ensuring consistent authentication across cameras.

Benefits of technology

This approach enhances the accuracy of person determination and tracking by ensuring that the authentication state can be continuously maintained without the need for repeated authentication actions, even in environments with varying camera frame rates and multiple individuals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023018060_21112024_PF_FP_ABST
    Figure JP2023018060_21112024_PF_FP_ABST
Patent Text Reader

Abstract

In this assessment method, a computer executes processes including: calculating the trajectory of a person appearing in each of a plurality of first images in a time series captured by a first camera at each first time, and the trajectory of a person appearing in each of a plurality of second images in a time series captured by a second camera at each second time, when the plurality of first images and the plurality of second images are acquired; determining a third time in accordance with the time at which each of the plurality of first images is acquired from the first camera and the time at which each of the plurality of second images is acquired from the second camera; and assessing whether the persons for which the trajectory was calculated from each of the image corresponding to the determined third time among the plurality of first images and the image corresponding to the determined third time among the plurality of second images are the same person. 
Need to check novelty before this filing date? Find Prior Art

Description

Determination method, determination program, and information processing device

[0001] The present invention relates to a determination method, a determination program, and an information processing device.

[0002] A technology has been disclosed that uses images captured by multiple cameras to track a person and continue authentication (see, for example, Japanese Patent Laid-Open No. 2003-222294).

[0003] JP 2013-206140 A

[0004] For example, when tracking and authenticating a person using multiple cameras with different shooting intervals, a predetermined interval is set so that the authentication process is performed using images captured by each camera at the same time, and the person tracking and authentication process is performed at the set interval. However, in this case, images captured at intervals shorter than the set interval are not used in the person tracking process, which deteriorates the accuracy of person tracking and causes a decrease in the accuracy of person determination.

[0005] In one aspect, the present invention aims to provide a determination method, a determination program, and an information processing device that can improve the accuracy of person determination.

[0006] In one aspect, the determination method is characterized in that, when a first camera acquires a plurality of first images in a time series taken every first hour and a plurality of second images in a time series taken every second hour by a second camera, a computer executes a process to calculate a trajectory of a person appearing in each of the plurality of first images and a trajectory of a person appearing in each of the plurality of second images, determine a third time based on the time at which each of the plurality of first images is acquired from the first camera and the time at which each of the plurality of second images is acquired from the second camera, and determine whether the person whose trajectory is calculated from both an image among the plurality of first images corresponding to the determined third time and an image among the plurality of second images corresponding to the determined third time are the same person.

[0007] This can improve the accuracy of determining people.

[0008] FIG. 1 is a diagram illustrating continuous authentication. FIG. 2 is a diagram illustrating details of Re-ID. FIG. 3 is a diagram illustrating details of Re-ID. FIG. 4 is a diagram for explaining an overview of a first embodiment. FIG. 5 is a diagram for explaining an overview of a first embodiment. (a) is a block diagram illustrating an overall configuration of a biometric authentication system according to a first embodiment, and (b) is a functional block diagram of an information processing device. (b) is a flowchart illustrating Re-ID processing executed by an information processing device. (c) is a diagram illustrating determination of an execution interval for second image processing. (d) is a diagram illustrating images to be saved in a queue. (e) is a diagram illustrating retrieval of images. (f) is a block diagram illustrating a hardware configuration of an information processing device.

[0009] Biometric authentication is a technology that verifies identity using biometric features such as fingerprints, faces, and veins. In biometric authentication, when identity verification is required, biometric feature data for verification acquired by a biometric sensor is compared (matched) with pre-registered biometric feature data, and identity verification is performed by determining whether the similarity is equal to or exceeds an identity verification threshold. Biometric authentication is used in various fields such as bank ATMs and access control, and has recently begun to be used in cashless payments at supermarkets, convenience stores, and other locations.

[0010] These biometric authentication methods are "point" authentication performed at specific authentication spots, such as in front of an authentication machine. However, with "point" authentication, the authentication state is interrupted when the user leaves the authentication spot, and if the user wishes to receive a service again or at a location where authentication is required multiple times, authentication must be performed each time. Therefore, there is a demand for continuous authentication technology that eliminates the need for repeated authentication and allows the user to continue receiving services with a single authentication.

[0011] Here, we will provide an overview of continuous authentication technology. Continuous authentication mainly involves the following authentication steps:

[0012] The first is authentication at check-in. At the gate, the user performs highly accurate authentication using palm vein authentication or fingerprint authentication, and when authentication is successful, a camera captures a photo of the user's appearance, which is then linked to the user's ID and the characteristic information acquired by the camera and registered.

[0013] Next is line authentication. The authentication status is maintained by tracking the same person across multiple cameras in the authentication space and performing authentication processing. This makes it possible to provide personalized services anywhere within the authentication space. In addition, since there is no need to perform authentication operations each time, services can be provided at the optimal timing.

[0014] For example, as illustrated in FIG. 1, multiple cameras 110 are installed at different positions in an authentication space. The cameras 110 acquire images in time series at predetermined time intervals. A person can be detected from the acquired multiple images. Furthermore, feature information can be extracted from the detected person, and the person can be tracked within the same image or by identifying the person across multiple images (Re-ID). Furthermore, by comparing the feature information of the tracked person with registered feature information, the authentication status of the tracked person can be maintained.

[0015] For example, suppose that person A moves within the authentication space. In this case, the authentication state of the person being tracked by the first camera 110 can be continued. Next, as person A moves, person A appears in the images acquired by the first camera 110 and the second camera 110. If it is determined that person A being tracked by the first camera 110 and continuing to be authenticated and person A being tracked by the second camera 110 and continuing to be authenticated are the same person, the tracking of person A can be integrated.

[0016] 2 and 3 are diagrams illustrating the details of Re-ID. Re-ID is executed repeatedly at predetermined time intervals. As illustrated in FIG. 2, multiple images are acquired in chronological order from each camera 110. For example, multiple images are acquired in chronological order within a specified time range from the start time of Re-ID. For example, images are acquired within the time range from the start time of Re-ID to the start time of the next Re-ID.

[0017] The acquired images are stored in the queue 120. Images stored during the previous Re-ID are discarded.

[0018] Next, as illustrated in FIG. 3 , image processing is performed on each of the images stored in the queue 120 that were acquired from each camera 110 at the same time. That is, the image processing is performed synchronously. Specifically, a person is detected, feature information is extracted from the detected person, and tracking is performed. Tracking is a tracking process that calculates the trajectory of the same person. Known tracking techniques may be used. If the trajectory can be calculated, each person can be continuously tracked even if multiple people are included in one image. For example, it is possible to track which person in the image has been identified as which person.

[0019] Next, it is determined which ID each detected person corresponds to. In this case, it is determined that the person has the ID with the highest similarity between the feature information extracted from the image and the registered feature information. In this way, it is possible to continue authenticating the person being tracked.

[0020] However, the frames per second (fps) of each camera 110 may differ. For example, there may be a mixture of high-fps cameras and low-fps cameras. In such cases, the arrival times of images differ, making it impossible to determine whether or not each person whose image was captured by multiple cameras and whose trajectories were calculated is the same person at the same time.

[0021] Therefore, for example, by saving images acquired from the camera 110 in chronological order and extracting and processing the images at the lowest fps, it becomes possible to determine whether the people photographed by multiple cameras and whose trajectories have been calculated are the same person. However, if image processing is performed at a low fps, the person's trajectory will be calculated at a low fps. In this case, the time interval for trajectory calculation becomes longer, resulting in a decrease in person tracking accuracy. When multiple people appear in a single image, tracking accuracy decreases. In particular, when multiple people appear in a single image and their movement directions differ, tracking accuracy decreases. As a result, even when Re-ID is performed, there is a risk that the accuracy of person identification will decrease.

[0022] Therefore, in the following embodiments, a determination method, a determination program, and an information processing device that can improve the accuracy of person determination will be described.

[0023] An overview of Example 1 will be described. In this example, attention is paid to the fact that tracking processing for calculating a person's trajectory can be performed using images acquired from a single camera. In other words, attention is paid to the fact that tracking processing is possible without using images acquired from multiple cameras.

[0024] As an example, consider two cameras. A first camera captures images every first time (first fps) and acquires multiple first images. A second camera captures images every second time (second fps). The image capturing range of the first camera and the image capturing range of the second camera overlap at least partially. This allows the first camera and the second camera to capture images of the same person at the same time in the overlapping areas.

[0025] As shown in FIG. 4, images are acquired from the first camera and the second camera according to the fps of each camera. Because the fps differs between the first camera and the second camera, the number of images acquired from each camera per unit time differs. Next, a third time is determined based on the time required to acquire each of the multiple first images from the first camera and the time required to acquire each of the multiple second images from the second camera. The third time is the interval between executions of the second image processing described below.

[0026] Next, as illustrated in Fig. 5, a first image processing is performed on each image acquired by each camera. In the first image processing, a person is detected, feature information is extracted from the detected person, and a tracking process is performed. This allows the trajectories of the person included in the multiple first images to be calculated, and the trajectories of the person included in the multiple second images to be calculated. The images after the first image processing are stored in a queue 50.

[0027] Next, from the queue 50, an image corresponding to the determined third time is retrieved from the plurality of first images. Also, from the queue 50, an image corresponding to the determined third time is retrieved from the plurality of second images. Next, a second image process is performed on the images retrieved from the queue 50. In the second image process, it is determined which person's ID each detected person matches. That is, in the second image process, it is determined that the detected person is the person with the ID that has the highest similarity between the feature information extracted from that person and the registered feature information. This makes it possible to determine whether the people whose trajectories are calculated from the plurality of first images and the plurality of second images are the same person.

[0028] By doing so, the tracking process can be performed regardless of the third time, and therefore high person tracking accuracy can be maintained. Since Re-ID is performed while high tracking accuracy is maintained, the person determination accuracy can be improved.

[0029] The details of this embodiment will be described below. Fig. 6(a) is a block diagram illustrating an example of the overall configuration of a biometric authentication system 300 according to the first embodiment. As illustrated in Fig. 6(a), the biometric authentication system 300 includes an information processing device 100, a plurality of cameras 200, and the like. These devices are connected wirelessly or by wire.

[0030] The multiple cameras 200 are cameras provided in the authentication space, and are installed at positions where it is easy to acquire characteristic information of a person. For example, they are installed on the ceiling or the like so that it is easy to track a person. At least one of the multiple cameras 200 may have a different fps from the other cameras.

[0031] Fig. 6(b) is a functional block diagram of the information processing device 100. As illustrated in Fig. 6(b), the information processing device 100 functions as an image acquisition unit 10, an execution interval determination unit 20, a first image processing unit 30, an image storage unit 40, a queue 50, an image extraction unit 60, a second image processing unit 70, and the like.

[0032] Next, each process executed by the information processing device 100 will be described.

[0033] 7 is a flowchart illustrating the Re-ID process executed by the information processing device 100. The Re-ID process is executed repeatedly at predetermined time intervals. The authentication process will be described below with reference to the flowchart in FIG.

[0034] First, the image acquisition unit 10 acquires multiple images in chronological order from each camera 200 (step S1). For example, the image acquisition unit 10 acquires multiple images in chronological order from each camera 200 within a specified time range from the start time of Re-ID. For example, images are acquired within a time range from the start time of Re-ID to the start time of the next Re-ID. The image acquisition unit 10 may extract images from the video received from each camera 200 in accordance with the fps. Alternatively, each camera 200 may extract an image from the video and transmit it to the image acquisition unit 10 as an image.

[0035] Next, the execution interval determination unit 20 determines the execution interval of the second image processing according to the time interval at which images are acquired from each camera 200 (step S2). For example, the execution interval determination unit 20 determines the smallest value among the fps of each camera 200 as the execution interval. The fps may be set for the video, but it may also be calculated from the image generation interval, the image arrival interval, or the like. Alternatively, it may be determined in advance based on the performance of the cameras 200 and the network in the authentication space. As an example, in the example of FIG. 8, if the fps of each camera 200 are 5 fps, 15 fps, and 30 fps, the execution interval is determined to be the smallest value of 5 fps.

[0036] Next, the first image processing unit 30 performs the first image processing on all images acquired in step S1 (step S3). By performing step S3, it is possible to calculate and track the trajectory of the same person. If necessary, the processing result of step S3 may be stored in the image storage unit 40.

[0037] Next, the queue 50 stores the image after the first image processing in step S3 (step S4). For example, as illustrated in FIG. 9, if a next image is acquired from a high-fps camera before the second image processing is performed, the image stored during the previous Re-ID may be discarded by overwriting it with the new image, or may be stored as past data. For example, for camera A with a 5-fps frame rate, an image stored at 12:00:00.025 on December 19, 2022 is stored. For camera B with a 15-fps frame rate, images stored at 12:00:00.021 on December 19, 2022 and 12:00:00.028 on December 19, 2022 are stored. For camera C, images taken at 12:00:00.020 on 12 / 19 / 2022, 12:00:00.023 on 12 / 19 / 2022, 12:00:00.027 on 12 / 19 / 2022, and 12:00:00.030 on 12 / 19 / 2022 are stored.

[0038] Next, the image retrieval unit 60 retrieves images from the queue 50 at the execution interval of the second image processing determined in step S2 (step S5). For example, the image closest to the time specified by the execution interval of the second image processing may be retrieved, or the most recent image after the time specified by the execution interval of the second image processing may be retrieved. For example, images are retrieved as shown in FIG. 10 . Images crossed out indicate that they have not been retrieved. In the example of FIG. 10 , an image retrieved from camera A at 12:00:00.025 on December 19, 2022 is retrieved. An image retrieved from camera B at 12:00:00.028 on December 19, 2022 is retrieved. An image retrieved from camera C at 12:00:00.030 on December 19, 2022 is retrieved. If there are no images to be retrieved, it is assumed that there are no images from the target camera. For example, there may be cases where there are no images to be extracted for a camera 200 with a lower fps than the execution interval. There may also be cases where there are no images to be extracted when a delay occurs due to the influence of the network.

[0039] Next, the second image processing unit 70 performs second image processing on the image extracted in step S6 (step S6), as illustrated in FIG. 7 . This makes it possible to determine whether the people photographed by the multiple cameras 200 and whose trajectories have been calculated are the same person. Note that the second image processing may be performed using the results of the first image processing stored in the image storage unit 40. For example, by using the results of the first image processing stored in the image storage unit 40, it is possible to utilize the tracking results of the person identified by the previous Re-ID.

[0040] In the above example, all images stored in the queue 50 are processed as the first image processing, but the first image processing does not necessarily have to be performed on all images. However, it is preferable to set the execution interval for the first image processing to be shorter than the execution interval for the second image processing. For example, when images are acquired at 30 fps, the execution interval for the second image processing may be set to 5 fps, and the execution interval for the first image processing may be set to 10 fps, which is shorter than the execution interval for the second image processing.

[0041] Furthermore, the interval at which the first image processing is performed may be changed for each camera. For example, in a location near an entrance or exit where there is a lot of pedestrian movement, it is difficult to maintain high person tracking accuracy. Therefore, for cameras installed in such locations, the interval at which the first image processing is performed may be set to a high frame rate, such as 30 frames per second.

[0042] Alternatively, the interval at which the first image processing is executed may be changed depending on the state of the acquired image. For example, person detection may be executed on the acquired image, and the more people detected, the shorter the interval at which the first image processing is executed may be.

[0043] In the above example, the latest image stored in each camera is extracted and processed as the second image processing, but an image acquired recently may be selected, extracted, and processed. In this case, the previous image may be retained from the time it is extracted by the image acquisition unit 10 until it is processed.

[0044] Furthermore, the execution interval determination unit 20 may determine not only the execution interval of the second image processing but also the execution time when the acquisition timing is approaching.

[0045] FIG. 11 is a block diagram illustrating an example of the hardware configuration of an information processing device 100. As illustrated in FIG. 11, the information processing device 100 includes a CPU 101, a RAM 102, a storage device 103, and the like. The CPU (Central Processing Unit) 101 is a central processing unit. The RAM (Random Access Memory) 102 is a volatile memory that temporarily stores programs executed by the CPU 101, data processed by the CPU 101, and the like. The storage device 103 is a non-volatile storage device. For example, the storage device 103 may be a read-only memory (ROM), a solid-state drive (SSD) such as a flash memory, or a hard disk driven by a hard disk drive. The CPU 101 executes a determination program stored in the storage device 103 to realize the functions of each unit of the information processing device 100. Note that the functions of each unit of the information processing device 100 may be configured using dedicated circuits or the like.

[0046] In each of the above examples, the first image processing unit 30 is an example of a calculation unit that, when acquiring a plurality of first images in a time series captured by a first camera every first hour and a plurality of second images in a time series captured by a second camera every second hour, calculates a trajectory of a person appearing in each of the plurality of first images and a trajectory of a person appearing in each of the plurality of second images. The execution interval determination unit 20 is an example of a determination unit that determines a third time based on a time at which each of the plurality of first images is acquired from the first camera and a time at which each of the plurality of second images is acquired from the second camera. The second image processing unit 70 is an example of a determination unit that determines whether a person whose trajectory is calculated from both an image among the plurality of first images corresponding to the determined third time and an image among the plurality of second images corresponding to the determined third time are the same person.

[0047] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims.

[0048] REFERENCE SIGNS LIST 10 Image acquisition unit 20 Execution interval determination unit 30 First image processing unit 40 Image storage unit 50 Queue 60 Image extraction unit 70 Second image processing unit 100 Information processing device 200 Camera 300 Biometric authentication system

Claims

1. A method for determining whether or not a person whose trajectory is calculated from an image among the plurality of first images corresponding to the determined third time and an image among the plurality of second images corresponding to the determined third time is the same person, wherein the method is performed by a computer to execute a process in which, when a first camera acquires a plurality of first images in a time series taken every first hour and a plurality of second images in a time series taken every second hour, a first camera acquires a plurality of first images in a time series taken every second hour, a trajectory of a person appearing in each of the plurality of first images and a trajectory of a person appearing in each of the plurality of second images, a third time is determined according to the time at which each of the plurality of first images is acquired from the first camera and the time at which each of the plurality of second images is acquired from the second camera, and a third time is determined according to the time at which each of the images among the plurality of first images corresponding to the determined third time and an image among the plurality of second images corresponding to the determined third time are the same person.

2. The method according to claim 1, wherein the third time is determined to be the longer of the first time and the second time.

3. The method of claim 1, characterized in that the third time is determined to be the longer of the time interval between acquiring each of the multiple first images from the first camera and the time interval between acquiring each of the multiple second images from the second camera.

4. A method for determining whether or not a trajectory exists, comprising the steps of: calculating the trajectory from the plurality of first images and the plurality of second images; extracting images from at least one of the plurality of first images and the plurality of second images so that the images are spaced apart by a time interval shorter than the third time; and calculating the trajectory from the images.

5. A method for determining whether or not a trajectory exists in a subject, comprising the steps of: calculating a trajectory from the plurality of first images and the plurality of second images; extracting images from at least one of the plurality of first images and the plurality of second images so that the more people contained in the image, the shorter the time interval at which the image is extracted; and calculating a trajectory from the extracted images.

6. A determination program that causes a computer to execute a process that, when a first camera acquires a plurality of first images in a time series taken every first hour and a plurality of second images in a time series taken every second hour by a second camera, calculates a trajectory of a person appearing in each of the plurality of first images and a trajectory of a person appearing in each of the plurality of second images; determines a third time according to the time at which each of the plurality of first images is acquired from the first camera and the time at which each of the plurality of second images is acquired from the second camera; and determines whether the persons whose trajectories are calculated from both an image among the plurality of first images corresponding to the determined third time and an image among the plurality of second images corresponding to the determined third time are the same person.

7. The judgment program according to claim 6, wherein the third time period is determined to be the longer of the first time period and the second time period.

8. The determination program according to claim 6, characterized in that the third time is determined to be the longer of the time interval between acquiring each of the plurality of first images from the first camera and the time interval between acquiring each of the plurality of second images from the second camera.

9. A determination program as described in any one of claims 6 to 8, characterized in that when calculating the trajectory from the multiple first images and the multiple second images, for at least one of the multiple first images and the multiple second images, images are extracted so that the time interval is shorter than the third time, and a trajectory is calculated.

10. A determination program as described in any one of claims 6 to 8, characterized in that when calculating the trajectory from the multiple first images and the multiple second images, for at least one of the multiple first images and the multiple second images, images are extracted so that the more people contained in the image, the shorter the time interval is, and the trajectory is calculated.

11. An information processing device comprising: a calculation unit that, when a first camera acquires a plurality of first images in a time series taken every first hour and a plurality of second images in a time series taken every second hour by a second camera, calculates a trajectory of a person appearing in each of the plurality of first images and a trajectory of a person appearing in each of the plurality of second images; a determination unit that determines a third time based on the time at which each of the plurality of first images is acquired from the first camera and the time at which each of the plurality of second images is acquired from the second camera; and a determination unit that determines whether the persons whose trajectories are calculated from both an image among the plurality of first images corresponding to the determined third time and an image among the plurality of second images corresponding to the determined third time are the same person.

12. The information processing apparatus according to claim 11, wherein the determination unit determines the longer of the first time and the second time as the third time.

13. The information processing device described in claim 11, characterized in that the determination unit determines the third time to be the longer of the time interval between acquiring each of the multiple first images from the first camera and the time interval between acquiring each of the multiple second images from the second camera.

14. An information processing device as described in any one of claims 11 to 13, characterized in that when calculating the trajectory from the multiple first images and the multiple second images, the calculation unit calculates the trajectory by extracting images from at least one of the multiple first images and the multiple second images so that the time interval is shorter than the third time.

15. An information processing device as described in any one of claims 11 to 13, characterized in that when calculating the trajectory from the multiple first images and the multiple second images, the calculation unit calculates the trajectory by extracting images for at least one of the multiple first images and the multiple second images such that the time interval is shorter the more people contained in the image.

Citation Information

Patent Citations

  • Moving object tracking device

    JP2013206140A

  • Tracking system, tracking method and tracking program

    JP2017028655A

  • Method and apparatus for people flow analysis using similar-image search

    JP2022062675A

  • Method and apparatus for stereo, multi-camera tracking and RF and video track fusion

    US20060028552A1

  • Time synchronized cameras for multi-camera event videos

    US20200396392A1