Determination method, determination program, and information processing device
By calculating the overlap of person images and dynamically adjusting authentication thresholds, the system improves the accuracy of biometric authentication in ReID systems, addressing issues of occlusion and lighting variations.
Patent Information
- Application Number
- PCT/JP2023/017686
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-14
AI Technical Summary
Biometric authentication systems using person re-identification (ReID) face challenges in maintaining accuracy due to occlusion and variations in lighting or clothing, leading to incorrect identification of registered individuals as new candidates.
The system calculates the degree of overlap between detected person images and determines whether partial images should be subjected to authentication, suppressing authentication when overlap is excessive to prevent incorrect identification, and adjusts threshold values dynamically to improve accuracy.
This approach enhances the accuracy of personal authentication by preventing incorrect identification due to occlusion and varying conditions, ensuring reliable user verification.
Smart Images

Figure JP2023017686_14112024_PF_FP_ABST
Abstract
Description
Determination method, determination program, and information processing device
[0001] The present invention relates to a determination method, a determination program, and an information processing device.
[0002] In services such as payment using biometric authentication, there is a demand for smooth services that use a video camera to constantly authenticate the individual, eliminating the need for user operation for authentication. A technology known as person re-identification is used for such services. In the following description, this person re-identification technology will be referred to as "ReID."
[0003] ReID calculates the similarity between a person image (query person image) acquired by detecting and tracking a person's image from video data obtained by a video camera and a registered image for authentication of each person candidate, and uses this similarity as the identity accuracy. ReID constantly performs identity authentication using this identity accuracy.
[0004] Several techniques related to such ReID are known (see, for example, Patent Documents 1 to 4).
[0005] For example, an image recognition device is known that is superior in real-time performance and capable of robust personal identification. This device acquires a relative orientation relationship between first and second input area images obtained by capturing images of a person from different orientations. Next, the device compares features of a first registration area image included in a group of registration area images obtained by capturing images of the person or other people from at least three orientations with features of the first input area image. The device also compares features of a second registration area image of the same person included in the group of registration area images with features of the second input area image to determine whether the person captured in the first and second input area images is the same person. The device then selects the first and second registration area images so that the relative orientation relationship between the first and second registration area images approaches the relative orientation relationship between the first and second input area images.
[0006] JP 2016-1447 A JP 2022-18808 A U.S. Patent Application Publication No. 2020 / 0349348 U.S. Patent Application Publication No. 2021 / 0374973
[0007] In ReID, if there is an occlusion in the query person image or if the lighting or clothing of the person is different between the query person image and the candidate person image, the similarity used as the identity probability will be low, which may result in treating a person who should already be registered as a candidate person as a new candidate person.
[0008] In one aspect, the present invention aims to improve the accuracy of determining which of the candidates a person in a query person image is.
[0009] In one proposal, a computer executes the following processing. When multiple people are detected in an image obtained by photographing a specified space, this processing calculates the degree of overlap between a first person among the multiple people and other people. This processing also determines whether or not a partial image included in the image, which indicates the first person, is to be subject to person determination, based on the calculated degree of overlap. This processing also determines which of the candidates the first person is, using the partial image, if it is determined that the first person is to be subject to person determination. This processing also suppresses the determination of which of the candidates the first person is, which is performed using the partial image, if it is determined that the first person is not to be subject to person determination.
[0010] According to one aspect, it is possible to easily guide a subject and a terminal to a desired positional relationship.
[0011] 1 is a diagram illustrating an overview of control based on overlap of person images. FIG. 1 is a diagram illustrating an example of the functional configuration of an authentication device. FIG. 2 is a diagram illustrating the functional configuration of a first example of an authentication system. FIG. 3 is a diagram illustrating a first example of a method for acquiring information about a person located in a predetermined space. FIG. 4 is a diagram illustrating a second example of a method for acquiring information about a person located in a predetermined space. FIG. 5 is a flowchart illustrating the processing content of a first example of authentication control processing. FIG. 6 is a diagram illustrating the functional configuration of a second example of an authentication system. FIG. 7 is a diagram illustrating a change in a threshold used to determine whether the degree of overlap is large or small. FIG. 8 is a diagram illustrating state transitions between execution and suppression of personal authentication due to a change in the threshold. FIG. 9 is a diagram illustrating the functional configuration of a third example of an authentication system. A flowchart illustrating the processing content of an example of tracking processing. FIG. 10 is a diagram illustrating the functional configuration of a fourth example of an authentication system. FIG. 11 is a diagram illustrating an example of installation of multiple imaging units. FIG. 12 is a diagram illustrating the operation of an authentication integration unit. A flowchart illustrating the processing content of an example of authentication integration processing. FIG. 13 is a diagram illustrating an example of the hardware configuration of an information processing device.
[0012] Hereinafter, an embodiment will be described with reference to the drawings.
[0013] In the embodiment described below, in an authentication system that implements ReID, if there is excessive overlap between a person image extracted from a video and another partial image, the person image is not treated as a query image. First, this control will be described using FIG. 1 .
[0014] In this embodiment, a rectangular frame (bounding box) surrounding an image of a person detected by a detection process on past time-series images is tracked to estimate the position of the frame in image 10. Then, the person detected by the detection process from the position on image 10 closest to this estimated position is associated with the person whose image is surrounded by the frame detected from the past time-series images as the same person.
[0015] In Fig. 1, image 10 is one frame of a video sequence (time-series images) obtained by capturing an image of a specific space to be authenticated. The example in Fig. 1 shows a situation in which person images 11, 12, 13, and 14 surrounded by rectangular frames (bounding boxes) are detected by combining a detection process for detecting a person's image from image 10 with a tracking process for the frame in a past video sequence.
[0016] 1, the person candidate storage unit 20 stores in advance image feature amounts for people who may appear in the image 10. In the example of FIG. 1, the person candidate storage unit 20 stores image feature amounts for each of person candidates 21, 22, 23, and 24. The person candidates 21, 22, 23, and 24 correspond to the people shown in person images 11, 12, 13, and 14, respectively.
[0017] First, in this embodiment, a case where ReID is performed using the person image 13 in the example of FIG. 1 as a query image will be described.
[0018] In this case, first, the similarity between the image feature amount obtained from the person image 13 and the image feature amount of each of the person candidates 21, 22, 23, and 24 stored in the person candidate storage unit 20 is calculated. The similarity calculated for each of the person candidates 21, 22, 23, and 24 becomes the person certainty, i.e., the certainty that the person shown in the person image 13 is the person himself.
[0019] 1, the person shown in person image 13 corresponds to person candidate 23, and therefore the identity accuracy for person candidate 23 indicates the highest value among the identity accuracy for each of person candidates 21, 22, 23, and 24. Therefore, the result of identity authentication is that the person shown in person image 13 is person candidate 23.
[0020] Next, a case where ReID is performed using the person image 14 in the example of FIG. 1 as a query image in this embodiment will be described.
[0021] In this case, first, the similarity between the image feature amount obtained from the person image 13 and the image feature amount of each of the person candidates 21, 22, 23, and 24 stored in the person candidate storage unit 20 is calculated. The similarity calculated for each of the person candidates 21, 22, 23, and 24 becomes the identity probability that indicates the likelihood that the person shown in the person image 14 is the person himself.
[0022] 1 , however, occlusion occurs between the person shown in person image 14 and the person shown in person image 11, and person image 14 largely overlaps with person image 11. With such person image 14, the identity accuracy, which is the degree of similarity of image features between the person shown in person image 14 and the person candidate 24 corresponding to the person shown in person image 14, becomes a small value. As a result of identity authentication, one of the other person candidates 21, 22, and 23 may be mistaken for the person shown in person image 14.
[0023] Therefore, in this embodiment, when there is a large overlap with the person image 14, the person image 14 is not treated as a query image, and the execution of personal authentication to determine whether the person in the person image 14 is one of the candidate persons 21, 22, 23, and 24 is suppressed. As a result, this improves the accuracy of personal authentication.
[0024] Next, a description will be given of Fig. 2. Fig. 2 shows an example of the functional configuration of the authentication device 30. This authentication device 30 is a device that controls personal authentication based on the overlap of person images described above.
[0025] The authentication device 30 includes an acquisition unit 31 , a specification unit 32 , a calculation unit 33 , a first determination unit 34 , an extraction unit 35 , a second determination unit 36 , and a suppression unit 37 .
[0026] The acquisition unit 31 acquires information about a person present in a predetermined space.
[0027] The identifying unit 32 identifies a person present in the predetermined space as a candidate for the person indicated by the query person image, based on the person information acquired by the acquiring unit 31 .
[0028] When the calculation unit 33 detects the reflection of multiple people in image 10, which is one frame of a moving image obtained by photographing a specified space where information on people entering and exiting is acquired by the acquisition unit 31, it calculates the degree of overlap between a first person among the multiple people and the other people.
[0029] The first determination unit 34 determines, based on the degree of overlap calculated by the calculation unit 33, whether or not a partial image included in the image 10 and showing the first person is to be subject to person determination.
[0030] In other words, the calculation unit 33 calculates the degree of overlap between the partial image showing the first person and the partial image showing another person as the overlap degree, and the first judgment unit 34 judges whether or not to treat the partial image showing the first person as a query person image based on this overlap degree.
[0031] The extraction of the partial image representing the first person from image 10 is performed by extraction unit 35. Extraction unit 35 performs this extraction based on a combination of the result of detecting the image of the person from image 10 and the result of estimating the position of the partial image in image 10 based on the position of the partial image in each of previous images 10 obtained by photographing earlier than image 10.
[0032] In this embodiment, the position of a frame (bounding box) surrounding an image of a person detected by detection processing on past time-series images is estimated using a motion vector determined from the position of the frame in image 10. Then, the image of the person detected by detection processing on image 10 is extracted from the position closest to this estimated position from image 10. Furthermore, the person detected by the detection processing is associated with the person surrounded by the frame detected from past time-series images as the same person, and the detected position of the person is set as the position of the frame on image 10.
[0033] Note that the process of estimating the position of an element included in image 10 based on a motion vector determined from the position of that element in each of the time-series images older than image 10 is a common technique. Therefore, by employing this technique, it is easily possible to perform tracking that estimates the position of a partial image on image 10 based on a motion vector determined from the position of that partial image in each of the time-series images older than image 10. In addition, the process of detecting a human figure from image 10 is also a common technique. In this embodiment, these techniques are used to realize the function of extraction unit 35.
[0034] The second determination unit 36 performs personal authentication to determine which of the candidates the first person is, using the partial image representing the first person as a query person image.
[0035] The suppression unit 37 suppresses the above-described personal authentication performed by the second determination unit 36 to determine which of the candidates the first person is, when the first determination unit 34 determines that the partial image showing the first person is not to be subject to person determination. On the other hand, when the first determination unit 34 determines that the partial image showing the first person is to be subject to person determination, the suppression unit 37 does not perform such suppression. Therefore, when the first determination unit 34 determines that the partial image showing the first person is to be subject to person determination, the second determination unit 36 performs personal authentication to determine which of the candidates the first person is.
[0036] The authentication device 30 has the above configuration.
[0037] The following describes an authentication system 100 that provides the same functions as the components of the authentication device 30.
[0038] First, a description will be given of a first example of the authentication system 100. FIG.
[0039] The authentication system 100 in Figure 3 includes an in-area person information acquisition unit 101, a person candidate storage unit 102, a photographing unit 103, a person image extraction unit 104, an identity accuracy estimation unit 105, an overlap degree calculation unit 106, an authentication control unit 107, and an authentication result output unit 108.
[0040] The area-specific person information acquisition unit 101 acquires information about people located in a specified space, records and stores the acquired person information in the person candidate storage unit 102, and provides functions equivalent to the acquisition unit 31 and identification unit 32 in the authentication device 30.
[0041] The person candidate storage unit 102 stores information about people acquired by the in-area person information acquisition unit 101. The people whose information is stored in the person candidate storage unit 102 are candidates for the person indicated by the query person image.
[0042] Here, a method for acquiring person information by the in-area person information acquisition unit 101 will be described.
[0043] FIG. 4A shows a first example of a method for acquiring information about people present in a predetermined space.
[0044] In this first example, an entrance management device 112 and an exit management device 113 serving as the in-area person information acquisition unit 101 are installed at the entrance and exit of a venue 111 including a target space 110, which is a predetermined space.
[0045] The entrance management device 112 is equipped with a camera that photographs entrants to the venue 111. The entrance management device 112 authenticates entrants to the venue 111 using biometric information of the entrant, such as face, palm veins, or fingerprints, or using an ID card or smartphone owned by the entrant. Note that ID is an abbreviation for identification. The entrance management device 112 assigns identification information to entrants who are successfully authenticated, and creates information that associates the identification information with image features obtained from a portrait image of the entrant photographed by the camera, and records and saves this information in the candidate storage unit 102 as candidate information.
[0046] The exit management device 113 performs the same authentication on a person exiting the venue 111 as the entrance management device 112, and associates the person with an entrance. The exit management device 113 then deletes information about the entrance candidate associated with the person exiting from the candidate person storage unit 102.
[0047] FIG. 4B shows a second example of a method for acquiring information about people present in a predetermined space.
[0048] 4B shows a screen 121 obtained by capturing a space including a target space 120, which is a predetermined space, using a camera provided in the in-area person information acquisition unit 101. The in-area person information acquisition unit 101 detects the entry and exit of people into the target space 120 in the edge region of the screen 121, which is the outer periphery of the target space 120, and authenticates the people who have entered using, for example, facial recognition or gait recognition. The in-area person information acquisition unit 101 then assigns identification information to people who have been successfully authenticated, creates information that associates the identification information with image features obtained from a captured image of the person, and records and saves this information as candidate information in the candidate storage unit 102. The in-area person information acquisition unit 101 also deletes information about candidates who have exited the target space 120 from the candidate storage unit 102.
[0049] The image capturing unit 103 is a camera that captures an image of a predetermined space, such as the target space 110 in Fig. 4A or the target space 120 in Fig. 4B, to obtain time-series images that are moving images. The camera serving as the image capturing unit 103 may also be used as a camera included in the in-area person information acquisition unit 101.
[0050] The person image extraction unit 104 extracts a person image from each of the time-series images acquired by the photographing unit 103, and provides a function equivalent to the extraction unit 35 in the authentication device 30. The person image extraction unit 104 extracts a person image from image 10, which is one frame of the time-series images, based on a combination of the result of detecting a person image from image 10 and the result of estimating the position of the person image in image 10. Note that the estimation of the position of the person image in image 10 is performed based on the position of the person image in each of the time-series images earlier than image 10.
[0051] Similar to the description of the extraction unit 35, the person image extraction unit 104 estimates the position of a frame (bounding box) in the image 10 surrounding an image of a person detected by a detection process on time-series images older than the image 10, using a motion vector determined from the position of the frame. The unit then extracts from the image 10 the image of the person detected by the detection process on the image 10 from the position closest to this estimated position. Furthermore, the person detected by the detection process is associated with the person surrounded by the frame detected from the previous time-series images, and the detected position of the person is set as the position of the frame on the image 10. In this embodiment, similar to the extraction unit 35, the function of the person image extraction unit 104 is realized by applying the general technology described above.
[0052] The identity accuracy estimation unit 105 estimates identity accuracy, which indicates the likelihood that each candidate whose image feature is stored in the candidate storage unit 102 is a person represented by the person image extracted by the person image extraction unit 104. More specifically, the identity accuracy estimation unit 105 calculates the similarity between the image feature of the person image and the image feature of each candidate, and sets the obtained similarity as the similarity.
[0053] The overlap degree calculation unit 106 calculates, as the overlap degree, the degree of overlap between the area within a frame surrounding a person extracted from the image 10 by the person image extraction unit 104 and the area within a frame surrounding another person extracted from the image 10. The overlap degree calculation unit 106 provides a function equivalent to that of the calculation unit 33 in the authentication device 30, and the overlap degree is an example of the aforementioned overlap degree.
[0054] The overlap degree calculation unit 106 may calculate, as the degree of overlap, for example, IoU, which is the ratio of the common part to the union of the two regions. IoU is an abbreviation for Intersection over Union. Alternatively, the overlap degree calculation unit 106 may acquire depth information of the position of each person in a predetermined space, and use the depth information to calculate IoU in three-dimensional space to use as the degree of overlap. Alternatively, the overlap degree calculation unit 106 may determine whether the two regions overlap, and calculate the degree of overlap for the two regions only if there is overlap.
[0055] The authentication control unit 107 uses the person image extracted by the person image extraction unit 104 to determine which of the candidates whose image features are stored in the person candidate storage unit 102 corresponds to the person shown in the person image (first person). More specifically, the authentication control unit 107 performs identity authentication by determining, by the identity accuracy estimation unit 105, that the candidate with the highest identity accuracy among the candidates is the first person. In other words, the authentication control unit 107 provides a function equivalent to the second determination unit 36 in the authentication device 30.
[0056] However, the authentication control unit 107 determines whether the overlap degree calculated by the overlap degree calculation unit 106 for the person image of the first person extracted by the person image extraction unit 104 is greater than a predetermined threshold. This threshold is set as the upper limit of the overlap degree when execution of personal authentication for the person image is permitted. In other words, the authentication control unit 107 also provides a function equivalent to the first determination unit 34 in the authentication device 30.
[0057] In the above-described determination, when it is determined that the calculated degree of overlap is greater than a predetermined threshold, the authentication control unit 107 suppresses determination of which of the candidates the person shown in the person image corresponds to. That is, in this case, the authentication control unit 107 does not perform the above-described personal authentication of the person shown in the person image extracted from the image 10. That is, the authentication control unit 107 also provides a function equivalent to the suppression unit 37 in the authentication device 30.
[0058] Here, a detailed description will be given of the authentication control process performed by the authentication control unit 107. Fig. 5 is a flowchart showing the processing contents of a first example of the authentication control process.
[0059] When the process of FIG. 5 starts, first, in S101, the person image extraction unit 104 extracts one person image from the image 10 and obtains the extracted person image.
[0060] Next, in S102, a process is performed to obtain the overlap degree calculated by the overlap degree calculation unit 106 for the person image obtained in the process of S101.
[0061] Next, in S103, a process is performed to determine whether the overlap degree of the person image acquired in S101, which was acquired in S102, is equal to or less than a threshold. In this determination process, if it is determined that the overlap degree is equal to or less than the threshold (if the determination result is YES), the process proceeds to S104. On the other hand, in this determination process, if it is determined that the overlap degree is greater than the threshold (if the determination result is NO), the process proceeds to S106. This determination process provides the function of the first determination unit 34 in the authentication device 30.
[0062] The processing of S104 and S105 is a processing of treating the person image acquired by the processing of S101 as a query person image, performing identity authentication on the person (first person) shown in the query person image, and outputting the authentication result.
[0063] In S104, a process is performed to identify the candidate with the highest identity probability as the person (first person) shown in the person image acquired by the process of S101, from among the candidates whose image features are stored in the candidate candidate storage unit 102. This identity probability uses a value estimated by the identity probability estimation unit 105. This process is a process of determining which of the candidates whose image features are stored in the candidate candidate storage unit 102 is the first person, and is a process that provides the function of the second determination unit 36 in the authentication device 30.
[0064] In S105, the candidate identified in the process of S104 is treated as the first person, and the authentication result output unit 108 is caused to output identification information about the candidate. After this process, the process proceeds to S107.
[0065] On the other hand, in S106, the personal authentication process (the processes of S104 and S105) for the first person shown in the person image acquired in the process of S101 is not executed, and a process is performed to prohibit the output of the result of personal authentication by the authentication result output unit 108. This process of S106 is a process to prevent the person image acquired in the process of S101 from being treated as a query person image, and is a process to provide the function of the suppression unit 37 in the authentication device 30.
[0066] In S107, a process is performed to determine whether all of the person images extracted from the image 10 by the person image extraction unit 104 have been acquired by the process of S101. If it is determined in this determination process that all of the person images have been acquired (if the determination result is YES), the authentication control process is terminated and the authentication control process for the next image 10 in the time-series images is awaited. On the other hand, if it is determined in this determination process that an unacquired person image remains (if the determination result is NO), the process returns to S101, and the processes from S101 onwards are performed again on the unacquired person images.
[0067] The above-described processing is a first example of the authentication control processing.
[0068] 3 includes the above-described components, and therefore when a person's image overlaps with another person's image, the person's image is not treated as a query image, and the person in the person's image is not authenticated. As a result, the accuracy of the person authentication is improved.
[0069] Next, a second example of the authentication system 100 will be described.
[0070] Fig. 6 shows the functional configuration of a second example of the authentication system 100. In Fig. 6, components that provide the same functions as those in the first example of Fig. 3 are denoted by the same reference numerals, and detailed description of these components will be omitted.
[0071] The second example in FIG. 6 has a configuration in which a threshold control unit 201 is added to the configuration of the first example shown in FIG.
[0072] As already explained with reference to Fig. 1, when there is a large overlap between a person's image and another person's image, the accuracy of personal authentication decreases. In addition, if this situation continues, the reliability of past information decreases, and the accuracy of person image extraction by the person image extraction unit 104 also decreases.
[0073] Therefore, the threshold control unit 201 performs control to change the threshold for the determination of the size of the overlap between a person image extracted from one frame of the time-series images and another person image, depending on the result of the determination. More specifically, when it is determined that the person image should not be treated as a query image, the threshold control unit 201 changes to a smaller value the reference threshold for the size determination of the person image extracted from the next time-series image following the time-series image from which the person image was extracted.
[0074] This change in threshold value will be described with reference to FIG.
[0075] The lower waveform in Fig. 7 shows an example of the change over time in the degree of overlap for a certain person image. At the start of this waveform, the reference threshold for determining whether the degree of overlap is large or small in the determination process at S103 in the flowchart shown in Fig. 5 is set to "Threshold 1."
[0076] Thereafter, when the degree of overlap increases and reaches "threshold 1," the control signal changes from "authentication ON" to "authentication OFF," as shown in the upper waveform. The control signal represents a signal that controls switching between performing and suppressing personal authentication. Note that "authentication ON" represents a control signal corresponding to a state in which personal authentication is performed, and "authentication OFF" represents a control signal corresponding to a state in which personal authentication is suppressed.
[0077] When the degree of overlap increases and reaches "Threshold 1," the authentication control unit 107 changes the control signal from "Authentication ON" to "Authentication OFF." In the flowchart shown in FIG. 5, the result of the determination process in S103 changes from YES to NO at this time, and the authentication control unit 107 executes the process in S106 to suppress the execution of personal authentication. Also, at this time, the threshold control unit 201 changes the threshold used as the reference for determining whether the degree of overlap is large or small in the determination process in S103 from "Threshold 1" to "Threshold 2," which is a value smaller than "Threshold 1."
[0078] Thereafter, for a period during which the degree of overlap is greater than "Threshold 2," the authentication control unit 107 continues to suppress the execution of personal authentication. However, when the degree of overlap eventually decreases and reaches "Threshold 2," the authentication control unit 107 changes the control signal from "Authentication OFF" to "Authentication ON." In the flowchart shown in FIG. 5, at this time, the result of the determination process in S103 changes from NO to YES, and the authentication control unit 107 executes the processes from S104 to S105 to perform personal authentication. Also, at this time, the threshold control unit 201 restores the threshold used as the reference for determining whether the degree of overlap is large or small in the determination process in S103 from "Threshold 2" to the previous "Threshold 1."
[0079] FIG. 8 shows the state transition between execution and suppression of personal authentication due to the change in the threshold value described above.
[0080] 8, the "authentication ON" state represents a state in which personal authentication is being performed by the authentication control unit 107, and the "authentication OFF" state represents a state in which the execution of personal authentication by the authentication control unit 107 is inhibited. As mentioned above, "threshold 2" is a smaller value than "threshold 1."
[0081] 8, in the "authentication ON" state, the "authentication ON" state continues as long as the overlap degree is equal to or less than "threshold 1" (arrow T1), but when the overlap degree becomes greater than "threshold 1", the state transitions to "authentication OFF" (arrow T2). In the "authentication OFF" state, the "authentication OFF" state continues as long as the overlap degree is greater than "threshold 2" (arrow T3), but when the overlap degree becomes greater than "threshold 2", the state transitions to "authentication ON" (arrow T4).
[0082] In this way, when it is determined that a person image showing the first person included in a first image, which is one of the time-series images, is not to be subject to person determination, the threshold control unit 201 changes the threshold used to determine whether or not to subject the person image to person determination to a value smaller than the previous value. Then, the authentication control unit 107 uses the changed threshold to determine whether or not to subject a partial image showing the first person included in a second image, which is a time-series image following the first image, to person determination. Thereafter, when it is determined that a partial image showing the first person included in the second image is to be subject to person determination, the threshold control unit 201 changes the threshold used to determine whether or not to subject the person image to person determination back to the previous value. Then, the authentication control unit 107 uses the changed threshold to determine whether or not to subject a partial image showing the first person included in a third image, which is a time-series image following the second image, to person determination.
[0083] In this way, once a decision has been made not to treat an extracted person image as a query image, the criteria for determining whether or not to treat a person image as a query image are increased, and the conditions for performing personal authentication are tightened, thereby improving the accuracy of personal authentication.
[0084] Instead of switching between two discrete values, the threshold value may be changed continuously using a smoothing filter, for example. Also, the overlap degree may be determined to be larger or smaller than the threshold value using a value obtained by smoothing the change over time.
[0085] Next, a third example of the authentication system 100 will be described.
[0086] Fig. 9 shows the functional configuration of a third example of the authentication system 100. In Fig. 9, the same components as those in the first example of Fig. 3 are denoted by the same reference numerals, and detailed description of these components will be omitted.
[0087] The third example in FIG. 9 has a configuration in which a tracking control unit 301 is added to the configuration of the first example shown in FIG.
[0088] As described above, the person image extraction unit 104 extracts a person image from the image 10. This extraction is performed based on a combination of the result of detecting a person image from the image 10 and the result of estimating the position of the person image in the image 10 based on the position of the person image in each of the time-series images older than the image 10. More specifically, the person image extraction unit 104 estimates the position of a frame (bounding box) surrounding an image of a person detected by a detection process on a time-series image older than the image 10, using a motion vector determined from the position of the frame in the image 10. Furthermore, the person detected by the detection process is associated with the person surrounded by the frame detected from the past time-series image, and the detected position of the person is set as the position of the frame on the image 10.
[0089] However, when there is a large overlap between an image of a person and another image of another person, the accuracy of the person detection by the detection process decreases. In this case, if the detected position of the detected person is set to the position of the frame in image 10, this will result in a decrease in the accuracy of the estimation process for the position of the frame in the next frame following image 10 in the time-series images.
[0090] Therefore, when there is a large overlap between a person image and another person image, the tracking control unit 301 sets the estimated position of the frame as the position of the frame on the image 10. In other words, in this case, the tracking control unit 301 does not compensate for the estimated position of the frame based on the position of the person image detected by the detection process, thereby suppressing the impact of a decrease in the estimation accuracy of the position of the frame in the next frame following the image 10.
[0091] Now, a description will be given of Fig. 10. Fig. 10 is a flowchart showing the processing contents of an example of tracking processing.
[0092] 10, the processes of S304 and S306 are characteristic processes in the third example of the authentication system 100. On the other hand, the other processes are also performed by the person image extraction unit 104 in the first and second examples described so far.
[0093] 10 starts, first, in S301, a process is performed to acquire the latest image 10 from the time-series images, which are moving images obtained by the imaging unit 103. Hereinafter, the image 10 acquired by the process of S101 will be referred to as the "current image."
[0094] In S302, a process is performed to obtain a frame (bounding box) surrounding a person image in the frame immediately before the current image obtained by the process of S301, from among the time-series images obtained by the imaging unit 103.
[0095] In S303, the position of the frame in the current image acquired in the process of S302 is estimated from the motion vector obtained from the position of the frame in each of the previous time-series images prior to the current image.
[0096] In S304, a process is performed to determine whether the overlap degree in the current image calculated by the overlap degree calculation unit 106 for the person image surrounded by the frame acquired in the process of S302 is greater than a predetermined threshold value. This threshold value is the same as the threshold value used by the authentication control unit 107 to determine whether or not to perform identity authentication for the person image.
[0097] In the judgment process of S304, if it is determined that the degree of overlap is greater than the threshold value (if the judgment result is YES), the process proceeds to S306, and if it is determined that the degree of overlap is less than the threshold value (if the judgment result is NO), the process proceeds to S305.
[0098] In S305, a process is performed in which the position closest to the estimated position of the frame in S303 is identified as the position of the frame in the current image among the detected positions of the human image detected in the current image by the detection process performed together with tracking in the human image extraction unit 104. Thereafter, the process proceeds to S307.
[0099] S306 is a process performed when the determination process of S304 determines that the person image surrounded by the frame acquired by the process of S302 should not be used for identity authentication in the current image. In S306, the estimated position of the frame obtained by the process of S303 is directly identified as the position of the frame in the current image. In other words, in S306, the estimated position of the frame is not compensated for by the detected position of the person image detected from the current image by the detection process performed by the person image extraction unit 104.
[0100] In S307, a process is performed to determine whether all of the frames in the frame immediately preceding the current image have been acquired by the process of S302. If it is determined in this process that all of the frames have been acquired (if the determination result is YES), the tracking process ends and the process waits for acquisition of a new image following the current image by the imaging unit 103. On the other hand, if it is determined in this process that there are frames that have not been acquired (if the determination result is NO), the process returns to S302, and the processes from S102 onwards are performed again for the frames that have not been acquired.
[0101] The above-described processing is the tracking processing. Note that the determination processing in S304 in Fig. 10 is processing performed by the tracking control unit 301 in Fig. 9, and the remaining processing is processing performed by the person image extraction unit 104 in Fig. 9.
[0102] As described above, in the third example of authentication system 100, when it is determined that a person image is not to be subject to person determination, the extraction of the person image from the second image following the first image from which the person image was extracted is performed using only the result of estimating the position of the person image in the second image. This estimation is performed based on the position of the person image in each of the time-series images earlier than the second image, and the result of detecting the person image in the second image is not used to extract the person image from the second image. By doing so, the third example of authentication system 100 suppresses a decrease in estimation accuracy in estimating the position of a partial image included in an image later than the second image in the time-series image.
[0103] Next, a fourth example of the authentication system 100 will be described.
[0104] Fig. 11 shows the functional configuration of a fourth example of the authentication system 100. In Fig. 11, components that provide the same functions as those in the first example of Fig. 3 are denoted by the same reference numerals, and detailed description of these components will be omitted.
[0105] 11 differs from the first example in that the fourth example includes a plurality of each of the photographing unit 103, the person image extraction unit 104, the identity accuracy estimation unit 105, the overlap degree calculation unit 106, and the authentication control unit 107, and in that an authentication integration unit 401 is added. For convenience, FIG. 11 illustrates three photographing units 103, three person image extraction units 104, three identity accuracy estimation units 105, three overlap degree calculation units 106, and three authentication control units 107.
[0106] The multiple image capturing units 103 are all cameras that capture images of a predetermined space to obtain time-series images that are moving images, and are arranged so as to capture images of the predetermined space from different image capturing positions.
[0107] Fig. 12 shows an example of the installation of multiple image capturing units 103, and is a schematic representation of an example of installation in a venue 111 including a target space 110, which is a predetermined space, as illustrated in Fig. 4A. In Fig. 12, image capturing units 103-1 and 103-2 are installed in different positions, and the target space 110 is captured from each of the installation positions.
[0108] When there is a large overlap between images 10 of a person captured by one of the image capturing units 103, the authentication integration unit 401 adopts and outputs the result of personal authentication performed on the image of the person using an image 10 captured by another of the image capturing units 103. The operation of the authentication integration unit 401 will be described with reference to FIG. 13 .
[0109] 13 shows example images obtained by capturing images of target space 110 using each of image capturing units 103-1 and 103-2, the installation example of which is shown in FIG. 12. Of these, in the image captured by image capturing unit 103-1, person image 14 largely overlaps with person image 11. On the other hand, there is no overlap in person image 14 in the image captured by image capturing unit 103-2, and the degree of overlap is zero.
[0110] In such a case, the authentication integration unit 401 adopts the result of the personal authentication performed using the image acquired by the image capturing unit 103-2 with a small overlap with the person image 14, and causes the authentication result output unit 108 to output the adopted result of the personal authentication. By doing so, even when the overlap of the person images is large, a decrease in the accuracy of the personal authentication is suppressed.
[0111] The following describes details of the authentication control process performed by the authentication integration unit 401. Fig. 14 is a flowchart showing an example of the integrated authentication process.
[0112] 14 starts, in S401, a process is performed in which the results of personal authentication for the same person and the overlap degree of the person images of the person are acquired from the multiple authentication control units 107. Note that the person images acquired from the multiple image capturing units 103 may be associated with each other, for example, using a distance sensor that acquires the distance from each image capturing unit 103 to the person, based on the distance acquired by the distance sensor and the installation position of each image capturing unit 103.
[0113] In S402, a process is performed in which the authentication result output unit 108 performs the result of personal authentication obtained by the process of S401 for the person in the person image with the smallest degree of overlap obtained by the process of S401, and then the process of Figure 14 is terminated.
[0114] The above-described processing is the integrated authentication processing.
[0115] As described above, in the fourth example of authentication system 100, a second image is acquired, which is different from a first image acquired by capturing an image of a predetermined space from a different capturing position from the capturing position of the first image. Then, the overlapping degree between the first person and other persons reflected in the first image is calculated as a first overlapping degree, and another overlapping degree between the first person and other persons reflected in the second image is calculated as a second overlapping degree.
[0116] In the fourth example, whether or not to use a person image (partial image) showing a first person included in the first image as a target for person determination is determined based on the first overlapping degree. If it is determined that the person image showing the first person included in the first image is a target for person determination, the person image showing the first person included in the first image is used to determine which of the candidates the first person is.
[0117] In a fourth example, it is further determined whether or not to use a person image (another partial image) showing a first person included in the second image as a subject of person determination, depending on the second overlap degree. If it is determined that the person image showing the first person included in the second image is a subject of person determination, it is determined which of the candidates the first person is, using the person image showing the first person included in the second image.
[0118] In a fourth example, if the second degree of overlap is smaller than the first degree of overlap, a determination result as to which of the candidates the first person is is output using a person image of the first person included in the second image, whereas if the first degree of overlap is smaller than the second degree of overlap, a determination result as to which of the candidates the first person is is output using a person image of the first person included in the first image.
[0119] In the fourth example of the authentication system 100, by doing so, even if there is a large overlap of person images, a decrease in the accuracy of personal authentication is suppressed.
[0120] Next, Fig. 15 will be described. Fig. 15 shows an example of the hardware configuration of an information processing device 40. This information processing device 40 can function as the person candidate storage unit 102, person image extraction unit 104, identity accuracy estimation unit 105, overlap degree calculation unit 106, authentication control unit 107, and authentication result output unit 108 in each example of the authentication system 100 described above. In addition, this information processing device 40 can also function as the threshold control unit 201 in the second example of the authentication system 100, the tracking control unit 301 in the third example of the authentication system 100, and the authentication integration unit 401 in the fourth example of the authentication system 100.
[0121] The information processing device 40 includes, as its components, the following hardware elements: a processor 41, a memory 42, a storage device 43, a reading device 44, a communication interface 46, and an input / output interface 47. These components are connected via a bus 48, allowing data to be exchanged between the components.
[0122] The processor 41 may be, for example, a single processor or a multi-processor and multi-core processor. The processor 41 executes programs using the memory 42, thereby functioning as each of the components described above in the authentication system 100.
[0123] The memory 42 is, for example, a semiconductor memory, and may include a RAM area and a ROM area. RAM is an abbreviation for Random Access Memory, and ROM is an abbreviation for Read Only Memory.
[0124] The storage device 43 is, for example, a semiconductor memory such as a flash memory or a hard disk drive, and stores the above-mentioned subject model and terminal model in advance.
[0125] The reader 44 accesses the removable storage medium 45 in accordance with instructions from the processor 41. The removable storage medium 45 is realized by, for example, a semiconductor device (such as a USB memory), a medium to which information is input and output by magnetic action (such as a magnetic disk), or a medium to which information is input and output by optical action (such as a CD-ROM or DVD). Note that USB is an abbreviation for Universal Serial Bus. CD is an abbreviation for Compact Disc. DVD is an abbreviation for Digital Versatile Disk.
[0126] The communication interface 46 transmits and receives data via a communication network (not shown) according to instructions from the processor 41, for example.
[0127] The input / output interface 47 provides an interface with, for example, the in-area person information acquisition unit 101 and the photographing unit 103. The input / output interface 47 also provides the function of an authentication result output unit 108 in combination with, for example, an output device such as a display device.
[0128] As described above, the hardware configuration of the information processing device 40 is similar to that of a standard computer. Therefore, a computer may be used as the information processing device 40.
[0129] The program executed by the processor 41 of the information processing device 40 is provided in the following form, for example.
[0130] (1) Pre-installed in the storage device 43. (2) Provided by the removable storage medium 45. (3) Provided to the communication interface 46 from a server such as a program server via a communication network (not shown).
[0131] The hardware configuration of the information processing device 40 is an example, and the embodiment is not limited to this. For example, some or all of the functions of the above-described functional units may be implemented as hardware such as an FPGA or SoC. FPGA is an abbreviation for Field Programmable Gate Array. SoC is an abbreviation for System-on-a-chip.
[0132] The authentication system 100 described above is useful when used in, for example, an airport. In an airport where many people come and go, there is a concern that the accuracy of person authentication may decrease due to occlusion. Using the authentication system 100 in such a situation can reduce the possibility of erroneous authentication of the person to be authenticated due to occlusion. Therefore, it is expected that inappropriate service provision, such as displaying boarding gate information for a person who has been mistakenly identified as the passenger on an electronic information board for passengers guiding them to their boarding gate, can be prevented.
[0133] While the disclosed embodiments and their advantages have been described in detail above, those skilled in the art may make various modifications, additions, and omissions without departing from the scope of the invention as clearly set forth in the claims.
[0134] For example, the threshold control unit 201 in the second example of the authentication system 100 may be added to the functional configuration of the third example of the authentication system 100 shown in Figure 9, and the threshold value used as the standard for determining whether the overlap is large or small may be changed in the third example in the same way as in the second example.
[0135] Furthermore, the threshold control unit 201 in the second example of the authentication system 100 may be added between each of the multiple overlap degree calculation units 106 and the corresponding authentication control unit 107 in the functional configuration of the fourth example of the authentication system 100 shown in Fig. 11. In other words, the threshold value used as the criterion for determining whether the overlap is large or small in each of the multiple authentication control units 107 may be changed in the same way as in the second example.
[0136] Furthermore, the tracking control unit 301 in the third example of the authentication system 100 may be added between each of the multiple overlap degree calculation units 106 and the corresponding person image extraction units 104 in the functional configuration of the fourth example of the authentication system 100 shown in Fig. 11. In other words, the tracking control unit 301 may be configured to control the tracking performed in each of the multiple person image extraction units 104.
[0137] 10 Image 11, 12, 13, 14 Person image 20, 102 Person candidate storage unit 21, 22, 23, 24 Person candidate 30 Authentication device 31 Acquisition unit 32 Identification unit 33 Calculation unit 34 First judgment unit 35 Extraction unit 36 Second judgment unit 37 Suppression unit 40 Information processing device 41 Processor 42 Memory 43 Storage device 44 Reading device 45 Removable storage medium 46 Communication interface 47 Input / output interface 48 Bus 100 Authentication system 101 In-area person information acquisition unit 103, 103-1, 103-2 Photography unit 104 Person image extraction unit 105 Identity accuracy estimation unit 106 Overlap calculation unit 107 Authentication control unit 108 Authentication result output unit 110, 120 Target space 111 Venue 112 Entrance management device 113 Exit management device 121 Screen 201 Threshold control unit 301 Tracking control unit 401 Authentication integration unit
Claims
1. A method for determining whether or not a partial image included in the image, which shows the first person, is to be subject to person determination according to the calculated overlapping degree; if it is determined that the first person is to be subject to person determination, the partial image is used to determine which of the candidates the first person is; and if it is determined that the first person is not to be subject to person determination, the partial image is used to suppress the determination of which of the candidates the first person is to be determined using the partial image.
2. The method according to claim 1, further comprising the steps of: acquiring information about a person present in the specified space; and identifying the candidate based on the information about the person.
3. The method of claim 1, wherein the image is a first image which is one of a time-series image obtained by continuously photographing the specified space, and in the process of determining whether or not the partial image showing the first person is to be a subject for person determination, if the degree of overlap is greater than a specified threshold, the computer determines that the partial image showing the first person is not to be a subject for person determination, and if the computer determines that the partial image showing the first person is not to be a subject for person determination, the computer changes the specified threshold to a value smaller than the value before the change, and then determines whether or not the partial image showing the first person, which is included in a second image which is the time-series image following the first image, is to be a subject for person determination.
4. The method of claim 3, characterized in that when it is determined that the partial image showing the first person contained in the second image is to be the subject of person determination, the computer further executes a process of changing the specified threshold back to the value before the change, and then determining whether or not the partial image showing the first person contained in a third image, which is the time-series image following the second image, is to be the subject of person determination.
5. The method of determining whether or not a person is a first image, which is one of a time-series image obtained by continuously photographing the specified space, the partial image included in the first image is extracted from the first image based on a combination of a result of detecting a person from the first image and a result of estimating the position of the partial image in the first image based on the position of the partial image in each of the time-series images older than the first image, and when a determination is made that the partial image showing the first person is not to be subject to person determination, the extraction of the partial image included in a second image, which is the time-series image following the first image, is performed using only the result of estimating the position of the partial image in the second image, out of the result of detecting a person from the second image and the result of estimating the position of the partial image in the second image based on the position of the partial image in each of the time-series images older than the second image.
6. The method of determining whether or not the computer further performs the steps of: calculating an overlap between the first person reflected in the other image, which is an image other than the image obtained by photographing the specified space from a shooting position different from that of the image, and another person reflected in the other image, as an overlap; and determining whether or not to use another partial image included in the other image, which shows the first person, as a subject for person determination, depending on the calculated overlap.
7. The method of judgment described in claim 6, characterized in that when it is determined that the other partial image is to be the subject of the person judgment, the computer further executes a process of using the other partial image to judge which of the candidates the first person is.
8. The method of determination described in claim 7, characterized in that the computer further performs the following processing: when the other degree of overlap is smaller than the degree of overlap, outputting a determination result as to which of the candidates the first person is determined using the other partial image; and when the degree of overlap is smaller than the other degree of overlap, outputting a determination result as to which of the candidates the first person is determined using the partial image.
9. A judgment program characterized by having a computer execute the following processes: when multiple people are detected in an image obtained by photographing a specified space, calculate the degree of overlap between a first person among the multiple people and other people; determine whether or not a partial image included in the image and showing the first person is to be subject to person judgment according to the calculated degree of overlap; when it is determined that the first person is to be subject to person judgment, use the partial image to determine which of the candidates the first person is; and when it is determined that the first person is not to be subject to person judgment, suppress the determination of which of the candidates the first person is to be determined using the partial image.
10. An information processing device comprising: a calculation unit which, when multiple people are detected to be reflected in an image obtained by photographing a specified space, calculates the degree of overlap between a first person among the multiple people and other people; a first judgment unit which judges whether or not a partial image contained in the image, which partial image indicates the first person, is to be subject to person judgment in accordance with the calculated degree of overlap; a second judgment unit which, when it is determined that the first person is to be subject to person judgment, uses the partial image to judge which of the candidates the first person is; and a suppression unit which, when it is determined that the first person is not to be subject to person judgment, suppresses the judgment, made using the partial image, of which of the candidates the first person is to be.
Citation Information
Patent Citations
Target detection method and device and electronic system
CN112613540A
Video streaming personnel online duration estimation method and corresponding system
CN112766230A
Device, method and program for object recognition
JP2021149687A
