Information processing device, information processing method, and program

Abstract

This invention provides an information processing device, an information processing method, and a program that improve the accuracy of person recognition. [Solution] An information processing device 100 that verifies the identity of a registered person by facial recognition when payment is made at an unmanned checkout counter in a store, comprising: an action detection unit 303 that detects suspicious behavior of a person detected from an image; a biometric recognition unit 310 that performs recognition processing on the detected person based on biometric information obtained from the image of the detected person; and a level change unit 306 that changes the security level related to the recognition processing by the biometric recognition unit 310 based on the detected suspicious behavior.

Description

Technical Field

[0001] The present invention relates to information processing technology used for personal identification.

Background Art

[0002] Personal identification using biometric information such as face and fingerprint has been increasingly used in recent years because it can recognize whether a person is the registered user without the need to carry an extra item such as an IC card. However, by obtaining the biometric information of the registered user (such as a face photo in the case of face recognition), it is possible to "impersonate" the registered user, and there is a risk of bypassing the personal identification. Therefore, methods and devices for enhancing security by detecting impersonation have been proposed. Taking face recognition as an example, items used for impersonation (hereinafter referred to as impersonation attack items) include printed materials with the face photo of the registered user, notepads with the face image of the registered user drawn, latex masks molded based on the face of the registered user, etc. By presenting any of these impersonation attack items in front of the camera that captures the face image to be processed for face recognition, it becomes possible for others to impersonate the registered user. Impersonation detection is a technology for detecting such impersonation attack items.

[0003] On the other hand, Patent Document 1 discloses a personal identification system that enhances security by performing an impersonation detection process for detecting whether it is an impersonation by distinguishing between living and non-living bodies a predetermined number of times, and increasing the number of personal identification attempts when impersonation is detected. Patent Document 2 discloses a method for detecting impersonation based on the movement of the line of sight of the target person. Further, Patent Document 3 discloses a method for detecting impersonation by asking the target person to move their face. In addition, as another method for enhancing security, Patent Document 4 discloses a method of registering the face image of a suspicious person who has performed suspicious actions in a list, and invalidating the password input for an electronic lock if the person who unlocks the electronic lock is included in the suspicious person list.

Prior Art Documents

Patent Documents

[0004] [Patent Document 1] Japanese Patent Publication No. 2015-82195 [Patent Document 2] Japanese Patent Publication No. 2008-15800 [Patent Document 3] Japanese Patent Publication No. 2008-305400 [Patent Document 4] Japanese Patent Publication No. 2009-59222 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, the aforementioned impersonation detection methods do not necessarily detect impersonating attackers as non-living entities, and if impersonation detection fails, there is a risk that the impersonator may be mistakenly identified as the registered user. In the method disclosed in Patent Document 1, for example, if an impersonating attacker that is an elaborate replica of the registered user's face is used, there is a possibility that the impersonation will not be detected even after performing impersonation detection a predetermined number of times. In the method disclosed in Patent Document 2, for example, if the eye area of ​​the mask is cut out to reveal the real eyes, there is a possibility that the impersonation will not be detected. In the method disclosed in Patent Document 3, if a mask with a three-dimensional shape rather than a flat surface is used as an impersonating attacker, there is a possibility that the impersonation will not be detected. In addition, in the method disclosed in Patent Document 4, for example, if someone else wears a latex mask, it may not be possible to determine whether they are a suspicious person, and an impersonation attack may be carried out. Thus, with existing technologies, there is a risk that a person impersonating the registered user may be recognized as the registered user, and the accuracy of identity recognition is not necessarily high.

[0006] Therefore, the present invention aims to improve the accuracy of person recognition. [Means for solving the problem]

[0007] The information processing apparatus of the present invention is characterized by comprising: behavior detection means for detecting suspicious behavior of a person detected from an image; recognition means for performing recognition processing on the detected person based on biometric information obtained from the image of the detected person; and modification means for changing the security level related to the recognition processing by the recognition means based on the detected suspicious behavior. [Effects of the Invention]

[0008] According to the present invention, the accuracy of person recognition can be increased. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of an application of an information processing device. [Figure 2] This figure shows an example of the hardware configuration of an information processing device. [Figure 3] This figure shows an example of the functional configuration of an information processing device. [Figure 4] This figure shows an example of a lookup table used to change security levels. [Figure 5] This is a flowchart illustrating the information processing flow according to the embodiment. [Figure 6] This is a flowchart that includes the learning process for the suspicious behavior classifier. [Figure 7] This diagram shows the information for each item stored in the behavioral database. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. The following embodiments are not intended to limit the present invention, and not all combinations of features described in these embodiments are essential to the solutions of the present invention. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the apparatus to which the present invention is applied and various conditions (usage conditions, usage environment, etc.). In addition, in the following embodiments, the same or similar configurations and processing steps are denoted by the same reference numerals, and redundant explanations are omitted.

[0011] <First Embodiment> In the first embodiment, an example will be given of a scenario in which the information processing device according to this embodiment verifies the identity of the registered person by facial recognition when payment is made at an unmanned checkout counter in a store. Figure 1 shows an example in which the information processing device 100 according to this embodiment is installed in a store equipped with an unmanned checkout counter. As shown in Figure 1, the store is equipped with the information processing device 100, a checkout counter camera 101, and a surveillance camera 102, and the checkout counter camera 101 and the surveillance camera 102 are connected to the information processing device 100 via a network. The surveillance camera 102 is used to photograph the inside of the store, and the checkout counter camera 101 is used to photograph the area in front of the checkout counter.

[0012] When the information processing device 100 detects a person 110 from the video acquired by the surveillance camera 102, it starts the information processing according to this embodiment and collects behavioral information (information representing the person's actions) while tracking the detected person 110. If, for example, multiple people are detected, the information processing device 100 tracks each person and collects behavioral information for each person. The information processing device 100 also performs the process of detecting a person 110 from the image acquired by the cash register camera 101. When the information processing device 100 determines that the person 110 it has been tracking has come to the cash register, or when it detects the person 110 from the image of the cash register camera 101, it starts the process of detecting suspicious behavior using the behavioral information collected about that person 110. It is assumed that the actions to be detected as suspicious behavior are predetermined. Details of the process for detecting suspicious behavior will be described later.

[0013] When the information processing device 100 detects suspicious behavior based on the action information collected about the person 110 in front of the cash register, it changes the security level related to the recognition process for that person 110. The details of the process of changing the security level related to the recognition process will be described later. After that, when the information processing device 100 can recognize, through the execution of the recognition process, that the person 110 is the registered person himself / herself registered in advance, it permits the person to settle the payment for the goods. On the other hand, when it recognizes that the person 110 is not the registered person, it does not permit the settlement.

[0014] FIG. 2 is a diagram showing an example of the hardware configuration of the information processing device 100. In addition to the information processing device 100, FIG. 2 also shows a camera 2, a network 208, an input device 210, and an output device 211. The information processing device 100 includes a CPU 200, a ROM 201, a RAM 202, a secondary storage device 203, an output device IF (interface) 204, an input device IF 205, a communication IF 206, and a system bus 207.

[0015] The CPU 200 executes instructions according to the programs stored in the ROM 201 and the RAM 202. The ROM 201 is a non-volatile memory and stores programs and data necessary for controlling each part. The RAM 202 is a volatile memory and stores image data for each frame of the video and various temporary data used in the recognition process. The secondary storage device 203 is a rewritable secondary storage device such as a hard disk drive or a flash memory, and stores image data for each frame of the video, the information processing program according to this embodiment, various setting contents, etc. The information processing program and data according to this embodiment are transferred to the RAM 202, and the CPU 200 executes the information processing program and uses the data. Note that the information processing program according to this embodiment may be stored in the ROM 201. The processing of each step in each flowchart described below and the functional configuration shown in FIG. 3 are realized by expanding the information processing program according to this embodiment stored in the secondary storage device 203 or the ROM 201 into the RAM 202 and executed by the CPU 200.

[0016] The output device IF 204 transmits information such as the result of the recognition process to the output device 211. The input device IF 205 receives the information input from the input device 210. The communication IF 206 is a modem or a LAN or the like that connects to a network 208 such as the Internet or an intranet. The system bus 207 connects these components to perform data input / output with each other.

[0017] The camera 209 is a photographing device configured to include an imaging lens, an imaging sensor such as a CCD or a CMOS, an image signal processing unit, etc., and outputs an image for each frame of a moving image. In the case of this embodiment, the camera 209 corresponds to the monitoring camera 102 or the pre-registration camera 101 in FIG. 1. The image for each frame output from the camera 209 is transmitted to the information processing device 100 via the network 208, and the information processing device 100 acquires the image for each frame transmitted from the camera 209 via the communication IF 206. Note that the camera 209 may be directly connected to the information processing device 100 without going through the network 208.

[0018] The input device 210 is, for example, a device that reads biometric information such as a fingerprint, a keyboard or a touch panel for inputting a password, etc., and enables input from a user. The output device 211 is, for example, a display device that displays the result of the recognition process, a cash register that receives the result of the recognition process by the information processing device 100 and performs settlement based on the result, etc.

[0019] FIG. 3 is a functional block diagram showing an example of the functional configuration of the information processing device 100. The image acquisition unit 300 acquires frame-by-frame images of video captured by cameras 209 (surveillance camera 102 is the cash register camera 101) connected via a network not shown in Figure 3. The person detection unit 301 detects people from images acquired by the image acquisition unit 300. That is, the person detection unit 301 detects people from video images taken by the surveillance camera 102 and images taken by the cash register camera 101. Hereafter, the people detected by the person detection unit 301 from the images will be referred to as detected people. The tracking unit 302 tracks the person detected by the person detection unit 301 frame by frame of the video.

[0020] The behavior detection unit 303 collects behavioral information of the person being tracked by the tracking unit 302 and detects suspicious behavior by the detected person based on that behavioral information. For example, based on the behavioral information of the detected person, the behavior detection unit 303 obtains a likelihood that the detected person's behavior is suspicious, and if that likelihood is greater than or equal to a predetermined likelihood threshold, it detects that the detected person's behavior is suspicious. Furthermore, the behavior detection unit 303 also has the function of detecting one or more of the following types of suspicious behaviors, including direct acts of impersonation, indirect acts of impersonation, and signs of pre-crime activity, based on the behavioral information of the detected person.

[0021] A direct act of impersonation is the act of having an impersonation object (an object used for impersonation) photographed by a camera (held in front of the camera). Direct acts of impersonation include, for example, the action of a person taking out an impersonation object such as a mask or tablet from a bag or pocket, or the action of putting something on. An example of a person putting something on is the action of putting an impersonation object such as a latex mask on their face (i.e., putting it on). The behavior detection unit 303 has the function of detecting suspicious behavior and determining the type of impersonation object that the detected person took out of their bag or elsewhere.

[0022] Indirect acts of impersonation are not direct actions that constitute impersonation, but are behaviors that are likely to be commonly performed by people attempting to impersonate others and serve as signs of such behavior. Examples of indirect acts of impersonation include "putting one's hand into a bag without looking at one's hands" and "abnormally reduced facial movement due to fear of facial recognition failure."

[0023] Criminal preparatory behaviors are not limited to impersonation, but are actions that are highly likely to be commonly performed by people with the intention to commit a crime. Examples of criminal preparatory behaviors include "a person wandering around," "looking around nervously," "avoiding being seen," and "body movements that are said to appear when committing a crime." Body movements that appear when committing a crime refer to abnormal amplitude and magnitude of body vibrations that are different from normal, and in this embodiment, this refers to cases where a person's body vibrations exceed a predetermined magnitude and amplitude.

[0024] In this embodiment, known detection methods will be used to detect direct acts of impersonation, indirect acts of impersonation, and signs of preparatory actions for a crime. Note that any method capable of detecting direct acts of impersonation, indirect acts of impersonation, and signs of preparatory actions for a crime may be used, even if it is a different action than those exemplified.

[0025] The biometric recognition unit 310 acquires biometric information from the image of the detected person and performs recognition processing on the detected person based on that biometric information. The biometric recognition unit 310 consists of an impersonation detection unit 304 and a person recognition unit 305. The impersonation detection unit 304 determines whether the person detected by the person detection unit 301 is a living being or not, and detects impersonation based on the result of that determination. In this embodiment, the impersonation detection unit 304 determines whether the detected person is a living being or a non-living being such as a photograph, based on the facial image of the person detected from the image of the camera 101 in front of the cash register, and determines that impersonation is occurring if it is determined that the person is a non-living being. The personal identification unit 305 acquires facial feature quantities as biometric information from images of predetermined biological parts of a person detected by the person detection unit 301 from the image of the camera 101 in front of the cash register, compares these feature quantities with the facial feature quantities of registered persons, and performs personal identification to determine whether the detected person is a registered person or not.

[0026] In this embodiment, if suspicious behavior is detected by the behavior detection unit 303, the level change unit 306, described later, changes at least one of the security levels related to impersonation detection and the security level related to identity recognition in the biometric recognition unit 310.

[0027] In other words, the impersonation detection unit 304 changes the security level related to impersonation detection according to the security level set by the level change unit 306, which will be described later. As will be described in detail later, for example, if suspicious behavior of a detected person is detected, the level change unit 306 changes the security level related to impersonation detection for that detected person to an increased level. In this embodiment, increasing the security level related to impersonation detection means changing the method used for impersonation detection (impersonation detection method) or making impersonation detection stricter.

[0028] Furthermore, the identity recognition unit 305 changes the security level related to identity recognition according to the security level set in the level change unit 306, which will be described later. As will be described in detail later, for example, if suspicious behavior of a detected person is detected, the level change unit 306 changes the security level related to identity recognition for that detected person to an increased level. In this embodiment, increasing the security level related to identity recognition means changing the method used for identity recognition (identity recognition method) or making identity recognition stricter.

[0029] When suspicious behavior is detected, the level change unit 306 changes at least one of the security levels related to impersonation detection performed by the impersonation detection unit 304 and the security level related to identity recognition performed by the identity recognition unit 305 to an increased level. For example, when changing the security level related to impersonation detection, the level change unit 306 changes at least one of the methods of impersonation detection or makes the impersonation detection stricter, depending on the detected suspicious behavior. Also, for example, when changing the security level related to identity recognition, the level change unit 306 changes at least one of the methods of identity recognition or makes the identity recognition stricter, depending on the detected suspicious behavior. If, for example, identity recognition for a detected person is refused when suspicious behavior is detected, the identity recognition result cannot be obtained, which would reduce user convenience. In contrast, in this embodiment, when suspicious behavior is detected, the methods and strictness of impersonation detection and identity recognition are changed, and impersonation detection and identity recognition themselves are not refused, so user convenience is maintained.

[0030] More specifically, the level change unit 306 changes the security level based on the type of suspicious behavior detected by the behavior detection unit 303, using one or more security level change methods selected from, for example, several different security level change methods. For example, the first method for changing the security level is a method for increasing the security level related to the impersonation detection described above. In other words, the first method for changing the security level is increased by tightening the threshold used to determine whether or not a person is a biological being during impersonation detection, or by switching to a more accurate impersonation detection method. An example of a more accurate impersonation detection method is a method that requires the person being recognized to move their face. By using the first method for changing the security level, the security level can be further enhanced.

[0031] The second method for changing the security level is to enhance the security level related to the facial recognition used for identity verification, as described above. Specifically, the second method for changing the security level involves, for example, tightening the similarity threshold when determining the similarity with registered individuals, or switching to a more accurate facial recognition method. Using the second method for changing the security level can further enhance the security level.

[0032] The third method for changing the security level is another example of a method for increasing the security level related to personal identification. In the third method for changing the security level, in addition to facial recognition, personal identification is made possible by requiring the use of other biometric information obtained from images of other biometric parts other than the face, such as veins, fingerprints, and irises. When using the third method for changing the security level, the security level can be further enhanced by performing additional biometric recognition such as veins, fingerprints, and irises.

[0033] The fourth method for changing the security level is an example of a method for increasing the security level related to personal identification. In the fourth method for changing the security level, the security level related to personal identification is increased by requiring non-biometric recognition in addition to biometric recognition by facial recognition. Examples of non-biometric recognition in this case include personal identification using non-biometric information such as SMS recognition, questions to identify the person, and PIN entry. When the fourth method for changing the security level is used, both biometric and non-biometric recognition are performed, so the security level can be increased.

[0034] The fifth method for changing the security level is another example of a method for increasing the security level related to personal identification. In the fifth method, personal identification is switched to personal identification using biometric information other than facial recognition. Examples of biometric identification other than facial recognition include biometric information such as veins, fingerprints, and irises. For example, if suspicious behavior equivalent to an impersonation attack against facial recognition is detected, the system switches from facial recognition to biometric identification other than facial recognition. By using the fifth method for changing the security level, the security level can be further enhanced by switching from facial recognition to biometric identification other than facial recognition.

[0035] The sixth method for changing the security level is another example of a method for increasing the security level related to identity verification. In the sixth method, for example, biometric recognition such as facial recognition is disabled and switched to non-biometric recognition. Examples of non-biometric recognition that can be switched to by disabling biometric recognition include password entry, QR code recognition, IC card recognition, and identity verification by store staff. By using the sixth method for changing the security level, the security level can be increased by switching from biometric recognition to non-biometric recognition.

[0036] In this embodiment, the level change unit 306 selects one or more of the aforementioned security level change methods by referring to a lookup table as shown in Figure 4, based on the type of suspicious behavior detected by the behavior detection unit 303. As shown in Figure 4, the lookup table is a table that shows the correspondence between each of the multiple types of suspicious behavior detected by the behavior detection unit 303 and each of the multiple security level change methods.

[0037] For example, if the suspicious behavior detected by the behavior detection unit 303 is the action of putting something on, the level change unit 306 will select a security level change method that uses a recognition method other than facial recognition. That is, if the action of putting something on is detected as suspicious behavior, the level change unit 306 will select, for example, the fifth security level change method or the sixth security level change method. The choice of which recognition method other than facial recognition to use may be made by displaying these recognition methods as options on the screen of a touch panel having the functions of an input device 210 and an output device 211, and the user selecting from these options. Alternatively, for example, the level change unit 306 or the user recognition unit 305 may randomly select from among multiple security level change methods that use recognition methods other than facial recognition.

[0038] For example, if the suspicious behavior detected by the behavior detection unit 303 is the action of taking out a fraudulent attack object from a bag, the level change unit 306 tightens at least one of the thresholds for fraud detection and face recognition. In other words, if the action of taking out a fraudulent attack object from a bag is detected, the level change unit 306 selects, for example, the first security level change method or the second security level change method.

[0039] For example, if the behavior detection unit 303 detects suspicious behavior such as taking a spoofing attack object out of a bag, and also identifies the type of spoofing attack object, the level change unit 306 may select a security level change method appropriate to the type of spoofing attack object. For example, if the type of spoofing attack object is a 3D mask such as a latex mask, the level change unit 306 may select the fifth security level change method and change the biometric recognition method to a biometric recognition method other than facial recognition. For example, if the type of spoofing attack object is a tablet, the level change unit 306 may select the first security level change method as a more accurate method of detecting spoofing, and may, for example, request the detected person (recognized person) to move their face.

[0040] For example, if the behavior detection unit 303 detects suspicious behavior and obtains a likelihood that the behavior is suspicious, the level change unit 306 may change at least one of the biometric recognition method or strictness according to the likelihood of the suspicious behavior. For example, if the likelihood is well above the threshold, the level change unit 306 may select a fourth security level change method that performs non-biometric recognition in addition to facial recognition to make identity recognition stricter. Alternatively, if the likelihood is well above the threshold, the level change unit 306 may select a sixth security level change method that disables facial recognition and performs non-biometric recognition to change the identity recognition method. For example, if the likelihood is below the threshold but close to it, there is a possibility that the behavior is suspicious, so the first security level change method may be selected to tighten the threshold for impersonation detection, or the second security level change method may be selected to tighten the threshold for identity recognition.

[0041] Figure 5 is a flowchart showing the information processing flow related to the information processing device 100 according to the first embodiment, namely, the flow from person detection and tracking to detection of suspicious behavior and impersonation based on the detected person's behavioral information, and further to person recognition. The flowchart in Figure 5(a) mainly shows the flow related to person detection, and the flowchart in Figure 5(b) shows the flow from tracking to collection of behavioral information and person recognition, which is performed in accordance with each detected person. The processing in the flowchart in Figure 5 starts when the store opens for business or in response to a start instruction from the store manager, etc., and ends in response to a stop instruction from the store manager, etc.

[0042] First, as part of step S101 in the flowchart of Figure 5(a), the image acquisition unit 300 of the information processing device 100 acquires images of the store's video footage from one of the cameras 209, which is the surveillance camera 102. Next, in step S102, the person detection unit 301 performs a process to detect a person from the image acquired by the image acquisition unit 300. Next, in step S103, the person detection unit 301 determines whether or not a person was detected in the image in step S102. If a person is detected, the process proceeds to step S104, which is performed by the tracking unit 302. On the other hand, if no person is detected, the information processing device 100 returns to step S101.

[0043] If the process proceeds to step S104, the tracking unit 302 determines whether the person detected in step S102 is a person who has already been tracked. If it is determined that the person has already been tracked, the information processing device 100 returns to step S101. On the other hand, if it is determined that the person has not already been tracked, the tracking unit 302 proceeds to step S105.

[0044] When the process proceeds to step S105, the tracking unit 302 identifies the person detected by the person detection unit 301 in step S102 as a new person to be tracked (referred to as the tracked person), starts tracking that person, and proceeds to step S201 in Figure 5(b). In addition, the information processing device 100, as part of the process in step S106 in Figure 5(a), determines whether there is a termination instruction from, for example, a store manager, and terminates the process in the flowchart of Figure 5 if there is a termination instruction. On the other hand, if there is no termination instruction, the information processing device 100 returns to step S101.

[0045] If the process proceeds to step S201 in Figure 5(b), the tracking unit 302 starts the process of tracking the subject between frames in the video acquired by the image acquisition unit 300. Next, in step S202, the action detection unit 303 collects action information of the person being tracked. The tracking method is not limited to matching frames; it is sufficient to collect action information while tracking the person being tracked, and to continue tracking until the point when face recognition is performed on that person by the information processing device 100.

[0046] Next, in step S203, the behavior detection unit 303 determines whether the person being tracked (i.e., the detected person) is about to start the payment procedure at the register. For example, if the behavior detection unit 303 detects from the image of the surveillance camera 102 that the person being tracked has come to the front of the register, it determines that the person is about to start the payment procedure. Alternatively, for example, if the behavior detection unit 303 detects the person being tracked from the image of the camera in front of the register 101, it determines that the person is about to start the payment procedure. Furthermore, for example, if the identity recognition unit 305 starts the identity recognition process to determine whether the person being tracked is the registered person, it determines that the person is about to start the payment procedure. If the behavior detection unit 303 has not determined that the payment procedure has started, the information processing device 100 returns to step S201. On the other hand, if it has determined that the payment procedure has started, the behavior detection unit 303 proceeds to step S204.

[0047] When the process proceeds to step S204, the behavior detection unit 303 analyzes the content of the behavior information collected in step S202. Next, in step S205, the level change unit 306 changes at least one of the security levels for impersonation detection and identity recognition using one or more security level change methods selected according to the results of the analysis of behavioral information by the behavior detection unit 303.

[0048] Next, in step S206, the impersonation detection unit 304 executes an impersonation detection process according to the security level related to impersonation detection set by the level change unit 306 in step S205. Furthermore, in the next step S207, the identity recognition unit 305 performs identity recognition processing according to the security level related to identity recognition set by the level change unit 306. Then, the information processing device 100 completes the payment for the goods, etc., if it successfully recognizes that the person who initiated the payment procedure is the registered person. Note that the information processing device 100 may perform the identity recognition processing in step S207 only if no impersonation is detected in the impersonation detection processing in step S206.

[0049] As described above, the information processing device 100 according to this embodiment detects suspicious behavior of a detected person and changes the security level related to impersonation detection and identity recognition for a suspicious person attempting to impersonate someone based on the detection result of that suspicious behavior. As a result, the information processing device 100 of this embodiment can improve the accuracy of identity recognition of whether or not a person is the registered person, even in cases where existing impersonation detection methods cannot detect it as impersonation.

[0050] In the embodiment described above, if there are multiple detected individuals, each detected individual will be tracked and behavioral information collected for each individual. However, tracking by the tracking unit 302 is not essential if, for example, a detected individual can be identified without tracking, or if only the cash register camera 101 is installed in the store. An example of a case where a detected individual can be identified without tracking is when there is only one person in the store. Furthermore, in the embodiment described above, an example was given in which the impersonation detection unit 304 performs impersonation detection before the identity recognition unit 305 performs identity recognition, but impersonation detection is not mandatory. In other words, if suspicious behavior is detected, only the method of identity recognition or the security level of strictness may be changed. For example, in a system where impersonation detection and identity verification are performed, if suspicious behavior is detected, it is possible to change only the security level related to impersonation detection, without changing the security level related to identity verification. Furthermore, in the embodiment described above, information on the actions of individuals detected from images of the surveillance camera 102 was collected, but information on the actions of individuals detected from images of, for example, the cash register camera 101 may also be collected. In this case, the only camera installed in the store may be the cash register camera 101.

[0051] The above embodiment described an example of identity recognition using facial recognition in front of an unmanned checkout counter in a store, but it can be applied to other scenarios as well. For example, the information processing device of this embodiment can be applied to identity recognition for transactions at ATMs (automated teller machines), identity recognition for determining whether or not to pass through security gates, and identity recognition for determining whether or not to enter membership-based facilities. Furthermore, while the above-described embodiment used facial biometric information for identity recognition, it is not limited to facial biometric information and other biometric information may be used. For example, biometric information such as fingerprints, veins, or irises may be used. When identity recognition is performed using fingerprint biometric information, an example of a forgery attack object is conceivable, such as an artificial fingerprint molded from resin. When identity recognition is performed using vein biometric information, an example of a forgery attack object is conceivable, such as a printed vein pattern. When identity recognition is performed using iris biometric information, an example of a forgery attack object is conceivable, such as a printed iris.

[0052] <Second Embodiment> In the first embodiment, an example was given in which a predetermined suspicious behavior is detected and the security level related to impersonation detection and identity recognition is changed according to the detection result. In the second embodiment described below, an example is given in which the direct and indirect acts of impersonation described above are learned at the location where the information processing device is installed, and the learned model is used for suspicious behavior detection. In this embodiment, an example of learning the actions of direct and indirect acts of impersonation is given, but signs of pre-criminal behavior may also be learned in the same way. In the second embodiment as in the first embodiment described above, an example is given in which the information processing device 100 is installed in a store equipped with an unmanned checkout counter as shown in Figure 1. The hardware configuration of the information processing device 100 according to the second embodiment is the same as in Figure 2, and the functional configuration of the information processing device 100 is the same as in Figure 3, so their illustration and explanation are omitted.

[0053] Figure 6 is a flowchart showing the flow from tracking, which is performed in accordance with each detected person, to the collection of behavioral information and the person recognition process in the information processing device 100 according to the second embodiment. Note that the flow of the person detection process in the information processing device 100 of the second embodiment is the same as the flowchart in Figure 5(a) described above, so its illustration and explanation are omitted. Also, in the flowchart of Figure 6, the same processing steps as in the flowchart shown in Figure 5(b) after the start of the tracking process are denoted by the same reference numerals as in Figure 5(b), and their explanations are also omitted. The flowchart of Figure 6 includes the learning process of the suspicious behavior discriminator, and it is assumed that the information processing device 100 of the second embodiment has a learning function that performs the learning process of the suspicious behavior discriminator.

[0054] In the second embodiment, as part of the processing in step S301, the information processing device 100 determines whether there is a high probability that the person being tracked has committed impersonation, based on the result of the impersonation detection in step S206 and the information on the actions of the person being tracked after the impersonation detection. For example, if impersonation is detected in step S206 and the person being tracked gives up on paying, or if they attempt to impersonate another person to try to identify the person, the information processing device 100 determines that there is a high probability that they have committed impersonation.

[0055] Next, in step S302, the information processing device 100 associates a label indicating the result of the determination of the possibility of impersonation in step S301 with the behavioral information collected in step S202, and stores it in the behavioral database 601. Figure 7 is a diagram showing the information stored in the behavior database 601 by item. In this embodiment, behavior information includes, for example, the number of times a person performs multiple types of basic actions from person detection to recognition at the register, and the order in which these actions are performed. As shown in Figure 7, basic actions may include, for example, squatting, shaking the head, and raising the right hand. The information processing device 100 of this embodiment determines whether there is a high probability of impersonation (True) or a low probability of impersonation (False) based on the number of times and the order of these multiple types of basic actions exemplified in Figure 7.

[0056] Next, in step S303, the information processing device 100 uses the behavioral information stored in the behavioral database 601 as training data to train a suspicious behavior classifier that can determine whether or not the behavioral information is an impersonation. The training of the suspicious behavior classifier is performed using existing machine learning methods, and this training is carried out after a predetermined number of data has been collected in the behavioral database 601. In the second embodiment, the trained suspicious behavior classifier is used when analyzing the behavioral information in step S204.

[0057] In the second embodiment, an example was described in which a suspicious behavior classifier is obtained by collecting behavioral information of a person who is likely to have committed impersonation and learning from that behavior, but the invention is not limited to this example. For example, along with learning the suspicious behavior classifier, images of the possessions of a person who is likely to have committed impersonation (possessions that are likely to be impersonation attack items) may be collected, and an impersonation attack item detector may be learned using those images. According to the information processing device 100 of the second embodiment, it becomes possible to detect suspicious behavior other than predetermined suspicious behavior, thereby increasing the accuracy of person recognition compared to the example of the first embodiment.

[0058] The present invention can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, it can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. The above-described embodiments are merely examples of concrete implementations of the present invention, and the technical scope of the invention should not be limited by them. That is, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0059] Each embodiment of the disclosure includes the following configurations, methods, and programs. (Composition 1) A behavior detection means for detecting suspicious behavior of a person detected from an image, Recognition means that performs recognition processing on the detected person based on biometric information obtained from the image of the detected person, A modification means for changing the security level related to the recognition process in the recognition means based on the detected suspicious behavior, An information processing device characterized by having the following features. (Configuration 2) The information processing apparatus according to Configuration 1, characterized in that the modification means modifies at least one of the recognition method used in the recognition process and the strictness of the recognition process as a change in the security level related to the recognition process. (Composition 3) The aforementioned recognition means is The aforementioned impersonation detection means determines whether the detected person is a living or non-living being, and if it is a non-living being, it detects that impersonation has occurred. A personal identification means that performs personal identification to determine whether the detected person is a registered person or not, based on the biometric information obtained from the image of the detected person, It has, The information processing apparatus according to configuration 1 or 2, characterized in that the modification means modifies at least one of the security level related to the recognition process, namely the security level related to the impersonation detection in the impersonation detection means and the security level related to the identity recognition in the identity recognition means. (Composition 4) The information processing device according to configuration 3, characterized in that the identity recognition means performs identity recognition only when impersonation is not detected by the impersonation detection means. (Composition 5) The behavior detection means determines at least one of the following: the type of object used for impersonation that the detected person possesses, and the type of suspicious behavior performed by the detected person. The information processing apparatus according to configuration 3 or 4, characterized in that the modification means changes the security level related to the recognition process according to the determination result of the type. (Composition 6) The information processing apparatus according to configuration 5, characterized in that the modification means changes the security level related to the recognition process using one or more security level modification methods selected from a plurality of different security level modification methods according to the determination result of the type. (Composition 7) The aforementioned multiple different methods for changing security levels include: A first modification method that changes at least one of the methods for detecting impersonation or the strictness of the impersonation detection, A second method of modification involves changing at least one of the methods of identity recognition or the strictness of identity recognition, A third modification method that, in addition to recognizing a person using biometric information obtained from images of a predetermined biological part of a person, also performs recognition using other biometric information obtained from images of other biological parts other than the predetermined biological part, In addition to identity recognition using biometric information obtained from images of specific biological parts of a person, a fourth modification method is provided to perform identity recognition using non-biometric recognition. A fifth modification method for switching from self-recognition using biometric information obtained from images of predetermined biological parts of a person to self-recognition using other biometric information obtained from images of other biological parts other than the predetermined biological parts, A sixth modification method that disables self-recognition using biometric information obtained from images of a specific biological part of a person and switches to self-recognition using non-biometric recognition, The information processing device according to configuration 6, characterized in that it includes one or more of the above. (Composition 8) The aforementioned behavior detection means detects one or more of the following: direct acts of impersonation, indirect acts of impersonation, objects used in impersonation, and signs of preliminary actions. The information processing apparatus according to any one of configurations 3 to 7, characterized in that the modification means changes the security level related to the recognition process based on one or more detection results. (Composition 9) The direct act of impersonation includes at least one of the following: the action of a person putting on the object used for impersonation; the action of taking out the object used for impersonation; and the action of having the object used for impersonation photographed by the image-taking device. The aforementioned indirect act of impersonation includes at least one of the following: the action of a person reaching into a bag without looking at their hands, and the action of reducing facial movement. The information processing device according to configuration 8, characterized in that the aforementioned preliminary action signs include at least one of the following: a person wandering around; a person looking around nervously; a person avoiding being seen; and a vibration of the person's body exceeding a predetermined magnitude and amplitude. (Composition 10) The information processing device according to configuration 8 or 9, characterized in that the behavior detection means detects one or more of the direct acts of impersonation, indirect acts of impersonation, and impersonation attack objects using a discriminator learned at the location where the information processing device is installed. (Composition 11) The behavior detection means obtains the likelihood of the suspicious behavior of the detected person, The information processing apparatus according to any one of configurations 1 to 10, characterized in that the modification means changes the security level related to the recognition process based on the likelihood of the suspicious behavior. (Composition 12) The information processing apparatus according to any one of configurations 1 to 11, characterized in that the biological information is at least one of the following: face, iris, fingerprint, and vein. (Method 1) A behavior detection process that detects suspicious behavior of a person detected from an image, A recognition step which performs recognition processing on the detected person based on the biometric information obtained from the image of the detected person, A modification step, which modifies the security level related to the recognition process in the recognition step based on the detected suspicious behavior, An information processing method characterized by having the following features. (program) A program that causes a computer to function as an information processing device described in any one of configurations 1 through 11. [Explanation of Symbols]

[0060] 100: Information processing unit, 300: Image acquisition unit, 301: Person detection unit, 302: Tracking unit, 303: Behavior detection unit, 304: Impersonation detection unit, 305: Person recognition unit, 306: Level change unit, 310: Biometric recognition unit

Claims

1. A behavior detection means for detecting suspicious behavior of a person detected from an image, Recognition means that performs recognition processing on the detected person based on biometric information obtained from the image of the detected person, A modification means for changing the security level related to the recognition process in the recognition means based on the detected suspicious behavior, An information processing device characterized by having the following features.

2. The information processing apparatus according to claim 1, characterized in that the modification means modifies at least one of the recognition method used in the recognition process and the strictness of the recognition process as a change in the security level related to the recognition process.

3. The aforementioned recognition means is The aforementioned impersonation detection means determines whether the detected person is a living or non-living being, and if it is a non-living being, it detects that impersonation has occurred. A personal identification means that performs personal identification to determine whether the detected person is a registered person or not, based on the biometric information obtained from the image of the detected person, It has, The information processing apparatus according to claim 1, characterized in that the modification means modifies at least one of the security level related to the recognition process, namely the security level related to the impersonation detection in the impersonation detection means and the security level related to the identity recognition in the identity recognition means.

4. The information processing apparatus according to claim 3, characterized in that the identity recognition means performs identity recognition only when impersonation is not detected by the impersonation detection means.

5. The behavior detection means determines at least one of the following: the type of object used for impersonation that the detected person possesses, and the type of suspicious behavior performed by the detected person. The information processing apparatus according to claim 3, characterized in that the modification means changes the security level related to the recognition process according to the determination result of the type.

6. The information processing apparatus according to claim 5, characterized in that the modification means modifies the security level related to the recognition process using one or more security level modification methods selected from a plurality of different security level modification methods according to the determination result of the type.

7. The aforementioned multiple different methods for changing security levels include: A first modification method that changes at least one of the methods for detecting impersonation or the strictness of the impersonation detection, A second method of modification involves changing at least one of the methods of identity recognition or the strictness of identity recognition, A third modification method that, in addition to performing identity recognition using biometric information obtained from images of predetermined biological parts of a person, also performs identity recognition using other biometric information obtained from images of other biological parts other than the predetermined biological parts, In addition to identity recognition using biometric information obtained from images of specific biological parts of a person, a fourth modification method is provided to perform identity recognition using non-biometric recognition. A fifth modification method for switching from self-recognition using biometric information obtained from images of predetermined biological parts of a person to self-recognition using other biometric information obtained from images of other biological parts other than the predetermined biological parts, A sixth modification method that disables self-recognition using biometric information obtained from images of a predetermined biological part of a person and switches to self-recognition using non-biometric recognition, The information processing apparatus according to claim 6, characterized in that it includes one or more of the following.

8. The aforementioned behavior detection means detects one or more of the following: direct acts of impersonation, indirect acts of impersonation, objects used in impersonation, and signs of preliminary actions. The information processing apparatus according to claim 3, characterized in that the modification means changes the security level related to the recognition process based on one or more detection results.

9. The direct act of impersonation includes at least one of the following: the action of a person putting on the object used for impersonation; the action of taking out the object used for impersonation; and the action of having the object used for impersonation photographed by the image-taking device. The aforementioned indirect act of impersonation includes at least one of the following: the action of a person reaching into a bag without looking at their hands, and the action of reducing facial movement. The information processing apparatus according to claim 8, characterized in that the aforementioned preliminary action signs include at least one of the following: a person wandering around; a person looking around nervously; a person avoiding being seen; and a vibration of the person's body exceeding a predetermined magnitude and amplitude.

10. The information processing device according to claim 8, characterized in that the behavior detection means detects one or more of the direct acts of impersonation, indirect acts of impersonation, and impersonation attack objects using a discriminator learned at the location where the information processing device is installed.

11. The behavior detection means obtains the likelihood of the suspicious behavior of the detected person, The information processing apparatus according to claim 1, characterized in that the modification means changes the security level related to the recognition process based on the likelihood of the suspicious behavior.

12. The information processing apparatus according to any one of claims 1 to 11, characterized in that the biological information is at least one of the following: face, iris, fingerprint, and vein.

13. A behavior detection process that detects suspicious behavior of a person detected from an image, A recognition step which performs recognition processing on the detected person based on the biometric information obtained from the image of the detected person, A modification step, which modifies the security level related to the recognition process in the recognition step based on the detected suspicious behavior, An information processing method characterized by having the following features.

14. Computers, A behavior detection means for detecting suspicious behavior of a person detected from an image, Recognition means that performs recognition processing on the detected person based on biometric information obtained from the image of the detected person, A modification means for changing the security level related to the recognition process in the recognition means based on the detected suspicious behavior, A program that makes an information processing device function as having a certain feature.

Images