Information processing system, information processing method, and computer program

Abstract

An information processing system S1 comprising an acquisition unit 11 that sequentially acquires pieces of face information relating to a face area F detected from sequentially captured images, a storage unit 12 that sequentially stores the acquired pieces of face information, an extraction unit 13 that, if a predetermined condition is satisfied, performs an extraction operation of extracting features from pieces of extraction target face information that are at least a part of the pieces of face information stored in the storage unit 12, and an authentication unit 14 that performs face recognition by using the features.

Description

[Technical Field]

[0001] The present disclosure relates to the technical fields of an information processing system, an information processing method, and a recording medium. [Background technology]

[0002] Patent Document 1 describes a technology that includes multiple image input units to which images are input, detects an object area from the image input from the image input units using a detection unit, extracts features from the image of the object area detected by the detection unit, and controls and monitors the detection process and feature extraction process performed on the images input from the multiple image input units based on the object area detection results by the detection unit. Patent document 2 describes a technology that acquires object information of one or more objects for each frame image of a video, monitors the processing load of an image processing device, selects object information to be matched based on the processing load, calculates features using the selected object information, performs matching using the features, and matches objects in the video with a database. Patent document 3 describes a matching device for use at a gate equipped with a restriction unit that restricts the flow of people, the device comprising: a processing unit that performs a first candidate face image narrowed down by the result of a first face image matching using a first image taken of the first area, and a second face image matching using a second image taken of the second area, on a route where there is a flow of people from a first area toward a second area located upstream of a restriction unit; and a communication unit that outputs the result of the second face image matching. Patent Document 4 describes a technology that detects a subject to be matched from an image, identifies a method to be used for matching the subject from among multiple methods, determines whether the subject satisfies predetermined conditions according to the identified method, selects an image of the subject that is determined to satisfy the predetermined conditions, and transmits the image of the selected subject to an external device that can perform subject matching using multiple methods. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-070576 [Patent Document 2] Japanese Patent Publication No. 2020-009383 [Patent Document 3] Patent Publication No. 2021-131606 [Patent Document 4] Japanese Patent Publication No. 2022-026846 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of this disclosure is to provide an information processing system, an information processing method, and a recording medium that aim to improve upon the techniques described in prior art documents. [Means for solving the problem]

[0005] One aspect of the information processing system includes an acquisition means for sequentially acquiring face information of a face area detected from sequentially captured images, a storage means for sequentially storing the acquired face information, an extraction means for performing an extraction operation to extract features from each of at least some of the face information to be extracted that is stored in the storage means when a predetermined condition is satisfied, and an authentication means for performing face authentication using the features.

[0006] One aspect of the information processing method sequentially acquires face information of a face area detected from sequentially captured images, sequentially stores the acquired face information in a storage means, and when a predetermined condition is satisfied, performs an extraction operation to extract features from each of the face information to be extracted, at least a portion of the face information stored in the storage means, and performs face authentication using the features.

[0007] One aspect of the recording medium has recorded thereon a computer program for causing a computer to execute an information processing method that sequentially acquires face information of a face area detected from sequentially captured images, sequentially stores the acquired face information in a storage means, and, when predetermined conditions are met, performs an extraction operation to extract features from each of the face information to be extracted, at least a portion of the face information stored in the storage means, and performs face authentication using the features. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the configuration of an information processing system according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of an information processing system according to the second embodiment. [Figure 3] FIG. 3 is a flowchart showing the flow of information processing operations of the information processing system in the second embodiment. [Figure 4] FIG. 4 is a conceptual diagram of the information processing operation of the information processing system in the second embodiment. [Figure 5] FIG. 5 is a conceptual diagram of an information processing system according to the third embodiment. [Figure 6] FIG. 6 is a block diagram showing the configuration of an information processing system according to the third embodiment. [Figure 7] FIG. 7 is a flowchart showing the flow of information processing operations of the information processing system in the third embodiment. [Figure 8] FIG. 8 is a conceptual diagram of the information processing operation of the information processing system in the third embodiment. [Figure 9] FIG. 9 is a block diagram showing the configuration of an information processing system according to the fourth embodiment. [Figure 10] FIG. 10 is a flowchart showing the flow of information processing operations of the information processing system according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of an information processing system, an information processing method, and a recording medium will be described with reference to the drawings. [1: First embodiment]

[0010] An information processing system, an information processing method, and a recording medium according to a first embodiment will be described below. In the following, the information processing system, the information processing method, and the recording medium according to the first embodiment will be described using an information processing system S1 to which the information processing system, the information processing method, and the recording medium according to the first embodiment are applied. [1-1: Configuration of information processing system S1]

[0011] Fig. 1 is a block diagram showing the configuration of an information processing system S1 in the first embodiment. As shown in Fig. 1, the information processing system S1 includes an acquisition unit 11, a storage unit 12, an extraction unit 13, and an authentication unit 14.

[0012] The acquisition unit 11 sequentially acquires face information of face regions detected from sequentially captured images. The storage unit 12 sequentially stores the acquired face information. When a predetermined condition is satisfied, the extraction unit 13 performs an extraction operation to extract features from each of at least a portion of the face information to be extracted that is stored in the storage unit 12. The authentication unit 14 performs face authentication using the features. [1-2: Technical Effects of Information Processing System S1]

[0013] When a predetermined condition is satisfied, the information processing system S1 in the first embodiment extracts features from at least a portion of the facial information to be extracted that is stored in the storage unit 12. That is, before the extraction operation, the facial information is temporarily stored in the storage unit 12, thereby enabling batch processing of a large amount of facial information. This increases the number of feature extractions per unit time, thereby improving throughput. [2: Second embodiment]

[0014] Next, a second embodiment of the information processing system, the information processing method, and the recording medium will be described. Hereinafter, the second embodiment of the information processing system, the information processing method, and the recording medium will be described using an information processing system S2 to which the second embodiment of the information processing system, the information processing method, and the recording medium is applied.

[0015] The information processing system S2 in the second embodiment may be applied to a situation where a plurality of subjects P need to be authenticated one after another.

[0016] 2 is a block diagram of an information processing system S2 according to the second embodiment. As shown in FIG. 2, the information processing system S2 according to the second embodiment may include an information processing device 2 and an imaging device C. [2-1: Configuration of information processing device 2]

[0017] 2, the information processing device 2 includes a calculation device 21 and a storage device 22. The information processing device 2 may further include a communication device 23, an input device 24, and an output device 25. However, the information processing device 2 does not necessarily include at least one of the communication device 23, the input device 24, and the output device 25. The calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0018] The arithmetic device 21 includes at least a GPU (Graphics Processing Unit). The arithmetic device 21 may further include, for example, at least one of a CPU (Central Processing Unit) and an FPGA (Field Programmable Gate Array). The arithmetic device 21 reads a computer program. For example, the arithmetic device 21 may read a computer program stored in the storage device 22. For example, the arithmetic device 21 may read a computer program stored in a computer-readable, non-transitory recording medium using a recording medium reading device (e.g., an input device 24 described later) not shown in the drawings that is included in the information processing device 2. The arithmetic device 21 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the information processing device 2 via the communication device 23 (or another communication device). The arithmetic device 21 executes the read computer program. As a result, logical functional blocks for executing operations to be performed by the information processing device 2 are realized within the arithmetic device 21. That is, the arithmetic device 21 can function as a controller for realizing logical functional blocks for executing the operations (in other words, processing) that the information processing device 2 should perform.

[0019] FIG. 2 shows an example of logical functional blocks implemented within the arithmetic device 21 to perform information processing operations. As shown in FIG. 2, the arithmetic device 21 implements an acquisition unit 211, which is a specific example of "acquisition means" described in the appendix, a face information processing unit 212, an extraction unit 213, which is a specific example of "extraction means" described in the appendix, and an authentication unit 214, which is a specific example of "authentication means" described in the appendix. The acquisition unit 211 may include an image receiving unit 2111 and a face area detection unit 2112. The face information processing unit 212 may include a memory control unit 2121. Details of the respective operations of the image receiving unit 2111, the face area detection unit 2112, the memory control unit 2121, the extraction unit 213, and the authentication unit 214 will be described later with reference to FIGS. 3 and 4.

[0020] The storage device 22 can store desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic device 21. The storage device 22 may temporarily store data temporarily used by the arithmetic device 21 when the arithmetic device 21 is executing a computer program. The storage device 22 may store data to be stored long-term by the information processing device 2. The storage device 22 may include at least one of a random access memory (RAM), a read-only memory (ROM), a hard disk device, a magneto-optical disk device, a solid-state drive (SSD), and a disk array device. In other words, the storage device 22 may include a non-transitory recording medium. The storage device 22 may include a face information storage unit 221, a feature storage unit 222, and a registered information database DB, which are specific examples of the "storage means" described in the appendix below. However, the storage device 22 does not necessarily include at least one of the face information storage unit 221, the feature storage unit 222, and the registered information database DB. In this case, at least one of the face information storage unit 221, the feature amount storage unit 222, and the registration information database DB may be realized in another device.

[0021] The communication device 23 is capable of communicating with devices external to the information processing device 2 via a communication network (not shown). The information processing device 2 may transmit and receive signals to and from the imaging device C via the communication device 23.

[0022] The input device 24 is a device that accepts information input to the information processing device 2 from outside the information processing device 2. For example, the input device 24 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the information processing device 2. For example, the input device 24 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the information processing device 2.

[0023] The output device 25 is a device that outputs information to the outside of the information processing device 2. For example, the output device 25 may output information as an image. That is, the output device 25 may include a display device (a so-called display) that can display an image showing the information to be output. For example, the output device 25 may output information as sound. That is, the output device 25 may include an audio device (a so-called speaker) that can output sound. For example, the output device 25 may output information on paper. That is, the output device 25 may include a printing device (a so-called printer) that can print desired information on paper. [2-2: Information processing operation performed by information processing system S2]

[0024] Next, the information processing operation performed by the information processing system S2 in the second embodiment will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a flowchart showing the flow of the information processing operation performed by the information processing system S2 in the second embodiment. Fig. 3(a) shows the flow of the storage operation of face information FI, and Fig. 3(b) shows the flow of the extraction operation and authentication operation. The storage operation of face information FI shown in Fig. 3(a) and the extraction operation and face authentication operation shown in Fig. 3(b) may be performed in parallel. Fig. 4 is a conceptual diagram of the information processing operation performed by the information processing system S2 in the second embodiment.

[0025] 3(a), the image receiving unit 2111 acquires a captured image I (step S20). The imaging device C may sequentially capture the captured images I. The image receiving unit 2111 acquires the captured images I sequentially captured from the imaging device C via the communication device 23.

[0026] Note that the image receiving unit 2111 sequentially acquires captured images I from the imaging device C, but the operations from "start" to "end" shown in Fig. 3(a) may be operations related to one captured image I. For example, if the imaging device C is capturing a moving image, the operations from "start" to "end" shown in Fig. 3(a) may be operations for each frame.

[0027] The face area detection unit 2112 detects a face area F from the captured image I (step S21). The face area detection unit 2112 receives the captured image I from the image receiving unit 2111, and detects a face area F included in the captured image I. For example, when the image receiving unit 2111 acquires the captured image I illustrated in FIG. 4(a), the face area detection unit 2112 may detect a face area F3, a face area F4, a face area F5, and a face area F6.

[0028] The storage control unit 2121 sequentially acquires the face information FI of the detected face area F and stores it sequentially in the face information storage unit 221 (step S22). The face information storage unit 221 may be a storage unit that temporarily accumulates each piece of face information FI before batch processing by the extraction unit 213, which will be described later. The face information FI may include a face image of the face area F cut out from the captured image I. For example, when the face information storage unit 221 stores face information FI1 and face information FI2 as illustrated in FIG. 4(b), if the image receiving unit 2111 acquires the captured image I as illustrated in FIG. 4(a), the storage control unit 2121 may sequentially store face information FI3, face information FI4, face information FI5, and face information FI6 in the face information storage unit 221 as illustrated in FIG. 4(c). When the face information storage unit 221 stores face information FI of the subject P corresponding to the detected face information FI, the storage control unit 2121 may delete the face information FI of the subject P stored in the face information storage unit 221. For example, when face information FI1 and face information FI6 are face information of the same person, the storage control unit 2121 may delete the face information FI1 from the face information storage unit 221. That is, for face information FI of the same person, the storage control unit 2121 may discard older face information FI in order to store the latest face information FI. The storage control unit 2121 may determine whether the face information FI is of the same person using a technology such as a face tracking function, for example.

[0029] The face area detection unit 2112 determines whether all face areas F have been detected (step S23). If all face areas F have been detected (step S23: Yes), the operation for one captured image I ends. If all face areas F have not been detected (step S23: No), the process returns to step S21. In other words, if the captured image I includes multiple face areas F, the face area detection unit 2112 may detect all face areas F.

[0030] 3(b), it is determined whether the number of pieces of face information FI stored in the face information storage unit 221 has reached a predetermined number (step S24). The face information processing unit 212 determines whether the predetermined number of pieces of face information FI has been accumulated in the face information storage unit 221. If the number of pieces of face information FI stored in the face information storage unit 221 has reached the predetermined number (step S24: Yes), the process proceeds to step S26.

[0031] If the number of pieces of face information FI stored in the face information storage unit 221 has not reached the predetermined number (step S24: No), the face information processing unit 212 determines whether or not a predetermined time has elapsed since the previous extraction operation by the extraction unit 213 (step S25). If the predetermined time has not elapsed since the previous extraction operation by the extraction unit 213 (step S25: No), the process returns to step S24.

[0032] If a predetermined time has passed since the previous extraction operation by the extraction unit 213 (step S25: Yes), the process proceeds to step S26. That is, the process proceeds to step S26 in at least one of the following cases: the number of pieces of face information FI stored in the face information storage unit 221 reaches a predetermined number, and a predetermined time has passed since the previous extraction operation by the extraction unit 213. In other words, if the predetermined number of pieces of face information FI have not been accumulated, the information processing system S2 waits for the predetermined time to pass and then proceeds to step S26.

[0033] If the predetermined condition is satisfied (step S24: Yes or step S25: Yes), the face information processing unit 212 instructs the extraction unit 213 to perform extraction. The face information processing unit 212 may input at least a portion of the face information FI to be extracted from the face information FI stored in the face information storage unit 221 in a batch to the extraction unit 213. The face information processing unit 212 may input a predetermined number of pieces of face information FI in a batch. The face information processing unit 212 may instruct the extraction unit 213 to perform batch processing using GPU resources.

[0034] The extraction unit 213 performs an extraction operation to extract features from at least some of the face information FI to be extracted from the face information storage unit 221 (step S26). The extraction unit 213 may perform the extraction operation using GPU resources. The extraction unit 213 may receive multiple pieces of face information FI from the face information storage unit 221 via the face information processing unit 212 and collectively extract facial feature amounts FF from the face information FI using the GPU resources. The facial feature amounts FF extracted by the extraction unit 213 may be stored in the feature amount storage unit 222. The number of pieces of face information FI received by the extraction unit 213 from the face information storage unit 221 via the face information processing unit 212 may be a predetermined number. For example, if the predetermined number is "5," the face information processing unit 212 may determine face information FI1, face information FI2, face information FI3, face information FI4, and face information FI5 as at least some of the face information FI to be extracted from the face information storage unit 221, as illustrated in FIG. 4(d). In this case, as illustrated in Fig. 4(e), the feature amount storage unit 222 may store the collectively extracted facial feature amounts FF1, FF2, FF3, FF4, and FF5 through the extraction operation of the extraction unit 213. Here, the facial feature amount FF1 may be the facial feature amount FF extracted from the face image included in the face information FI1, the facial feature amount FF2 may be the facial feature amount FF extracted from the face image included in the face information FI2, the facial feature amount FF3 may be the facial feature amount FF extracted from the face image included in the face information FI3, the facial feature amount FF4 may be the facial feature amount FF extracted from the face image included in the face information FI4, and the facial feature amount FF5 may be the facial feature amount FF extracted from the face image included in the face information FI5. Furthermore, when the extraction operation is performed, the memory control unit 2121 may perform a deletion operation to delete each of the facial information FI to be extracted stored in the face information storage unit 221.

[0035] The authentication unit 214 performs face authentication using the facial feature amount FF (step S27). The authentication unit 214 may sequentially acquire the facial feature amount FF from the feature amount storage unit 222. The authentication unit 214 may perform face authentication by comparing the facial feature amount FF stored in the feature amount storage unit 222 with information registered in the registration information database DB. For example, if the information processing system S2 in the second embodiment is applied to gate opening and closing control, the gate may be sequentially opened and closed in response to successful authentication.

[0036] The authentication unit 214 determines whether face authentication has been performed for all the extracted facial feature amounts FF (step S28). If face authentication has been performed for all the extracted facial feature amounts FF (step S28: Yes), or if a predetermined condition is met (step S24 or step S25: Yes), the operation ends. If face authentication has not been performed for all the extracted facial feature amounts FF (step S28: No), the process returns to step S27. That is, if a plurality of facial feature amounts FF are extracted in step S26, the authentication unit 214 performs face authentication for all the facial feature amounts FF in sequence.

[0037] The extraction unit 213 performs the extraction operation using GPU resources, but at least one of the acquisition unit 211, face information processing unit 212, and authentication unit 214 may also perform their operation using GPU resources. At least one of the acquisition unit 211, face information processing unit 212, and authentication unit 214 may also perform their operation using at least one of GPU resources, CPU resources, and FPGA resources. [2-4: Technical Effects of Information Processing System S2]

[0038] The information processing system S2 in the second embodiment extracts features from each of at least some of the face information FI to be extracted from the face information FI stored in the face information storage unit 221 when at least one of the following occurs: the number of face information FI stored in the face information storage unit 221 reaches a predetermined number; or a predetermined time has elapsed since the previous extraction operation by the extraction unit 213. Furthermore, since the extraction unit 213 uses GPU resources for the extraction operation, it is possible to process a large number of pieces of face information FI in a batch. That is, since the face information FI is temporarily stored in the face information storage unit 221 before the extraction operation by the extraction unit 213, it is possible to process a large number of pieces of face information FI in a batch. This increases the number of cases processed in a single extraction operation, increases the number of cases processed per unit of time, and achieves high throughput. Furthermore, since the extraction processing time per case is shortened, it is possible to reduce the waiting time until the subject P is authenticated, thereby achieving a short response time. In a device that performs authentication, when the facial feature FF extraction operation is performed using GPU resources, the overall cost can be reduced compared to when the facial feature FF extraction operation is performed using CPU resources. [3: Third embodiment]

[0039] Next, a third embodiment of the information processing system, the information processing method, and the recording medium will be described. Hereinafter, the third embodiment of the information processing system, the information processing method, and the recording medium will be described using an information processing system S3 to which the third embodiment of the information processing system, the information processing method, and the recording medium is applied.

[0040] The information processing system S3 in the third embodiment may be applied to situations where it is necessary to authenticate multiple subjects P one after another, as well as situations where it is not clear until the last moment which subject P to authenticate first, and situations where the order of authentication is reversed. Hereinafter, the third embodiment will be described by taking as an example a case where the authentication result by the information processing system S3 is applied to the opening / closing control of a gate device G through which a target person P can pass. [3-1: Overall configuration of information processing system S3]

[0041] 5 is a conceptual diagram of an information processing system S3 in the third embodiment. As shown in FIG. 5, the information processing system S3 in the third embodiment may include an information processing device 3 and an imaging device C.

[0042] The gate device G is a device capable of controlling the passage of subjects P1, P2, P3, P4, P5, and P6 (referred to as subjects P when not distinguished). The gate device G may be opened or closed depending on the authentication result by the information processing system S3. The gate device G may include multiple flapper gates FG. The subject P may be able to pass through at least one of the multiple flapper gates FG. An imaging device C may be provided for each flapper gate FG. For example, an imaging device C1 provided near the flapper gate FG1 may be able to capture an image of the vicinity of the flapper gate FG1. Furthermore, an imaging device C2 provided near the flapper gate FG2 may be able to capture an image of the vicinity of the flapper gate FG2. Furthermore, an imaging device C3 provided near the flapper gate FG3 may be able to capture an image of the vicinity of the flapper gate FG3. The distance between the imaging device C and the subject P may correspond to the distance between the flapper gate FG and the subject P.

[0043] The second area A2 is a destination of the target person P, and may be an area that only authenticated target person P can enter. For the authenticated target person P, the flapper gate FG is controlled to open, and the target person P can move from the first area A1 to the second area A2 through the flapper gate FG.

[0044] The flapper gate FG is a member capable of controlling the passage of the subject P. Although a plate-shaped member is illustrated as the flapper gate FG, a rod-shaped member may also be used. The state of the flapper gate FG may be controlled by the information processing device 3 based on the information processing result of the subject P by the information processing device 3. Specifically, if the authentication of the subject P by the information processing device 3 is successful (i.e., it is determined that the subject P matches a registered person), the state of the flapper gate FG may be controlled by the information processing device 3 to be in an open state that allows the subject P to pass through the flapper gate FG. The registered person may be, for example, a person who is permitted to enter the second area A2. On the other hand, if the authentication of the subject P by the information processing device 3 is unsuccessful (i.e., it is determined that the subject P does not match a registered person), the state of the flapper gate FG may be controlled by the information processing device 3 to be in a closed state that prevents the subject P from passing through the flapper gate FG. [3-2: Configuration of Information Processing System S3]

[0045] The configuration of an information processing system S3 in the third embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing the configuration of the information processing system S3 in the third embodiment. As shown in Fig. 6, the information processing system S3 in the third embodiment may include an information processing device 3 and an imaging device C, similar to the information processing system S2 in the second embodiment.

[0046] As shown in FIG. 6 , the information processing device 3 in the third embodiment includes a calculation device 21 and a storage device 22, similar to the information processing device 2 in the second embodiment. Furthermore, the information processing device 3 in the third embodiment may include a communication device 23, an input device 24, and an output device 25, similar to the information processing device 2 in the second embodiment. However, the information processing device 3 does not necessarily include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 3 in the third embodiment differs from the information processing device 2 in the second embodiment in that the acquisition unit 211 included in the calculation device 21 further includes a distance calculation unit 3113, and the face information processing unit 212 included in the calculation device 21 further includes a priority determination unit 3122 and a selection unit 3123. Other features of the information processing device 3 may be the same as the other features of the information processing device 2 in the second embodiment. Therefore, hereinafter, differences from the embodiments already described will be described in detail, and descriptions of other overlapping features will be omitted as appropriate. [3-3: Information processing operations performed by information processing system S3]

[0047] As shown in FIG. 7, the information processing operation performed by the information processing system S3 differs from the information processing operation performed by the information processing system S2 shown in FIG. 3 in the operations in steps S21, S22, and S26.

[0048] 7(a), in step S21, the face area detection unit 2112 detects face areas F from the captured image I (step S211). The face area detection unit 2112 may detect the face areas F included in the captured image I in descending order of size.

[0049] The face area detection unit 2112 acquires time information indicating the time related to the detection of the face area F (step S212). The time is, for example, the time of day or the date and time, and the time information is, for example, time information or date and time information. The face area detection unit 2112 may detect the time when the face area F was detected from the captured image I. Alternatively, the face area detection unit 2112 may detect the time when the captured image I was acquired. Alternatively, the face area detection unit 2112 may detect the time when the captured image I was captured.

[0050] The distance calculation unit 3113 calculates distance information (step S213). The distance information is information indicating the distance from the location where images are sequentially captured to the location of the face area F. That is, the distance calculation unit 3113 calculates the distance between the image capture device C and the subject P corresponding to the face area F. The distance calculation unit 3113 may calculate the distance between the image capture device C and the subject P from the size of the face area F, the distance between the eyes included in the face area F, a comparison result with the size of a comparison object in the captured image I, etc. Alternatively, the distance calculation unit 3113 may calculate the distance between the image capture device C and the subject P from the measurement result by a distance measurement sensor.

[0051] In the third embodiment, the face information FI may include at least a face image, time information, and distance information of the face area F. The time information and distance information may be called meta information.

[0052] 7(b), in step S22, the storage control unit 2121 stores the face information FI in the face information storage unit 221 (step S221). That is, the storage control unit 2121 stores time information and distance information in association with the face image in the face information storage unit 221. The storage control unit 2121 may store in the face information storage unit 221 face information FI of the same subject P detected from captured images I captured by multiple image capture devices C in association with each other.

[0053] The priority determination unit 3122 determines the priority of the face information FI stored in the face information storage unit 221 according to the time information and the distance information (step S222). The priority determination unit 3122 may determine whether the subject P corresponding to the face information FI newly stored in the face information storage unit 221 is closer to the image capture device C than the subject P corresponding to the already-stored face information FI according to the time information and the distance information. When the priority determination unit 3122 determines that the subject P corresponding to the new face information FI is closer to the image capture device C than the subject P corresponding to the already-stored face information FI, it may increase the priority of the new face information FI.

[0054] 8(a), the face information storage unit 221 stores face information FI1, face information FI2, face information FI3, and face information FI4, and the image receiving unit 2111 acquires a captured image I including face areas F5, F6, F7, and F8 shown in FIG. 8(b). It is assumed that the face information FI1, face information FI2, face information FI3, and face information FI4 are detected in the following order: FI1, FI2, FI3, FI4. It is also assumed that the size of the face areas F corresponding to the face information FI1, FI2, FI3, and FI4 is equal to the size of the face area F5, and that the face area F6 is larger than the face area F5, the face area F7 is larger than the face area F6, and the face area F8 is larger than the face area F7. In this case, the priority determination unit 3122 may determine the priorities of the face information FI8 corresponding to the face area F8, the face information FI7 corresponding to the face area F7, and the face information FI6 corresponding to the face area F6 to be lower than the priority of the face information FI1 detected earlier, and higher than the priority of the face information FI2, the face information FI3, and the face information FI4. In Fig. 8, the face information FI with higher priority may be located higher and further to the left in the face information storage unit 221. That is, Fig. 8(c) may illustrate a case where the priority is highest in the order of face information FI1, face information FI8, face information FI7, face information FI6, face information FI2, face information FI3, face information FI4, and face information FI5.

[0055] 7(c), in step S26, the selection unit 3123 selects face information FI (step S261). Based on the priority determined in step S222, the selection unit 3123 selects face information FI to be included in the extraction target from the face information FI stored in the face information storage unit 221. For example, in the example shown in FIG. 8(c), the selection unit 3123 may select face information FI1, face information FI8, face information FI7, face information FI6, and face information FI2 stored in the face information storage unit 221 as the extraction target.

[0056] The extraction unit 213 detects a facial feature amount FF from each piece of face information FI selected by the selection unit 3123 in response to an instruction from the face information processing unit 212 (step S262). For example, as illustrated in Fig. 8(d), the feature amount storage unit 222 may store a facial feature amount FF1, a facial feature amount FF8, a facial feature amount FF7, a facial feature amount FF6, and a facial feature amount FF2 extracted from each piece of face information FI1, face information FI8, face information FI7, face information FI6, and face information FI2.

[0057] After step S26, similarly to the information processing operation in the second embodiment, the authentication unit 214 performs face authentication processing using the facial feature amount FF. If the authentication is successful, the information processing device 3 may control to open the gate corresponding to the successfully authenticated facial feature amount FF. [3-4: Technical Effects of Information Processing System S3]

[0058] In the information processing system S3 of the third embodiment, when face information FI corresponding to a subject P who is close to the imaging device C is detected later, the priority of the face information FI is increased so that the face information FI is subjected to an extraction operation first. That is, the priority of the face information FI corresponding to the subject P who should be authenticated first is increased so that the facial feature amount FF is extracted first. This can shorten the perceived time from the subject P's arrival at the location to be authenticated to the completion of authentication. When applied to a gate device G, the perceived time from the subject P's arrival at the gate to the unlocking can be shortened. [4: Fourth embodiment]

[0059] Next, a fourth embodiment of the information processing system, the information processing method, and the recording medium will be described. Hereinafter, the fourth embodiment of the information processing system, the information processing method, and the recording medium will be described using an information processing system S4 to which the fourth embodiment of the information processing system, the information processing method, and the recording medium is applied.

[0060] The information processing system S4 in the fourth embodiment, like the information processing system S3 in the third embodiment, may be applied to situations where it is necessary to authenticate multiple subjects P one after another, and it is not clear until the last moment which subject P to authenticate first, or where the order of authentication is reversed. In the fourth embodiment, similarly to the information processing system S3 in the third embodiment, the authentication result by the information processing system S4 may be applied to the opening / closing control of the gate apparatus G through which the target person P can pass. [4-1: Configuration of information processing system S4]

[0061] The configuration of an information processing system S4 in the fourth embodiment will be described with reference to Fig. 9. Fig. 9 is a block diagram showing the configuration of the information processing system S4 in the fourth embodiment. As shown in Fig. 9, the information processing system S4 in the fourth embodiment may include an information processing device 4 and an imaging device C, similar to the information processing system S2 in the second embodiment and the information processing system S3 in the third embodiment.

[0062] 9 , the information processing device 4 in the fourth embodiment includes a calculation device 21 and a storage device 22, similar to the information processing device 2 in the second embodiment and the information processing device 3 in the third embodiment. Furthermore, the information processing device 4 in the fourth embodiment may include a communication device 23, an input device 24, and an output device 25, similar to the information processing device 2 in the second embodiment and the information processing device 3 in the third embodiment. However, the information processing device 4 does not have to include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 4 in the fourth embodiment differs from the information processing device 2 in the second embodiment and the information processing device 3 in the third embodiment in that the face information processing unit 212 included in the calculation device 21 further includes a first change unit 4124, a second change unit 4125, and a third change unit 4126.

[0063] The first change unit 4124 changes the predetermined number according to the rate of change in the number of pieces of face information FI stored in the face information storage unit 221. The second change unit 4125 changes the predetermined time according to the time from when the previous extraction operation was performed by the extraction unit 213 until the number of pieces of face information FI stored in the face information storage unit 221 reaches the predetermined number. The third change unit 4126 compares the time from the start to the completion of the extraction operation with the predetermined time, and changes the imaging interval corresponding to the imaging interval for sequential imaging.

[0064] In the fourth embodiment, the predetermined number and the predetermined time are changed depending on the situation. The predetermined number and the predetermined time each have a settable lower limit and a settable upper limit.

[0065] Other features of the information processing device 4 may be the same as other features of the information processing device 2 in the second embodiment or the information processing device 3 in the third embodiment. Therefore, in the following, only the parts that differ from the embodiments already described will be described in detail, and descriptions of other overlapping parts will be omitted as appropriate. [4-2: Information processing operation performed by information processing system S4]

[0066] An information processing operation performed by the information processing system S4 in the fourth embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the flow of the information processing operation performed by the information processing system S4 in the fourth embodiment. The information processing operation shown in Fig. 10 may be an operation that is performed periodically at predetermined intervals.

[0067] The face information processing unit 212 monitors the rate of change in the number of pieces of face information FI stored in the face information storage unit 221. The face information processing unit 212 also monitors the time from the previous extraction operation performed by the extraction unit 213 until the number of pieces of face information FI stored in the face information storage unit 221 reaches a predetermined number.

[0068] 10, the face information processing unit 212 determines whether the time until the number of pieces of face information FI accumulated in the face information storage unit 221 reaches a predetermined number tends to be short, tends to be long, or is neither short nor long (step S40). For example, the face information processing unit 212 may make the determination by comparing with the average of the accumulation tendency of the face information FI over the past five minutes.

[0069] If it tends to take a long time for the number of pieces of face information FI accumulated in the face information storage unit 221 to reach the predetermined number (step S40:L), the first change unit 4124 reduces the predetermined number. Also, the second change unit 4125 shortens the predetermined time (step S41). By reducing the predetermined number and shortening the predetermined time, it is possible to prevent the waiting time until accumulation from being wasted.

[0070] The face information processing unit 212 may dynamically change and optimize at least one of the predetermined number and the predetermined time so as to maintain the relationship of the following formula. (Time to accumulate to a certain number + α) <= Time until a certain time has elapsed That is, the predetermined time may be dynamically changed and optimized so that the time difference between the time when the predetermined number is accumulated and the time when the predetermined time has elapsed falls within a certain range (α).

[0071] The face information processing unit 212 determines whether at least one of the predetermined number and the predetermined time has reached its settable lower limit (step S42). If at least one of the predetermined number and the predetermined time has reached its lower limit (step S42: Yes), it is predicted that the load on the entire device is lower than expected. Therefore, the third change unit 4126 changes the image capture interval to be shorter (step S43). The image receiving unit 2111 may gradually adjust the image capture interval of the image capture device C to the image capture interval changed by the third change unit 4126.

[0072] Furthermore, the third change unit 4126 compares the time from the start to the completion of the extraction operation with a predetermined time, and changes the imaging interval of the sequential imaging. That is, the third change unit 4126 may compare the time corresponding to the predetermined number with the predetermined time, and change the imaging interval of the sequential imaging.

[0073] Furthermore, the third change unit 4126 may change the imaging interval of the imaging device C according to congestion information corresponding to the imaging device C. The face area detection unit 2112 may calculate congestion information. The congestion information is information indicating the degree of congestion around the imaging device C when the face information FI is detected. The face area detection unit 2112 may calculate the congestion information from the number of detected face areas F included in the captured image I. The third change unit 4126 may change the imaging interval of a congested imaging device C by giving priority to increasing the imaging interval. When the area is congested, it is predicted that the subject P will move less, and it is expected that the impact of increasing the imaging interval and reducing the frame rate will be small. By increasing the imaging interval, the load is reduced, while delays are also reduced by prioritizing the processing of the face information FI based on the distance between the subject P and the imaging device C.

[0074] If at least one of the predetermined number and the predetermined time has not reached the lower limit (step S42: No), the information processing operation shown in FIG. 10 ends.

[0075] If the time it takes for the number of pieces of face information FI accumulated in the face information storage unit 221 to reach a predetermined number tends to be short, it can be determined that the system is congested, so it is more efficient to increase the number of items to be batch-processed using GPU resources. Therefore, if the time tends to be short (step S40:S), the first change unit 4124 increases the predetermined number. Furthermore, the second change unit 4125 lengthens the predetermined time (step S45). If the time it takes to reach the predetermined number tends to be short, it can be determined that the system is congested, so the process proceeds to step S45, where the predetermined number and the predetermined time are increased to increase the number of items to be batch-processed, thereby improving efficiency.

[0076] The face information processing unit 212 determines whether or not at least one of the predetermined number and the predetermined time has reached its upper limit (step S46). If the upper limit has been reached, it is predicted that the load on the entire device is higher than expected. If at least one of the predetermined number and the predetermined time has reached its upper limit (step S46: Yes), the third change unit 4126 lengthens the image capturing interval (step S47). The third change unit 4126 may instruct the image capturing device C to gradually lengthen the image capturing interval. If at least one of the predetermined number and the predetermined time has not reached its upper limit (step S46: No), the process proceeds to step S44.

[0077] In step S40, if there is neither a short-term trend nor a long-term trend (step S40:M), the face information processing unit 212 determines whether the load of at least one of the GPU resource and the CPU resource is high (step S44). For example, the trend of the usage rate of at least one of the GPU resource and the CPU resource may be compared with the average value for the past five minutes.

[0078] If the load on at least one of the GPU resource and the CPU resource is high (step S44: Yes), the third change unit 4126 changes the imaging interval to be longer (step S47). The image receiving unit 2111 may gradually adjust the imaging interval of the imaging device C to the imaging interval changed by the third change unit 4126.

[0079] If the load on at least one of the GPU resource and the CPU resource is not high (step S44: No), the process proceeds to step S43.

[0080] The face information processing unit 212 periodically monitors the trend in the number of pieces of face information FI stored in the face information storage unit 221, and adjusts the predetermined number and predetermined time in the face information storage unit 221 according to the load situation before batch processing by the extraction unit 213. In this way, control is performed to achieve optimal throughput and response for the environment. [4-3: Technical Effects of Information Processing System S4]

[0081] In the information processing system S4 according to the fourth embodiment, the first change unit 4124 dynamically changes the predetermined number depending on the load status, thereby enabling control to optimize throughput and response. When the first change unit 4124 lowers the predetermined number, extraction operations using GPU resources can be started immediately. When the first change unit 4124 increases the predetermined number, batch extraction operations using GPU resources can be more effectively utilized. Furthermore, the second change unit 4125 dynamically changes the predetermined time depending on the load status, thereby enabling control to optimize throughput and response. Furthermore, the third change unit 4126 dynamically changes the imaging interval depending on the load status, thereby enabling control to optimize throughput and response. In particular, when the imaging interval is lengthened, significant variation in the distance between the subject P and the imaging device C occurs depending on the timing at which the face area F is detected by the face area detection unit 2112. Therefore, changing the imaging interval is highly effective when, as in the third embodiment, the face information processing unit 212 performs a process of changing the priority so that face information FI corresponding to a subject P who is close to the imaging device C can be processed first when it is later stored in the face information storage unit 221. This is particularly effective when the number of pieces of face information FI stored in the face information storage unit 221 exceeds a predetermined number of pieces of face information to be processed collectively by GPU resources. [5: Note]

[0082] The following additional notes are provided regarding the above-described embodiment. [Appendix 1] an acquisition means for sequentially acquiring face information of a face area detected from sequentially captured images; a storage means for sequentially storing the acquired face information; an extraction unit that performs an extraction operation to extract features from each of the face information to be extracted, at least a portion of the face information stored in the storage unit, when a predetermined condition is satisfied; an authentication means for performing face authentication using the feature amount; An information processing system comprising: [Appendix 2] the extraction means uses a GPU resource for the extraction operation; The face information includes a face image of the face region. 2. The information processing system according to claim 1. [Appendix 3] The case where the predetermined condition is satisfied includes at least one of the case where the number of pieces of face information stored in the storage means reaches a predetermined number, and the case where a predetermined time has elapsed since the previous extraction operation by the extraction means. 3. The information processing system according to claim 1 or 2. [Appendix 4] The apparatus further includes a first change unit that changes the predetermined number in accordance with a change rate of the number of pieces of face information stored in the storage unit. 4. The information processing system according to claim 3. [Appendix 5] The apparatus further includes a second change unit that changes the predetermined time period in accordance with the time period from when the extraction unit performed the previous extraction operation until the number of pieces of face information stored in the storage unit reaches the predetermined number. 5. The information processing system according to claim 3 or 4. [Appendix 6] The imaging device further includes a third change unit that compares the time from the start to the completion of the extraction operation with the predetermined time and changes the imaging interval of the sequential imaging. 6. The information processing system according to claim 3. [Appendix 7] the face information includes a face image of the face area, time information indicating a time related to detection of the face area, and distance information indicating a distance from the location where the sequential images are captured to the location of the face area; a face information processing unit that determines a priority of the face information stored in the storage unit according to the time information and the distance information, and selects face information to be included in the extraction target from the face information stored in the storage unit based on the priority. 7. The information processing system according to claim 6. [Appendix 8] Sequentially acquiring face information of face areas detected from sequentially captured images; The acquired face information is sequentially stored in a storage means; When a predetermined condition is satisfied, an extraction operation is performed to extract a feature amount from each of the face information to be extracted, which is at least a part of the face information stored in the storage means; Perform face recognition using the features Information processing methods. [Appendix 9] On the computer, Sequentially acquiring face information of face areas detected from sequentially captured images; The acquired face information is sequentially stored in a storage means; When a predetermined condition is satisfied, an extraction operation is performed to extract a feature amount from each of the face information to be extracted, which is at least a part of the face information stored in the storage means; Perform face recognition using the features A recording medium on which a computer program for executing an information processing method is recorded.

[0083] At least some of the constituent elements of each of the above-described embodiments can be appropriately combined with at least some of the other constituent elements of each of the above-described embodiments. Some of the constituent elements of each of the above-described embodiments may not be used. Furthermore, to the extent permitted by law, the disclosures of all documents (e.g., published patent applications) cited in this disclosure are incorporated by reference as part of the description of this disclosure.

[0084] This disclosure may be modified as appropriate within the scope of the claims and the technical idea that can be read from the entire specification. Information processing systems, information processing methods, and recording media that involve such modifications are also included in the technical idea of ​​this disclosure. [Explanation of symbols]

[0085] S1, S2, S3, S4 Information Processing System 2,3,4 Information processing equipment 11,211 Acquisition Department 2111 Image receiving unit 2112 Face area detection unit 12 Storage section 212 Face information processing section 2121 Memory control unit 221 Face information memory unit 222 Feature memory unit 13,213 Extraction part 14,214 Authentication Department 3113 Distance calculation unit 3122 Priority determination unit 3123 Selection Section 4124 First Change Section 4125 Second Change Section 4126 Third Change Section C. Imaging device I Captured image F face area FI Facial Information FF facial features

Claims

1. an acquisition means for sequentially acquiring face information of a face area detected from sequentially captured images; a storage means for sequentially storing the acquired face information; an extraction unit that performs an extraction operation to extract features from each of the face information to be extracted, at least a portion of the face information stored in the storage unit, when a predetermined condition is satisfied; an authentication means for performing face authentication using the feature amount; Equipped with The case where the predetermined condition is satisfied includes at least one of the case where the number of pieces of face information stored in the storage means reaches a predetermined number, and the case where a predetermined time has elapsed since the previous extraction operation by the extraction means, The image forming apparatus further includes a first change unit that changes the predetermined number in accordance with a change rate of the number of pieces of face information stored in the storage unit. Information processing system.

2. An acquisition means for sequentially acquiring face information of a face area detected from sequentially captured images; a storage means for sequentially storing the acquired face information; an extraction unit that performs an extraction operation to extract features from each of the face information to be extracted, at least a portion of the face information stored in the storage unit, when a predetermined condition is satisfied; an authentication means for performing face authentication using the feature amount; Equipped with The case where the predetermined condition is satisfied includes at least one of the case where the number of pieces of face information stored in the storage means reaches a predetermined number, and the case where a predetermined time has elapsed since the previous extraction operation by the extraction means, The apparatus further includes a second change unit that changes the predetermined time period in accordance with the time period from when the extraction unit performed the previous extraction operation until the number of pieces of face information stored in the storage unit reaches the predetermined number. Information processing system.

3. the extraction means uses GPU resources for the extraction operation; The face information includes a face image of the face region.

3. The information processing system according to claim 1 or 2.

4. The imaging device further includes a third change unit that compares the time from the start to the completion of the extraction operation with the predetermined time and changes the imaging interval of the sequential imaging.

3. The information processing system according to claim 1 or 2.

5. the face information includes a face image of the face area, time information indicating a time related to detection of the face area, and distance information indicating a distance from the location where the sequential images are captured to the location of the face area; a face information processing unit that determines a priority of the face information stored in the storage unit according to the time information and the distance information, and selects face information to be included in the extraction target from the face information stored in the storage unit based on the priority. The information processing system according to claim 4 .

6. Sequentially acquiring face information of a face area detected from sequentially captured images; Sequentially storing the acquired face information in a storage means; performing an extraction operation to extract features from each of the face information to be extracted, which is at least a part of the face information stored in the storage means, when a predetermined condition is satisfied; Performing face authentication using the feature amount Including, The case where the predetermined condition is satisfied includes at least one of the case where the number of pieces of face information stored in the storage means reaches a predetermined number, and the case where a predetermined time has elapsed since the previous extraction operation, The method further includes changing the predetermined number in accordance with a rate of change in the number of pieces of face information stored in the storage means. A computer-implemented information processing method.

7. Sequentially acquiring face information of a face area detected from sequentially captured captured images; Sequentially storing the acquired face information in a storage means; performing an extraction operation to extract features from each of the face information to be extracted, which is at least a part of the face information stored in the storage means, when a predetermined condition is satisfied; Performing face authentication using the feature amount Including, The case where the predetermined condition is satisfied includes at least one of the case where the number of pieces of face information stored in the storage means reaches a predetermined number, and the case where a predetermined time has elapsed since the previous extraction operation, The method further includes changing the predetermined time period depending on the time period from the previous execution of the extraction operation until the number of pieces of face information stored in the storage means reaches the predetermined number. A computer-implemented information processing method.

8. On the computer, Sequentially acquiring face information of a face area detected from sequentially captured images; Sequentially storing the acquired face information in a storage means; performing an extraction operation to extract features from each of the face information to be extracted, which is at least a part of the face information stored in the storage means, when a predetermined condition is satisfied; Performing face authentication using the feature amount Including, The case where the predetermined condition is satisfied includes at least one of the case where the number of pieces of face information stored in the storage means reaches a predetermined number, and the case where a predetermined time has elapsed since the previous extraction operation, The method further includes changing the predetermined number in accordance with a rate of change in the number of pieces of face information stored in the storage means. A computer program for executing an information processing method.

9. A computer comprising: Sequentially acquiring face information of a face area detected from sequentially captured images; Sequentially storing the acquired face information in a storage means; performing an extraction operation to extract features from each of the face information to be extracted, which is at least a part of the face information stored in the storage means, when a predetermined condition is satisfied; Performing face authentication using the feature amount Including, The case where the predetermined condition is satisfied includes at least one of the case where the number of pieces of face information stored in the storage means reaches a predetermined number, and the case where a predetermined time has elapsed since the previous extraction operation, The method further includes changing the predetermined time period depending on the time period from the previous execution of the extraction operation until the number of pieces of face information stored in the storage means reaches the predetermined number. A computer program for executing an information processing method.

Images