Management method and management device for entry and exit

Abstract

One camera is used to detect the entry and exit of a management object with respect to a predetermined area having one entrance and exit. A face image and a whole body image of an arbitrary person passing through an entrance and exit are acquired using a camera image captured by one camera provided near the entrance and exit of a predetermined area. The approach of a management object to a predetermined area is detected by a face authentication process using a face image of an arbitrary person included in a camera image and a registered face image of the management object. A feature amount of a subject to be managed is extracted using a whole body image of the subject to be managed in which entry into a predetermined region is detected. A feature amount of an arbitrary person is extracted using a whole body image of the arbitrary person included in a camera image after a management object enters a predetermined region. A rerecognition process using the extracted feature amount of the object to be managed and the extracted feature amount of the arbitrary person detects a return of the object to be managed from the predetermined region.

Description

Technical Field

[0001] The present disclosure relates to a method and apparatus for managing the entry and exit of people from a regulated area having one entrance and exit. Background Art

[0002] Japanese Unexamined Patent Application Publication No. 2021-152738 discloses an authentication device for a person entering a regulated area having a first door and a second door. This existing authentication device performs facial authentication of a person using a facial image of a first camera disposed near the first door outside the regulated area. When the facial authentication is successful, the lock of the first door is released, and further, the whole body information of the subject person for whom the facial authentication is performed is obtained by the first camera. When obtaining the whole body information of the subject person, an instruction to assume a registration pose in front of the first camera is issued to the person for whom the facial authentication is successful. That is, the whole body information includes a whole body image of a person who has assumed the registration pose.

[0003] When the existing authentication device further obtains the whole body information of a person by a second camera disposed near the second door inside the regulated area, it compares the obtained whole body information with the whole body information already obtained by the first camera. If the person assumes the same pose near the second door as the registration pose assumed near the first door, whole body information including a whole body image of a person who has assumed the authentication pose is obtained. When the similarity between the registration pose and the authentication pose is equal to or greater than a threshold value, the lock of the second door is released.

[0004] As a document indicating the technical level of the technical field related to the present disclosure, in addition to Japanese Unexamined Patent Application Publication No. 2021-152738, Japanese Unexamined Patent Application Publication No. 2010-154134 can also be exemplified.

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2021-152738

[0006] Patent Document 2: Japanese Unexamined Patent Application Publication No. 2010-154134

[0007] Consider a case where the entry and exit of the same person with respect to a regulated area having a certain entrance and exit are managed. As a management method at this time, a method of detecting the entry of a certain management object into the regulated area and then detecting the exit of the management object from the regulated area can be cited. Specifically, a camera is disposed near the entrance and exit to obtain a facial image of a person, and facial authentication using the obtained facial image is performed. Thereby, it is possible to detect the entry of the management object into the regulated area and the exit of the management object from the regulated area.

[0008] However, in the case where there is only one entrance / exit, since the directions of people's movement are opposite when entering and exiting, the orientations of people's faces are almost never the same. Therefore, in face authentication using face images obtained from one camera installed near the entrance / exit, it may not be possible to detect the entry of the same person into a specified area and the exit from the specified area.

[0009] Regarding this point, if two or more cameras are installed near the entrance / exit, this problem can be solved. However, in cases where there are restrictions on the installation positions of the cameras, sometimes only one camera can be installed near the entrance / exit. Therefore, it is desired to develop a technology that uses one camera to detect the entry and exit of a management target with respect to a specified area having only one entrance / exit. Summary of the Invention

[0010] One object of the present disclosure is to provide a technology that can use one camera to detect the entry and exit of a management target with respect to a specified area having one entrance / exit.

[0011] The first aspect of the present disclosure is a method for managing the entry and exit of a management target with respect to a specified area having one entrance / exit, having the following features.

[0012] The above method includes the following steps, namely: using a camera image captured by one camera installed near the above entrance / exit to obtain a face image and a full-body image of any person passing through the above entrance / exit; performing face authentication processing using the face image of any person included in the above camera image and the registered face image of the above management target to detect the entry of the above management target into the above specified area; using the full-body image of the above management target detected to enter the above specified area in the full-body image of any person included in the above camera image to extract the feature amount of the management target; using the full-body image of any person included in the above camera image after the above management target enters the above specified area to extract the feature amount of the any person; and performing re-identification processing using the feature amount of the above management target extracted based on the above camera image and the feature amount of the any person extracted based on the above camera image to detect the exit of the above management target from the above specified area.

[0013] The second aspect of the present disclosure is a device for managing the entry and exit of a management target with respect to a specified area having one entrance / exit, having the following features.

[0014] The above device is equipped with a processor for performing various processes. The above processor is configured to: use the camera image captured by one camera disposed near the above entrance / exit to obtain the facial image and full-body image of any person passing through the above entrance / exit, perform facial authentication processing using the facial image of any person included in the above camera image and the registered facial image of the above management target, thereby detecting the entry of the above management target into the above specified area, use the full-body image of the above management target, which is included in the full-body image of any person in the above camera image and whose entry into the above specified area is detected through the above facial authentication processing, to extract the feature quantity of the management target, use the full-body image of any person included in the above camera image after the above management target enters the above specified area to extract the feature quantity of any person, and perform re-identification processing using the extracted feature quantity of the management target and the extracted feature quantity of any person, thereby detecting the exit of the above management target from the above specified area.

[0015] According to the present disclosure, the facial image and full-body image of any person passing through the entrance / exit are obtained by one camera disposed near the entrance / exit of a specified area. Moreover, facial authentication processing using the obtained facial image of any person is performed to detect the entry of the management target into the specified area. In addition, extraction of the feature quantity using the obtained full-body image of any person and re-identification processing using the feature quantity are performed. The obtained full-body image of any person includes the full-body image of the management target whose entry into the specified area is detected and the full-body image of any person after the management target enters the specified area. Therefore, by performing re-identification processing, the exit of the management target from the specified area is detected. Thus, it is possible to use one camera to implement detection of the entry and exit of the management target to / from a specified area having only one entrance / exit. Description of the Drawings

[0016] Figure 1 It is a diagram for explaining a structural example of the management device according to the first embodiment and a structural example of the specified area to which it is applied.

[0017] Figure 2 It is a diagram for explaining the point of view of the first embodiment.

[0018] Figure 3 It is a diagram for explaining the features of the first embodiment.

[0019] Figure 4 It is a block diagram showing a functional structural example of the management device according to the first embodiment.

[0020] Figure 5 It is a diagram for explaining the features of the second embodiment.

[0021] Figure 6 It is a block diagram showing an example of the functional structure of the management device according to the second embodiment.

[0022] Figure 7 It is a block diagram showing an example of the functional structure of the management device according to the third embodiment.

[0023] Figure 8 It is a diagram for explaining the features of the fourth embodiment.

[0024] Explanation of reference numerals:

[0025] 10... Management device; 11... Processor; 12... Storage device; 20... Specified area; 21... Entrance / exit; 22, 23... Cameras; PS1, PS2, PS3... Any person; PT... Management target; PU... Uncertified person; VD1, VD2... Images; IMF_PS1, IMF_PS3... Facial images; IMB_PS1, IMB_PS2, IMB_PS3... Full-body images. Detailed implementation manners

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In addition, in each figure, the same or corresponding parts are denoted by the same reference numerals, and the description thereof is simplified or omitted.

[0027] 1. First embodiment

[0028] 1-1. Structural example

[0029] Figure 1 It is a diagram for explaining an example of the structure of the management device for entry and exit according to the first embodiment and an example of the structure of the specified area to which it is applied. Figure 1 The management device 10 shown is the management device according to the first embodiment. The management device 10 includes at least one processor 11 and at least one storage device 12. The processor 11 executes various processes. Examples of the processor 11 include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), ASICs (Application Specific Integrated Circuits), an FPGA (Field-Programmable Gate Array), etc. The storage device 12 stores various information. Examples of the storage device 12 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc.

[0030] The management device 10 is configured to be able to communicate with a camera 22 disposed near an entrance / exit 21 in a specified area 20. In the present disclosure, the specified area 20 refers to a space having a certain width. As a space having a certain width, a room provided in a facility (such as a childcare facility or an educational facility) can be cited. A room and a passage connected to the room are also an example of a space having a certain width. A space having a certain width may also include a plurality of rooms. However, in the first embodiment, only one camera 22 is provided at the entrance / exit 21. As the entrance / exit 21, an entrance of a facility, an entrance / exit of a room provided in the facility, and an entrance / exit of a passage connected to the room can be cited. In addition, for the communication line network connecting the management device 10 and the camera 22, a wired and wireless network is used.

[0031] In Figure 1 In the example shown, the camera 22 is disposed inside the specified area 20. The camera 22 is installed, for example, on the top surface or side wall surface of the specified area 20. The pointing direction of the camera 22 is a direction from the inside of the specified area 20 toward the outside. For the shooting range of the camera 22, the entire entrance / exit 21 and the ground near the entrance / exit 21 are included. With such a camera 22, a front image of any person passing through the entrance / exit 21 and entering the specified area 20 can be obtained.

[0032] 1-2. Features of the First Embodiment

[0033] Figure 2 is a diagram for explaining the point of view of the first embodiment. As described with reference to Figure 1 As described above, with the camera 22, a front image of any person entering the specified area 20 can be obtained. Therefore, in the first embodiment, a face image IMF_PS1 of an arbitrary person PS1 is obtained from the camera image constituting the video VD1 obtained by the camera 22. Then, this face image IMF_PS1 is used for face authentication processing. In the face authentication processing, the face image IMF_PS1 is compared with a pre-registered face image IMF_PT. The face image IMF_PT represents the face image of a pre-registered person (hereinafter also referred to as "management target PT"). When the face image IMF_PS1 and the face image IMG_PT match (for example, when the similarity between the two is equal to or greater than a threshold value), it can be detected that the management target PT has entered the specified area 20.

[0034] However, the traveling direction of the person PS1 when passing through the entrance / exit 21 and entering the specified area 20 is exactly opposite to the traveling direction when passing through the entrance / exit 21 and leaving the specified area 20. Therefore, for the purpose of detecting that the management target PT has left the specified area 20 by performing the above facial authentication process, a prescribed action (such as a review action, an action of looking at the camera 22, etc.) for providing a facial image to be compared with the facial image IMF_PT is imposed on the person passing through the entrance / exit 21.

[0035] Therefore, in the first embodiment, in order to detect the situation that the management target PT has left the specified area 20, a person re-identification process is performed. The person re-identification process (Re ID entification processing) is a technique for identifying the same person from among multiple images. In the person re-identification process, feature amounts of a person extracted from the images can be used. This feature amount is also referred to as a ReID feature amount. For example, the extraction of the ReID feature amount is performed by applying a set of bounding boxes to a ReID model based on machine learning, where the above bounding boxes are associated over multiple time steps to represent the same person. In addition, the extraction of the ReID feature amount itself is a well-known technique, and the extraction method applied to the first embodiment is not particularly limited.

[0036] Figure 3 This is a diagram illustrating the features of the first embodiment. In the first embodiment, in order to extract the ReID feature amount of the management target PT, a full-body image IMB_PT of the management target PT detected in the facial authentication process when entering the specified area 20 is determined. The full-body image IMB_PT can be determined based on the camera image of the video VD1 when the management target PT enters the specified area 20 (for example, the passing moment when the management target TP passes through the entrance / exit 21, the moment immediately before or after this passing moment). By extracting the ReID feature amount of the management target PT, the management target PT can be identified.

[0037] In the first embodiment, in addition, the ReID feature amount of an arbitrary person PS2 leaving the specified area 20 is also extracted. Then, a re-identification process is performed, in which the ReID feature amount of the management target PT extracted when the management target PT enters the specified area 20 and the ReID feature amount of the person PS2 extracted after the management target PT enters the specified area 20 (for example, after the passing moment when the management target TP passes through the entrance / exit 21) are used. Thereby, the situation that the management target PT has left the specified area 20 can be detected.

[0038] Thus, according to the first embodiment, in addition to the face authentication process, a person re-identification process is also performed. Therefore, it is possible to use a single camera 22 to detect the entry and exit of the management target PT with respect to the specified area 20 having only one entrance / exit 21.

[0039] 1-3. Example of functional structure

[0040] Figure 4 represents Figure 1 a block diagram showing an example of the functional structure of the management device 10 shown. In Figure 4 the example shown, the management device 10 includes a person detection unit 31, a face detection unit 32, a face authentication unit 33, a face image management unit 34, an entry detection unit 35, a tracking unit 36, a feature quantity extraction unit 37, a feature quantity management unit 38, and an exit detection unit 39. These functions are actually installed, for example, in a circuit or processing circuit including a general-purpose processor, a special-purpose processor, an integrated circuit, ASICs, a CPU, an existing type of circuit, and / or a combination thereof that is programmed to implement these functions.

[0041] Here, the processor includes transistors and other circuits and is regarded as a circuit or processing circuit. The processor may also be a program processor that executes a program stored in a memory. In the present disclosure, a circuit, a unit, and an organization are hardware programmed to implement the described functions or hardware that executes the functions. The hardware may also be all the hardware disclosed in this specification or all the hardware known as hardware programmed to implement the above functions or hardware that executes the above functions. When the hardware is a processor regarded as a circuit type, the circuit, organization, or unit is a combination of the hardware and the software used to constitute the hardware and / or the processor.

[0042] The image VD1 acquired by the camera 22 is input to the person detection unit 31. The person detection unit 31 performs a person detection process of detecting a person PS1 among the camera images of a plurality of time steps included in the image VD1. In the person detection process, a bounding box is assigned to the person PS1 among each camera image. The bounding box indicates the position of the person PS1 detected in the camera image. In the person detection process, information on the bounding box assigned to the person PS1 among each camera image is acquired. The bounding box is assigned to an image of the face part of the person PS1 or an image of the whole body of the person PS1. In addition, the person detection process is a well-known technique, and its method is not particularly limited. For example, YOLOX is applied to the person detection unit 31.

[0043] The information of the bounding box is input from the human detection unit 31 to the face detection unit 32. The face detection unit 32 performs face detection processing for detecting the face image of the person PS1 based on the information of the bounding box. When the image of the face part of the person PS1 is given a bounding box in the human detection processing, in the face detection processing, this bounding box information becomes the flow. When the image of the whole body of the person PS1 is given a bounding box in the human detection processing, in the face detection processing, the image of the face part is extracted from the image of the whole body.

[0044] The face image of the person PS1 (i.e., the face image IMF_SP1) is input from the face detection unit 32 to the face authentication unit 33. The face authentication unit 33 performs face authentication processing using this face image and the face image of the management target TP (i.e., the face image IMF_TP) stored in the face image management unit 34 (e.g., the storage device 12). In the face authentication processing, the face image IMF_SP1 is compared with the face image IMF_TP. When both are consistent, it is determined that the person PS1 and the management target TP are the same person. The face authentication unit 33 sends the verification result to the entry detection unit 35. In the verification result, it includes the judgment information for the face image IMF_SP1 and the attached information of this face image IMF_SP1. When it is determined that the person PS1 is the management target TP, in the judgment information, it includes the identification information of the management target TP. In the attached information, it includes the identification information of the camera 22 that has obtained the face image IMF_SP1, the coordinate information of the face image IMF_SP1, and the timestamp information.

[0045] The verification result from the face authentication unit 33 is input to the entry detection unit 35. The entry detection unit 35 detects the entry of the management target TP based on the judgment information included in the verification result. When the entry of the management target TP is detected, the entry detection unit 35 also outputs the detection information of the entry of the management target TP and the attached information included in the verification result to the feature quantity management unit 38 together.

[0046] The information of the bounding box is input from the human detection unit 31 to the tracking unit 36. The tracking unit 36 performs tracking processing of the person PS1 based on the information of the bounding box. The tracking processing refers to the technology of automatically tracking the same person included in the camera image based on the tracking algorithm. Specifically, in the tracking processing, multiple bounding boxes representing the same person (i.e., the person PS1) in multiple time steps are associated with each other. Thus, information representing the time series of multiple bounding boxes is generated. In addition, the tracking processing itself is a well-known technology, and its method is not particularly limited.

[0047] A set of multiple bounding boxes that represent the same person and are related to each other is input from the tracking unit 36 to the feature quantity extraction unit 37. The feature quantity extraction unit 37 performs an extraction process for extracting the ReID feature quantity of the same person (i.e., person PS1) based on this set of bounding boxes. The extraction process is performed using, for example, a ReID model. The ReID model is, for example, a model based on a Transformer.

[0048] Information on the ReID feature quantity is input from the feature quantity extraction unit 37 to the feature quantity management unit 38. The feature quantity management unit 38 stores the input information from the feature quantity extraction unit 37 in the storage device 12. In addition, detection information on the entry of the management target TP and additional information included in the verification result are input from the entry detection unit 35 to the feature quantity management unit 38. The feature quantity management unit 38 determines, based on the input information from the entry detection unit 35, the ReID feature quantity among the ReID feature quantities included in the input information from the feature quantity extraction unit 37 that corresponds to the ReID feature quantity of the management target TP. The determination of the ReID feature quantity of the management target TP can be performed, for example, using the additional information included in the verification result, the coordinate information of the bounding box that is the object for extracting the ReID feature quantity, and the timestamp information. Information on the determined ReID feature quantity of the management target TP is stored in the storage device 12.

[0049] Information on the ReID feature quantity of person SP2 is input from the feature quantity extraction unit 37 to the exit management unit 39. The exit management unit 39 performs a re-identification process for the management target TP using the input information from the feature quantity extraction unit 37 and the ReID feature quantity of the management target TP stored in the feature quantity management unit 38 (storage device 12). In this re-identification process, the ReID feature quantity of person SP2 that has been input from the feature quantity extraction unit 37 is compared with the ReID feature quantity of the management target TP stored in the feature quantity management unit 38. When the two match (for example, when the similarity between the two is equal to or greater than a threshold value), the exit management unit 39 detects the exit of the management target TP.

[0050] 2. Second Embodiment

[0051] 2-1. Features of the Second Embodiment

[0052] Figure 5This is a diagram showing the features of the second embodiment of the present disclosure. In the first embodiment, face authentication processing using the face image IMF_PS1 and the face image IMF_PT is performed, thereby detecting the entry of the management target TP into the specified area 20. However, when the face image IMF_PS1 is unclear, the face authentication processing cannot be performed correctly. In this case, a situation occurs where the entry of the management target TP into the specified area 20 cannot be detected regardless of whether the management target TP has entered or not. In addition, if the face authentication processing cannot be performed correctly, the determination of the ReID feature amount of the management target TP based on the verification result of the face authentication processing cannot be performed either. Therefore, a situation occurs where the exit of the management target TP from the specified area 20 cannot be detected either.

[0053] Therefore, in the second embodiment, a camera 23 different from the camera 22 is used as a sub-camera, and a face image IMF_PU of an unauthenticated person PU is obtained from the camera images constituting the image VD2 acquired by the camera 23. Similar to the camera 22, the camera 23 is provided inside the specified area 20. The pointing direction of the camera 23 is inside the specified area 20. A part of the shooting range of the camera 23 may overlap with the shooting range of the camera 22. The total number of cameras 23 is at least one.

[0054] The unauthenticated person PU is a person PS1 who has not been authenticated in the face authentication processing using the camera images constituting the image VD1. The determination of the unauthenticated person PU is performed by person re-identification processing. In this re-identification processing, the ReID feature amount of the person PS1 extracted from the camera images constituting the image VD1 and the ReID feature amount of any person PS3 extracted from the camera images constituting the image VD2 are compared. When both match (for example, when the similarity between both is equal to or greater than the threshold), it is determined that the person PS1 and the person PS3 are the same person.

[0055] When it is determined that the person PS1 corresponds to the unauthenticated person PU and the person PS1 and the person PS3 are the same person, the person PS3 corresponds to the unauthenticated person PU. Therefore, in the second embodiment, the face image IMF_PS3 of the person PS3 is used as the face image IMF_PU, thereby performing face authentication processing. In this face authentication processing, the face image IMF_PU is verified with the face image IMF_PT. Thus, in the second embodiment, additional re-identification processing and additional face authentication processing using the camera images constituting the image VD2 are performed.

[0056] Also, in the second embodiment, when it is detected, as a result of the additional face authentication process, that the management target TP has entered the specified area 20, the full-body image IMB_PT is determined based on the camera image of the video image VD2 that constitutes the time of this detection (for example, the time before or after the entry of the management target TP is detected). Then, the ReID feature amount of the management target PT is extracted from this full-body image IMB_PT, thereby performing a person re-identification process. This re-identification process is the same as the process performed in the first embodiment.

[0057] Thus, according to the second embodiment, an additional re-identification process using the camera image that constitutes the video image VD2, an additional face authentication process, and the determination of the full-body image IMB_PT required for the re-identification process for detecting the exit of the management target TP are performed. Therefore, even in the case where the face authentication process of the management target TP using the camera image that constitutes the video image VD1 fails, the entry and exit of the management target PT with respect to the specified area 20 can be detected.

[0058] 2-2. Example of Functional Structure

[0059] Figure 6 It is a block diagram showing an example of the functional structure of the management device 10 related to the second embodiment. In Figure 6 the example shown, in addition to Figure 4 the person detection units 31 to the feature amount management unit 38 shown, the management device 10 further includes an unauthenticated person determination unit 41. For ease of explanation, the exit detection unit 39 is omitted. Figure 4 The difference from Figure 6 is that the unauthenticated person determination unit 41 is added, and the video images VD1 and VD2 are input to the person detection unit 31. However, various processes such as the face authentication process and the tracking process using the video image VD2 are basically the same as the various processes using the video image VD1 described in Figure 4 . Therefore, hereinafter, the functions particularly related to the second embodiment will be described.

[0060] The verification result from the face authentication unit 33 is input to the entry detection unit 35. The entry detection unit 35 detects the entry of the management target TP based on the judgment information included in the verification result. So far, it is the same as the first embodiment. In the second embodiment, when there is no face image IMF_TP in the face image management unit 34 that is consistent with the face image IMF_SP1, information indicating that the person SP1 is equivalent to the unauthenticated person PU is added to the judgment information of the face image IMF_SP1. When this information of the unauthenticated person PU is included in the judgment information, the entry detection unit 35 outputs this information of the unauthenticated person PU and the additional information included in the verification result to the feature amount management unit 38.

[0061] The feature quantity management unit 38 determines, based on the input information from the entry detection unit 35, the ReID feature quantity among the ReID feature quantities included in the input information from the feature quantity extraction unit 37 that corresponds to the ReID feature quantity of the unauthenticated person PU. The determination of the ReID feature quantity of the unauthenticated person PU can be performed, for example, using the attached information included in the verification result, the coordinate information of the bounding box that is the object for extracting the ReID feature quantity, and the timestamp information. The information on the determined ReID feature quantity of the unauthenticated person PU is stored in the storage device 12.

[0062] The unauthenticated person determination unit 41 performs a re-identification process of the unauthenticated person PU using the ReID feature quantity of the unauthenticated person PU stored in the feature quantity management unit 38 (storage device 12) and the ReID feature quantity of the person PS3 stored in the feature quantity management unit 38. In this re-identification process, the ReID feature quantity of the unauthenticated person PU is compared with the ReID feature quantity of the person PS3. When both match, the unauthenticated person determination unit 41 determines that the person PS3 corresponds to the unauthenticated person PU. Then, the unauthenticated person determination unit 41 outputs an instruction for the face authentication process using the face image IMF_PS3 to the face authentication unit 33.

[0063] When an instruction for the face authentication process has been input, the face authentication unit 33 performs a face authentication process using the face image IMF_PS3 (i.e., the face image IMF_TU) and the face image of the management target TP (i.e., the face image IMF_TP) stored in the face image management unit 34 (e.g., the storage device 12). In the face authentication process, the face image IMF_SP3 is verified against the face image IMF_TP. When both match, it is determined that the person PS3 and the management target TP are the same person.

[0064] 3. Third Embodiment

[0065] 3-1. Features of the Third Embodiment

[0066] In the first embodiment, the ReID feature quantity of the management target TP was extracted from the camera image of the video VD1 when the management target TP entered the specified area 20, and the ReID feature quantity of an arbitrary person PS2 who exited the specified area 20 was extracted from the camera image of the video VD1 after the management target TP entered the specified area 20. In the first embodiment, a re-identification process using the ReID feature quantity of the management target TP and the ReID feature quantity of the person PS2 was further performed. Therefore, by comparing these ReID feature quantities, it is possible to detect that the management target PT has exited the specified area 20.

[0067] However, even if the management target TP and the person PS2 are the same person, there are cases where the similarity of the ReID feature amount becomes low. For example, when the management target TP changes clothes within the specified area 20, the clothes of the management target TP are different when entering and leaving the area. In this case, it becomes a situation where it is impossible to detect that the management target PT has exited the specified area 20.

[0068] Therefore, in the third embodiment, the re-identification process of the management target TP is performed using the ReID feature amount of the management target TP extracted from the camera image constituting the image VD1 and the ReID feature amount of the person PS3 extracted from the camera image constituting the image VD2. In this re-identification process, the ReID feature amount of the management target TP is compared with the ReID feature amount of the person PS3. When both match (for example, when the similarity of both is equal to or greater than the threshold value), it is determined that the person PS3 and the management target TP are the same person. In addition, since the case where the similarity of the ReID feature amount becomes low is assumed, the threshold value used in the re-identification process of the management target TP can also be set to a value lower than the threshold value in the re-identification process performed in the exit management of the first embodiment.

[0069] By performing the re-identification process of the management target TP in this way, it is possible to continuously identify the management target TP within the specified area 20. Therefore, even when the management target TP changes clothes within the specified area 20, the management target TP within the specified area 20 is still tracked, so that it is possible to detect the situation where the management target PT has exited the specified area 20.

[0070] 3 - 2. Example of functional structure

[0071] Figure 7 It is a block diagram showing an example of the functional structure of the management device 10 related to the third embodiment. In Figure 7 the example shown, in addition to Figure 4 the person detection unit 31 to the exit detection unit 39 shown, the management device 10 further includes a management target tracking unit 51. Figure 4 The difference from Figure 7 is the addition of the management target tracking unit 51 and the matter of inputting the images VD1 and VD2 to the person detection unit 31. However, various processes such as the face authentication process and the tracking process using the image VD2 are basically the same as the various processes using the image VD1 described in Figure 4 . Therefore, hereinafter, the functions particularly related to the third embodiment will be described.

[0072] Information on the ReID feature amount of person SP3 is input from the feature amount extraction unit 37 to the management object tracking unit 51. The management object tracking unit 51 performs re-identification processing of the management object TP using the input information from the feature amount extraction unit 37 and the ReID feature amount of the management object TP stored in the feature amount management unit 38 (storage device 12). In this re-identification processing, the ReID feature amount of person SP3 that has been input from the feature amount extraction unit 37 is compared with the ReID feature amount of the management object TP stored in the feature amount management unit 38. When both match, it is determined that person PS3 and the management object TP are the same person. When it is determined that person PS3 and the management object TP are the same person, the management object tracking unit 51 outputs the determination information to the feature amount management unit 38.

[0073] The determination information from the management object tracking unit 51 is input to the feature amount management unit 38. The feature amount management unit 38 stores the input information from the management object tracking unit 51 in the storage device 12. When it is determined that person PS3 and the management object TP are the same person, the ReID feature amount of person SP3 is determined as the ReID feature amount of the management object TP and stored in the storage device 12.

[0074] Information on the ReID feature amount of person SP2 is input from the feature amount extraction unit 37 to the exit management unit 39. The exit management unit 39 performs re-identification processing of the management object TP using the input information from the feature amount extraction unit 37 and the ReID feature amount of the management object TP stored in the feature amount management unit 38 (storage device 12). In this re-identification processing, the ReID feature amount of person SP2 that has been input from the feature amount extraction unit 37 is compared with the ReID feature amount of the management object TP stored in the feature amount management unit 38. When it is determined that person PS3 and the management object TP are the same person, the information on the ReID feature amount of the management object TP stored in the feature amount management unit 38 is updated with the ReID feature amount of person SP3.

[0075] 4. Fourth Embodiment

[0076] Figure 8 This is a diagram for explaining the features of the fourth embodiment of the present disclosure. In the third embodiment, in order to identify the management object TP within the specified area 20, the ReID feature amount of person PS3 extracted from the camera images constituting the image VD2 was extracted. Then, when the ReID feature amount of this person PS3 matches the ReID feature amount of the management object TP extracted from the camera images constituting the image VD1, it is determined that person PS3 and the management object TP are the same person.

[0077] However, when there are many candidates who may be the same person as the management target TP, it is difficult to determine the management target TP through the re-identification process of the management target TP within the specified area 20. Especially in facilities such as childcare facilities and educational facilities, it is expected that the age of the person P3 and the management target TP is small. In this case, it may be misjudged that the person PS3 and the management target TP are the same person.

[0078] Therefore, in the fourth embodiment, the possession OB (such as a personal locker) of the management target TP provided within the specified area 20 is photographed by the camera 23. By making the shooting range of the camera 23 include the installation position of the possession OB, the image of the possession OB is included in the camera image IMG_VD2 that constitutes the image VD2 obtained by the camera 23. In the fourth embodiment, when the image of the person P3 is included in the camera image IMG_VD2, the distance between the representative coordinates of the image of the person P3 and the representative coordinates (known) of the image of the possession OB is calculated. Then, when the distance between the representative coordinates is within the specified distance, it is inferred that the person P3 photographed by the camera 23 and the management target TP are the same person.

[0079] In this way, in the fourth embodiment, the inference process of the person P3 is performed based on the distance between the representative coordinates of the image of the possession OB and the representative coordinates of the image of the person P3 on the camera image IMG_VD2. If the inference of the person P3 is performed, the candidates who may be the same person as the management target TP are narrowed down. Therefore, by adding the result of this inference process to the information of the ReID feature amount of the person SP3, it is possible to suppress the situation of misjudging that the person PS3 and the management target TP are the same person.

Claims

1. A method for managing entry and exit is a method for managing the entry and exit of a management object relative to a specified area with one entrance and exit. The method for managing entry and exit is characterized in that it includes the following steps, namely: Using a camera image captured by a camera installed near the entrance, a facial image and a full-body image of an arbitrary person passing through the entrance are obtained; performing a facial authentication process using the facial image of the arbitrary person included in the camera image and the registered facial image of the managed object, thereby detecting the entry of the managed object into the prescribed area; extracting a feature value of the managed object by using the full-body image of the managed object whose entry into the predetermined area is detected by the facial recognition process, among the full-body images of the arbitrary person included in the camera image; extracting a feature value of the arbitrary person using a full-body image of the arbitrary person included in the camera image after the managed object enters the predetermined area; as well as Re-recognition processing is performed using the feature amount of the managed object extracted based on the camera image and the feature amount of the arbitrary person extracted based on the camera image, thereby detecting the exit of the managed object from the predetermined area.

2. The method for managing entry and exit according to claim 1, characterized in that: The entry and exit management method further comprises the following steps, namely: using a sub-camera image captured by at least one sub-camera disposed inside the predetermined area to obtain a full-body image of an arbitrary person present inside the predetermined area; extracting a feature quantity of the arbitrary person using the full-body image of the arbitrary person included in the sub-camera image; extracting a feature amount of an unauthenticated person using a full-body image of an unauthenticated person who is not authenticated in the facial authentication process, among the full-body images of the arbitrary person included in the camera image; performing a re-identification process using a feature quantity of an arbitrary person extracted based on the sub-camera image and a feature quantity of an unauthenticated person extracted based on the camera image to identify the unauthenticated person existing inside the predetermined area; as well as When the unauthenticated person has been identified, and when the sub-camera image includes a facial image of the identified unauthenticated person, an additional facial authentication process using the facial image of the unauthenticated person and a registered facial image of the management target is performed. In the additional facial authentication process, when the facial image of the identified unauthenticated person matches the registered facial image of the managed object, entry of the managed object into the predetermined area is detected.

3. The method for managing entry and exit according to claim 1, characterized in that: The entry and exit management method further comprises the following steps, namely: using a sub-camera image captured by at least one sub-camera disposed inside the predetermined area to obtain a full-body image of an arbitrary person present inside the predetermined area; extracting a feature value of the arbitrary person using a full-body image of the arbitrary person included in the sub-camera image after the managed object enters the predetermined area; performing a re-recognition process using the feature quantity of the managed object extracted based on the camera image and the feature quantity of the arbitrary person extracted based on the sub-camera image to identify the managed object existing inside the predetermined area; as well as In the case where the management object has been determined, the determined management object is tracked.

4. The method for managing entry and exit according to any one of claims 1 to 3, characterized in that: The entry and exit management method further comprises the following steps, namely: using a sub-camera image captured by at least one sub-camera disposed inside the predetermined area to obtain a full-body image of an arbitrary person present inside the predetermined area; as well as inferring the arbitrary person based on the coordinates of the image of the arbitrary person included in the sub-camera image after the managed object enters the prescribed area on the sub-camera image, The photographing range of the sub-camera includes the occupied property of the management target located inside or outside the predetermined area, When the distance from the coordinates of the image of the arbitrary person on the sub-camera image to the coordinates of the setting position of the occupied object in the sub-camera image is less than a predetermined distance, it is inferred that the arbitrary person is a management target corresponding to the occupied object.

5. An entry and exit management device is a device for managing the entry and exit of a management object relative to a specified area with one entrance and exit, The entry and exit management device is characterized in that it has a processor for performing various processing. The processor is composed of: Using a camera image captured by a camera installed near the entrance, a facial image and a full-body image of an arbitrary person passing through the entrance are acquired, performing a facial authentication process using the facial image of the arbitrary person included in the camera image and the registered facial image of the managed object, thereby detecting the entry of the managed object into the prescribed area, extracting a feature value of the managed object by using the full-body image of the managed object whose entry into the predetermined area is detected by the facial recognition process, among the full-body images of the arbitrary person included in the camera image, extracting a feature value of the arbitrary person using a full-body image of the arbitrary person included in the camera image after the managed object enters the predetermined area, A re-recognition process using the extracted feature amount of the managed object and the extracted feature amount of the arbitrary person is performed to detect the exit of the managed object from the predetermined area.

Images