Person re-identification system, person re-identification method, management device, and person re-identification program

Abstract

To detect a disconnect of a camera or a reference object, even when the reference object can move.SOLUTION: A person re-identification system comprises at least one camera that generates a ranging image by imaging and ranging at least a part of re-identification area including at least one of a plurality of markers for defining a person re-identification area, and a management device that performs face re-identification of a person passing through the re-identification area by using the ranging image. The management device determines which of the marker and the camera deviates from an initial state, by using the ranging image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a person matching system, a person matching method, a management device, and a person matching program. [Background technology]

[0002] Patent Document 1 discloses a monitoring device that detects a deviation in the angle of view of a monitoring camera and adjusts the amount of the deviation in the angle of view to enable continuous monitoring under the same settings. Specifically, Patent Document 1 discloses that in a monitoring device equipped with an image analysis server that issues an alarm when an alarm-inducing object that has entered an intrusion detection area is detected based on a camera image captured by the camera, the image analysis server acquires a camera image, sets a fixed object area corresponding to a fixed object included in a reference image, acquires a camera image, extracts a fixed object area corresponding to a fixed object included in a live image, calculates the amount of deviation in the angle of view based on a comparison result with the reference area and the reference position of the reference area, and generates and displays a monitoring screen that includes an adjustment screen that guides an operation to eliminate the deviation in the angle of view. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-35448 Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 detects deviations in the camera's angle of view using a fixed object as a reference, and therefore cannot be applied to cases where the reference object can move.

[0005] An object of the present disclosure is to provide a technology for detecting a shift of a camera or a reference object even when the reference object may move. [Means for solving the problem]

[0006] A person matching system according to one embodiment of the present disclosure is a person matching system including at least one camera that captures and measures the distance to at least a portion of a matching area, including at least one of a plurality of markers for defining the matching area, and generates a distance measurement image, and a management device that uses the distance measurement image to perform facial matching of people passing through the matching area, and the management device uses the distance measurement image to determine whether the marker or the camera has shifted from its initial state.

[0007] In a person matching method according to one embodiment of the present disclosure, at least one camera captures and measures the distance to at least a portion of a matching area, including at least one of a plurality of markers for defining the matching area of ​​a person, to generate a ranging image, and a management device uses the ranging image to perform facial matching of a person passing through the matching area, and uses the ranging image to determine whether the marker or the camera has shifted from its initial state.

[0008] A management device according to one embodiment of the present disclosure includes an interface that receives a ranging image from at least one camera that captures and measures the distance to at least a portion of a person's matching area, including at least one of a plurality of markers for defining the matching area, and generates a ranging image; and a processor that uses the ranging image to perform facial matching of people passing through the matching area, and the processor uses the ranging image to determine whether the marker or the camera has shifted from its initial state.

[0009] A person matching program according to one embodiment of the present disclosure causes a computer to perform the following steps: capture and measure the distance of at least a portion of a matching area including at least one of a plurality of markers for defining the matching area of ​​a person; receive the distance image from at least one camera that generates the distance image; use the distance image to perform facial matching of a person passing through the matching area; and use the distance image to determine whether the marker or the camera has shifted from its initial state.

[0010] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0011] According to the present disclosure, even when the reference object is movable, it is possible to detect a deviation of the camera or the reference object. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram illustrating a configuration example of a person matching system according to a first embodiment; [Figure 2] FIG. 1 is a block diagram showing a configuration example of a management device according to a first embodiment; [Figure 3] 1 is a flowchart showing an example of advance preparation and operation processing related to person matching processing according to the first embodiment; [Figure 4] FIG. 1 is a diagram for explaining initial calibration according to the first embodiment. [Figure 5] FIG. 1 is a diagram for explaining a deviation handling process according to the first embodiment; [Figure 6] FIG. 1 is a diagram showing an example of a state before and after a marker is displaced according to the first embodiment; [Figure 7] FIG. 10 shows examples of distance measurement images before and after the distance measurement camera according to the first embodiment is shifted; [Figure 8] FIG. 10 is a diagram showing an example of a screen in a maintenance mode according to the first embodiment; [Figure 9] A diagram showing an example of the configuration of a person matching system using one distance measuring camera. [Figure 10] Flowchart showing an example of processing in a person matching system using a single ranging camera DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0014] (Embodiment 1) <System configuration> FIG. 1 is a schematic diagram showing an example of the configuration of a person verification system 1 according to the first embodiment.

[0015] The person verification system 1 is a system that performs facial verification of a person 3 passing through a verification area 2 and determines whether or not the person 3 has the right to pass through the verification area 2.

[0016] For example, as shown in Fig. 1, poles 4A and 4C are connected by belt 5A, poles 4B and 4D are connected by belt 5B, and the rectangular area connecting poles 4A, 4B, 4D, and 4C is defined as verification area 2. Person verification system 1 verifies person 3 passing through verification area 2 between belt 5A and belt 5B.

[0017] For ease of explanation, as shown in Figure 1, the base of pole 4A is defined as the origin O, the axis extending from origin O to pole 4B is defined as the X-axis, the axis extending from origin O to pole 4C is defined as the Y-axis, and the axis extending in the height direction from origin O is defined as the Z-axis. For ease of explanation, the positive direction of the Z-axis may be referred to as "up," the negative direction of the Z-axis as "down," the direction of the X-axis as "the width direction of the passage," and the direction of the Y-axis as "the length direction of the passage." These directional expressions are used for ease of explanation and are not intended to limit the orientation of the structure during actual use.

[0018] The person matching system 1 includes at least one ranging camera 10 (10A, 10B), at least one monitor 11 (11A, 11B), and a management device 20 (see Figure 2) to which the ranging cameras 10 (10A, 10B) and the monitor 11 (11A, 11B) are connected.

[0019] The distance measuring camera 10 is a camera that captures an object (including person 3) to generate an RGB image, and also measures the distance to the object to generate a depth image. Hereinafter, the RGB image and the depth image are collectively referred to as a distance measuring image. However, the distance measuring image may include only a depth image or only an RGB image. Alternatively, the distance measuring image may be used as an RGB image for face recognition, and when measuring the distance to an object (including person 3), the depth image may be used as the main image and the RGB image may be used as a sub image.

[0020] The distance measuring camera 10 and the monitor 11 may be configured as a set (hereinafter referred to as a camera monitor set 12). Fig. 1 shows an example in which two camera monitor sets 12A and 12B are arranged.

[0021] The poles 4A, 4B, 4C, and 4D and the camera monitor sets 12A and 12B are portable and movable, allowing the operator of the person verification system 1 to set the verification area 2 in any location and perform person verification operations.

[0022] The management device 20 uses the ranging image generated by the ranging camera 10 to track the position of the person 3 within the matching area 2 and perform face matching of the person 3. A calibration operation is performed before starting the person matching operation in order to create conversion information for converting distance information to an object indicated by the depth image output from the ranging camera 10 into coordinates in real space (hereinafter referred to as world coordinates).

[0023] For example, in the calibration process, a predetermined calibration chart (hereinafter referred to as chart 6) is first placed in a position where the distance measuring camera 10 can capture an image, and the distance measuring camera 10 captures an image of the chart 6. Next, the management device 20 calculates the correspondence between the distance information to the captured chart 6 and world coordinates, and generates conversion information based on the calculation result. This allows the management device 20 to identify the position on the world coordinates of an object (including person 3) captured by the distance measuring camera.

[0024] However, because the pole 4 and camera monitor set 12 are movable, the pole 4 or the ranging camera 10 may shift from its initial position after the initial calibration on the chart 6. In this case, if the position of an object in the ranging image captured by the ranging camera is converted into world coordinates using the conversion information generated initially using the conventional method, the calculated world coordinates will be shifted from the actual position in real space. As a result, the person verification system will no longer be able to correctly detect and track the person 3 present in the verification area 2.

[0025] Therefore, in this embodiment, a person verification system 1, a management device 20, a person verification method, and a person verification program will be described that can detect the occurrence of misalignment when at least one of the pole 4 and the distance measuring camera 10 is misaligned, and can continue person verification operation depending on the situation even when a misalignment occurs. Note that the chart 6 shown in Fig. 1 is used for calibration work, and may be removed during person verification operation.

[0026] <Configuration of management device> FIG. 2 is a block diagram showing an example of the configuration of the management device 20 according to the first embodiment.

[0027] The management device 20 includes, as hardware, a processor 21, a memory 22, a storage 23, a communication I / F 24, an input I / F 25, and an output I / F 26. Note that "I / F" is an abbreviation for interface.

[0028] The processor 21 executes computer programs in cooperation with the memory 22 and storage 23 to realize the functions of the management device 20. The processor 21 transmits and receives data via a communication I / F 24. The processor 21 receives instructions from a user via an input I / F 25. The processor 21 outputs information, images, etc. via an output I / F 26.

[0029] The memory 22 stores computer programs and data and is configured by a volatile storage medium (e.g., Random Access Memory (RAM)) and / or a non-volatile storage medium (e.g., Read Only Memory (ROM), flash memory, etc.).

[0030] The storage 23 stores computer programs and data and is configured by a non-volatile storage medium (for example, a flash memory, a solid state drive (SSD), a hard disk drive (HDD), etc.).

[0031] The communication I / F 24 is an interface for communicating with the ranging camera 10 and the monitor 11 via a network or a cable. Examples of networks include a wired LAN, a wireless LAN, a mobile communication network (e.g., LTE, 4G, 5G, etc.), the Internet, etc. Examples of cables include a LAN cable, a USB cable, an HDMI (registered trademark) cable, etc.

[0032] Input devices are connected to the input I / F 25. Examples of the input devices include a keyboard, a mouse, a touch panel, and a microphone.

[0033] An output device is connected to the output I / F 26. Examples of the output device include a display, a speaker, and a lamp.

[0034] The management device 20 has, as its functions, a person DB 31, an image processing unit 32, a person recognition unit 33, a face matching unit 34, a passerby determination unit 35, a matching result output unit 36, and a space recognition unit 37. These functions may be realized by the processor 21 executing a computer program in cooperation with the memory 22, etc.

[0035] The person DB 31 registers facial images of persons 3 who have the right to pass through the verification area 2. Hereinafter, facial images of persons registered in the person DB 31 will be referred to as registered facial images.

[0036] The image processor 32 processes the distance measurement images received from the distance measurement camera 10. The distance measurement images may be moving images or a plurality of still images captured at a predetermined interval.

[0037] The person recognition unit 33 recognizes a person 3 present in the verification area 2 from the distance measurement image.

[0038] The face matching unit 34 matches the face image of the person 3 recognized by the person recognition unit 33 with the registered face images in the person DB 31, and determines whether they are the same person. Note that a known face matching method may be used to determine whether the face image and the registered face image are the same person.

[0039] Based on the matching result by the face matching unit 34, the passerby determination unit 35 determines whether the person 3 recognized by the person recognition unit 33 has the right to pass through the matching area 2. For example, if the passerby determination unit 35 succeeds in matching the face of the person 3, it determines that the person 3 has the right to pass through the matching area 2, and if the passerby determination unit 35 fails to match the face of the person 3, it determines that the person 3 does not have the right to pass through the matching area 2.

[0040] The matching result output unit 36 ​​outputs the matching result for each person 3 passing through the matching area 2 to the monitor 11. For example, the matching result output unit 36 ​​outputs a facial image of a person 3 who does not have the right to pass through the matching area 2 to the monitor 11.

[0041] The spatial recognition unit 37 recognizes and manages the positional relationship between the person 3 captured by the distance measuring camera 10 and the verification area 2 in a three-dimensional world coordinate space based on the distance measuring image received from the distance measuring camera 10. The spatial recognition unit 37 includes a first spatial processing unit 41, a second spatial processing unit 42, and an integration processing unit 43.

[0042] The first spatial processing unit 41 analyzes and processes distance information and position information of an object based on the distance image received from the distance measuring camera 10 A. The first spatial processing unit 41 manages the relative positional relationship between the world coordinates and the distance measuring camera 10 A.

[0043] The second spatial processing unit 42 analyzes and processes distance information and position information of an object based on the distance image received from the distance measuring camera 10 B. The second spatial processing unit 42 manages the relative positional relationship between the world coordinates and the distance measuring camera 10 B.

[0044] The integrated processing unit 43 analyzes and processes the data analyzed and processed by the first spatial processing unit 41 and the second spatial processing unit 42, as well as other information, in an integrated manner. For example, the integrated processing unit 43 shares, connects, and compares the world coordinates of the range finding cameras 10A and 10B, and recognizes and manages the positions of the poles 4A, 4B, 4C, and 4D that make up the verification area 2.

[0045] <Flowchart of person matching process> FIG. 3 is a flowchart showing an example of advance preparation and operation processing related to the person matching processing according to the first embodiment.

[0046] <<Installation and Initial Calibration>> An operator sets up poles 4A, 4B, 4C, and 4D and camera monitor sets 12A and 12B (S101). For example, the operator sets up poles 4A, 4B, 4C, and 4D so as to define a verification area 2, and adjusts the placement of camera monitor sets 12A and 12B and the orientation (PTZ (pan, tilt, zoom)) of ranging cameras 10A and 10B so that the verification area 2 can be imaged.

[0047] The worker and the management device 20 perform an initial calibration of the installed distance measuring camera 10 in the verification area 2 (S102).

[0048] Next, the above steps S101 and S102 will be described in detail with reference to Fig. 4. Fig. 4 is a diagram for explaining the initial calibration according to the first embodiment.

[0049] For example, as shown in Figure 4, a worker places poles 4A, 4B, 4C, and 4D at points A, B, C, and D, respectively, at the four corners of the verification area 2, and connects poles 4A and 4C with belt 5A and poles 4B and 4D with belt 5B to form an aisle in the verification area 2. Here, point A is defined as the origin O, the width between points A and B (and between points C and D) is W, and the length between points A and C (and between points B and D) is L.

[0050] Next, the worker places the chart 6 on the floor of the verification area 2. The chart 6 may be removed after the calibration is completed.

[0051] The distance measuring camera 10 captures an image of at least one of the chart 6 and the pole 4 to generate a distance measuring image, and transmits the image to the management device 20.

[0052] The management device 20 calculates the correspondence between the coordinates (0,0,0) of point A, the coordinates (W,0,0) of point B, the coordinates (0,L,0) of point C, and the coordinates (W,L,0) of point D, which are stored in advance as a setting file, and the positions of the chart 6 and poles 4 in the ranging image received from the ranging camera 10, and identifies the world coordinates in real space of each of the poles 4A, 4B, 4C, and 4D. Hereinafter, the world coordinates of each pole 4 identified in the initial calibration may be referred to as the "initial world coordinates of each pole 4." The integration processing unit 43 of the management device 20 stores the initial world coordinates of each pole 4 as initial setting information.

[0053] <<Marker placement>> The worker places markers 7 (7A, 7B, 7C, 7D) on the apex of each pole 4 (4A, 4B, 4C, 4D) installed in step S101 (S103). The markers 7 may be any marker that the management device 20 can detect from the distance measurement image. Examples of the markers 7 include two-dimensional codes, specific characters, specific illustrations, specific shapes, etc. Note that the markers 7 are not limited to being placed on the apex of the poles 4, and may be placed on structures that indicate the boundaries of the verification area 2, for example. Furthermore, the markers 7 are not limited to being visible to humans, and may be invisible printed materials that are invisible to humans but can be detected by the distance measurement camera 10. This allows the markers 7 to blend into the surrounding environment.

[0054] The distance measuring camera 10 takes a distance measuring image and transmits it to the management device 20 (S104).

[0055] The management device 20 determines, from the distance measurement image received from the distance measurement camera 10, whether the marker 7 is inside or outside the angle of view of the distance measurement camera 10 (S105).

[0056] If the marker 7 is outside the angle of view of the ranging camera 10 (S105: outside angle of view), the process proceeds to step S108. If the marker 7 is within the angle of view of the ranging camera 10 (S105: inside angle of view), the management device 20 calculates the world coordinates of the marker 7 from the position of the marker 7 included in the ranging image (S106).

[0057] The management device 20 updates the world coordinates of the four corners of the verification area 2 from the initial world coordinates of each pole 4 to the world coordinates of each marker 7 calculated in step S106 (S107). Next, an example of the processing of steps S106 and S107 will be described.

[0058] The management device 20 calculates the world coordinates of the markers 7A, 7B, 7C, and 7D on each of the poles 4A, 4B, 4C, and 4D based on distance information obtained from the distance measurement images received from the distance measurement camera 10. For example, the management device 20 calculates the coordinates (Xma, Yma, H) of marker 7A, the coordinates (Xmb, Ymb, H) of marker 7B, the coordinates (Xmc, Ymc, H) of marker 7C, and the coordinates (Xmd, Ymd, H) of marker 7D. If the X and Y coordinates of the markers 7A, 7B, 7C, and 7D deviate from the X and Y coordinates of the initial world coordinates of each pole 4 by a predetermined threshold or more, the management device 20 updates the world coordinates of the four corners of the verification area 2 held in the space recognition unit 37 to the X and Y coordinates of the markers 7A, 7B, 7C, and 7D.

[0059] The lines connecting points A, B, C, and D are the boundary lines that define the matching area 2, and are expressed as functions on the XY plane. For example, in the case of marker 7A (Xa, Ya), marker 7B (Xb, Yb), marker 7C (Xc, Yc), and marker 7D (Xd, Yd), the line segments AB, BC, CD, and DA are expressed by the following functions. Line segment AB:y=((Yb-Ya) / (Xb-Xa))*x+(Ya-((Yb-Ya) / (Xb-Xa))*Xa) Line segment BC: (same as line segment AB) Line segment CD: (same as line segment AB) Line segment DA: (same as line segment AB)

[0060] Returning to the explanation of Fig. 3, the integration processing unit 43 of the management device 20 holds the world coordinates of the four corners of the verification area 2 (S108).

[0061] The integration processing unit 43 of the management device 20 recalculates the boundary of the verification area 2 using the world coordinates of the four corners of the verification area 2 in step 108 (S109).

[0062] <<Automatic adjustment of the matching area after operation>> After the preparation processes of steps S101 to S109 described above are completed, person verification system 1 starts the operation of person verification in verification area 2 (S111).

[0063] The management device 20 periodically identifies the position of the marker 7 using the distance measurement image received from the distance measurement camera 10 (S112).

[0064] The management device 20 determines whether the position of the marker 7 identified in step 112 is displaced by a predetermined threshold or more compared to the original position of the marker 7 (S113). That is, the management device 20 determines whether the positional displacement of the marker 7 is a predetermined threshold or more.

[0065] If the positional deviation of the marker 7 is less than the predetermined threshold (S113: NO), the management device 20 returns the process to step S112 and continues the operation of person matching.

[0066] If the positional deviation of the marker 7 is equal to or greater than a predetermined threshold (S113: YES), the management device 20 performs deviation handling processing (S114) and then continues or stops the person matching operation. The deviation handling processing will be described in detail later.

[0067] <Misalignment processing> 5 is a diagram for explaining the deviation handling process according to Embodiment 1. The process explained using FIG. 5 corresponds to the details of the process in step S114 in FIG.

[0068] <<When operating with one ranging camera>> The following describes the process of dealing with deviations when performing person verification using a configuration in which one distance measuring camera 10 covers the imaging of four markers 7 (poles 4).

[0069] When the integrated processing unit 43 detects a displacement of one specific marker 7 or a plurality of markers 7 but not all of them from a distance measurement image received from one distance measurement camera 10, it determines that the marker 7 (pole 4) has been displaced, and executes a displacement response process for the (correspondence A) marker 7. The details of the displacement response process for the (correspondence A) marker 7 will be described later.

[0070] When the integrated processing unit 43 detects displacement of all markers from the distance measurement image received from one distance measurement camera 10, it determines that the marker 7 (pole 4) has displaced, and executes displacement response processing for the (correspondence A) marker 7. Details of the (correspondence A) marker displacement response processing will be described later.

[0071] <<When using multiple ranging cameras>> The following describes a process for dealing with deviations when performing person verification using a configuration in which multiple distance measuring cameras 10 cover the imaging of four markers 7 (poles 4).

[0072] The integrated processing unit 43 detects a deviation of a specific marker 7 or a plurality of markers 7 from a distance measurement image received from one of the distance measurement cameras 10 among the distance measurement images received from each of the multiple distance measurement cameras 10, and if the marker 7 does not appear in the distance measurement images received from the other distance measurement cameras 10, it determines that the marker 7 (pole 4) has deviated, and executes a (correspondence A) marker 7 deviation response process. The (correspondence A) marker 7 deviation response process will be described in detail later.

[0073] If the integration processing unit 43 detects a misalignment of one specific marker 7 or some but not all of the markers 7 in a distance measurement image received from one of the distance measurement cameras 10, but does not detect a misalignment of the marker 7 in distance measurement images received from the other distance measurement cameras 10 (i.e., there is no consistency in the misalignment of the markers 7 among the multiple distance measurement images), the integration processing unit 43 determines that the installation configuration of the pole 4 and the distance measurement cameras 10 has been fundamentally changed, and executes a recalibration process (Correspondence C). The details of the recalibration process (Correspondence C) will be described later.

[0074] When the integration processing unit 43 detects displacement of all markers in a distance measurement image received from one distance measurement camera 10 among the distance measurement images received from the multiple distance measurement cameras 10, it determines that the one distance measurement camera 10 has displaced, and executes (correspondence B) displacement response processing for the distance measurement camera 10. Details of (correspondence B) displacement response processing for the distance measurement camera 10 will be described later.

[0075] The integration processing unit 43 detects misalignment of a specific marker 7 or of some but not all of the markers 7 from the distance measurement images received from two or more of the distance measurement cameras 10, and if the marker misalignment is consistent between the distance measurement images, it determines that the marker 7 (pole 4) has misaligned, and executes (correspondence A) marker 7 misalignment response processing. Details of the (correspondence A) marker misalignment response processing will be described later.

[0076] The integration processing unit 43 detects misalignment of a specific marker 7 or of some but not all of the markers 7 from the distance measurement images received from two or more of the distance measurement cameras 10, and if there is no consistency in the misalignment of the markers 7 between the multiple distance measurement images, it determines that the installation configuration of the pole 4 and the distance measurement cameras 10 has been fundamentally changed, and executes a (correspondence C) recalibration process. The (correspondence C) recalibration process will be described in detail later.

[0077] When the integration processing unit 43 detects misalignment of all of the markers 7 from the distance measurement images received from two or more of the distance measurement cameras 10, it determines that the installation configuration of the pole 4 and the distance measurement cameras 10 has been fundamentally changed, and executes a recalibration process (Correspondence C). The details of the recalibration process (Correspondence C) will be described later.

[0078] <<(Response A) Marker misalignment processing>> When the integration processing unit 43 determines through the above-described processing that the marker 7 (pole 4) has shifted, it executes, for example, the following processing as processing for dealing with the shift of the marker 7. Fig. 6 is a diagram showing an example of the situation before and after the marker 7 (pole 4) according to the first embodiment has shifted.

[0079] The integration processing unit 43 uses the coordinates of the marker 7 after the displacement to update the world coordinates of the four corners (vertices) of the rectangle that defines the verification area 2, and stores the updated world coordinates in the memory 22. Next, the integration processing unit 43 recalculates the division of the verification area 2 based on the updated world coordinates of the four corners of the rectangle, and stores the recalculated division in the memory 22. As a result, even if the marker 7 (pole 4) is displaced, the verification area 2 is updated to reflect the displacement, and person verification can continue without interruption.

[0080] During person verification, the management device 20 continues to calculate and acquire the world coordinates of markers 7A, 7B, 7C, and 7D in the same manner as immediately after placing the marker 7 described above. If the calculated X and Y coordinates of markers 7A, 7B, 7C, and 7D deviate by a predetermined threshold or more compared with the X and Y coordinates of points A, B, C, and D stored in the spatial recognition unit 37, the management device 20 updates the coordinates of the four corners of the verification area 2 stored in the spatial recognition unit 37 to the latest X and Y coordinates of marker 7. Note that the boundary line ABDC defining the verification area 2 can be expressed by a function, as shown in FIG. 6, in the same way as after placing the marker. For example, if marker 7B deviates by dx in the X direction and dy in the Y direction as shown in FIG. 6, the boundary lines AB, BD, DC, and CA after the deviation can be expressed by the following functions, respectively. Boundary AB: y=(dy / (W+dx))*x Boundary BD:y=(-1)*((L-dy) / dx)*x+(((L-dy) / dx)*W+L) Boundary DC:y=L Boundary CA:x=0

[0081] <<(Solution B) Processing to deal with misalignment of the ranging camera>> When the integration processing unit 43 determines through the above-described processing that the ranging camera 10 has shifted, it executes, for example, the following processing as processing to deal with the shift of the ranging camera 10. Fig. 7 is a diagram showing an example of ranging images before and after the ranging camera 10 according to the first embodiment has shifted.

[0082] The integration processing unit 43 calculates parameters (hereinafter referred to as displacement correction parameters) for correcting the coordinates of all markers 7 included in the distance measurement images output by the distance measurement camera 10 in which a deviation has occurred to the coordinates of all markers 7 included in the distance measurement images output by the distance measurement camera 10 before the deviation occurred, and stores the calculated parameters in the memory 22. Next, the integration processing unit 43 corrects the coordinates of objects (including person 3) detected in the distance measurement images received from the distance measurement camera 10 in which a deviation has occurred using the displacement correction parameters before using the corrected coordinates. As a result, even if the distance measurement camera 10 is displaced, the coordinates of objects (including person 3) measured by the distance measurement camera 10 can be corrected to match the displacement, and person matching operations can be continued without interruption.

[0083] The integrated processing unit 43 may perform the following processing as a process for dealing with the deviation of the distance measuring camera 10. Fig. 8 is a diagram showing an example of a screen in the maintenance mode according to the first embodiment.

[0084] A case where the misalignment of the ranging camera 10 is manually corrected will be described. The management device 20 displays a maintenance mode screen shown in FIG. 8. On the maintenance mode screen, the management device 20 superimposes the position of the marker 7 at the time of initial calibration (hereinafter referred to as the initial marker position 8) on the ranging image. This allows the worker to visually confirm the misalignment between the current position of the displaced marker 7 (pole 4) displayed in the ranging image captured by the ranging camera 10 and the initial marker position 8. The worker can easily return the ranging camera 10 to the state before it was displaced by manually adjusting the position and orientation of the ranging camera 10 so that the current position of the displaced marker 7 (pole 4) displayed in the ranging image overlaps with the initial marker position 8.

[0085] A case where the displacement of the ranging camera 10 is automatically corrected will be described. If the ranging camera 10 has a function for automatically adjusting the PTZ, the following processing may be performed. The management device 20 calculates the difference between the 3D coordinates of the marker 7 at the time of initial calibration (i.e., the initial marker position 8) and the 3D coordinates of the marker 7 (pole 4) in the ranging image after displacement, and automatically adjusts the PTZ of the ranging camera 10 so that the difference becomes smaller. This allows the displaced ranging camera 10 to be automatically returned to the state before the displacement.

[0086] Furthermore, by using the initial marker position superimposed display in the maintenance mode described in this embodiment, the positions of the pole 4 and marker 7 can be moved to match the superimposed display, thereby achieving the same effect as when adjusting the camera installation.

[0087] <<(Response C) Recalibration process>> If the integrated processing unit 43 determines through the above-described processing that the installation configuration of the pole 4 and the ranging camera 10 has been fundamentally changed, it executes, for example, the following processing as a processing to deal with the deviation of the ranging camera 10.

[0088] The integration processing unit 43 warns the operator that it is necessary to perform initial calibration again. For example, the integration processing unit 43 displays an image indicating the warning on a display connected to the output I / F 26, or outputs a sound indicating the warning from a speaker connected to the output I / F 26. For example, the integration processing unit 43 displays an image and information indicating the warning on the monitor 11 of the camera monitor set 12. In this case, the integration processing unit 43 may stop person matching because there is a possibility that correct person matching may not be performed.

[0089] (Supplementary and Modified Examples) Some supplementary and modified examples of the first embodiment will be described below.

[0090] <What to do if the four corner markers cannot be captured within the angle of view of a single ranging camera> In a configuration in which the angle of view of one distance measuring camera 10 is not sufficient to capture images of the markers 7 at the four corners, the management device 20 may adopt any one of the following measures 1 to 3.

[0091] (Response 1) Multiple ranging cameras 10 that share world coordinates are arranged so as to cover the four corner markers 7. The integration processing unit 43 integrates the ranging images received from these ranging cameras 10 to calculate the world coordinates of the four corner markers 7.

[0092] (Countermeasure 2) The management device 20 uses preset values ​​for the coordinates of the markers 7 that are not shown in the distance measurement image. In this way, the management device 20 identifies the world coordinates of the markers 7 at the four corners.

[0093] (Response 3) The positional relationship of the boundary lines AC / DB is given in advance as a relational expression to the management device 20, and the coordinates of the markers 7 not captured by the ranging camera 10 are estimated from the coordinates of the markers 7 captured by the ranging camera 10 and the relational expression of the positional relationship of the boundary lines AC / DB. For example, if the shape of the matching area 2 is defined as a rectangle, point symmetric, and clockwise, with points A (Xa, Ya), C (Xc, Yc), D (Xd, Yd), and B (Xb, Yb), and only point B is not captured by the ranging camera 10, the position (Xb, Yb) of point B can be expressed by the following calculation formula. Xb=Xa-Xc+Xd Yb=Ya-Yc+Yd

[0094] <What to do when the marker disappears from the ranging camera> When the marker 7 is not captured by the distance measuring camera 10 for a predetermined period of time or longer, the management device 20 may determine that the marker 7 has shifted out of the angle of view of the distance measuring camera 10.

[0095] If a marker 7 that has disappeared from the ranging camera 10 reappears within a predetermined period of time and the amount of deviation of the marker 7 is less than a predetermined threshold, the management device 20 may determine that the marker 7 has been temporarily hidden by, for example, a person 3, and that no deviation of the marker 7 has occurred, and may continue the person matching operation as is.

[0096] When a marker 7 that was not captured by the distance measuring camera 10 during the initial calibration is now captured by the distance measuring camera 10, the management device 20 may determine that the marker 7 has shifted.

[0097] <Notification to administrator or operator when marker position shift occurs> When the amount of positional displacement of the marker 7 is equal to or greater than a predetermined threshold, the management device 20 may notify the administrator (management room) or operator of information indicating that a positional displacement of the marker 7 has occurred. Examples of notification methods include displaying a warning image, outputting an alarm sound, sending an email, and sending a push notification to a predetermined application.

[0098] When the amount of positional deviation of one marker 7 is equal to or greater than a predetermined threshold, the management device 20 may notify information indicating that a positional deviation has occurred in the marker 7 (pole 4) that defines the verification area 2. This allows the administrator or operator to immediately notice that a positional deviation of the marker 7 (pole 4) has occurred and take prompt action.

[0099] When the amount of positional deviation of all the markers 7 is equal to or greater than a predetermined threshold, the management device 20 may notify information indicating that a deviation has occurred in the position or orientation (posture) of the ranging camera 10. This allows the administrator or operator to quickly notice that a deviation has occurred in the position or orientation (posture) of the ranging camera 10 and take action accordingly.

[0100] The management device 20 may stop person matching when the administrator or operator who has received the notification instructs the person matching to stop. Alternatively, the management device 20 may automatically stop person matching together with the notification when the positional deviation of the marker 7 is equal to or greater than a predetermined threshold.

[0101] <Notification to end users or passersby when a marker position shift occurs> If the amount of positional misalignment of the marker 7 is equal to or greater than a predetermined threshold, the management device 20 may notify the end user or passersby in the verification area 2 of information indicating that the positional misalignment of the marker 7 has occurred. Examples of notification methods include displaying an image on the monitor 11 and making an audio announcement. The management device 20 may set multiple thresholds in stages and change the content of the notification in stages depending on whether the amount of positional misalignment is equal to or greater than each threshold. Alternatively, the management device 20 may change the notification method as follows depending on whether the state is (Response A), (Response B), or (Response C) in FIG. 5. If the verification area 2 can be automatically adjusted in (Response A) or (Response B), the management device 20 does not issue a notification and continues operation. If (Response B) is true and manual adjustment of the verification area 2 is required, the management device 20 notifies the user that the verification area 2 is being adjusted. · In the case of (Response C) where adjustment of the verification area 2 is not possible, the management device 20 notifies that the operation of person verification is currently suspended.

[0102] <Example of using one ranging camera to identify people passing between poles 4E and 4F> 9 is a diagram showing an example of the configuration of a person matching system 1 using one distance measuring camera 10. FIG. 10 is a flowchart showing an example of processing in the person matching system 1 using one distance measuring camera 10.

[0103] As shown in Fig. 9, when a walk-through type person verification system 1 is configured with one ranging camera 10, there is no need to share the space of the verification area 2 among multiple ranging cameras 10. In this case, the person verification system 1 may automatically set the verification area 2 using the above-mentioned post-operation automatic adjustment function without performing initial calibration. For example, the person verification system 1 executes the following process as shown in Fig. 10.

[0104] The worker sets up the pole 4 and the camera monitor set 12 (S201).

[0105] The worker provides a rough draft of the world coordinate transformation formula to the space recognition unit 37 (S202). For example, the worker provides the space recognition unit 37 with provisional setting values ​​of the world coordinates that will serve as a rough draft. In other words, the worker provides position information (top-bottom direction, matrices and vectors indicating provisional XYZ directions) that corresponds to the initial calibration to the space recognition unit 37 using a setting file or the like. The detailed positional relationship of this position information may be inaccurate as long as the actual layout and top-bottom of the structure are correct.

[0106] As shown in FIG. 9, the worker places the markers 7E and 7F on the apexes of the poles 4E and 4F that indicate the passage route (S203).

[0107] The distance measuring camera 10 takes a distance measuring image and transmits it to the management device 20 (S204).

[0108] The management device 20 calculates the world coordinates of the markers 7E and 7F (S205).

[0109] The integration processing unit 43 of the management device 20 holds the world coordinates of the markers 7E and 7F calculated in step S205 (S206).

[0110] The management device 20 automatically sets the verification area 2 (boundary line PQRS) using the world coordinates of the markers 7E and 7F as a reference, for example, by the following process (S207).

[0111] The operator provides the spatial recognition unit 37 in advance, using a setting file or the like, with a calculation formula indicating the relative positional relationship between the position of point PQRS and markers 7E and 7F. Here, the world coordinates of marker 7E are (Xa, Ya), and the world coordinates of marker 7F are (Xb, Yb). Also, the distance between point S and point P is 2.0 m, the distance between point R and point Q is 2.0 m, the distance between marker 7E and point P is 1.0 m, and the distance between marker 7F and point Q is 1.0 m. In this case, the point P(Xp, Yp) is calculated as follows: Xp=Xa+(Xa-Xb)*(1 / ((Xa-Xb) 2 +(Ya-Yb) 2 ) 0.5)*1.0[m] Yp=Ya+(Ya-Yb)*(1 / ((Xa-Xb) 2 +(Ya-Yb) 2 ) 0.5 )*1.0[m] Furthermore, the point S (Xs, Ys) is calculated as follows. Xs=Xp+(±1)*(Ya-Yb)*(1 / ((Xa-Xb) 2 +(Ya-Yb) 2 ) 0.5 )*2.0[m] Ys=Yp+(±1)*(Xa-Xb)*(1 / ((Xa-Xb) 2 +(Ya-Yb) 2 ) 0.5 )*2.0[m]

[0112] Returning to the description of Fig. 10, the management device 20 recalculates the boundary line of the verification area 2 based on the coordinates of the point PQRS calculated in step S207 (S208).

[0113] After the preparation process of steps S201 to S208 described above is completed, person verification system 1 starts the operation of person verification in verification area 2 (S211).

[0114] The management device 20 periodically identifies the position of the marker 7 using the distance measurement image received from the distance measurement camera 10 (S212).

[0115] The management device 20 determines whether the position of the marker 7 identified in step 212 is displaced by a predetermined threshold or more compared to the original position of the marker 7 (S213). That is, the management device 20 determines whether the positional displacement of the marker 7 is a predetermined threshold or more.

[0116] If the positional deviation of the marker 7 is less than the predetermined threshold (S213: NO), the management device 20 returns the process to step S212 and continues the operation of person matching.

[0117] If the positional deviation of the marker 7 is equal to or greater than the predetermined threshold (S213: YES), the space recognition unit 37 of the management device 20 updates the world coordinates of the marker 7 to the latest ones identified in step S212 and stores them (S214).

[0118] The management device 20 recalculates the boundary line of the verification area 2 using the latest world coordinates of the marker 7 updated in step S214 (S215), and returns the process to step S212.

[0119] In this way, even if the position of the marker 7 (pole 4) is shifted, the person verification system 1 can correct the verification area in accordance with the position shift and continue the operation of person verification.

[0120] (Summary of the first embodiment) The above description of the first embodiment discloses the following techniques.

[0121] <Technology 1> The person matching system (1) according to the first embodiment includes at least one camera (e.g., a ranging camera 10) that captures and measures the distance to at least a portion of a matching area (2) including at least one of a plurality of markers (7) for defining the matching area (2) of a person (3) and generates a ranging image, and a management device (20) that uses the ranging image to perform facial matching of people passing through the matching area, and the management device uses the ranging image to determine whether the marker or the camera has shifted from its initial state. This allows the management device to determine whether the marker defining the matching area and / or the camera capturing and measuring the distance to the matching area has shifted from its initial state in cases where the marker and / or the camera may move.

[0122] <Technology 2> In the person verification system described in Technique 1, the management device determines whether the marker or the camera has shifted from its initial state based on the movement of the marker included in the distance measurement image from its initial position. This allows the management device to determine whether the marker defining the matching area and / or the camera capturing and measuring the distance to the matching area has shifted from its initial state in cases where the marker and / or the camera may move.

[0123] <Technology 3> In the person verification system described in Technique 2, when the position of at least one marker but not all of the plurality of markers has moved from its initial position, the management device determines that the marker has shifted from its initial state. This allows the management device to detect that the marker has shifted from its initial state.

[0124] <Technology 4> In the person verification system described in Technique 2 or 3, the management device determines that the camera has shifted from its initial state when the positions of the plurality of markers have moved from their initial positions. This allows the management device to detect that the camera has shifted from its initial state.

[0125] <Technology 5> In the person matching system described in Technology 4, the number of cameras is multiple, and the management device determines that a camera, among the multiple cameras, that has generated the ranging image in which the positions of the multiple markers have moved from their initial positions has shifted from its initial state. This allows the management device to detect a misaligned camera.

[0126] <Technology 6> In the person verification system according to any one of Techniques 3 to 5, when the management device determines that the marker has shifted from its initial state, the management device redefines the verification area based on the position of the shifted marker. This allows the management device to continue operation even if the marker is displaced, by redefining the collation area in accordance with the displacement of the marker.

[0127] <Technology 7> In the person verification system according to any one of Techniques 3 to 6, the management device issues a predetermined notification when it determines that the marker has shifted from its initial state. This allows the management device to notify the administrator, operator, passersby, etc. that a marker has shifted.

[0128] <Technology 8> In the person matching system described in any one of Techniques 4 to 7, when the management device determines that the camera has shifted from its initial state, the management device corrects the shifted initial state of the camera based on the shifts of the positions of the multiple markers from their initial positions. This allows the management device to correct the camera misalignment and continue operation even if the camera is misaligned.

[0129] <Technology 9> In the person verification system according to any one of the fourth to eighth techniques, the management device issues a predetermined notification when it determines that the camera has shifted from its initial state. This allows the management device to notify the administrator, operator, passersby, etc. that a camera misalignment has occurred.

[0130] <Technology 10> In the person matching system described in any one of Techniques 2 to 9, the number of cameras is multiple, and when an inconsistency occurs in the deviation of the position of at least one of the markers from its initial position between the respective ranging images of the multiple cameras, the management device stops the operation of person face matching in the matching area and outputs a predetermined warning. This allows the management device to prevent continuing operation of face matching using the matching area in a state where it cannot be operated normally, and can notify the administrator, operator, passersby, etc. that face matching operation has been stopped.

[0131] <Technology 11> In the person matching method according to the first embodiment, at least one camera (e.g., a ranging camera 10) captures and measures the distance to at least a part of a matching area (2) of a person (3), including at least one of a plurality of markers (7) for defining the matching area, and generates a ranging image. A management device (20) uses the ranging image to perform facial matching of people passing through the matching area, and uses the ranging image to determine whether the marker or the camera has shifted from its initial state. This allows the management device to determine whether the marker defining the matching area and / or the camera capturing and measuring the distance to the matching area has shifted from its initial state in cases where the marker and / or the camera may move.

[0132] <Technology 12> The management device (20) according to the first embodiment includes an interface (e.g., a communication I / F 24) that receives a distance measurement image from at least one camera (e.g., a distance measurement camera 10) that captures and measures the distance of at least a portion of a matching area (2) including at least one of a plurality of markers (7) for defining the matching area (2) of a person (3) and generates a distance measurement image, and a processor (21) that uses the distance measurement image to perform facial matching of a person passing through the matching area, and the processor uses the distance measurement image to determine whether the marker or the camera has shifted from its initial state. This allows the management device to determine whether the marker defining the matching area and / or the camera capturing and measuring the distance to the matching area has shifted from its initial state in cases where the marker and / or the camera may move.

[0133] <Technology 13> The person matching program according to the first embodiment causes a computer (e.g., a management device 20) to perform the following operations: capture an image of and measure the distance to at least a portion of a matching area (2) of a person (3), including at least one of a plurality of markers (7) for defining the matching area; receive the distance image from at least one camera (e.g., a distance-measuring camera 10) that generates the distance image; use the distance image to perform facial matching of a person passing through the matching area; and use the distance image to determine whether the marker or the camera has shifted from its initial state.

[0134] Although the embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components in the above-described embodiments may be combined in any manner without departing from the spirit of the invention. [Industrial Applicability]

[0135] The technology of the present disclosure is useful for person matching using a camera. [Explanation of symbols]

[0136] 1. Person matching system 11, 11A, 11B Monitor 2. Matching Area 3 people 4, 4A, 4B, 4C, 4D, 4E, 4F pole 5A, 5B Belt 6 Charts 7,7A,7B,7C,7D,7E,7F markers 10, 10A, 10B Range finding camera 12, 12A, 12B Camera Monitor Set 20 Management device 21 processors 22 Memory 23. Storage 24 Communication I / F 25 Input I / F 26 Output I / F 31 Person DB 32 Video Processing Unit 33 Person recognition section 34 Face matching unit 35 Passerby Judgment Department 36 Matching result output section 37 Spatial recognition part 41 First spatial processing section 42 Second spatial processing section 43 Integrated Processing Unit

Claims

1. at least one camera that captures and measures distance to at least a part of the verification area, the camera including at least one of a plurality of markers for defining a verification area of ​​a person, and generates a distance measurement image; a management device that performs face matching of a person passing through the matching area using the distance measurement image, the management device determines whether the marker or the camera has shifted from an initial state using the distance measurement image; Person matching system.

2. the management device determines whether the marker or the camera has shifted from its initial state based on a movement of the marker included in the distance measurement image from its initial position; The person verification system according to claim 1 .

3. the management device determines that the marker has shifted from its initial state when the position of at least one marker but not all of the plurality of markers has moved from its initial position; The person verification system according to claim 2 .

4. the management device determines that the camera has shifted from its initial state when the positions of the plurality of markers have moved from their initial positions; The person verification system according to claim 2 .

5. the number of cameras is plural, the management device determines that a camera, among the plurality of cameras, that has generated the ranging image in which the positions of the plurality of markers have moved from their initial positions has shifted from its initial state; The person verification system according to claim 4 .

6. When the management device determines that the marker has shifted from its initial state, the management device redefines the verification area based on the position of the shifted marker. The person verification system according to claim 3 .

7. the management device issues a predetermined notification when it determines that the marker has shifted from its initial state. The person verification system according to claim 3 .

8. When the management device determines that the camera has deviated from its initial state, the management device corrects the deviated initial state of the camera based on the deviation of the positions of the plurality of markers from their initial positions. The person verification system according to claim 4 or 5.

9. the management device issues a predetermined notification when it determines that the camera has deviated from its initial state; The person verification system according to claim 4 or 5.

10. the number of cameras is plural, the management device, when an inconsistency occurs in the deviation of at least one of the marker positions from an initial position among the respective ranging images of the plurality of cameras, stops the operation of face matching of the person in the matching area and outputs a predetermined warning. The person verification system according to claim 2 .

11. at least one camera captures and measures distance to at least a part of the verification area, including at least one of a plurality of markers for defining the verification area of ​​the person, to generate a distance measurement image; The management device Using the distance measurement image, face matching is performed on a person passing through the matching area; using the distance measurement image, determining whether the marker or the camera has shifted from an initial state; Person matching method.

12. an interface for receiving a ranging image from at least one camera that captures and measures distance to at least a portion of a verification area including at least one of a plurality of markers for defining the verification area of ​​a person, and generates the ranging image; a processor that performs face matching of a person passing through the matching area using the distance measurement image, the processor uses the distance measurement image to determine whether the marker or the camera has shifted from an initial state; Management device.

13. receiving a ranging image from at least one camera that captures and measures distance to at least a portion of a verification area including at least one of a plurality of markers for defining a verification area of ​​a person, and generates the ranging image; Using the distance measurement image, face matching is performed on a person passing through the matching area; determining whether the marker or the camera has shifted from an initial state using the distance measurement image; Person matching program.

Images