Information processing device, information processing system, information processing method, and information processing program

Abstract

The present invention provides a device, system, method, and program that can appropriately detect and identify a person attempting to illegally pass through an automatic ticket gate from captured images. [Solution] The information processing device includes a communication unit that acquires captured images from an imaging unit that images a gate for selectively permitting or restricting passage to an area restricted to persons who meet specific conditions, and a control unit that detects a person passing through a first area and a predetermined object present in a second area from the captured images acquired by the communication unit, and the control unit determines whether the person is making an unauthorized entry into the area based on the change in the combination of the detection result of the person passing through the first area and the detection result of the predetermined object present in the second area.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing system, an information processing method, and an information processing program.

Background Art

[0002] In facilities such as stations, airports, event venues, and office buildings, restricted areas are provided where only people (users) who meet specific conditions (authorizations) can access. Gates for entry are provided in such areas, and confirmation of entry (entry management) of entrants is performed by an authentication system, an automatic ticket gate, etc. for payment confirmation, prevention of unauthorized entry, and ensuring safety.

[0003] Conventionally, a method has been known for quickly obtaining information on unauthorized users who have illegally breached an automatic ticket gate (gate) using face recognition technology. For example, in Patent Document 1, a monitoring center is connected to each station, and the automatic ticket gates and surveillance cameras at each station monitor for unauthorized use. The face recognition unit performs image processing on the recorded face images and assigns an authentication code to each person. Also, when it is detected by a sensor that an automatic ticket gate has been illegally breached, the control unit records the time when the surveillance camera took the picture and the face image of the person. Then, the aggregation processing unit creates a report including the station where the act of illegally breaching the automatic ticket gate was performed and the face image of the unauthorized user based on the authentication code, and monitors and records the illegal acts occurring at each station. By such means, a system is configured that can quickly identify and track unauthorized users.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the aforementioned system, for individuals including fraudulent users, (1) a facial recognition unit recognized the person's facial image and assigned an authentication code, and (2) sensors installed in the automatic ticket gate detected whether or not the person had passed through the ticket gate.

[0006] Therefore, the facial recognition technology described in (1) had the problem that facial recognition could not be performed properly and people could not be identified if a person concealed their face while passing through, if face detection was not possible due to clothing or other items, or in environments with insufficient lighting or backlighting. Furthermore, the sensor-based person detection method described in (2) had the problem that a person could not be detected until they entered the ticket gate passage, and that people could not be identified or specified using only sensors.

[0007] On the other hand, technologies for detecting the entire human body, such as skeletal detection (hereinafter referred to as "human body detection"), have been known for some time. However, no method for using this human body detection to detect users (people) at ticket gates was known, and there was room for consideration as to how to use it to detect users at ticket gates.

[0008] Therefore, technologies are needed to solve the aforementioned problems.

[0009] In non-limiting embodiments of this disclosure, it is possible to provide an information processing device, information processing system, information processing method, and information processing program that can detect a person who is a user of a ticket gate from a captured image and appropriately determine an unauthorized user attempting to pass through the ticket gate illegally. [Means for solving the problem]

[0010] An information processing device according to one embodiment of the present disclosure includes a communication unit that acquires captured images from an imaging unit that images a gate for selectively permitting or restricting passage to an area restricted to persons who meet specific conditions; and a control unit that detects a person passing through a first area and a predetermined object present in a second area from the captured images acquired by the communication unit, wherein the control unit determines whether the person is making an unauthorized entry into the area based on a change in the combination of the detection result of the person passing through the first area and the detection result of the predetermined object present in the second area.

[0011] These comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or recording media, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media. [Effects of the Invention]

[0012] According to non-limiting embodiments of this disclosure, by combining the detection of a person passing through the ticket gate and the detection of the ticket gate door (flap) from captured images, it is possible to accurately identify a person attempting to pass through the ticket gate illegally. Furthermore, in the embodiments of this disclosure, illegal passage can be detected using only camera images, offering high flexibility in installation and making it effective as a simple surveillance system. In addition, by performing human body detection in the embodiments of this disclosure, it is possible to achieve highly accurate person detection with flexibility and robustness.

[0013] Further advantages and effects of one embodiment of this disclosure will be made apparent from the specification and drawings. Such advantages and / or effects are provided by several embodiments and features described in the specification and drawings, but not all of them are necessarily provided in order to obtain one or more identical features. [Brief explanation of the drawing]

[0014] [Figure 1]It is a diagram showing an example of a system configuration for detecting unauthorized passage according to an embodiment of the present disclosure. [Figure 2] It is a diagram showing an example of the installation of a camera for photographing a ticket gate. [Figure 3] It is a diagram showing an example of an image photographed by the installed camera. [Figure 4] It is a diagram showing an example of an area set in the ticket gate. [Figure 5] It is a diagram showing an example of a detection result of detecting the skeleton of a person. [Figure 6] It is a diagram showing an example of a state where the flap of the ticket gate is closed. [Figure 7] It is a diagram showing an example of a matrix showing the results of person detection and object detection. [Figure 8] It is a diagram showing an example of a photographed image when a person approaches the ticket gate. [Figure 9] It is a diagram showing an example of a matrix showing the results of person detection and object detection in FIG. 8. [Figure 10] It is a diagram showing an example of a photographed image when a person enters the ticket gate. [Figure 11] It is a diagram showing an example of a matrix showing the results of person detection and object detection in FIG. 10. [Figure 12] It is a diagram showing an example of a photographed image when a person approaches the closing point of the ticket gate. [Figure 13] It is a diagram showing an example of a matrix showing the results of person detection and object detection in FIG. 12. [Figure 14] It is a diagram showing an example of a photographed image when a person approaches the exit line of the ticket gate. [[ID=X]] [Figure 15] It is a diagram showing an example of a matrix showing the results of person detection and object detection in FIG. 14. [Figure 16] It is a diagram showing an example of a photographed image when a person approaches a one-way ticket gate. [Figure 17] It is a diagram showing an example of a matrix showing the results of person detection and object detection in FIG. 16. [Figure 18]This is a diagram showing an example of a captured image when a person approaches the closing point of the ticket gate with the flap of the ticket gate closed. [Figure 19] This is a diagram showing an example of a matrix indicating the results of person detection and object detection in FIG. 18 and an example of state variables. [Figure 20] This is a diagram showing an example of a captured image when a person forcefully passes through the closed flap of the ticket gate. [Figure 21] This is a diagram showing an example of a matrix indicating the results of person detection and object detection in FIG. 20 and an example of state variables. [Figure 22] This is a diagram showing an example of the state where the flap of the ticket gate is closed after a person has passed through the ticket gate normally. [Figure 23] This is a diagram showing an example of a captured image when a person approaches the closing point of the ticket gate with the flap of the ticket gate closed. [Figure 24] This is a diagram showing an example of a captured image when a passerby crosses in front of a person in the state of FIG. 23. [Figure 25] This is a diagram showing an example of face authentication performed on the person and the passerby in FIG. 24. [Figure 26] This is a diagram showing an example of calculating the movement vector for the person and the passerby in FIG. 24. [Figure 27] This is a diagram showing an example of a situation where a part of the captured image is blocked with the flap of the ticket gate open. [Figure 28] This is a diagram showing an example of a situation where a part of the captured image is blocked with the flap of the ticket gate closed. [Figure 29] This is a diagram showing an example of a situation where a part of the captured image is blocked while a person is passing through the ticket gate. [Figure 30] This is a diagram showing an example of the result of performing complementary processing for skeleton detection of a person. [Figure 31] This is a diagram showing an example of providing an offset for the monitoring area. [Figure 32] This is a diagram showing an example of detecting the contour of a person by an image segmentation method. [Figure 33]This figure shows an example of converting lines connecting parts detected by skeletal detection into rectangles. [Figure 34] This is a flowchart showing the operation of the main processing unit of the information processing device. [Figure 35] This is a flowchart showing the operation of the person information processing device. [Figure 36] This is a flowchart showing the operation of the information processing device's unauthorized passage detection process. [Figure 37] This is a flowchart showing the operation of the information processing device's person list processing. [Figure 38] This flowchart shows the operation of the information processing device's illegal passage detection process in the case of the previous frame during normal passage. [Figure 39] This flowchart shows the operation of the information processing device's fraudulent passage detection process in the case of a post-frame during normal traffic. [Figure 40] This flowchart shows the operation of the information processing device's illegal passage detection process in the case of an illegal passage in the preceding frame. [Figure 41] This flowchart shows the operation of the information processing device's illegal passage detection process in the case of a post-frame incident involving illegal passage. [Modes for carrying out the invention]

[0015] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same function are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0016] <Configuration of the information processing system> Figure 1 shows an example of the system configuration of the information processing system 10 according to this embodiment. The information processing system 10 has the function of detecting and determining a person who is attempting to pass through an automatic ticket gate illegally. The information processing system 10 consists of an information processing device 1 and a camera 2, and the camera 2 is connected to the information processing device 1 by wire and / or wireless. The information processing device 1 may be connected to a ticket gate 3, etc. via a network so that it can acquire linked data from the ticket gate 3, etc.

[0017] The information processing device 1 comprises a control unit 11, a communication unit 12, an input device 13, an output device 14, a main memory unit 15, and an auxiliary memory unit 16. The information processing device 1 may be implemented using cloud computing, where computing resources are provided via the internet. Alternatively, the information processing device 1 may be implemented using information processing functions provided within a miniature camera.

[0018] The control unit 11 is composed of a central processing unit (CPU) and other components, and controls the overall operation of the system. The control unit is a unit that controls its operation according to a program, acquiring and decoding instructions to be processed, issuing instructions to other hardware at the appropriate time to execute the processing, and overseeing instruction execution, data processing flow management, arithmetic processing, and the operation of each functional unit.

[0019] The control unit 11 may also be referred to as an arithmetic unit, processor, or controller. The control unit 11 further comprises various functional units such as a person detection unit 111, an object detection unit 112, and a determination unit 113. These functional units work together to detect and determine whether a person is attempting to pass through an automatic ticket gate illegally. The operation of each function will be described later in the section on <Operation of Information Processing Device>.

[0020] The communication unit 12 connects to a network interface, the Internet, or other devices. This enables the communication unit 12 to exchange data with the camera 2 as well as other devices and systems inside and outside the system (for example, the ticket gate 3 described later). The communication unit 12 may include wireless communication such as LAN (Local Area Network), Wi-Fi, Bluetooth, or wired communication (such as Ethernet).

[0021] The input device 13 is a device for inputting data from an external source (e.g., text, audio, images, sensor information, etc.) in a format that the information processing device 1 can process. For example, a keyboard, mouse, or touch panel may be connected as the input device 13. The user inputs information to the information processing device 1 using the input device 13.

[0022] The output device 14 is a device for transmitting data processed by the information processing device 1 to the user or an external system, and outputs the processing results of the information processing device 1 visually, audibly, or physically. For example, a display (monitor) or speaker may be connected as the output device 14.

[0023] The main memory unit 15 is composed of main memory (RAM: Random Access Memory) and other components, enabling high-speed access and temporarily storing programs and data used within the computer system. Programs and data are loaded into the main memory unit 15, and the CPU of the control unit 11 processes these programs and data to realize the program's functionality. The main memory unit 15 also stores variables and data structures used when the program is executed.

[0024] The auxiliary storage unit 16 is a storage device that consists of a hard disk drive (HDD) or a solid state drive (SSD), and is used to store large amounts of data for a long period of time. Any storage device that can retain data even when the computer is turned off will suffice, and optical discs (CDs, DVDs), USB memory sticks, magnetic tapes, etc., may be used as the auxiliary storage unit 16.

[0025] Camera 2 is equipped with an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor) sensor inside the camera. Camera 2 uses an imaging lens to form an image of the object onto the surface of the image sensor, converts the formed image into an electrical signal using the image sensor, and outputs it as digital data to the information processing device 1.

[0026] [Camera installation] As shown in Figure 2 as an example, camera 2 is installed above the front of the ticket gate 3 and captures an overhead view of the entire ticket gate 3. Camera 2 can be any existing surveillance camera capable of recording video, and there are no limitations on the type or number of cameras. Camera 2 is not limited to the example in Figure 2; it may be installed as part of the ticket gate 3, or around the entrance and exit of the ticket gate 3. For example, if camera 2 captures an image of a person 4, who is a user attempting to enter the ticket gate 3 from the front, it will capture an image like that in Figure 3. Camera 2 transmits the captured image (hereinafter referred to as "captured image") to the information processing device 1.

[0027] <Operation of the Information Processing Device> Next, we will explain the operation of the information processing device, specifically "person detection," "object detection," and "decision processing."

[0028] [Person Detection] As shown in Figure 1, the information processing device 1 acquires images captured by the camera 2 via the communication unit 12. The control unit 11 of the information processing device 1 performs image processing on the acquired images. The person detection unit 111 of the information processing device 1 has the function of detecting people 4 from the entire acquired images. The person detection unit 111 detects people throughout the image from the images acquired from the camera 2 using a person detection algorithm. For example, the person detection unit 111 may sequentially detect people present in the image using a known library (a known image recognition library such as OpenCV or YOLO).

[0029] Here, we will explain how the person detection unit 111 detects an area from the acquired captured image. As shown in Figure 4, within the area of ​​the acquired captured image, the entrance to the ticket gate 3 is defined by the "entrance line 31," and the exit to the ticket gate 3 is defined by the "exit line 33." The area enclosed by the entrance line 31 and the exit line 33 is called the "monitoring passage 32." The person detection unit 111 detects that person 4 is located within this "monitoring passage 32" (hereinafter referred to as "person detection").

[0030] The person detection unit 111 detects a person in the captured image and then calculates the coordinates (in the image) of the person 4. For example, the person detection unit 111 may use a skeleton detection library (a known image recognition library such as OpenPose or PoseNet) to perform skeleton detection (human body detection) and, for example, as shown in Figure 5, detect each part of the person 4 present in the captured image. The number and location of the detected feature points will differ depending on the type and method of the library used and the options set, so it is not necessarily limited to the form shown in Figure 5. The skeleton detection library does not require an external GPU and can be processed relatively quickly even on a general-purpose PC. From the detection result of the person 4, the person detection unit 111 obtains the coordinates of the ankles (parts 10 and 13) and calculates the leg coordinates 45 where the person 4 is located from the obtained ankle coordinates.

[0031] The person detection unit 111 calculates the leg coordinates 45 of person 4 in the captured image, and then detects that person 4 is located within the monitoring passage 32 based on the calculated leg coordinates 45 of person 4 and the position of the monitoring passage 32.

[0032] Furthermore, the person detection unit 111 detects whether person 4 is overlapping the monitoring area 35 of the ticket gate 3 in the captured image in order to determine whether person 4 is passing through illegally. The person detection unit 111 may also detect that person 4 has crossed the closing point 36 of the ticket gate 3.

[0033] [Object Detection] The object detection unit 112 of the information processing device 1 has the function of detecting an object (flap 34 or door 34) from the acquired captured image.

[0034] As shown in Figure 6, the ticket gate 3 is a flap-type automatic ticket gate equipped with a flap (flapper or door) 34. The flap 34 is a movable plate-shaped or wing-shaped structure installed in the monitoring passage 32 of the ticket gate 3, and has the function of allowing or blocking the passage of users. When the ticket gate 3 successfully authenticates the user's IC card, magnetic ticket, QR ticket (ticket), or biometric authentication (e.g., facial recognition, iris recognition, vein recognition, etc.), it opens the flap 34 to allow the user to pass through. On the other hand, if the authentication of the user's IC card or ticket, biometric authentication, etc. fails, or if unauthorized passage or fraudulent passage is detected, the ticket gate 3 closes the flap 34 to block the user's passage. In addition, the ticket gate 3 also closes the flap 34 to prevent users from passing in the opposite direction when the direction of passage is set to one-way.

[0035] As shown in Figure 6, the ticket gate 3 closes its flaps 34 when blocking a user's passage. When the flaps 34 are closed in this manner, the area in which both flaps 34 are visible is called the "monitoring area 35". The position in which the flaps 34 of the ticket gate 3 close is called the "closed point 36".

[0036] The object detection unit 112 monitors the monitoring area 35 of the ticket gate 3 from the acquired captured image and detects that the flaps 34 are closed (hereinafter referred to as "object detection"). On the other hand, the object detection unit 112 does not detect that the flaps 34 are closed when the flaps 34 of the ticket gate 3 are open and there are no two closed flaps 34 in the monitoring area 35. Also, the object detection unit 112 does not detect that the flaps 34 are closed if a person 4 is standing in front of the two closed flaps 34. In this case, the result of "object detection" will be "none".

[0037] For example, the object detection unit 112 may monitor the monitoring area 35 within the captured image and detect whether the flap 34 of the ticket gate 3 is closed by extracting predetermined image features from the captured image using methods such as background difference comparison, line detection, contrast comparison, and pattern matching (template matching), or by comparing the captured image with a predetermined pattern and detecting matching parts. The object detection unit 112 is not limited to the examples described above, and any method that can recognize and identify the target object may be used. By narrowing the area, detection parameters and judgment criteria can be determined quickly and with high accuracy, and the processing time itself can be kept short.

[0038] [Person detection / object detection matrix and decision processing] The determination unit 113 of the information processing device 1 acquires the person detection result from the person detection unit 111 and the object detection result from the object detection unit 112, and classifies the person's movement status into four patterns. Figure 7 is an example of showing the results of the "person detection" and "object detection" described above in a matrix. For example, consider the case where the person detection unit 111 and the object detection unit 112 perform detection processing on a captured image like the one in Figure 6. Since person 4 is not in the monitoring passage 32, the result of "person detection" is "none". Also, since the person is in the monitoring area 35 with the flap 34 closed, the result of "object detection" is "present". In this case, the pattern of the person detection / object detection matrix is ​​classified into pattern (3) as shown in Figure 7. A matrix in which the results of "person detection" and "object detection" performed by the person detection unit 111 and the object detection unit 112 are shown as patterns is called the "person detection / object detection matrix". The determination unit 113 uses this person detection / object detection matrix to pattern and track the progress of person 4 passing through ticket gate 3, and determines whether unauthorized passage has occurred.

[0039] Next, we will explain how the determination unit 113 classifies the progress of a person passing through the ticket gate 3 into patterns and determines whether it is illegal passage, using an example of a person detection / object detection matrix.

[0040] (Under normal circumstances) When authentication of the user's IC card or ticket, biometric authentication, etc., is successful and the ticket gate 3 permits the user to pass through, the person detection / object detection matrix for person 4, who is the user passing through the ticket gate 3, will be described below in chronological order.

[0041] Figure 8 shows an example of an image captured when person 4 approaches the ticket gate 3. In the image in Figure 8, person 4 has not yet entered the monitoring passage 32, so the "person detection" result is "none". Also, the flaps 34 of the ticket gate 3 are stored in the storage spaces on both sides in the open position, so the flaps 34 are not detected in the monitoring area 35. As a result, the "object detection" result is "none". Therefore, the person detection / object detection matrix in this case is classified into pattern (4), as shown in Figure 9.

[0042] Figure 10 shows an example of an image captured when person 4 crosses the entry line 31 of the ticket gate 3 and enters the monitoring passage 32. In the image captured in Figure 10, person 4 has entered the monitoring passage 32, so the "person detection" result is "present". Also, the flaps 34 of the ticket gate 3 are stored in the storage spaces on both sides in the open position, and the flaps 34 are not detected in the monitoring area 35, so the "object detection" result is "absent". Therefore, the person detection / object detection matrix in this case is classified into pattern (2), as shown in Figure 11.

[0043] Figure 12 shows an example of an image captured when person 4 approaches the closing point 36 of the ticket gate 3. In the image captured in Figure 12, person 4 has entered the monitoring passage 32, so the "person detection" result is "present". Also, the flaps 34 of the ticket gate 3 are stored in the storage spaces on both sides in the open position, and the flaps 34 are not detected in the monitoring area 35, so the "object detection" result is "absent". Therefore, the person detection / object detection matrix in this case is classified into pattern (2), as shown in Figure 13.

[0044] Figure 14 shows an example of an image taken when person 4 has passed the closing point 36 of the ticket gate 3 and is approaching the exit line 33. In the image taken in Figure 14, person 4 is still inside the monitoring passage 32, so the "person detection" result is "present". Also, the flaps 34 of the ticket gate 3 are stored in the storage spaces on both sides in the open position, and the flaps 34 are not detected in the monitoring area 35, so the "object detection" result is "absent". Therefore, the person detection / object detection matrix in this case is classified into pattern (2), as shown in Figure 15.

[0045] The above description shows the person detection / object detection matrix (Figures 9, 11, 13, 15) for a person 4 (Figures 8, 10, 12, 14) passing through the ticket gate 3 during normal passage. As explained above, during normal passage, the determination unit 113 does not determine that passage is illegal.

[0046] (In the case of normal traffic in a one-way street) Furthermore, if the ticket gate 3 is in a one-way (one-way, single-direction, or one-way) state, the image taken before person 4 enters will be as shown in Figure 16, for example. In the image taken in Figure 16, person 4 has not yet entered the monitoring passage 32, so the "person detection" result is "none". Also, the flap 34 of the ticket gate 3 is closed, and the flap 34 is detected in the monitoring area 35, so the "object detection" result is "present". Therefore, the person detection / object detection matrix in this case is classified into pattern (3), as shown in Figure 17.

[0047] When ticket gate 3 is in a one-way state, if person 4 crosses the entry line 31 of ticket gate 3 and enters the monitoring passage 32, a sensor installed inside ticket gate 3 is activated, the one-way state is released, and the system transitions to the state shown in Figure 10. After this, if authentication of person 4's IC card or ticket, biometric authentication, etc. is successful, the subsequent states will transition in the same manner as described in Figures 12 to 15.

[0048] The above explanation describes the determination process in the case of normal passage in a one-way situation, and the person detection / object detection matrix for person 4 (Figures 16, 10, 12, 14) passing through ticket gate 3 (Figures 17, 11, 13, 15).

[0049] (In case of illegal passage) Let's consider a scenario where, despite the user's IC card or ticket authentication, biometric authentication, etc., failing and the ticket gate 3 closing its flap 34 to block the user's passage, the user 4 forcibly passes through the closing point 36 of the ticket gate 3. In such a case, the person detection / object detection matrix and the determination of unauthorized passage for person 4, the user passing through the ticket gate 3, will be explained below in chronological order.

[0050] The behavior and state transitions when Person 4 approaches the ticket gate 3, and when Person 4 crosses the entry line 31 of the ticket gate 3 and enters the monitoring passage 32, are the same as those described in Figures 8-11 for normal passage. After this, the behavior and state transitions when Person 4's IC card or ticket authentication, biometric authentication, etc., fails and Person 4 approaches the closing point 36 of the ticket gate 3 will be described sequentially below.

[0051] Figure 18 shows an example of an image captured when person 4 approaches the closing point 36 of the ticket gate 3. In the image captured in Figure 18, person 4 has entered the monitoring passage 32, so the "person detection" result is "present". Also, authentication of person 4's IC card or ticket, biometric authentication, etc., has failed, the flap 34 of the ticket gate 3 is closed, and the flap 34 is detected in the monitoring area 35, so the "object detection" result is "present". Therefore, the person detection / object detection matrix in this case is classified into pattern (1) as shown in Figure 19.

[0052] The pattern shown in Figure 19 (1) above means that person 4 has entered the monitoring passage 32 despite the flap 34 being closed. The determination unit 113 determines that the state in which the flap 34 is closed and person 4 has entered the monitoring passage 32 is a state in which unauthorized passage is possible, and sets the "possibility of unauthorized passage flag" to "ON" as shown in Figure 19.

[0053] Figure 20 shows an example of an image taken when, starting from the state of the image taken in Figure 18, person 4 crosses the closing point 36 of the ticket gate 3 by stepping over, jumping over, or forcibly passing through by pushing open the closed flap 34. The person detection unit 111 detects that person 4 is overlapping within the monitoring area 35 of the ticket gate 3, so the determination unit 113 recognizes "passage through the closing point 36" (see Figure 21). In the image taken in Figure 20, person 4 is inside the monitoring passage 32, so the result of "person detection" is "yes". Also, in the monitoring area 35 of the image taken in Figure 20, a person standing in front of the flap 34 is detected, so the result of "object detection" is "no". Therefore, the person detection / object detection matrix for the image taken in Figure 20 is classified into pattern (2) as shown in Figure 21.

[0054] The transition from Figure 18 to Figure 20, as described above, means that after the "possibility of unauthorized passage flag" was set to "ON," person 4 forcibly passed through the closing point 36 of the ticket gate 3 ("passing through closing point 36") (see Figure 21). In such a case, the determination unit 113 determines that person 4 has committed unauthorized passage.

[0055] The above explanation describes how illegal passage is detected and shows the person detection / object detection matrix for person 4 (Figures 8, 10, 18, 20) passing through ticket gate 3 (Figures 9, 11, 19, 21).

[0056] (In the case of illegal passage in a one-way street) This section describes how to pass through illegally when ticket gate 3 is in a one-way state. When person 4 approaches ticket gate 3, the situation is the same as in Figure 16. After person 4 crosses the entry line 31 of ticket gate 3 and enters the monitoring passage 32, a sensor installed inside ticket gate 3 is activated, and the one-way state is released as shown in Figure 10. Subsequently, if authentication of person 4's IC card or ticket, biometric authentication, etc., fails, the situation transitions as described in Figures 18 to 21.

[0057] The above explanation describes how illegitimate passage is detected in a one-way situation, and the person detection / object detection matrix for person 4 (Figures 16, 10, 18, 20) passing through ticket gate 3 (Figures 17, 11, 19, 21).

[0058] (If you turn back from the closing point 36) The behavior and state transitions when person 4 approaches the ticket gate 3, and when person 4 crosses the entry line 31 of the ticket gate 3 and enters the monitoring passage 32, are the same as those described in Figures 8 to 11. Subsequently, if authentication of person 4's IC card or ticket, biometric authentication, etc., fails, and person 4 approaches the closing point 36 of the ticket gate 3 as shown in Figure 18 above, the flap 34 closes. Now, let's consider the case where person 4 leaves the closing point 36 of the ticket gate 3 and turns back. For example, suppose person 4 returns from the state in Figure 18 to the state in Figure 8 above. The determination unit 113 initializes the "possibility of unauthorized passage flag" to "OFF" when person 4 leaves the monitoring passage 32 and turns back beyond the entry line 31, as shown in Figure 8. Therefore, if person 4 turns back beyond the entry line 31 from the closing point 36, it is not determined that unauthorized passage occurred.

[0059] The above explanation describes the determination process when turning back from the closing point 36, and the person detection / object detection matrix for a person 4 (Figures 8, 10, 18, 8) passing through the ticket gate 3 (Figures 9, 11, 19, 9).

[0060] (If you turn back and then attempt to pass through illegally) The behavior and state transitions when person 4 approaches the ticket gate 3, and when person 4 crosses the entry line 31 of the ticket gate 3 and enters the monitoring passage 32, are the same as those described in Figures 8 to 11. Subsequently, the behavior and state transitions when authentication of person 4's IC card or ticket, biometric authentication, etc., fails and person 4 approaches the closing point 36 of the ticket gate 3 are the same as those described in Figures 18 to 19.

[0061] From this state, we assume that person 4 leaves the closing point 36 of the ticket gate 3, turns back, and then jumps over, steps over, or pushes open the flap 34 of the ticket gate 3 to forcibly pass through.

[0062] The behavior and state transitions when Person 4 leaves the closing point 36 of the ticket gate 3 and turns back are the same as in the case of "turning back from the closing point 36" described above. For example, Person 4 turns back to the state shown in Figure 8.

[0063] Subsequently, Person 4 takes a running start and jumps over, steps over, or pushes open the flap 34 of the ticket gate 3 to forcibly pass through. As explained earlier, in the state transitioning from Figure 18 to Figure 20, the determination unit 113 recognizes that Person 4 has passed the "closed point 36" of the ticket gate 3 after the "possibility of unauthorized passage flag" is set to "ON" (see Figure 21), and therefore determines that Person 4 has committed unauthorized passage.

[0064] The above explanation describes how the system is judged when a person turns back and then attempts to pass through illegally, and shows the person detection / object detection matrix for person 4 (Figures 8, 10, 18, 8, 10, 18, 20) passing through ticket gate 3 (Figures 9, 11, 19, 9, 11, 19, 21).

[0065] <Measures to improve accuracy> Next, we will explain matters related to preventing misrecognition and improving accuracy in the "judgment processing" based on the "person detection" and "object detection" described above.

[0066] (To prevent incorrect decisions after the game has started) As described above in Figures 8-15, we assume that person 4 passes through the ticket gate 3 normally. We assume that after person 4 has passed through the ticket gate 3 normally and exited the exit line 33, the flap 34 of the ticket gate 3 closes, as shown in Figure 22.

[0067] In such a case, the object detection unit 112 detects the flap 34 in the monitoring area 35, which could lead the judgment unit 113 to mistakenly determine that the person 4 who has stepped outside the exit line 33 has committed illegal passage. To avoid such a misjudgment, a "passage line," such as the exit line 33, is installed at a position beyond the monitoring area 35 and the closing point 36. Then, even if the flap 34 of the ticket gate 3 closes and "object detection" occurs after the person 4 has crossed the "passage line," the judgment unit 113 will not perform any illegal passage determination processing for the person 4.

[0068] As explained above, even if object detection occurs after person 4 has crossed the "pass line," misidentification can be prevented by not performing any fraudulent judgment processing for person 4.

[0069] (Distinguishing between users and passersby) As shown in Figure 23, let's assume that the flap 34 of the ticket gate 3 closes while person 4 (user) is passing through the ticket gate 3. Immediately after this, as shown in Figure 24, let's assume that passerby 5 crosses in front of camera 2 and this is captured on camera. In such a case, since passerby 5 is detected overlapping within the monitoring area 35, the determination unit 113 may mistakenly recognize passerby 5 in the captured image in Figure 24 as person 4 who committed illegal passage.

[0070] As a solution in such cases, for example, the control unit 11 of the information processing device 1 acquires the face portion 41 of person 4 when person 4 enters the ticket gate 3 and stores the characteristic information of person 4's face. Then, as shown in Figure 25, the control unit 11 detects the face portion 41 of person 4 and the face portion 42 of passerby 5, respectively, in the captured image. The control unit 11 performs one-to-one facial recognition (matching) on ​​the detected face portion 41 of person 4 and the face portion 42 of passerby 5. Because the control unit 11 distinguishes between person 4 and passerby 5 through this recognition, misidentification can be prevented.

[0071] For example, a known facial recognition library may be used for the facial recognition processing performed by the control unit 11. The control unit 11 may use the facial recognition library to detect the face portion 41 of person 4 and the face portion 42 of passerby 5 from the captured image, perform facial recognition on the detected face portions, and distinguish between person 4 and passerby 5 by comparing the recognition results.

[0072] Furthermore, as shown in Figure 26, the control unit 11 of the information processing device 1 may acquire the movement vectors (motion vectors) of the person 4 and the passerby 5 detected from the captured image, and perform processing to distinguish between the person 4 and the passerby 5.

[0073] For example, the control unit 11 may use techniques such as Optical Flow or utilize known image processing libraries for the motion vector acquisition process.

[0074] Furthermore, the control unit 11 may combine the facial recognition process and the motion vector processing described above as necessary.

[0075] (Detection assistance) As shown in Figures 27-28, let's assume that person 6 is standing in front of camera 2. In this case, because person 6 obstructs part of the captured image, the object detection unit 112 will be unable to detect the aforementioned "monitoring area 35". Also, person detection unit 111 will be unable to detect the "monitoring passage 32".

[0076] In such cases, the object detection unit 112 may set the indicator lights or warning lights (37, 38) of the ticket gate 3, as shown in Figures 27 and 28, as the "monitoring area" instead of the "monitoring area 35". The object detection unit 112 may perform "object detection" using the display patterns or lighting patterns of the indicator lights or warning lights (37, 38) of the ticket gate 3. The object detection unit 112 determines from the display patterns or lighting patterns of the indicator lights or warning lights (37, 38) of the ticket gate 3 whether the flap 34 of the ticket gate 3 is closed (passage is restricted).

[0077] For example, the example shown in Figure 27 is an example of a person 4 passing through normally. The object detection unit 112 determines from the state of the indicator lights and warning lights (37, 38) that the flap 34 of the ticket gate 3 is open (passage is not restricted).

[0078] For example, the example shown in Figure 28 is an example where the flap 34 of the ticket gate 3 is closed, blocking the passage of person 4. The object detection unit 112 determines from the state of the indicator lights and warning lights (37, 38) that the flap 34 of the ticket gate 3 is closed (passage is restricted).

[0079] Furthermore, even if the person detection unit 111 cannot detect the "monitoring passage 32," it may still detect the person's position and movement based on the person's movement vector (motion vector) and its positional relationship with the entry / exit lines 31 and 33, and perform "person detection." For example, the person detection unit 111 can detect that person 4 has exited if person 4 moves in the direction of travel beyond the vicinity of the exit line 33 of the ticket gate 3 and moves away from the ticket gate 3.

[0080] (Prediction and interpolation of leg coordinates) As shown in Figure 29, if a part of person 4's body in the captured image is obscured by person 6, the person detection unit 111 cannot detect the obscured part of person 4's body. In such cases, the person detection unit 111 may perform interpolation processing by using a skeleton detection library to perform skeleton detection (human body detection) and estimating the position of the obscured part. Furthermore, in addition to being obscured by person 6, etc., part of person 4's body may be obscured or out of frame and not captured due to the camera's installation environment or the person 4's physique (individual differences). In such cases as well, the person detection unit 111 may perform interpolation processing in the same manner.

[0081] As shown in Figure 30, the person detection unit 111 performs skeletal detection (human body detection) using a skeletal detection library, and estimates and interpolates the distance and coordinates of obstructed parts connected to the detected parts from the coordinates and distances of the detected parts. For example, in the example in Figure 30, the person detection unit 111 interpolates by drawing a perpendicular line from the coordinates of part 8 by a multiplier applied to the distance between part 8 and part 2 to estimate the coordinates of part 9. Similarly, the person detection unit 111 estimates the coordinates of part 10, part 12, and part 13. The person detection unit 111 interpolates by estimating the leg coordinates 45 from the estimated coordinates of part 10 and part 13.

[0082] If the detection results of the library used do not include the parts necessary for detecting a person, the person detection unit 111 may perform interpolation by estimating the coordinates of the missing parts. For example, if interpolation of the fingertips is required, the tilt of the arm may be taken into consideration, and the person detection unit 111 may perform interpolation by estimating the position of part 9 as the value obtained by multiplying the distance between part 6 and part 7 by a factor, on the extension of the coordinates along the tilt of part 6 and part 7, from the coordinates of part 6. The distance, factor, and coordinate direction (tilt) can be freely set. Note that the coordinates of the legs may exceed the image size.

[0083] In the skeleton detection (human body detection) performed by the person detection unit 111, a score value is calculated for each detection result, and results that do not reach a reliable score value may not be adopted. If the legs are cut off, the score may fall below a threshold, making it impossible to perform the intended interpolation process. In such cases, a process to increase the score value only for the legs may be performed. For example, a correction score may be increased, and this correction score may be used to decide whether to accept or reject the detection result.

[0084] As described above, even if the person detection unit 111 cannot detect the "monitoring passage 32", it can estimate the position of person 4 by predicting and interpolating the leg coordinates 45, thus enabling "person detection".

[0085] (Processing to determine overlap between the surveillance area and the detected person) As mentioned above in the "Person Detection" section, the person detection unit 111 performs a process to detect person 4 within the monitoring area in the captured image in order to determine whether person 4 is passing through illegally. For example, as shown in Figure 31, the person detection unit 111 detects person 4 within the monitoring area 39 by detecting the overlap between the monitoring area 39 and person 4. In this case, the parts of person 4 (part 7, part 19) detected by skeletal detection (human body detection) are located inside the actual outline of the human body. Therefore, even if person 4 has entered the monitoring area 39, the parts of person 4 (part 7, part 19) may be determined to be located outside the monitoring area 39, and a correct determination may not be possible. To prevent such misdetermination, the following methods (1) to (3) may be adopted.

[0086] (1) The person detection unit 111 sets an offset 40 in the monitoring area 39 to adjust the difference between the contour and the skeleton, and detects whether there is an overlap between the parts of person 4 (part 7, part 19) and the offset 40 (see Figure 31).

[0087] (2) The person detection unit 111 accurately detects the contour 43 of person 4 using an image segmentation method and detects whether there is an overlap of the contour 43 of person 4 with respect to the monitoring area 39 (see Figure 32).

[0088] (3) The person detection unit 111 converts the lines connecting the parts detected by skeletal detection (human body detection) into rectangles 42 that are enlarged by a predetermined width or a magnification for each part, and uses these rectangles 44 as parts of the person 4 to detect whether there is any overlap with the monitoring area 39 (see Figure 33).

[0089] Image segmentation is a technique that divides an image into multiple segments (regions). For example, by assigning predefined categories to pixels in an image, it is possible to distinguish between people and the background in detail. By using this image segmentation technique, the person detection unit 111 can extract and identify specific objects or regions within an image. The person detection unit 111 can achieve the same functionality as skeleton detection (human body detection) by using image segmentation instead of skeleton detection (human body detection).

[0090] The above methods may be combined and adopted as needed.

[0091] The above description outlines the processing methods for improving accuracy. The control unit 11 may combine the above-described processes as needed.

[0092] <Flowchart of the operation of the information processing device> Next, the operation of the information processing device 1 described above will be explained below using a flowchart.

[0093] [Main process] Figure 34 is a flowchart illustrating an example of the operation of the main processing of the information processing device 1. The information processing device 1 repeatedly executes the processing shown in the flowchart of Figure 34, for example, at a fixed interval.

[0094] In step S1, the control unit 11 acquires the captured image from the camera 2 via the communication unit 12.

[0095] In S2, the person detection unit 111 detects people from the entire captured image acquired from the camera 2. The person detection unit 111 registers the detected people in the "detection list" of the main memory unit 15. If the person detection unit 111 detects one or more people (S2 is "number of detections ≥ 1"), the flow proceeds to S3. On the other hand, if the person detection unit 111 does not detect any people (S2 is "number of detections = 0"), the flow proceeds to S7.

[0096] In S3, if the entire body of the person detected in S2 is not captured, the person detection unit 111 estimates and fills in the missing parts of the person (see Figure 30).

[0097] In S4, the person detection unit 111 refers to the "detection list" in the main memory unit 15 where the people detected in the main process S2 are registered, and detects people in the monitoring passage 32 from the leg coordinates 45 of the people (see Figure 4). As a result, if the person detection unit 111 detects one or more people in the monitoring passage 32 (S4 is "number of detections ≥ 1"), the flow proceeds to S5. On the other hand, if the person detection unit 111 does not detect any people in the monitoring passage 32 (S4 is "number of detections = 0"), the flow proceeds to S7.

[0098] In S5, the determination unit 113 determines whether there is an unauthorized passerby based on the detection results of the person detection unit 111 and the object detection unit 112 within the monitored passageway 32. If the determination unit 113 determines that there is an unauthorized passerby (S5 is "Determination Result = "Unauthorized Passerby"), the flow proceeds to S6. On the other hand, if the determination unit 113 does not determine that there is an unauthorized passerby (S5 is "Determination Result = "Normal"), the flow proceeds to S7.

[0099] In S6, the determination unit 113 outputs an "illegal passage alert" to the output device 14, and the processing of the flowchart ends.

[0100] In S7, the determination unit 113 outputs the result "normal" to the output device 14, and the processing of the flowchart ends.

[0101] [Character completion processing] Figure 35 is a flowchart showing an example of the operation of the information processing device 1 during the "person detection result completion processing" of S3 in the main processing described above.

[0102] In S31, the person detection unit 111 selects person 4 to be processed from the "detection list" in the main memory unit 15 where the person detected in the main processing S2 is registered. The person detection unit 111 uses the skeleton detection library to detect each part of person 4 (see Figure 5). If the person detection unit 111 detects ankles (parts 10, parts 13) for each part of person 4 detected, it obtains the ankle coordinates and calculates the leg coordinates 45 where person 4 is located. If the calculation result for the leg coordinates 45 of person 4 is not available and the leg coordinates 45 are missing (S31 is "Yes"), the flow proceeds to S32. On the other hand, if the calculation result for the leg coordinates 45 of person 4 is available and the leg coordinates 45 are not missing (S31 is "No"), the flow proceeds to S34.

[0103] In S32, if the person detection unit 111 detects shoulders (part 2, part 5) and waist (part 8 and part 11) for each part of person 4 detected in S31, and coordinates for the shoulders and waist exist (S32 is "Yes"), the flow proceeds to S33. On the other hand, if there are no detection results for shoulders (part 2, part 5) and waist (part 8 and part 11) for each part of person 4, and coordinates for the shoulders and waist do not exist (S32 is "No"), the flow proceeds to S34.

[0104] In S33, the person detection unit 111 estimates and fills in the coordinates of the missing knee (part 9, part 12) or ankle (part 10, part 13) of the leg from the coordinates of the shoulder (part 2, part 5) and waist (part 8 and part 11) of person 4. The person detection unit 111 calculates the leg coordinates 45 from the estimated and filled-in ankle (part 10, part 13) coordinates. The person detection unit 111 stores the filled-in coordinates and the leg coordinates 45 in the "detection list" of the main memory unit 15.

[0105] In S34, if the person detection unit 111 does not detect the fingertips (parts 18 and 19) for each part of person 4 detected in S31, and the fingertips coordinates are missing (S34 is "Yes"), the flow proceeds to S35. On the other hand, if the person detection unit 111 detects the fingertips (parts 18 and 19) for each part of person 4, and the fingertips coordinates are not missing (S34 is "No"), the flow proceeds to S37.

[0106] In S35, if the person detection unit 111 detects elbows (part 3, part 6) and wrists (part 4, part 7) for each part of person 4 detected in S31, and coordinates for the elbows and wrists exist (S35 is "Yes"), the flow proceeds to S36. On the other hand, if there are no detection results for elbows (part 3, part 6) and wrists (part 4, part 7) for each part of person 4, and coordinates for the elbows and wrists do not exist (S35 is "No"), the flow proceeds to S37.

[0107] In S36, the person detection unit 111 estimates and completes the coordinates of the missing fingertips (parts 18 and 19) from the coordinates of the elbows (parts 3 and 6) and wrists (parts 4 and 7) of person 4. The person detection unit 111 stores the completed coordinates in the "detection list" of the main memory unit 15.

[0108] In S37, the person detection unit 111 refers to the "detection list" in the main memory unit 15 where the people detected in the main process S2 are registered, and determines whether processing for all of the detected people has been completed. If processing for all of the detected people has been completed (S37 is "Yes"), the processing of the flowchart ends. On the other hand, if processing for all of the detected people has not been completed (S37 is "No"), the flowchart proceeds to S31.

[0109] As mentioned above, the number and location of the detected feature points differ depending on the type and method of the library used and the options set. Therefore, in the "person completion processing," appropriate skeletal information may be selected according to the number and location of the feature points of each skeletal detection library, and the completion processing described above may be performed.

[0110] [Illegal passage detection process] Figure 36 is a flowchart showing an example of the operation of the information processing device 1 during the "illegal passage determination process" in S5 of the main processing described above.

[0111] In S51, if the person detection unit 111 detects two or more people in the monitoring passage 32 in the main process S4, it temporarily stores them in the "processing waiting list" of the main memory unit 15 and sequentially checks those multiple people. If the person detected in the monitoring passage 32 in the main process S4 is not stored in the "person list" of the main memory unit 15, the person detection unit 111 assigns a unique number or symbol (identifier) ​​to the person and stores it in the "person list" of the main memory unit 15, setting that person as the "target person" (person 4) for processing.

[0112] In S52, the object detection unit 112 monitors the monitoring area 35 of the ticket gate 3 from the acquired captured image and performs object detection. If the object detection unit 112 finds that the flaps 34 of the ticket gate 3 are open and does not detect the two closed flaps 34 in the monitoring area 35, the object detection result will be "none". Also, if a person is standing in front of the two closed flaps 34, the object detection result will be "none". If the object detection unit 112 does not detect the flaps 34 in a closed state (S52 is "none"), the flow proceeds to S53. On the other hand, if the object detection unit 112 detects an object (S52 is "present"), the flow proceeds to S56.

[0113] In S53, if the person detection unit 111 detects that person 4 is overlapping within the monitoring area 35 of the ticket gate 3 ("present" in S53), the flow proceeds to S54. On the other hand, if the person detection unit 111 does not detect person 4 within the monitoring area 35 of the ticket gate 3 ("absent" in S53), the flow proceeds to S57. Alternatively, the person detection unit 111 may make a determination by detecting that person 4 has crossed the closing point 36 of the ticket gate 3, instead of detecting person 4 within the monitoring area 35 of the ticket gate 3.

[0114] In S54, the determination unit 113 determines the state of the "possibility of unauthorized passage flag" corresponding to the currently set "target person". If the determination result shows that the state of the "possibility of unauthorized passage flag" is "ON" (S54 is "ON"), the flow proceeds to S55. On the other hand, if the state of the "possibility of unauthorized passage flag" is "OFF" (S54 is "OFF"), the flow proceeds to S57.

[0115] In S55, the determination unit 113 changes the currently set "target person" to "illegal passerby".

[0116] In S56, the determination unit 113 turns on the "possibility of unauthorized passage flag" in the main memory unit 15, which is set for the currently set "target person".

[0117] In S57, the determination unit 113 changes the currently set "target person" to "normal passerby".

[0118] In S58, the determination unit 113 stores the determination result of the currently set "target person" in the "person list".

[0119] In S59, the determination unit 113 checks whether there are any unprocessed individuals remaining in the "processing waiting list" of the main memory unit 15 and confirms whether all processing has been completed. If the confirmation finds that there are no unprocessed individuals remaining in the "processing waiting list" and all processing has been completed (S59 is "Yes"), the flow proceeds to S60. On the other hand, if there are unprocessed individuals remaining in the "processing waiting list" and all processing has not been completed (S59 is "No"), the flow proceeds to S51.

[0120] In S60, the determination unit 113 counts the number of "illegal passersby" in the "person list". If the number of "illegal passersby" in the "person list" is "0" (S60 is "0"), the flow proceeds to S60. On the other hand, if the number of "illegal passersby" in the "person list" is "1" or more (S60 is "1" or more), the flow proceeds to S61.

[0121] In S61, the determination unit 113 determines that there is an unauthorized passerby (determination result = "unauthorized passerby"), and the processing of the flowchart ends.

[0122] In S62, the determination unit 113 does not determine that there is an unauthorized passerby (determination result = "normal"), and the processing of the flowchart ends.

[0123] [Processing the list of people] Figure 37 is a flowchart showing an example of the operation of the information processing device 1 in the "person list processing in the monitored passage" step S51 during the unauthorized passage detection process described above.

[0124] In S511, the person detection unit 111 checks whether the person in the monitoring passage 32 detected in S4 of the main process already exists in the "person list" of the main memory unit 15. If two or more people are detected in the monitoring passage 32 in S4 of the main process, the person detection unit 111 temporarily stores them in the "processing waiting list" of the main memory unit 15 and checks those multiple people sequentially. If the check reveals that the person in the monitoring passage 32 already exists in the "person list" of the main memory unit 15 (S511 is "Yes"), the flow proceeds to S512. On the other hand, if the person in the monitoring passage 32 does not exist in the "person list" of the main memory unit 15 (S511 is "No"), the flow proceeds to S513.

[0125] In S512, the person detection unit 111 sets the person detected in the monitoring passage 32 in the main process S4 as the "target person" (person 4) for the process, and the processing of the flowchart ends.

[0126] In S513, the person detection unit 111 adds the person detected in the monitoring passage 32 in the main processing S4 to the "person list" in the main memory unit 15. The person detection unit 111 assigns a unique number or symbol (identifier) ​​to the person in the monitoring passage 32 and stores it in the "person list" in the main memory unit 15. The person detection unit 111 manages information for each person separately using the "person list" in the main memory unit 15 that has been set up in this way. For example, the person detection unit 111 may link IC card information from the ticket gate 3 acquired via the communication unit 12, magnetic tickets, QR tickets, facial recognition results of the face portion 41 acquired by the control unit 11, other biometric authentication results, or motion vector detection results acquired by the control unit 11 to the "person list" in the main memory unit 15.

[0127] In S514, the person detection unit 111 sets the "potential fraud flag" of the person added to the "person list" in the main memory unit 15 to the "OFF" state (initializes it).

[0128] In S515, the person detection unit 111 sets the person added to the "person list" in the main memory unit 15 as the "target person" (person 4) for processing, and the processing of the flowchart ends.

[0129] [Processing based on the previous and subsequent frames] The flowchart illustrating the operation of the "illegal passage determination process" in S5 during the main processing by the aforementioned information processing device 1 will be explained separately for each case (normal / illegal passage, before / after passing the closing point 36). Here, the scene before passing the closing point 36 is called the "pre-frame," and the scene after passing the closing point 36 is called the "post-frame."

[0130] (In the case of the "front frame" during normal traffic flow) If authentication of the user's IC card or ticket, biometric authentication, etc., is successful and the ticket gate 3 permits the user to pass, the person 4 proceeds to pass through normally. In the case of such normal passage, the operation of the information processing device 1 in the "pre-frame" stage, which is the state before the person 4 crosses the entrance line 31 of the ticket gate 3 and passes the closing point 36 of the ticket gate 3, is as shown in Figure 38, for example. In the "pre-frame" stage of normal passage, the steps shown by the solid lines are executed sequentially.

[0131] (In the case of "rear frame" during normal traffic) In the case of normal passage, the operation of the information processing device 1 during the "post-frame" stage, when person 4 has passed the closing point 36 of the ticket gate 3, is as shown in Figure 39, for example. In the case of the "post-frame" during normal passage, the steps shown by the solid lines are executed sequentially.

[0132] (In the case of the "previous frame" when illegal passage occurs) This scenario assumes a situation where authentication of the user's IC card or ticket, biometric authentication, etc., fails, and the ticket gate 3 closes its flap 34 to block the user's passage, but the user 4 forcibly passes through the closing point 36 of the ticket gate 3 (illegal passage). In such an illegal passage case, the operation of the information processing device 1 in the "pre-frame" stage, which is the state before person 4 crosses the entry line 31 of the ticket gate 3 and passes through the closing point 36 of the ticket gate 3, is as shown in Figure 40, for example. In the "pre-frame" stage of illegal passage, the steps shown by the solid lines are executed sequentially.

[0133] (In the case of "post-frame" during illegal passage) In the case of unauthorized passage, the operation of the information processing device 1 during the "post-frame" stage, when person 4 has passed the closing point 36 of the ticket gate 3, is as shown in Figure 41, for example. In the case of unauthorized passage during the "post-frame," the steps shown by the solid lines are executed sequentially.

[0134] (Variations) The embodiments of this disclosure may be modified as follows:

[0135] The information processing device 1 may record the video of person 4, who has been determined to be "illegally entering," in the auxiliary storage unit 16 along with the date and time of shooting and other identification information (such as magnetic tickets, QR tickets, IC card information from the ticket gate 3, characteristic information of skeletal detection (human body detection) results, or characteristic information of facial recognition results or other biometric authentication results), and make it possible to output based on the identification information, thereby enabling efficient collection and acquisition of information on illegal entrants (illegible users). The information processing device 1 may be controlled remotely via the communication unit 12. The information processing device 1 may also be installed using cloud computing, where computing resources are provided via the internet.

[0136] According to the person detection function of the information processing device 1, even if there are individual differences (differences in physique) among people, and even in situations where face detection (face recognition) is difficult, such as insufficient lighting or backlighting, person detection is possible, so it can also be installed as a means to supplement conventional face detection systems. Camera 2 can be installed above or to the side separately from the one used for face recognition to create an environment that improves the accuracy of the conventional face detection system. Furthermore, if person 4 is wearing clothing or other items with a person's face printed on it, the person detection function of the information processing device 1 can determine the presence or absence of a person based on the presence or absence of body parts and the overall size, thereby preventing malfunctions (such as misinterpreting one person as two people entering).

[0137] The person detection unit 111 of the information processing device 1 can identify the number of people and their size (adult or child) from the moment they approach the ticket gate, enabling accurate detection even when multiple people enter at once (e.g., with a child, cutting in line). The information processing device 1 may also be used to accurately determine the number of people passing through. Furthermore, the detected information on the size of the people (adult or child) may be used for pre-processing to prevent age falsification at IC card ticket gates.

[0138] In the object detection unit 112 of the information processing device 1, the design pattern of the monitoring area 35 may be freely changed to improve detection accuracy. The design of the flap 34 may be made into a striped pattern or other form suitable for object detection. In addition, the threshold may be arbitrarily changed based on the agreement rate when comparing images, the slope of the line, intercept, length, color information, etc.

[0139] The object detection unit 112 of the information processing device 1 may use a trained machine learning model (such as OpenCV or YOLO, which are known image recognition libraries) to monitor the monitoring area 35 in the captured image and detect whether the flap 34 of the ticket gate 3 is closed.

[0140] For object detection performed by the object detection unit 112 of the information processing device 1, a TOF (time of flight) sensor may be used instead of the camera 2. When using a TOF sensor, the person 4 and the flap 34 may be distinguished based on the height and shape of the detected object.

[0141] Instead of the object detection unit 112 of the information processing device 1, the information processing device 1 may be connected to the ticket gate 3, and the control unit 11 may acquire sensor information and door closing information from the ticket gate 3 via the communication unit 12. Alternatively, a separate sensor may be installed on the ticket gate 3, and passage information and door closing information may be acquired via the communication unit 12. The control unit 11 may also acquire passage right information such as magnetic tickets, QR tickets, IC card information, or biometric authentication information (e.g., facial recognition, iris recognition, vein recognition, etc.) from the ticket gate 3 via the communication unit 12, and determine whether or not it is normal passage.

[0142] Multiple cameras 2 may be installed depending on the shooting angle and shooting range. A wide area may be covered by installing cameras with wide-angle lenses or omnidirectional lenses. In this case, the control unit 11 may perform distortion correction and field of view control on the captured images. Alternatively, cameras 2 may be installed to capture a specific narrow range, and a portion of the captured image may be cut off. In this case, a process of combining (stitching) multiple captured images may be performed.

[0143] This disclosure can be implemented using software, hardware, or software integrated with hardware.

[0144] <Effects> As shown in this disclosure, this system performs skeletal detection (human body detection) on the entire body, resulting in a large detectable size and a large number of feature points, enabling human detection even in environments with insufficient lighting or backlighting. Furthermore, even if part of the body is not captured, the system can accurately determine the location of the person by interpolating and calculating the coordinates of the feet. In addition, the processing of skeletal detection (human body detection) and object detection can be calculated relatively quickly on a general-purpose PC, thus reducing the system's weight. In this way, this system possesses flexibility and robustness compared to conventional facial recognition systems, enabling highly accurate and flexible human detection.

[0145] Furthermore, this system does not require integration with ticket gates and can detect and identify individuals attempting to illegally pass through automatic ticket gates with high accuracy using only camera images. Therefore, it offers a high degree of flexibility in installation and is useful as a simple surveillance system. As such, it can be applied to existing surveillance cameras and is useful for detecting and deterring illegal use at unmanned stations and other locations.

[0146] Furthermore, by linking face detection, person detection, and object detection functions, this system can be made more lightweight and new functions can be added, such as determining "who," "where," and "in what state" someone is present. In addition, because the face detection, person detection, and object detection functions of this system collect information from captured images, they do not require sensors in ticket gates, eliminating the need for synchronization processing and time lag processing between sensors, and enabling high-precision linking of face detection, person detection, and object detection functions. Further applications include application to photoelectric sensorless ticket gates.

[0147] While embodiments have been described above with reference to the drawings, this disclosure is not limited to such examples. It will be apparent to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims. Such modifications or alterations are also understood to fall within the technical scope of this disclosure. Furthermore, the components in the embodiments may be combined in any way without departing from the spirit of this disclosure.

[0148] In the above-described embodiment, the notation "...part" used for each component may be replaced with other notations such as "...circuitry," "...assembly," "...device," "...unit," or "...module." Calculation may be read as calculation.

[0149] Each functional block used in the description of the above embodiments may be implemented partially or entirely as an integrated circuit (LSI), and each process described in the above embodiments may be controlled partially or entirely by a single LSI or a combination of LSIs. An LSI may consist of individual chips, or it may consist of a single chip that includes some or all of the functional blocks. An LSI may have data inputs and outputs. Depending on the degree of integration, LSIs may be referred to as ICs, system LSIs, super LSIs, or ultra LSIs.

[0150] The method of integration is not limited to LSIs; it may also be implemented using dedicated circuits, general-purpose processors, or dedicated processors. Furthermore, FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI manufacturing, or reconfigurable processors that allow for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used. This disclosure may be implemented as digital or analog processing.

[0151] Furthermore, if advancements in semiconductor technology or related technologies lead to the emergence of integrated circuit technologies that replace LSIs, then naturally, these technologies can be used to integrate functional blocks. The application of biotechnology, for example, is a possible possibility.

[0152] This disclosure is applicable to all types of devices, systems, and equipment having communication capabilities (collectively referred to as communication equipment). Communication equipment may include a radio transceiver and a processing / control circuit. A radio transceiver may include a receiver and a transmitter, or both as functions. A radio transceiver (transmitter, receiver) may include an RF (Radio Frequency) module and one or more antennas. The RF module may include an amplifier, an RF modulator / demodulator, or similar. Non-exclusive examples of communication devices include telephones (mobile phones, smartphones, etc.), tablets, personal computers (PCs) (laptops, desktops, notebooks, etc.), cameras (digital still / video cameras, etc.), digital players (digital audio / video players, etc.), wearable devices (wearable cameras, smartwatches, tracking devices, etc.), game consoles, digital book readers, telehealth / telemedicine devices, vehicles or mobile transport with communication capabilities (cars, airplanes, ships, etc.), and combinations of the above-mentioned devices.

[0153] Communication devices are not limited to portable or movable devices, but also include all kinds of non-portable or fixed devices, devices, and systems, such as smart home devices (appliances, lighting equipment, smart meters or measuring instruments, control panels, etc.), vending machines, and any other "things" that may exist on an IoT (Internet of Things) network.

[0154] Furthermore, in recent years, Cyber-Physical Systems (CPS), a new concept in IoT (Internet of Things) technology that creates new added value through information linkage between the physical and cyber spaces, has been attracting attention. This CPS concept can also be adopted in the above-described embodiment.

[0155] In other words, as a basic configuration of CPS, for example, edge servers located in physical space and cloud servers located in cyberspace are connected via a network, and processing can be distributed and performed by the processors installed on both servers. Here, it is preferable that each processing data generated on the edge server or cloud server is generated on a standardized platform, and by using such a standardized platform, it is possible to improve efficiency when building systems that include various diverse groups of sensors and IoT application software.

[0156] Communication includes data communication via cellular systems, wireless LAN systems, and communication satellite systems, as well as data communication using combinations of these.

[0157] Furthermore, the communication device also includes devices such as controllers and sensors that are connected to or linked to a communication device that performs the communication functions described in this disclosure. For example, this includes controllers and sensors that generate control signals and data signals used by the communication device that performs the communication functions of the communication device.

[0158] Furthermore, communication equipment includes infrastructure facilities such as base stations, access points, and any other devices, devices, and systems that communicate with or control the aforementioned non-limited types of equipment.

[0159] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of this disclosure. Furthermore, the components in the above embodiments may be combined in any way without departing from the spirit of the disclosure.

[0160] The specific examples of this disclosure have been described in detail above, but these are merely illustrative and do not limit the scope of the claims. The technologies described in the claims include various modifications and changes to the specific examples described above. [Industrial applicability]

[0161] One embodiment of this disclosure is suitable for detecting unauthorized passage at a ticket gate from captured images. [Explanation of symbols]

[0162] 1. Information Processing Device 2 cameras 3 Ticket gates 4 Person (user) 5 Passersby 6 people 10. Information Processing Systems 11 Control Unit 12 Communications Department 13 Input device 14 Output device 15 Main memory 16 Auxiliary storage 31 Entry Line 32 Surveillance Passageway 33 Qualifying Line 34 Flap (Flappa) 35 Monitoring area 36. Door closing point 37, 38 Indicator light, warning light 39. Monitoring area (different from 35) 40 Offset 41. The face of a person 42. Faces of passersby 43 Outline of a person 44. Rectangle for person detection 45 Leg coordinates 111 Personnel Investigation Department 112 Object detection unit 113 Judgment section

Claims

1. A communication unit acquires captured images from an imaging unit that images a gate used to selectively permit or restrict access to an area where only persons meeting specific conditions are allowed to enter, and A control unit detects a person passing through a first area and a predetermined object present in a second area from the captured image acquired by the communication unit, Equipped with, The control unit is an information processing device that determines whether or not the person is making an unauthorized entry into the area based on a change in the combination of the detection result of the person passing through the first area and the detection result of the predetermined object present in the second area.

2. The control unit, Based on the combination and temporal changes of the detection result of the person passing through the first area, the detection result of the predetermined object present in the second area, and the detection result of the person detected overlapping the second area on the first area, it is determined that the person is making the unauthorized passage. The information processing apparatus according to claim 1.

3. The control unit, When the person passing through the first area is detected and the predetermined object present in the second area is detected, the first flag is turned on. After the first flag is turned on, if the person is detected overlapping the second area on the first area, it is determined that the person is engaging in unauthorized passage. The information processing apparatus according to claim 1.

4. The control unit detects the person using skeletal detection or image segmentation. The information processing apparatus according to claim 1.

5. If, in the captured image, a part of the person's body is obscured and the obscured part of the person's body cannot be detected, the control unit estimates the coordinates of the obscured part of the person's body from the coordinates of the person's body parts detected using skeletal detection, and then fills in the obscured part of the person's body. The information processing apparatus according to claim 1.

6. The control unit detects a closed door in the second region of the captured image by extracting predetermined image features from the captured image, or by comparing the captured image with a predetermined pattern and detecting matching portions. The information processing apparatus according to claim 1.

7. The control unit, By adjusting the difference between the parts of the person detected by skeletal detection and the outline of the person's body, by detecting the outline of the person by image segmentation, or by converting the lines connecting the parts of the person detected by skeletal detection into rectangles enlarged to a predetermined width or by a magnification for each part, and using the rectangles as parts of the person, it is possible to detect whether there is an overlap between the parts of the person and the second region, and to detect the person who is present in the second region. The information processing apparatus according to claim 1.

8. The control unit, In the captured image, whether a person detected overlapping the second region on the first region is the same person passing through the first region is identified by facial recognition or motion vector of the person. The information processing apparatus according to claim 3.

9. The control unit, If the second region is obstructed in the captured image and cannot be detected, the state of a third region different from the second region is detected to determine whether or not passage through the gate is restricted. The information processing apparatus according to claim 1.

10. The first region is the area within the passageway enclosed by an entrance for entering the gate and an exit for exiting the gate. The information processing apparatus according to claim 1.

11. The aforementioned predetermined object is a closed door for restricting passage through the gate, The second area is the area in which the closed door for restricting passage through the gate is visible. The information processing apparatus according to claim 1.

12. An imaging unit that images a gate for selectively permitting or restricting access to an area where only persons meeting specific conditions are allowed to enter, A control unit detects a person passing through a first region and a predetermined object present in a second region from the image captured by the imaging unit, Equipped with, The control unit is an information processing system that determines whether or not the person is illegally entering the area based on a change in the combination of the detection result of the person passing through the first area and the detection result of the predetermined object present in the second area.

13. Information processing device, Capture images of gates that selectively permit or restrict access to restricted areas, allowing only individuals who meet specific conditions to enter. From the aforementioned captured image, a person passing through the first area is detected. From the aforementioned captured image, a predetermined object present in the second region is detected. An information processing method for determining whether a person is illegally entering an area based on a change in the combination of the detection result of a person passing through the first area and the detection result of a predetermined object present in the second area.

14. In an information processing device, The steps include: acquiring a photographic image of a gate used to selectively permit or restrict access to an area where only persons meeting specific conditions are allowed to enter; The steps include detecting a person passing through a first area from the aforementioned captured image, The steps include detecting a predetermined object in a second region from the captured image, The system is configured to perform the step of determining whether the person is illegally entering the area based on a change in the combination of the detection result of the person passing through the first area and the detection result of the predetermined object present in the second area. Information processing program.

Images