Monitoring device, monitoring method, and program

Abstract

This invention provides a monitoring device, a monitoring method, and a program capable of accurately detecting suspicious activity. [Solution] The monitoring device includes an image acquisition unit that acquires at least one of a face image taken of the subject person's face and a hand image taken of the subject person's hands; a body temperature acquisition unit that acquires body temperature information indicating the subject person's body temperature; a state information generation unit that generates state information indicating the state of the subject person using at least one of the acquired face image and hand image of the subject person and the identified subject person's body temperature information; and an output processing unit that performs a predetermined output using the generated state information of the subject person.

Description

Technical Field

[0001] The present invention relates to a monitoring device, a monitoring method, and a program.

Background Art

[0002] In order to relieve congestion in the inspection area, the immigration inspection at international airports is being automated using biometric authentication technologies such as fingerprint authentication. Patent Document 1 describes a device that monitors a wide area, such as an event venue, where it is difficult to narrow down in advance where suspicious behavior will occur, using multiple cameras, and detects the occurrence of suspicious behavior in areas that are blind spots of the cameras.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the device described in Patent Document 1 above, since the monitoring range is wide, it is necessary to image a wide area, and the accuracy of detecting suspicious actions with only a part of the body is low. Therefore, for example, it is highly likely that the device described in Patent Document 1 cannot detect suspicious actions that are performed only at hand level.

Means for Solving the Problems

[0005] A monitoring device according to one aspect of the present disclosure is image acquisition means for acquiring at least one of a face image obtained by imaging the face of a target person and a hand image obtained by imaging the hand of the target person; body temperature acquisition means for acquiring body temperature information indicating the body temperature of the target person; A state information generation means generates state information indicating the state of the target person using at least one of the acquired face image and hand image of the target person and the identified body temperature information of the target person, An output processing means that uses the generated state information of the target person to perform a predetermined output, It is equipped with.

[0006] One aspect of this disclosure is the monitoring method, One or more computers, At least one of the following is obtained: a facial image of the subject person and a hand image of the subject person. Obtain body temperature information indicating the body temperature of the aforementioned person, Using at least one of the facial image and hand image of the target person obtained, and the body temperature information of the identified target person, state information indicating the state of the target person is generated. This includes using the generated status information of the target person to produce a predetermined output.

[0007] One aspect of this disclosure is that the program is On the computer, Image acquisition process that acquires at least one of a face image taken of the target person's face and a hand image taken of the target person's hands. A body temperature acquisition process that obtains body temperature information indicating the body temperature of the aforementioned person. A state information generation process that generates state information indicating the state of the target person using at least one of the acquired face image and hand image of the target person and the identified body temperature information of the target person. Using the generated state information of the target person, an output process is executed that produces a predetermined output. [Effects of the Invention]

[0008] According to one example of this disclosure, a monitoring device, a monitoring method, and a program capable of accurately detecting suspicious activity can be obtained. [Brief explanation of the drawing]

[0009] [Figure 1] It is a functional block diagram showing a configuration example of a monitoring device according to the present disclosure. [Figure 2] It is a flowchart showing an example of a processing operation of a monitoring device according to the present disclosure. [Figure 3] It is a diagram conceptually showing a system configuration example of a monitoring system according to the present disclosure. [Figure 4] It is a diagram showing an example of a configuration of a computer that realizes a monitoring device according to the present disclosure. [Figure 5] It is a diagram for explaining a face imaging unit of an operation terminal. [Figure 6] It is a diagram for explaining a hand imaging unit of an operation terminal. [Figure 7] It is a diagram for explaining a thermometer unit of an operation terminal. [Figure 8] It is a partial functional block diagram showing a main part configuration example of a state information generation unit. [Figure 9] It is a partial functional block diagram showing another main part configuration example of a state information generation unit. [Figure 10] It is a partial functional block diagram showing yet another main part configuration example of a state information generation unit. [Figure 11] It is a flowchart showing an operation example of a state information generation unit. [Figure 12] It is a flowchart showing another operation example of a state information generation unit. [Figure 13] It is a flowchart showing yet another operation example of a state information generation unit. [Figure 14] It is a flowchart showing an operation example of an output processing unit. [Figure 15] It is a flowchart for explaining a usage scene of an operation terminal by a target person. [Figure 16] It is a flowchart showing an operation example of a state information generation unit using a machine learning model. [Figure 17] It is a flowchart showing another operation example of a state information generation unit using a machine learning model.

Mode for Carrying Out the Invention

[0010] Hereinafter, in the present disclosure, the drawings are associated with one or more embodiments. Also, in all the drawings, the same components are denoted by the same reference numerals, and the description thereof will be omitted as appropriate. Further, in each of the following figures, the configuration of portions not related to the essence of the present disclosure is omitted and not shown.

[0011] In the present disclosure, "acquisition" includes at least one of the act of the self-device going to obtain data or information stored in another device or storage medium (active acquisition), and the act of the self-device inputting data or information output from another device (passive acquisition). Examples of active acquisition include requesting or inquiring another device and receiving its reply, and accessing another device or storage medium to read it out. Examples of passive acquisition include receiving information distributed (or transmitted, push-notified, etc.). Further, "acquisition" may be selecting and acquiring from the received data or information, or selecting and receiving the distributed data or information.

[0012] <Functional configuration example> As shown in FIG. 1, the monitoring device 100 includes an image acquisition unit 102, a body temperature acquisition unit 104, a state information generation unit 106, and an output processing unit 108. The image acquisition unit 102 acquires at least one of a face image of the target person and a hand image of the target person. The body temperature acquisition unit 104 acquires body temperature information indicating the body temperature of the target person. The state information generation unit 106 generates state information indicating the state of the target person using at least one of the acquired face image and hand image of the target person and the identified body temperature information of the target person. The output processing unit 108 performs a predetermined output using the generated state information of the target person.

[0013] <Operation example> As shown in the flowchart of FIG. 2, the monitoring device 100 operates. First, the image acquisition unit 102 acquires at least one of the following: a face image captured from the subject's face and a hand image captured from the subject's hands (step S101). Then, the body temperature acquisition unit 104 acquires body temperature information indicating the subject's body temperature (step S103). The state information generation unit 106 generates state information indicating the state of the target person using at least one of the acquired face image and hand image of the target person and the identified target person's body temperature information (step S105). The output processing unit 108 uses the generated status information of the target person to perform a predetermined output (step S107).

[0014] According to this monitoring device 100, the state information generation unit 106 generates state information indicating the state of the target person using at least one of the acquired facial image and hand image of the target person, along with the identified target person's body temperature information, thereby enabling accurate detection of suspicious behavior.

[0015] The following describes a detailed example of the monitoring device 100.

[0016] (First Embodiment) <System Overview> As shown in Figure 3, the monitoring system 1 includes a monitoring device 100. The monitoring device 100 is connected to the operating terminal 200 via a communication network 3. The monitoring device 100 is a computer such as a personal computer or a server computer. The monitoring device 100 includes a storage device 120. The storage device 120 may be located inside the monitoring device 100 or outside of it. In other words, the storage device 120 may be hardware integrated with the monitoring device 100 or hardware separate from the monitoring device 100.

[0017] The operating terminal 200 is a device used for immigration and customs checks at the airport. Person U includes individuals undergoing immigration and customs procedures at airports.

[0018] The monitoring device 100 has the function of preventing fraudulent activity, for example, by detecting fraudulent activity in an operating terminal 200 that automatically performs immigration and customs procedures at an international airport, and by stopping, warning, or reporting the procedure. In this disclosure, the operating terminal 200 is described using an example of an operating terminal 200 that performs immigration and customs procedures at an international airport, but is not limited to this. The operating terminal 200 is a device that performs fingerprint authentication, vein authentication, etc., and is not limited to any device in which fraud may occur in the personal authentication procedure. For example, the operating terminal 200 may be an ATM (Automatic Teller Machine) of a financial institution, or an authentication device at a gate to a designated space.

[0019] The operating terminal 200 includes an operating unit 202, a face imaging unit 204, a body temperature measurement unit 206, a hand imaging unit 208, and a display 210. The operating terminal 200 may further include audio input / output devices such as a microphone and speaker, a code reader, etc. (not shown).

[0020] The operation unit 202 includes a fingerprint reader 212. The operation unit 202 consists of a touch panel, touchpad, operation buttons, and keyboard, and accepts user input for immigration and customs procedures. The fingerprint reader 212 includes a device for reading the fingerprints of the subject person U for identity verification during immigration and customs procedures. The facial imaging unit 204 includes a camera that captures a facial image of the person U who is subject to immigration and customs procedures. The body temperature measurement unit 206 includes a sensor that measures the body temperature of the subject person U. The hand imaging unit 208 includes a camera that images the hand of the subject U as he reads his fingerprints using the fingerprint reader unit 212 of the operating terminal 200. The display 210 is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. The display 210 displays the screen necessary for immigration and customs procedures. The display 210 may also be a touch panel including an operation unit 202 that accepts user input.

[0021] The cameras in the face imaging unit 204 and the hand imaging unit 208 are equipped with an image sensor such as a lens and a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and are network cameras such as IP (Internet Protocol) cameras. The network camera has, for example, a wireless LAN (Local Area Network) communication function and is connected to the monitoring device 100 via a relay device (not shown) such as a router of the communication network 3. The camera may also be equipped with a mechanism that tracks the movement of the target person U and controls the orientation of the camera body and lens, zoom control, and focus.

[0022] The camera (face imaging unit 204 and hand imaging unit 208) provided in the operating terminal 200 of this disclosure is a so-called 2D camera and generates an image with two-dimensional information in horizontal and vertical (X,Y) coordinates.

[0023] The images generated by the camera are preferably captured in real time and transmitted to the monitoring device 100. However, the images transmitted to the monitoring device 100 do not have to be transmitted directly from the camera; they may be images with a predetermined time delay. The images captured by the camera may be temporarily stored in another storage device, and the monitoring device 100 may read them sequentially or at predetermined intervals from the storage device. Furthermore, the images transmitted to the monitoring device 100 are preferably moving images, but they may also be frame images at predetermined intervals or still images.

[0024] <Example Hardware Configuration> The monitoring device 100 described herein is implemented by the computer 1000 shown in Figure 4. The operating terminal 200 shown in Figure 3 is also implemented by the computer 1000. Furthermore, the functions of the monitoring device 100 may be shared between the computer 1000 of the operating terminal 200 and the computer 1000 of the monitoring device 100. In addition, warning devices, communication devices, etc., which will be described later, are also implemented by the computer 1000.

[0025] Computer 1000 has a bus 1010, a processor 1020, memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0026] Bus 1010 is a data transmission path for the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060 to send and receive data to and from each other. However, the method of connecting the processor 1020 and the other components to each other is not limited to bus connection.

[0027] The 1020 processor is a processor implemented in components such as the CPU (Central Processing Unit) and GPU (Graphics Processing Unit).

[0028] Memory 1030 is a main memory device implemented using RAM (Random Access Memory), etc.

[0029] The storage device 1040 is an auxiliary storage device implemented as an HDD (Hard Disk Drive), SSD (Solid State Drive), memory card, or ROM (Read Only Memory). The storage device 1040 stores program modules that implement each function of the monitoring device 100 (for example, the image acquisition unit 102, the body temperature acquisition unit 104, the state information generation unit 106, the output processing unit 108, the facial expression recognition unit 110, the tension state detection unit 112, the hand movement tracking unit 114, the suspicious movement detection unit 116, the body temperature change identification unit 118, etc.). The processor 1020 reads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to that program module. The storage device 1040 may also function as the storage device 120 of the monitoring device 100.

[0030] The program module may be recorded on a recording medium. The recording medium on which the program module is recorded includes a non-temporary, tangible medium usable by a computer 1000, and program code that can be read by the computer 1000 (processor 1020) may be embedded in that medium.

[0031] The input / output interface 1050 is an interface for connecting the computer 1000 to various input / output devices. The input / output interface 1050 also functions as a communication interface for short-range wireless communication such as Bluetooth (registered trademark) and NFC (Near Field Communication). Furthermore, the input / output interface 1050 also functions as a wired communication interface such as USB (Universal Serial Bus) and HDMI (High Definition Multimedia Interface) (registered trademark).

[0032] The network interface 1060 is an interface for connecting the computer 1000 to the communication network 3. This communication network 3 is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method by which the network interface 1060 connects to the communication network 3 may be a wireless connection or a wired connection. Furthermore, the network interface 1060 also functions as a communication interface for communication using mobile communication systems such as CDMA (Code Division Multiple Access), LTE (Long Term Evolution), 4th generation communication (4G), 5th generation communication (5G), and 6th generation communication (6G) and later, via a mobile communication network.

[0033] The computer 1000 then connects to the necessary equipment (for example, the display, touch panel, operation buttons, touchpad, keyboard, mouse, speaker, microphone, camera, printer, etc. of the monitoring device 100; the operation unit 202 of the operation terminal 200, such as the touch panel, touchpad, operation buttons, and keyboard; the sensors of the face imaging unit 204, hand imaging unit 208, and body temperature measurement unit 206, the display 210, mouse, speaker, microphone, code reader, etc.) via the input / output interface 1050 or the network interface 1060.

[0034] Each component of the monitoring device 100 in each figure of this disclosure can be realized by any combination of the hardware and software of the computer 1000 in Figure 4. It will be understood by those skilled in the art that there are various modifications to the implementation method and apparatus. The functional block diagrams showing the monitoring device 100 in each figure of this disclosure show blocks of logical functional units, not hardware-level configurations.

[0035] <Example of Functional Configuration> As shown in Figure 1, the monitoring device 100 includes an image acquisition unit 102, a body temperature acquisition unit 104, a state information generation unit 106, and an output processing unit 108. The image acquisition unit 102 acquires at least one of the following: a face image, which is an image of the subject person's face, and a hand image, which is an image of the subject person's hands. The face image of subject U is acquired from the face imaging unit 204. The hand image of subject U's hands while they are operating the control terminal 200 is acquired from the hand imaging unit 208.

[0036] Here, the target person U is the person who operates the operation terminal 200 and is the person whose identity is to be verified, such as through fingerprint authentication. The operation terminal 200 may, for example, detect when the target person U stands in front of the operation terminal 200 (entering a predetermined area around the operation terminal 200) using a motion sensor. Alternatively, the operation terminal 200 may detect the presence of the target person U when the target person U performs a predetermined operation on the operation screen of the operation terminal 200. The monitoring device 100 executes processing when the target person U operates the operation terminal 200. In other words, the monitoring device 100 starts processing when the presence of the target person U is detected in front of the operation terminal 200.

[0037] After detecting the presence of the target person U, it is preferable that the face imaging unit 204, the body temperature measurement unit 206, and the hand imaging unit 208 start imaging and measurement processing, and the image acquisition unit 102 and the body temperature acquisition unit 104 start image acquisition processing and body temperature acquisition processing, respectively. The body temperature acquisition unit 104, the body temperature measurement unit 206, and the hand imaging unit 208 measure or acquire the body temperature, face image, and hand image of the target person U while the target person U is operating the operation terminal 200.

[0038] As shown in Figure 5, the face imaging unit 204 is located, for example, in the upper center of the display 210 of the operation terminal 200. It captures the face of a target person U who is facing the display 210 to operate the operation terminal 200, and generates a face image of the target person U. The face image of the target person U is processed by the state information generation unit 106 and used to recognize changes in the facial expression of the target person U. Therefore, the face image is preferably a moving image, but it may also be a frame image at predetermined intervals.

[0039] As shown in Figure 6, the hand imaging unit 208 is mounted, for example, on an arm above the operating terminal 200, and is positioned to image the hands of the target person U operating the operating terminal 200 from above. The output processing unit 108 images the hands of the target person U and generates a hand image of the target person U. The hand image of the target person U is processed by the state information generation unit 106 and used to track the movement of the target person U's hands and to identify whether the hand movement is suspicious or not. For this reason, the hand image and face image are preferably moving images, but they may also be frame images at predetermined intervals.

[0040] The body temperature acquisition unit 104 acquires body temperature information indicating the body temperature of the target person U from the body temperature measurement unit 206. The body temperature measurement unit 206 is, in one example, a thermosensor, a device that detects far-infrared radiation emitted by the object being measured and measures the temperature of the object. For example, the body temperature measurement unit 206 measures the body temperature of a predetermined local location within the body of the target person U, such as the forehead or cheeks of the target person U. In another example, the body temperature measurement unit 206 is, for example, a thermographic camera, which generates and outputs a thermograph showing the heat distribution in a predetermined area within the body of the target person U, such as the face or upper body.

[0041] As shown in Figure 7, the body temperature measurement unit 206 is mounted, for example, on an arm above the operating terminal 200, and is installed so as to measure the body temperature of the person U operating the operating terminal 200 from above. The body temperature measurement unit 206 measures the body temperature of the person U at predetermined intervals. The body temperature acquisition unit 104 sequentially acquires the body temperature of the person U from the body temperature measurement unit 206.

[0042] However, the body temperature information acquired by the body temperature acquisition unit 104 may be a measured value, but it may also be information indicating the degree of temperature increase in stages, such as high temperature (e.g., 37.5 degrees Celsius or higher), normal temperature (e.g., 36 degrees Celsius or higher but less than 37.5 degrees Celsius), or low temperature (e.g., less than 36 degrees Celsius). Furthermore, as described above, the body temperature acquisition unit 104 may acquire thermography (heat distribution) from the body temperature measurement unit 206.

[0043] The state information generation unit 106 generates state information indicating the state of the target person U using at least one of the acquired face image and hand image of the target person U, and the identified target person's body temperature information.

[0044] The following describes in detail the process of generating state information indicating the state of subject U (changes in facial expression, hand movements, and body temperature) using the subject U's facial image, hand image, and body temperature.

[0045] <Generating state information based on changes in facial expression> The state information generation unit 106 processes the acquired face image to recognize the facial expression of the target person U, detects the tension state of the target person U using the recognized facial expression, and generates state information indicating that the tension state of the target person U has been detected. In this way, the state information generation unit 106 generates state information indicating that the target person U is in a state of tension based on changes in facial expression. Then, if the generated status information of the target person U indicates that a state of tension has been detected in the target person U, the output processing unit 108 outputs an alert to a predetermined device as a predetermined output. The output processing will be described later.

[0046] In detail, as shown in Figure 8, the state information generation unit 106 includes an facial expression recognition unit 110, a tension state detection unit 112, and a tension state information DB 130. The tension state information DB 130 is contained in the storage device 120.

[0047] The image acquisition unit 102 acquires a facial image of the target person U from the facial imaging unit 204 of the operation terminal 200 and passes it to the state information generation unit 106. As described above, the operation terminal 200 and the monitoring device 100 start the facial image acquisition process of the target person U when it is detected that the target person U has started using the operation terminal 200. These processes continue to be executed while the target person U is using the operation terminal 200, at least until the fingerprint authentication process is completed, that is, during the period in which there is a possibility that the target person U may be committing fraudulent acts.

[0048] The facial expression recognition unit 110 processes the facial image of the target person U to recognize the time-series changes in the person's facial expression. The tension state detection unit 112 compares the recognized changes in the target person U's facial expression with a predetermined tension state pattern to detect that the target person U is in a state of tension.

[0049] The facial expression recognition unit 110, as an example, uses MediaPipe FaceMesh, part of the MediaPipe open-source machine learning (ML) framework provided by Google, to recognize facial expressions. By using MediaPipe FaceMesh, the facial expression recognition unit 110 estimates 468 3D facial landmarks from the captured image and recognizes changes in the facial expression of the target person U. With this configuration, the facial expression recognition unit 110 can perform highly accurate recognition processing.

[0050] The tension detection unit 112 acquires time-series coordinate data of 468 3D facial landmarks estimated by the facial recognition unit 110. Using the acquired time-series coordinate data of facial expressions, the tension detection unit 112 recognizes changes in the facial expressions of the target person U and detects that the mental state of the target person U using the operation terminal 200 is one of tension. The tension detection unit 112 refers to the tension information DB 130 and determines whether the acquired time-series coordinate data of facial expressions corresponds to a facial expression pattern indicating tension stored in the tension information DB 130. If the tension detection unit 112 determines that the acquired time-series coordinate data of facial expressions corresponds to a facial expression pattern in the tension information DB 130, it detects that the mental state of the target person U using the operation terminal 200 is one of tension.

[0051] The tension state information DB130 stores tension state identification information for identifying the tension state of the target person U on the operating terminal 200. The tension state identification information includes facial expression information that shows patterns of facial expressions when a person is in a state of tension, such as facial rigidity, increased blinking, unfocused and wandering eyes, and frequently looking around.

[0052] <Generating state information based on user actions> The state information generation unit 106 processes the acquired hand image to detect suspicious behavior of the target person U and generates state information indicating that suspicious behavior of the target person U has been detected. In this way, the state information generation unit 106 generates state information indicating that the target person U is performing suspicious behavior based on hand movements. Then, if the generated status information of the target person U indicates that suspicious behavior by the target person U has been detected, the output processing unit 108 outputs an alert to a predetermined device as a predetermined output. The output processing will be described later.

[0053] In detail, as shown in Figure 9, the state information generation unit 106 includes a hand movement tracking unit 114, a suspicious movement detection unit 116, and a suspicious movement information DB 140. The suspicious movement information DB 140 is contained in the storage device 120.

[0054] The image acquisition unit 102 acquires an image of the target person U's hands from the hand imaging unit 208 of the operation terminal 200 and passes it to the state information generation unit 106. As described above, the operation terminal 200 and the monitoring device 100 start the image acquisition and acquisition processes of the target person U's hands when it is detected that the target person U has started using the operation terminal 200. These processes continue to be executed while the target person U is using the operation terminal 200, at least until the fingerprint authentication process is completed, that is, during the period in which there is a possibility that the target person U may be committing fraudulent acts.

[0055] The hand movement tracking unit 114 processes images of the hands of the target person U to recognize and track the movements of the target person U's hands. The suspicious movement detection unit 116 compares the recognized hand movements of the target person U with predetermined suspicious movement patterns to detect that the hand movements of the target person U are suspicious.

[0056] The hand motion tracking unit 114, as an example, uses MediaPipe Hands, a part of the open-source machine learning (ML) framework MediaPipe provided by Google, to perform tracking. By using MediaPipe Hands, the hand motion tracking unit 114 infers 21 3D hand landmarks from a single captured frame and recognizes (tracks) the hand movements of the target person U. With this configuration, the hand motion tracking unit 114 can perform highly accurate recognition processing.

[0057] The suspicious action detection unit 116 acquires time-series coordinate data of 21 3D finger landmarks that were estimated when the hand motion tracking unit 114 tracked the hand movements of the target person U. Using the acquired time-series coordinate data of the hand movements, the suspicious action detection unit 116 recognizes the hand movements of the target person U and detects that the target person U, who is using the operation terminal 200, is performing a suspicious action. Specifically, the suspicious action detection unit 116 refers to the suspicious action information DB 140 and determines whether the acquired time-series coordinate data of the hand movements corresponds to a suspicious action pattern in the suspicious action information DB 140. If the suspicious action detection unit 116 determines that the acquired time-series coordinate data of the hand movements corresponds to a suspicious action pattern in the suspicious action information DB 140, it detects that the target person U, who is using the operation terminal 200, is performing a suspicious action.

[0058] A suspicious action may be an action that suggests the target person U of the operating terminal 200 is attempting to commit fraud (for example, a preliminary action), or it may be the fraudulent act itself, or it may include both.

[0059] The suspicious activity information DB140 stores suspicious activity patterns, such as patterns for removing suspicious objects, unauthorized authentication of fingerprint tapes, and unauthorized authentication of forged fingers. The suspicious object removal pattern is pattern information of actions taken by the subject person U to remove a suspicious object, and includes, for example, the action of the subject person U putting their hand into their pocket immediately before fingerprint authentication, and the action of removing the suspicious object from their pocket. The fingerprint tape fraudulent authentication pattern is pattern information of actions taken to fraudulently bypass fingerprint authentication using fingerprint tape, and includes, for example, the action of the subject person U putting tape or the like on their finger immediately before fingerprint authentication. The forged finger fraudulent authentication pattern is pattern information of actions taken to bypass fingerprint authentication using a forged finger, and includes, for example, the action of trying to authenticate by placing an object other than the recognized (tracked) hand or finger on the fingerprint reader 212.

[0060] <Generating state information based on changes in body temperature> The state information generation unit 106 uses the acquired information indicating the body temperature of the target person U to identify the time-series change in the body temperature of the target person U. Then, if the identified change in the body temperature of the target person U is on an upward trend, the state information generation unit 106 detects that the target person U is in a state of tension and generates state information indicating that the target person U is in a state of tension. In this way, the state information generation unit 106 generates state information indicating the tension state of the target person U based on the change in body temperature. Then, if the generated status information of the target person U indicates that a state of tension has been detected in the target person U, the output processing unit 108 outputs an alert to a predetermined device as a predetermined output. The output processing will be described later.

[0061] In detail, the state information generation unit 106 includes a body temperature change identification unit 118 and a tension state detection unit 112, as shown in Figure 10. The body temperature acquisition unit 104 acquires the body temperature information of the target person U from the body temperature measurement unit 206 of the operation terminal 200 and passes it to the status information generation unit 106. As described above, the operation terminal 200 and the monitoring device 100 start the body temperature measurement and acquisition process of the target person U when it is detected that the target person U has started using the operation terminal 200. These processes continue to be executed while the target person U is using the operation terminal 200, at least until the fingerprint authentication process is completed, that is, during the period in which there is a possibility that the target person U may be committing fraudulent acts.

[0062] The body temperature change identification unit 118 uses the body temperature information of the target person U to identify changes in the body temperature of the target person U. A person's body temperature rises due to psychological stress. Using this characteristic, the tension detection unit 112 determines whether the temperature change of the subject person U is showing an upward trend. If the tension detection unit 112 determines that the temperature change of the subject person U is showing an upward trend, it detects that the subject person U, who is using the operation terminal 200, is in a state of tension.

[0063] For example, there are various methods for identifying body temperature changes by the body temperature change identification unit 118. If the body temperature acquired by the body temperature acquisition unit 104 is the body temperature value of a local location (e.g., the forehead) of the subject person U, the body temperature change identification unit 118 determines whether the difference between the body temperature at the beginning and end of a predetermined period is an increase and whether the amount of increase is greater than or equal to a threshold. If the difference between the body temperature at the beginning and end of a predetermined period is an increase and the amount of increase is greater than or equal to a threshold, the body temperature change identification unit 118 identifies that the body temperature change of the subject person U is showing an upward trend. Alternatively, the body temperature change identification unit 118 may obtain a differential value indicating the body temperature change of the subject person U and determine whether the differential value is greater than or equal to a positive threshold. If the differential value of the time-series data of the subject person U's body temperature is greater than or equal to a threshold (i.e., the amount of change in body temperature increase is large), the body temperature change identification unit 118 identifies that the body temperature change of the subject person U is showing an upward trend.

[0064] As another example, if the body temperature acquisition unit 104 acquires body temperature data via thermography, the body temperature change identification unit 118 may determine whether the area of ​​the region showing a temperature above a predetermined value in the thermography at the beginning and end of a predetermined period has increased. If the area of ​​the region showing a temperature above a predetermined value in the thermography at the beginning and end of the predetermined period has increased, the body temperature change identification unit 118 identifies that the body temperature change of the subject person U is showing an upward trend. Alternatively, the body temperature change identification unit 118 may calculate the differential value representing the change in the area of ​​the region showing a temperature above a predetermined value in the thermography and determine whether the differential value is above a positive threshold. If the differential value of the time-series data of the area of ​​the region showing a temperature above a predetermined value for the subject person U is above the threshold (i.e., the increase in the area of ​​the region showing a temperature above a predetermined value is large), the body temperature change identification unit 118 identifies that the body temperature change of the subject person U is showing an upward trend.

[0065] Whether or not the body temperature of subject U is on an upward trend may be determined using the stress level information DB130 described above. In that case, the stress level identification information stored in the stress level information DB130 includes body temperature change information that shows a pattern of rising body temperature caused by psychological stress.

[0066] Furthermore, the configuration of the state information generation unit 106 shown in Figures 8 to 10 can be combined in multiple ways. In other words, the state information generation unit 106 can include a facial expression recognition unit 110, a tension state detection unit 112, a hand movement tracking unit 114, a suspicious movement detection unit 116, and a body temperature change identification unit 118.

[0067] <Output Processing> The output processing unit 108 uses the generated status information of the target person U to produce a predetermined output. An example of a predetermined output is an alert output to an alarm device (not shown). This status information of the target person U includes at least one of the following status information. - State information indicating the tension state of the target person U, detected from facial expression changes identified based on the facial image of the target person U by the facial expression recognition unit 110 and the tension state detection unit 112. - The hand movement tracking unit 114 and the suspicious movement detection unit 116 detect status information indicating suspicious movements of the target person U based on the hand image of the target person U. • The body temperature change identification unit 118 and the tension state detection unit 112 detect the tension state of the target person U based on the body temperature information of the target person U.

[0068] The alarm system can be various, but for example, it could be the main operating terminal 200, a monitoring room within the airport, a security guard room, a communication device carried by an airport employee or inspector, security guard, or an external monitoring device connected to the monitoring device 100 via a communication network 3. For example, the alarm system consists of a computer 1000 (Figure 4) including a display device and storage device, such as a personal computer or a server computer.

[0069] The alert information output includes information indicating that the target person U using the operating terminal 200 is in a state of tension or exhibiting suspicious behavior. Furthermore, if there are multiple operating terminals 200, the alert information includes information indicating which operating terminal 200 the target person U is using. Additionally, the alert information includes information indicating the date and time the state was detected.

[0070] Other examples of the specified output include status information of the subject person U, or a record of the alert information mentioned above. This record may also include information indicating that the status of the subject person U is normal (associated with information that can identify the operation terminal 200 and date and time information), even if the status information of the subject person U indicates that it is in a normal state.

[0071] There are various ways to generate alerts, as exemplified below, but they are not limited to these. Furthermore, multiple methods can be combined. (1) A warning message is displayed on the display 210 of the operating terminal 200. (2) A warning message is output from the speaker (not shown) of the operating terminal 200. For example, the output processing unit 108 may display a warning message such as "Suspicious activity has been detected, so the procedure will be stopped" on the display 210, or output an audio message from the speaker of the operation terminal 200. Furthermore, the monitoring device 100 may suspend the immigration inspection procedure performed by the operation terminal 200. (3) Activate the operating terminal 200 or a rotating light (not shown) installed near the operating terminal 200. These rotating lights are installed in a position and at a height that can be seen (visually or audibly) by the supervisor. (4) The alert information is displayed on the display (not shown) of the monitoring device 100. (5) Alert information is output from the speaker (not shown) of the monitoring device 100. (6) The alert information is recorded in the storage device 120 of the monitoring device 100. (7) Alert information is sent to communication devices (not shown) carried by inspectors, security guards, and other observers. (8) Activate a rotating light (not shown) that can be seen (visually or audibly) by inspectors, security guards, or other observers.

[0072] The warning messages and alert information displayed on the aforementioned display may be output in various forms, such as text and images. Furthermore, the warning messages and alert information output from the speaker may be output in various forms, such as voice and alert sounds.

[0073] <Example of operation> The operation of the monitoring device 100 will be explained below using Figure 2. As described above, first, as shown in Figure 2, the image acquisition unit 102 acquires at least one of the following: a face image captured of the face of the target person U and a hand image captured of the hands of the target person U (step S101). The image acquisition unit 102 acquires a face image of the target person U from the face imaging unit 204 and a hand image of the target person U from the hand imaging unit 208.

[0074] Then, the body temperature acquisition unit 104 acquires body temperature information indicating the body temperature of the target person U (step S103). The body temperature acquisition unit 104 acquires body temperature information indicating the body temperature of the target person U from the body temperature measurement unit 206. As described above, the body temperature information includes measured values ​​indicating body temperature, thermographic images, etc.

[0075] The state information generation unit 106 generates state information indicating the state of the target person U using at least one of the acquired face image and hand image of the target person U, and the identified body temperature information of the target person U (step S105). The output processing unit 108 uses the generated state information of the target person U to perform a predetermined output (step S107). The processes in steps S105 and S107 are described in detail below.

[0076] Next, we will explain the details of the state information generation process in step S105 of Figure 2. Each process of generating state information indicating the state of the target person U (changes in facial expression, hand movements, and changes in body temperature) using the target person U's face image, hand image, and body temperature will be explained using Figures 11 to 13.

[0077] <Generating state information based on changes in facial expression> In the state information generation unit 106, the facial expression recognition unit 110 processes the acquired facial image to recognize the time-series changes in the facial expression of the target person U (step S111). Specifically, the facial expression recognition unit 110 estimates 468 3D facial landmarks from the captured image, for example, by using MediaPipe FaceMesh, recognizes the changes in the facial expression of the target person U, and outputs time-series coordinate data.

[0078] Then, the tension detection unit 112 acquires time-series coordinate data of 468 3D facial landmarks estimated by the facial recognition unit 110 (step S113). The tension detection unit 112 compares the time-series coordinate data showing the facial expression changes of the recognized target person U with a predetermined tension pattern (step S115).

[0079] The tension detection unit 112 then refers to the tension information database (hereinafter referred to as "DB") 130 and determines whether the acquired time-series coordinate data of the facial expression corresponds to an facial expression pattern indicating a state of tension stored in the tension information DB 130 (step S117). If the tension detection unit 112 determines that the acquired time-series coordinate data of the facial expression corresponds to an facial expression pattern in the tension information DB 130 (YES in step S117), it detects that the mental state of the target person U using the operation terminal 200 is one of tension (step S119). The state information generation unit 106 generates state information indicating that the mental state of the target person U is one of tension. Then, the processing of this flow returns to the processing shown in Figure 2.

[0080] On the other hand, if it is determined that the acquired time-series coordinate data of facial expressions does not correspond to any facial expression pattern in the tension state information DB130 (NO in step S117), step S119 is bypassed and the process returns to Figure 2. In other words, the tension state detection unit 112 does not detect that the mental state of the target person U using the operation terminal 200 is one of tension. The state information generation unit 106 may also generate state information indicating that the mental state of the target person U is not one of tension.

[0081] In this way, the state information generation unit 106 can generate state information indicating the state of the target person U using the acquired facial image of the target person U.

[0082] <Generating state information based on user actions> In the state information generation unit 106, the hand motion tracking unit 114 processes the hand image of the target person U to recognize and track the hand movements of the target person U (step S121). Specifically, the hand motion tracking unit 114 uses MediaPipe Hands to estimate 21 3D finger landmarks from one captured frame, recognizes (tracks) the hand movements of the target person U, and outputs time-series coordinate data.

[0083] Then, the suspicious movement detection unit 116 acquires time-series coordinate data of 21 3D finger landmarks that were estimated when the hand movement tracking unit 114 tracked the hand movements of the target person U (step S123). Then, the suspicious movement detection unit 116 compares the time-series coordinate data of the recognized hand movements of the target person U with a predetermined suspicious movement pattern (step S125).

[0084] The suspicious action detection unit 116 refers to the suspicious action information database (hereinafter referred to as "DB") 140 and determines whether the acquired time-series coordinate data of the user's hand movements matches a suspicious action pattern in the suspicious action information DB 140 (step S127). If the suspicious action detection unit 116 determines that the acquired time-series coordinate data of the user's hand movements matches a suspicious action pattern in the suspicious action information DB 140 (YES in step S127), it detects that the target person U using the operation terminal 200 is performing a suspicious action (step S129). In this case, the state information generation unit 106 generates state information indicating that the target person U is performing a suspicious action. Then, the processing of this flow returns to the processing shown in Figure 2.

[0085] On the other hand, if the acquired time-series coordinate data of the user's actions is determined not to match any suspicious action pattern in the suspicious action information DB140 (NO in step S127), step S129 is bypassed and the process returns to Figure 2. In other words, the suspicious action detection unit 116 does not detect that the target person U using the operation terminal 200 is performing a suspicious action. The state information generation unit 106 may also generate state information indicating that the target person U is not performing a suspicious action.

[0086] In this way, the state information generation unit 106 can generate state information indicating the state of the target person U using the acquired image of the target person U's hands.

[0087] <Generating state information based on changes in body temperature> In the state information generation unit 106, the body temperature change identification unit 118 identifies the change in the body temperature of the target person U using the target person U's body temperature information (step S131). The tension state detection unit 112 determines whether the change in the target person U's body temperature is showing an upward trend (step S133). If the tension state detection unit 112 determines that the change in the target person U's body temperature is showing an upward trend (YES in step S133), it detects that the target person U using the operation terminal 200 is in a state of tension (step S135). The state information generation unit 106 generates state information indicating that the target person U's mental state is one of tension. Then, the processing of this flow returns to the processing shown in Figure 2.

[0088] On the other hand, if it is determined that the body temperature of the subject person U is not showing an upward trend (NO in step S133), step S135 is bypassed and the process returns to Figure 2. In other words, the tension state detection unit 112 does not detect that the mental state of the subject person U using the operation terminal 200 is one of tension. The state information generation unit 106 may generate state information indicating that the mental state of the subject person U is not one of tension.

[0089] In this way, the state information generation unit 106 can generate state information indicating the state of the target person U using the acquired body temperature information of the target person U.

[0090] The flowcharts in Figures 11 to 13 allow for the parallel execution of multiple flowchart processes.

[0091] Using the state information indicating the state of the target person U generated as described above, the output processing unit 108 operates as shown in Figure 14. This flow shows the detailed processing of step S107 in Figure 2. The output processing unit 108 determines whether the status information indicating the state of the target person U indicates that suspicious behavior has been detected (step S141). If the status information of the target person U indicates that suspicious behavior has been detected (YES in step S141), the output processing unit 108 outputs an alert indicating that the status information of the target person U indicates that suspicious behavior has been detected (step S145).

[0092] On the other hand, if the status information of the target person U does not indicate that suspicious behavior has been detected (NO in step S141), the output processing unit 108 determines whether the status information indicating the status of the target person U indicates that a state of tension has been detected (step S143). If the status information of the target person U indicates that a state of tension has been detected (YES in step S143), the output processing unit 108 outputs an alert indicating that the status information of the target person U has detected a state of tension (step S145). Then, the processing of this flow returns to the processing shown in Figure 2.

[0093] On the other hand, if the status information of the target person U does not indicate that a state of tension has been detected (NO in step S143), step S145 is bypassed and the process returns to Figure 2. In other words, no alert is output by the output processing unit 108.

[0094] Note that the processing order of steps S141 and S143 is not limited to the order shown in this diagram, and may be in the reverse order.

[0095] The operation of the monitoring system 1 will be explained below using Figure 15, following the usage scenario of the operation terminal 200 by the target person U. First, the person U who will use the operation terminal 200 stands in front of the operation terminal 200 (step ST1). The monitoring system 1 detects the presence of person U using a human presence sensor or the like (step S201). When the monitoring system 1 detects a person in front of the operation terminal 200, it identifies that person as person U and begins monitoring their state. Specifically, the face imaging unit 204 starts capturing images of person U's face and generates a face image, and the state information generation unit 106 starts recognizing person U's facial expression and monitoring changes in facial expression (step S203). In addition, the body temperature measurement unit 206 starts measuring person U's body temperature, and the state information generation unit 106 starts monitoring changes in person U's body temperature (temperature rise) (step S205). Furthermore, the hand imaging unit 208 starts capturing images of person U's hands and generates hand images, and the state information generation unit 106 starts monitoring for suspicious actions by person U (step S207). Steps S203 to S207 may be executed in parallel.

[0096] Then, the state information generation unit 106 detects that the state information of the target person U indicates a state of tension or suspicious behavior based on at least one of the target person U's facial expression changes, hand images, and body temperature changes (step S209). Then, the output processing unit 108 causes the warning device to output alert information indicating that the state information of the target person U indicates a state of tension or suspicious behavior (step S211). As described above, the state information of the target person U includes information indicating suspicious behavior detected using hand images, and information indicating a state of tension detected using the target person U's facial expression changes and body temperature changes (increase in body temperature).

[0097] As described above, according to this embodiment, the monitoring device 100 comprises an image acquisition unit 102, a body temperature acquisition unit 104, a state information generation unit 106, and an output processing unit 108. The image acquisition unit 102 acquires at least one of a face image, which is an image of the target person's face, and a hand image, which is an image of the target person's hands. The body temperature acquisition unit 104 acquires body temperature information, which indicates the body temperature of the target person. The state information generation unit 106 generates state information, which indicates the state of the target person, using at least one of the acquired face image and hand image of the target person, and the identified body temperature information of the target person. The output processing unit 108 performs a predetermined output using the generated state information of the target person.

[0098] In this way, the monitoring device 100 generates state information indicating the state of the target person using at least one of the acquired facial image and hand image of the target person, along with the identified target person's body temperature information, thereby enabling accurate detection of suspicious behavior.

[0099] For example, if image processing is used to identify attribute information of a target person U and detect suspicious behavior based on changes in those attributes, it is necessary to image a wide area around the target person U. As a result, it may not be possible to detect suspicious behavior in only a part of the body, for example, it may not be possible to detect suspicious behavior that is only performed with the hands.

[0100] According to the monitoring device 100 of this disclosure, suspicious actions performed only locally can be detected with high accuracy.

[0101] Furthermore, when tracking hand movements based on distance images generated by a 3D camera to detect suspected shoplifting by customers or fraudulent activity by part-time employees (suspicious behavior), only hand movement information is used. Therefore, if the hands are hidden or partially obscured in the captured image, detection may fail, or false detections may occur.

[0102] According to the monitoring device 100 of this disclosure, state information indicating the state of the subject person is generated using at least one of the acquired facial image and hand image of the subject person, along with the identified subject person's body temperature information. Therefore, even if the hands of subject person U are not visible in the hand image, suspicious behavior of subject person U can be detected with high accuracy.

[0103] In particular, the monitoring device 100 of this disclosure makes it possible to detect fraudulent activity in fingerprint authentication for automated immigration screening. Furthermore, by detecting changes in facial expressions and body temperature resulting from psychological stress during fraudulent activity, the monitoring device 100 can prevent omissions that could not be detected by hand tracking, and can also improve accuracy by combining these methods.

[0104] The embodiments of the present invention have been described above with reference to the drawings, but these are merely examples of the present invention, and various other configurations can also be adopted. (Other embodiments) For example, in the above embodiment, the state information generation unit 106 generated state information using the tension state information DB 130 and the suspicious behavior information DB 140. The state information generation unit 106 may generate state information using a machine learning (ML) model that has been used to study the target's actions, etc., with respect to the tension state information DB 130 and the suspicious behavior information DB 140. Alternatively, the state information generation unit 106 may output the judgment result as a similarity score. In that case, thresholds may be set for changes in facial expression, changes in body temperature, and suspicious behavior, and the output processing unit 108 may decide whether or not to output an alert.

[0105] An example of detecting suspicious behavior as state information will be explained using Figure 16. In step S105 of Figure 2, the hand movement tracking unit 114 processes the hand image of the target person U to recognize and track the hand movements of the target person U (step S121). Then, the suspicious behavior detection unit 116 acquires time-series coordinate data of 21 3D finger landmarks that were estimated when the hand movement tracking unit 114 tracked the hand movements of the target person U (step S123). Then, the suspicious behavior detection unit 116 inputs the time-series coordinate data of the recognized hand movements of the target person U into a learning model that has learned suspicious behavior and obtains the similarity (step S151). The suspicious behavior detection unit 116 determines whether the similarity is above a threshold (step S153). If the similarity is above a threshold (YES in step S153), the suspicious behavior detection unit 116 detects that the target person U is in a state of performing a suspicious behavior (step S129). If the similarity is below the threshold (NO in step S153), step S129 is bypassed and the process returns to the flow shown in Figure 2. In other words, no state information is generated indicating that the target person U is performing suspicious actions. The state information generation unit 106 may also generate state information indicating that the target person U is not performing suspicious actions (i.e., is not performing suspicious actions).

[0106] Next, an example of detecting changes in facial expression as state information will be explained using Figure 17. In step S105 of Figure 2, the facial expression recognition unit 110 processes the facial image acquired by the image acquisition unit 102 from the face imaging unit 204 to recognize the time-series changes in the facial expression of the target person U (step S111). Then, the tension state detection unit 112 inputs the time-series coordinate data of the 468 3D facial landmarks estimated by the facial expression recognition unit 110 into a learning model that has learned the changes in facial expression, and obtains the similarity (step S161). The tension state detection unit 112 determines whether the similarity is above a threshold (step S163). If the similarity is above a threshold (YES in step S163), the tension state detection unit 112 detects that the target person U is in a state of tension (step S117). The state information generation unit 106 generates state information indicating that the target person U is in a state of tension. If the similarity is below the threshold (NO in step S163), step S117 is bypassed, and the process returns to the flow shown in Figure 2. In other words, no state information indicating that the subject person U is in a state of tension is generated. The state information generation unit 106 may also generate state information indicating that the subject person U is not in a state of tension.

[0107] Furthermore, the learning model may be trained by images of suspicious actions actually performed on the operating terminal 200. For example, the monitoring device 100 may acquire images of the hands of the target person U when the state information generated by the state information generation unit 106 indicates that a suspicious action has been performed or when a state of tension is detected, and input these images into the learning model to train it as a suspicious action. Alternatively, the monitoring device 100 may input images of hands simulating a new type of suspicious action into the learning model to train it as a suspicious action.

[0108] With this configuration, state information is generated using a learning model, which increases the likelihood of accurately detecting even new types of suspicious behavior.

[0109] Alternatively, the functions of the status information generation unit 106 of the monitoring device 100 may be delegated to the computer 1000 of the operation terminal 200. In other words, the operation terminal 200 may perform the status information generation process, transmit the generated status information to the monitoring device 100, and the output processing unit 108 may use the acquired status information to perform output processing.

[0110] Other examples of the disclosure may include a program that causes at least one computer to execute the method disclosed above, or a recording medium that is readable by a computer on which such a program is recorded. This recording medium may include non-temporary tangible media. This computer program includes computer program code that, when executed by a computer, causes the computer to perform a monitoring method on a monitoring device.

[0111] Furthermore, any combination of the above components, as well as any conversion of the expressions of this disclosure between methods, apparatus, systems, recording media, computer programs, etc., are also valid as aspects of this disclosure.

[0112] Furthermore, the various components of this disclosure do not necessarily have to be independent entities; multiple components may be formed as a single member, one component may be formed from multiple members, one component may be part of another component, or a part of one component may overlap with a part of another component, and so on.

[0113] Furthermore, although the methods and computer programs disclosed herein describe multiple steps in sequence, the order in which these steps are described does not limit the order in which they must be performed. Therefore, when implementing the methods and computer programs disclosed herein, the order of the steps can be changed to the extent that it does not impair the content.

[0114] Furthermore, the methods and computer programs described herein are not limited to being executed at individually different times. For example, other procedures may be executed while one procedure is being executed, or the timing of one procedure's execution may overlap with the timing of another procedure's execution in whole or in part.

[0115] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate. Furthermore, any information regarding the subject of this disclosure that is obtained or used will be done in a lawful manner.

[0116] Furthermore, while the flowcharts used in the above description show multiple steps (processes) in sequence, the execution order of the steps performed in each embodiment is not limited to the order in which they are described. In each embodiment, the order of the illustrated steps can be changed to the extent that it does not impede the content.

[0117] Some or all of the above embodiments may also be described as follows, but are not limited to the following: 1. Image acquisition means for acquiring at least one of a face image taken of the face of a target person and a hand image taken of the hands of the target person, A body temperature acquisition means for acquiring body temperature information indicating the body temperature of the aforementioned person, A state information generation means generates state information indicating the state of the target person using at least one of the acquired face image and hand image of the target person and the identified body temperature information of the target person, An output processing means that uses the generated state information of the target person to perform a predetermined output, A monitoring device equipped with the following features. 2. In the monitoring device described in 1., The state information generation means is By processing the acquired facial image, the facial expression of the subject person is recognized. Using the facial expression of the recognized subject person, the subject person's state of tension is detected. The system generates the status information indicating that the subject's state of tension has been detected. The output processing means is A monitoring device that, when the generated status information of the target person indicates that the target person's state of tension has been detected, outputs an alert to a predetermined device as the predetermined output. 3. In the monitoring device described in 1. or 2., The state information generation means is By processing the acquired hand image, suspicious behavior of the subject is detected. The system generates the status information indicating that suspicious behavior has been detected in the person concerned. The output processing means is A monitoring device that, when the generated status information of the target person indicates that suspicious behavior by the target person has been detected, outputs an alert to a predetermined device as the predetermined output. 4. In any one of the monitoring devices described in 1. to 3., The state information generation means is Using the acquired information indicating the body temperature of the subject, the time-series changes in the subject's body temperature are identified. If the body temperature of the identified subject is on an upward trend, it is detected that the subject is in a state of tension. The system generates the status information indicating that the subject's state of tension has been detected. The output processing means is A monitoring device that, when the generated status information of the target person indicates that the target person's state of tension has been detected, outputs an alert to a predetermined device as the predetermined output. 5. In any one of the monitoring devices described in 1. to 4., The aforementioned target individuals include those undergoing immigration and customs procedures at airports, as well as surveillance devices. 6. In any of the monitoring devices described in 1. to 5., The monitoring device executes a process when the target person operates the operating terminal. The aforementioned operating terminal is A fingerprint reader that reads the fingerprints of the aforementioned person, A first camera that captures images of the hands of the aforementioned person, A second camera for capturing an image of the subject person's face, It has a sensor for measuring the body temperature of the person concerned, The image acquisition means acquires an image of the subject person's hands from the first camera while the subject person is performing the fingerprint reading operation using the fingerprint reader, The image acquisition means acquires the face image of the subject person from the second camera while the subject person is operating the operating terminal. The body temperature acquisition means is a monitoring device that acquires the body temperature of the subject person from the sensor while the subject person is operating the operation terminal. 7. In the monitoring device described in 2. The state information generation means is By processing the facial image of the aforementioned person, the time-series changes in the person's facial expression are recognized. A monitoring device that detects that a person is in a state of tension by comparing the facial expression changes of the recognized person with a predetermined tension state pattern. In the monitoring device described in 8.3, The state information generation means is By processing the hand images of the subject person, the movements of the subject person's hands are recognized. A monitoring device that detects whether a hand movement of a recognized person is suspicious by comparing it with a predetermined suspicious movement pattern. 9. One or more computers, At least one of the following is obtained: a facial image of the subject person and a hand image of the subject person. Obtain body temperature information indicating the body temperature of the aforementioned person, Using at least one of the facial image and hand image of the target person obtained, and the body temperature information of the identified target person, state information indicating the state of the target person is generated. A monitoring method that uses the generated status information of the target person to produce a predetermined output. 10. To the computer, Image acquisition process that acquires at least one of a face image taken of the target person's face and a hand image taken of the target person's hands. A body temperature acquisition process that obtains body temperature information indicating the body temperature of the aforementioned person. A state information generation process that generates state information indicating the state of the target person using at least one of the acquired face image and hand image of the target person and the identified body temperature information of the target person. A program that performs an output process that produces a predetermined output using the generated state information of the target person. 11. To the computer, Image acquisition process that acquires at least one of a face image taken of the target person's face and a hand image taken of the target person's hands. A body temperature acquisition process that obtains body temperature information indicating the body temperature of the aforementioned person. A state information generation process that generates state information indicating the state of the target person using at least one of the acquired face image and hand image of the target person and the identified body temperature information of the target person. A computer-readable recording medium containing a program that causes the system to execute an output process that produces a predetermined output using the generated state information of the target person.

[0118] Furthermore, some or all of the configurations described in Appendices 2 to 8, which are subordinate to Appendice 1 (monitoring device) as described above, may also be subordinate to Appendice 9 (monitoring method), Appendice 10 (program), and Appendice 11 (recording medium) in the same way as in Appendices 2 to 8. Moreover, not limited to Appendices 9, 10, and 11, some or all of the configurations described as appendices may also be subordinate to various hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above. [Explanation of symbols]

[0119] 1. Monitoring System 3. Communication Network 100 Monitoring equipment 102 Image acquisition unit 104 Body temperature acquisition section 106 State Information Generation Unit 108 Output Processing Unit 110 Facial expression recognition unit 112 Tension state detection unit 114 Handheld motion tracking unit 116 Suspicious Activity Detection Unit 118 Body Temperature Change Identification Section 120 Storage device 130 Tension Information Database 140 Suspicious Activity Information Database 200 Operating terminals 202 Operation section 204 Face Imaging Unit 206 Body Temperature Measurement Unit 208 Handheld Imaging Unit 210 displays 212 Fingerprint reading unit 1000 computers 1010 Bus 1020 Processor 1030 memory 1040 Storage Devices 1050 Input / Output Interface 1060 Network Interfaces

Claims

1. Image acquisition means for acquiring at least one of a face image captured of the target person's face and a hand image captured of the target person's hands, A body temperature acquisition means for acquiring body temperature information indicating the body temperature of the aforementioned person, A state information generation means generates state information indicating the state of the target person using at least one of the acquired face image and hand image of the target person and the identified body temperature information of the target person, An output processing means that uses the generated state information of the target person to perform a predetermined output, A monitoring device equipped with the following features.

2. In the monitoring device according to claim 1, The state information generation means is By processing the acquired facial image, the facial expression of the subject person is recognized. Using the facial expression of the recognized subject person, the subject person's state of tension is detected. The system generates the status information indicating that the subject's state of tension has been detected. The output processing means is A monitoring device that, when the generated status information of the target person indicates that the target person's state of tension has been detected, outputs an alert to a predetermined device as the predetermined output.

3. In the monitoring device according to claim 1 or 2, The state information generation means is By processing the acquired hand image, suspicious behavior of the subject is detected. The system generates the status information indicating that suspicious behavior has been detected in the person concerned. The output processing means is A monitoring device that, when the generated status information of the target person indicates that suspicious behavior by the target person has been detected, outputs an alert to a predetermined device as the predetermined output.

4. In the monitoring device according to claim 1 or 2, The state information generation means is Using the acquired information indicating the body temperature of the subject, the time-series changes in the subject's body temperature are identified. If the body temperature of the identified subject is on an upward trend, it is detected that the subject is in a state of tension. The system generates the status information indicating that the subject's state of tension has been detected. The output processing means is A monitoring device that, when the generated status information of the target person indicates that the target person's state of tension has been detected, outputs an alert to a predetermined device as the predetermined output.

5. In the monitoring device according to claim 1 or 2, The aforementioned target individuals include those undergoing immigration and customs procedures at airports, as well as surveillance devices.

6. In the monitoring device according to claim 1 or 2, The monitoring device executes a process when the target person operates the operating terminal. The aforementioned operating terminal is A fingerprint reader that reads the fingerprints of the aforementioned person, A first camera that captures images of the hands of the aforementioned subject, A second camera that captures an image of the subject person's face, It has a sensor for measuring the body temperature of the person concerned, The image acquisition means acquires an image of the subject person's hands from the first camera while the subject person is performing the fingerprint reading operation using the fingerprint reader. The image acquisition means acquires the face image of the target person from the second camera while the target person is operating the operating terminal. The body temperature acquisition means is a monitoring device that acquires the body temperature of the subject person from the sensor while the subject person is operating the operation terminal.

7. In the monitoring device described in claim 2, The state information generation means is By processing the facial image of the aforementioned person, the time-series changes in the person's facial expression are recognized. A monitoring device that detects that a person is in a state of tension by comparing the facial expression changes of the recognized person with a predetermined tension state pattern.

8. In the monitoring device described in claim 3, The state information generation means is By processing the hand images of the subject person, the movements of the subject person's hands are recognized. A monitoring device that detects whether a hand movement of a recognized person is suspicious by comparing it with a predetermined suspicious movement pattern.

9. One or more computers, At least one of the following is obtained: a facial image of the subject person and a hand image of the subject person. Obtain body temperature information indicating the body temperature of the aforementioned person, Using at least one of the facial image and hand image of the target person obtained, and the body temperature information of the identified target person, state information indicating the state of the target person is generated. A monitoring method that uses the generated status information of the target person to produce a predetermined output.

10. On the computer, Image acquisition process that acquires at least one of a face image taken of the target person's face and a hand image taken of the target person's hands. A body temperature acquisition process that obtains body temperature information indicating the body temperature of the aforementioned person. A state information generation process that generates state information indicating the state of the target person using at least one of the acquired face image and hand image of the target person and the identified body temperature information of the target person. A program that performs an output process that produces a predetermined output using the generated state information of the target person.

Images