Event Detection System, Event Detection Method, and Program

Abstract

An event detection system (10) comprises: a calculation unit (16) that calculates the degree of similarity between at least part of skeletal information extracted from a captured image of a user visiting an ATM and at least part of registration skeletal information, which is extracted from a registration image showing a person's calling motion and is registered in a motion database; and a determination unit (17) that determines that an event associated with the ATM has been detected if the degree of similarity is at least equal to a prescribed threshold value. This makes it possible to easily detect problematic events at ATMs while protecting privacy.

Description

Technical Field

[0001] The present disclosure relates to an event detection system, a monitoring system, an event detection method, and a non-transitory computer-readable medium.

Background Art

[0002] In recent years, transfer fraud in the form of instructing to transfer money from an ATM (Automatic Teller Machine) over the phone has become a problem. At ATMs with few or no staff, it is required to automatically alert transfer fraudsters who perform transfer operations while on the phone. Therefore, technologies for monitoring the actions of transfer fraudsters and detecting call operations have been developed. For example, Patent Document 1 discloses a call determination device that identifies the imaging area of a hand based on the position of a face detected from imaging information, and determines whether the person being imaged is on a call based on the amount of change in the pixel values of the imaging area of the hand.

[0003] Here, from the perspective of privacy protection, it is required to detect call operations without retaining specific pixel information of the photographed information. For example, Patent Document 2 discloses an action analysis device that generates a skeletal image of a person and a behavior image showing the temporal change of the skeleton from a plurality of original images, and generates a model capable of learning and inferring an action pattern based on the original image, the skeletal image, and the behavior image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the learning-based behavior analysis method described in Patent Document 2 mentioned above has a problem in that a large amount of learning data is required, and it takes time and cost to prepare the learning data.

[0006] An object of the present disclosure is to provide an event detection system, a monitoring system, an event detection method, and a non-transitory computer-readable medium that can easily detect an event that causes a problem at an ATM while protecting privacy in view of the above-described problems.

Means for Solving the Problems

[0007] An event detection system according to an aspect of the present disclosure includes calculating means for calculating a similarity between at least a part of skeleton information extracted from a captured image of a user who has visited an ATM and at least a part of registered skeleton information extracted from a registration image showing a call operation of a person and registered in an operation database; determining means for determining that an event related to the ATM has been detected when the similarity is equal to or greater than a predetermined threshold value and includes.

[0008] A monitoring system according to an aspect of the present disclosure includes an ATM, an event detection device that detects an event related to the ATM and includes, wherein the event detection device has calculating means for calculating a similarity between at least a part of skeleton information extracted from a captured image of a user who has visited an ATM and at least a part of registered skeleton information extracted from a registration image showing a call operation of a person and registered in an operation database; and determining means for determining that the event has been detected when the similarity is equal to or greater than a predetermined threshold value and has.

[0009] An event detection method according to an aspect of the present disclosure includes At least a part of skeleton information extracted from a captured image of a user who visited an ATM, and at least a part of registered skeleton information extracted from a registration image showing a person's call operation and registered in an operation database are used to calculate a similarity, When the similarity is equal to or greater than a predetermined threshold value, it is determined that an event related to the ATM has been detected.

[0010] A non-transitory computer-readable medium according to an aspect of the present disclosure stores a calculation process for calculating a similarity between at least a part of skeleton information extracted from a captured image of a user who visited an ATM and at least a part of registered skeleton information extracted from a registration image showing a person's call operation and registered in an operation database, a determination process for determining that an event related to the ATM has been detected when the similarity is equal to or greater than a predetermined threshold value, and a program for causing a computer to execute the processes.

Advantages of the Invention

[0011] According to the present disclosure, it is possible to provide an event detection system, a monitoring system, an event detection method, and a non-transitory computer-readable medium that can easily detect an event that causes a problem at an ATM while protecting privacy.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Mode for Carrying Out the Invention

[0013] Hereinafter, the present disclosure will be described through embodiments, but the disclosure according to the claims is not limited to the following embodiments. Also, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. In each drawing, the same elements are denoted by the same reference numerals, and duplicate explanations are omitted as necessary.

[0014] <Embodiment 1> First, Embodiment 1 of the present disclosure will be described. FIG. 1 is a block diagram showing the configuration of an event detection system 10 according to Embodiment 1. The event detection system 10 is a computer system that detects events related to an ATM (Automatic Teller Machine). An event related to an ATM is an event that causes problems in the ATM, for example, an event suspected of transfer fraud. The above event includes at least the execution of a call operation by an ATM access user. Hereinafter, an event related to an ATM may be simply referred to as an event. The event detection system 10 includes a calculation unit 16 and a determination unit 17.

[0015] The calculation unit 16 is also called a calculation means. The calculation unit 16 compares at least a part of the skeleton information extracted from the captured image with at least a part of the registered skeleton information, and calculates the similarity between them. The captured image is a captured image of a user who has visited the ATM. The registered skeleton information is extracted from a registered image showing a person's call operation and registered in an operation database (DB).

[0016] The determination unit 17 is also called a determination means. When the similarity is equal to or greater than a predetermined threshold value, the determination unit 17 determines that an event has been detected.

[0017] Then, in response to detecting an event, the event detection system 10 executes a predetermined process. For example, in response to detecting an event, the event detection system 10 may transmit warning information to the ATM or a bank management device (not shown) and cause it to output. Also, for example, in response to detecting an event, the event detection system 10 may record the determination history of event detection.

[0018] According to Embodiment 1 as described above, since the event detection system 10 uses skeletal information for event detection, privacy can be ensured. Further, since the event detection system 10 uses the similarity calculated based on the comparison of skeletal information to detect a calling operation, a large amount of learning data is not required. Therefore, the event detection system 10 can easily detect an event.

[0019] <Embodiment 2> Next, Embodiment 2 of the present disclosure will be described. FIG. 2 is a diagram for explaining an event according to Embodiment 2. The event to be detected at least includes a user U who has visited the ATM 100 making a call using a mobile phone P. For example, the event may be only the calling operation by the user U, or may be the user U making a call and performing an input operation on the ATM 100.

[0020] A camera 150 is disposed above the ATM 100. The camera 150 is disposed at a position and an angle that can capture at least a part of the body of the user U who has visited the ATM 100. In the present Embodiment 2, the camera 150 is configured to capture the upper body of the user U, but alternatively, it may be configured to capture only the face area.

[0021] FIG. 3 is a block diagram showing the configuration of the monitoring system 1 according to Embodiment 2. The monitoring system 1 is a computer system that monitors a user U who has visited the ATM 100 and executes a predetermined process in response to detecting a target event. The monitoring system 1 includes a camera 150, an ATM 100, a server 200, and a bank management device 300. Each device may be connected to a network N. The network N may be wired or wireless.

[0022] (ATM100) The ATM100 is a computer device that realizes cash withdrawal, deposit, and transfer based on the input operations of user U. Also, the ATM100 transmits the video data captured by the camera 150 to the server 200. The ATM100 includes a communication unit 101, a control unit 102, an input unit 103, and a display unit 104.

[0023] The communication unit 101 is a communication interface with the network N. The input unit 103 is an input device that accepts inputs. The display unit 104 is a display device. The input unit 103 and the display unit 104 may be integrally configured like a touch panel.

[0024] The control unit 102 controls the hardware of the ATM100. Based on the input operations from user U received by the input unit 103, the control unit 102 executes the normal ATM100 processes (processes such as cash withdrawal, deposit, and transfer).

[0025] Also, the control unit 102 acquires video data from the camera 150 via the communication unit 101. And the control unit 102 transmits the video data to the server 200 via the network N at a predetermined timing.

[0026] For example, the control unit 102 starts transmitting video data to the server 200 in response to the input unit 103 receiving a predetermined first operation. Note that the transmission of video data may be the transmission of a series of video data including multiple frame images, or the transmission of frame images in frame units. The first operation may be an operation by user U to start various services (cash withdrawal, deposit, or transfer) using the ATM (such as activating the screen), or an operation by user U to start the transfer service (such as selecting "transfer"). Also, the first operation may be a specific operation to receive various services. As an example, the first operation may be the operation of inserting a cash card or a passbook, or the operation of entering a PIN.

[0027] For example, when the input unit 103 receives a predetermined second operation, the control unit 102 terminates the transmission of video data to the server 200. The second operation is different from the first operation. The second operation may be an operation for ending various services at the ATM (such as selecting end), or an operation by the user U for ending the transfer service (such as selecting "start the next transaction"). Note that the trigger for ending the transmission of video data to the server 200 may be that no operation is received within a predetermined time instead of receiving the second operation, or that the ATM 100 has executed a process of returning the cash card or passbook.

[0028] Also, when the control unit 102 receives warning information from the server 200 via the communication unit 101, the control unit 102 causes the display unit 104 to display the warning information. At this time, the control unit 102 may also cause the warning information to be output as sound by a voice output unit (not shown).

[0029] (Server 200) The server 200 is a computer device that detects an event based on the video data received from the ATM 100. That is, the server 200 is an example of the above-described event detection system 10. Also, when the server 200 detects an event, the server 200 transmits warning information to the ATM 100 or the bank management device 300. The server 200 includes a registration information acquisition unit 201, a registration unit 202, an operation DB 203, an image acquisition unit 204, an extraction unit 205, a calculation unit 206, a determination unit 207, and an output control unit 208.

[0030] The registration information acquisition unit 201 is also called the registration information acquisition means. The registration information acquisition unit 201 acquires a registration image showing at least the call operation of a person in response to a registration request from the bank management device 300 or by the operation of the administrator of the server 200. The registration image may be an image of a person simply making a call, or an image showing a person making a call and performing an input operation on the ATM. Note that the registration image may be a still image (a single frame image) or a moving image including a series of multiple frame images. The registration information acquisition unit 201 supplies the acquired registration image to the registration unit 202.

[0031] The registration unit 202 is also called the registration means. The registration unit 202 supplies the registration image to the extraction unit 205 described later, and acquires from the extraction unit 205 the skeleton information extracted from the registration image as the registration skeleton information R. Then, the registration unit 202 registers the acquired registration skeleton information R in the operation DB 203 as an operation included in the event to be detected.

[0032] The operation DB 203 is a storage device that stores a plurality of registration skeleton information Rs corresponding to the operations included in the event to be detected. Note that the number of the registration skeleton information Rs stored in the operation DB 203 is not limited to a plurality and may be 1.

[0033] The image acquisition unit 204 is also called the image acquisition means. The image acquisition unit 204 acquires the video data received from the ATM 100 and the frame image (captured image) included in the video data. That is, the image acquisition unit 204 acquires the frame image in response to detecting a first operation signal for which the ATM 100 is subjected to the first operation. Here, the first operation signal is a signal indicating that the input unit 103 has received the first operation from the user U to the ATM 100. The image acquisition unit 204 supplies the acquired frame image to the extraction unit 205.

[0034] The extraction unit 205 is also called an extraction means. The extraction unit 205 detects an image area (body area) of a person's body from a frame image and extracts (for example, cuts out) it as a body image. Then, the extraction unit 205 uses a skeleton estimation technique using machine learning to extract at least partial skeleton information of the person's body based on features such as joints of the person recognized in the body image. The skeleton information is information composed of "keypoints" which are characteristic points such as joints and "bones (bone links)" indicating links between the keypoints. The extraction unit 205 may use a skeleton estimation technique such as OpenPose. The extraction unit 205 supplies the extracted skeleton information to the calculation unit 206.

[0035] The calculation unit 206 is an example of the above-described calculation unit 16. The calculation unit 206 calculates the similarity between the extracted skeleton information and each registered skeleton information R registered in the motion DB 203. Note that the calculation target of the calculation unit 206 may be, instead of the above similarity, the similarity between at least a part of the extracted skeleton information and each registered skeleton information R, or the similarity between the extracted skeleton information and at least a part of each registered skeleton information, or the similarity between at least a part of the extracted skeleton information and at least a part of each registered skeleton information.

[0036] Note that the calculation unit 206 may calculate the above similarity directly using the skeleton information or indirectly using it. For example, the calculation unit 206 may convert at least a part of the extracted skeleton information and at least a part of each registered skeleton information R registered in the motion DB 203 into other formats, respectively, and calculate the similarity between the converted information to calculate the above similarity. In this case, the above similarity may be the similarity itself between the converted information or a value calculated using the similarity between the converted information. The conversion method may be normalization of the size of the skeleton information, conversion into a feature amount using the angle formed by each bone (that is, the degree of bending of the joint), or conversion into three-dimensional pose information based on a machine learning model learned in advance.

[0037] The determination unit 207 is an example of the determination unit 17 described above. The determination unit 207 identifies the number of registration skeleton information Rs for which the similarity calculated by the calculation unit 206 is equal to or greater than a predetermined threshold. Then, based on the number of corresponding registration skeleton information Rs, the determination unit 207 determines whether an event has been detected. In the second embodiment, when there is even one registration skeleton information R for which the similarity is equal to or greater than the predetermined threshold, the determination unit 207 determines that an event has been detected, and when there is no registration skeleton information R for which the similarity is equal to or greater than the predetermined threshold, the determination unit 207 does not determine that an event has been detected. However, alternatively, the determination unit 207 may determine that an event has been detected when the number of registration skeleton information Rs for which the similarity is equal to or greater than the predetermined threshold is equal to or greater than a predetermined number, and may not determine that an event has been detected when the number of registration skeleton information Rs for which the similarity is equal to or greater than the predetermined threshold is less than the predetermined number. The determination unit 207 supplies the determination result to the output control unit 208.

[0038] When it is determined by the determination unit 207 that an event has been detected, the output control unit 208 transmits warning information to the ATM 100 and the bank management device 300. Note that the transmission destination of the warning information may be either the ATM 100 or the bank management device 300.

[0039] (Bank management device 300) The bank management device 300 is a computer device used by bank staff. The bank management device 300 transmits a registration request for the call operation to the operation DB 203 to the server 200. At this time, the registration request includes a registration image. Further, in response to receiving warning information from the server 200, the bank management device 300 outputs the warning information by display or voice output to notify bank staff of the occurrence of an event. By grasping the occurrence of the event, bank staff can take actions such as rushing to the scene.

[0040] FIG. 4 is a diagram showing the skeletal information extracted from the frame image 500 according to Embodiment 2. The frame image 500 includes an image area of the upper body of the user U who is on the phone with the mobile phone P. The skeletal information shown in FIG. 4 includes a plurality of key points and a plurality of bones detected from the upper body. As an example, in FIG. 4, the key points include the right eye A11, the left eye A12, the head A2, the neck A3, the right shoulder A41, the left shoulder A42, the right elbow A51, the left elbow A52, the right hand A61, the left hand A62, and the waist A7.

[0041] The server 200 compares such skeletal information with the registered skeletal information R corresponding to the upper body and determines whether they are similar, thereby detecting the call operation. Note that for detecting the call operation, it is important whether the hand is located near the head. Therefore, the server 200 may calculate the similarity by weighting the positional relationship between the right hand A61 and the right eye A11 or the head A2, and the positional relationship between the left hand A62 and the left eye A12 or the head A2. Alternatively, the server 200 may use only the skeletal information related to the right eye A11, the left eye A12, the head A2, the right hand A61, and the left hand A62 among the extracted skeletal information for calculating the similarity. In addition to the form of holding the mobile phone P by hand while making a call, there is a form of making a call by bringing the ear close to the mobile phone P while supporting the mobile phone P with the shoulder in the call operation. In this case, the positional relationship between the right shoulder A41 and the right eye A11 or the head A2, and the positional relationship between the left shoulder A42 and the left eye A12 or the head A2 may be added to the weighting target. Alternatively, the server 200 may add the skeletal information related to the right shoulder A41 and the left shoulder A42 to the skeletal information related to the right eye A11, the left eye A12, the head A2, the right hand A61, and the left hand A62 as the skeletal information used for calculating the similarity.

[0042] FIG. 5 is a flowchart showing the flow of a method for transmitting video data by the ATM 100 according to Embodiment 2. First, the control unit 102 of the ATM 100 determines whether or not a first operation signal related to a first operation of the ATM 100 has been detected (S20). When the control unit 102 determines that the first operation signal has been detected (Yes in S20), it starts transmitting the video data acquired from the camera 150 to the server 200 (S21). On the other hand, when the control unit 102 determines that the first operation signal has not been detected (No in S20), it repeats the process shown in S20. Next, the control unit 102 determines whether or not a second operation signal related to a second operation of the ATM 100 has been detected (S22). When the control unit 102 determines that the second operation signal has been detected (Yes in S22), it ends the transmission of the video data acquired from the camera 150 to the server 200 (S23). On the other hand, when the control unit 102 determines that the second operation signal has not been detected (No in S22), it repeats the process shown in S22.

[0043] Note that in the above flowchart, the triggers for starting and ending the transmission of video data are the operation signals of the ATM 100, but it is not limited to this. For example, the control unit 102 may detect the triggers for starting and ending the transmission of video data by analyzing the video data acquired from the camera 150. For example, when the control unit 102 first detects a body area of a person different from the previous one in the video data, it may start transmitting the video data. Also, when the control unit 102 first detects that the body area of that person has disappeared after detecting a body area of a person different from the previous one in the video data, it may end the transmission of the video data.

[0044] In this way, by limiting the transmission period of the video data between a predetermined start trigger and end trigger, the communication data volume can be minimized. Also, outside the period, since the event detection process in the server 200 can be omitted, computing resources can be saved.

[0045] FIG. 6 is a flowchart showing the flow of a method for registering in the operation DB by the server 200 according to Embodiment 2. First, the registration information acquisition unit 201 of the server 200 receives a registration request including a registration image from the bank management device 300 (S30). Next, the registration unit 202 supplies the registration image to the extraction unit 205. The extraction unit 205 that has acquired the registration image extracts a body image from the registration image (S31). Next, the extraction unit 205 extracts registration skeletal information R from the body image (S32). At this time, the extraction unit 205 may use all the skeletal information extracted from the body image as the registration skeletal information R, or only a part of the skeletal information (for example, the skeletal information related to the right eye, left eye, head, right hand, and left hand) as the registration skeletal information R. Next, the registration unit 202 acquires the registration skeletal information R from the extraction unit 205 and registers the registration skeletal information R in the operation DB 203 (S33).

[0046] FIG. 7 is a flowchart showing the flow of an event detection method by the server 200 according to Embodiment 2. When the image acquisition unit 204 of the server 200 acquires video data and frame images included in the video data from the ATM 100 (Yes in S40), the extraction unit 205 extracts a body image from the frame images (S41). Next, the extraction unit 205 extracts skeletal information from the body image (S42). The calculation unit 206 calculates the similarity between at least a part of the extracted skeletal information and each registration skeletal information R registered in the operation DB 203 (S43). Next, the determination unit 207 determines whether there is registration skeletal information R whose similarity is equal to or greater than a predetermined threshold (S44). If there is registration skeletal information R whose similarity is equal to or greater than the predetermined threshold (Yes in S44), the determination unit 207 determines that an event has been detected (S45). Then, the output control unit 208 transmits warning information indicating event detection to the bank management device 300 and the ATM 100 (S46) and returns the process to S40. On the other hand, if there is no registration skeletal information R whose similarity is equal to or greater than the predetermined threshold (No in S44), the determination unit 207 does not determine that an event has been detected and returns the process to S40.

[0047] On the display unit (not shown) of the bank management apparatus 300 that has received the warning information, the display screen 600 shown in FIG. 8 may be displayed. FIG. 8 is a diagram showing an example of the display screen 600 of the bank management apparatus 300 according to Embodiment 2. For example, on the display screen 600, a message such as "There may be a risk of transfer fraud." may be displayed together with the location or identification information of the ATM 100 where the event was detected. As a result, bank staff can take actions such as rushing to the site where the event occurred, and it becomes possible to prevent transfer fraud in advance.

[0048] Also, on the display unit 104 of the ATM 100 that has received the warning information, the display screen 700 shown in FIG. 9 may be displayed. FIG. 9 is a diagram showing an example of the display screen 700 of the ATM 100 according to Embodiment 2. For example, on the display screen 700, a message such as "Please be careful of transfer fraud!" may be displayed together with an input area for selecting whether to continue or end the service. As a result, the user U can recognize that he / she is involved in transfer fraud, and it becomes possible to prevent transfer fraud in advance.

[0049] In addition, the ATM 100 that has received the warning information may, instead of or in addition to the display of the display screen 700, deliberately slow down the processing speed or give the user U a waiting time to make the user U confirm the presence or absence of transfer fraud. This can also prevent transfer fraud in advance.

[0050] As described above, according to Embodiment 2, since the server 200 uses the skeleton information for detecting events, it can ensure privacy. Also, since the server 200 detects the call operation using the similarity based on the comparison of the skeleton information, a large amount of learning data is not required. Therefore, the server 200 can easily detect events.

[0051] As described above, in Embodiment 2, if events are detected excessively and warnings are issued, the monitoring burden on bank staff may increase, and the psychological burden on user U may also increase. Therefore, the event detection conditions may be made stricter. As an example of stricter event detection conditions, the determination unit 207 of the server 200 may detect an event when user U is performing a call operation and user U has a predetermined attribute. That is, the determination unit 207 may determine that an event has been detected when the similarity of the skeletal information is equal to or greater than a predetermined threshold value and user U is determined to have a predetermined attribute. The predetermined attribute may be being an elderly person, having a deposit amount equal to or greater than a predetermined amount, having a history of a predetermined disease, living alone, or having been involved in fraud in the past, etc.

[0052] The server 200 may acquire the attribute information of the user based on the information read by the ATM 100. As an example, first, the ATM 100 reads the user ID (such as an account number or name) recorded on the passbook or cash card, and queries the bank management device 300 for the attribute information of user U. Then the ATM 100 transmits the attribute information of user U to the server 200. As another example, first, the ATM 100 reads the user ID recorded on the passbook or cash card, and transmits the user ID to the server 200. Then the server 200 uses the user ID to query the bank management device 300 for the attribute information of user U. Thereby, the server 200 can easily acquire the attribute information of user U. In addition, when the predetermined attribute can be estimated from external appearance such as age, the server 200 may estimate the attribute information of user U by image analysis from the video data acquired from the ATM 100 instead of the information read by the ATM 100.

[0053] FIG. 10 is a flowchart showing the flow of the event detection method by the server 200 according to a modification of Embodiment 2. The steps shown in FIG. 10 include S50 to S51 in addition to S40 to S46 in FIG. 7.

[0054] First, the server 200 executes the same processing as S40 to S44 in FIG. 7. When there is registered skeleton information R with a similarity equal to or higher than a predetermined threshold (Yes in S44), the determination unit 207 acquires the attribute information of the user U by the method described above (S50). Next, the determination unit 207 determines whether the user U has a predetermined attribute (S51). When the user U has a predetermined attribute (Yes in S51), the determination unit 207 determines that an event has been detected (S45). Then, the output control unit 208 transmits warning information indicating event detection to the bank management device 300 and the ATM 100 (S46), and returns the process to S40. On the other hand, when the user U does not have a predetermined attribute (No in S51), the determination unit 207 returns the process to S40 without determining that an event has been detected.

[0055] As another example of stricter event detection conditions, the determination unit 207 of the server 200 may determine that an event has been detected when the similarity of the skeleton information is equal to or higher than a predetermined threshold and a predetermined operation signal to the ATM 100 is detected. The predetermined operation signal may be an operation signal in a transfer service or any other arbitrary operation signal. Thereby, it is possible to avoid detecting, as an event, a situation where the user U is simply making a call in front of the ATM 100 instead of receiving a transfer instruction by phone.

[0056] By making the event detection conditions stricter in this way, excessive warnings can be avoided. Thereby, the monitoring burden on bank staff can be reduced, and the psychological burden on the user U can be reduced.

[0057] In addition, when it is determined that the user U has the predetermined attributes described above, the registration unit 202 of the server 200 may inquire of the bank management device 300 whether to newly register the skeleton information extracted from the frame image in the operation DB 203. FIG. 11 is a diagram showing an example of a display screen 601 according to a modification of Embodiment 2. The display screen 601 shown in FIG. 11 may be displayed on a display unit (not shown) of the bank management device 300. For example, on the display screen 601, a message such as "Do you want to register the extracted skeleton information as registration skeleton information?" may be displayed together with the extracted skeleton information. Although the skeleton information is superimposed on the frame image in the display screen 601 of FIG. 11, the frame image may be omitted from the viewpoint of privacy protection. When the bank staff selects "Register", a registration request is sent from the bank management device 300 to the registration unit 202 of the server 200. The registration unit 202 that has received the registration request registers the skeleton information in the operation DB 203 as the registration skeleton information R.

[0058] Therefore, the server 200 can determine event detection based on various call operations performed by a person with an attribute to be monitored during operation. Thereby, the determination accuracy can be improved during operation.

[0059] In addition, the skeleton information to be newly registered is not limited to the skeleton information related to the call operation, and may be the skeleton information related to a new operation that was not assumed at the time of registration of the registration skeleton information R. In this case, even when a person with an attribute to be monitored performs a new operation during operation, the skeleton information of the operation can be newly registered as the registration skeleton information R. Thereby, the detection target range can be expanded compared to the time of registration.

[0060] In addition, the conditions for making a new registration inquiry to the bank management device 300 are not limited to the user U having the predetermined attributes described above. For example, when the registration unit 202 of the server 200 detects that the same person appears multiple times in a plurality of video data with different shooting times, the bank management device 300 may be inquired whether to newly register the extracted skeleton information in the operation DB 203. Thereby, the actions peculiar to suspicious persons can be registered as a watch list and used for the determination of event detection.

[0061] <Embodiment 3> Next, Embodiment 3 of the present disclosure will be described. Embodiment 3 is characterized in that the extraction process of skeleton information is executed at an ATM. FIG. 12 is a block diagram showing the configuration of the monitoring system 1a according to Embodiment 3. The monitoring system 1a is different from the monitoring system 1 in that it includes an ATM 100a and a server 200a instead of the ATM 100 and the server 200.

[0062] (ATM100a) The ATM 100a is different from the ATM 100 in that it includes a control unit 102a instead of the control unit 102. The control unit 102a executes the normal processing of the ATM 100 in the same manner as the control unit 102, but includes an image acquisition unit 114 and an extraction unit 115.

[0063] The image acquisition unit 114 acquires video data from the camera 150. The image acquisition unit 114 supplies the frame images included in the acquired video data to the extraction unit 115.

[0064] Similar to the extraction unit 205, the extraction unit 115 extracts a body image from the frame image and extracts skeletal information of at least a part of the body of the user U from the body image. The extraction unit 115 transmits the extracted skeletal information to the server 200a via the network N. At this time, the extraction unit 115 may start the above-described body image extraction process or skeletal information extraction process triggered by the detection of the first operation signal. Further, the extraction unit 115 may end the above-described body image extraction process or skeletal information extraction process triggered by the detection of the second operation signal. Thereby, the calculation load can be minimized. Alternatively, the extraction unit 115 may start transmitting the skeletal information triggered by the detection of the first operation signal and end transmitting the skeletal information triggered by the detection of the second operation signal. Thereby, the amount of communication data can be minimized and the processing in the server 200 can be omitted, so that the calculation resources can be saved.

[0065] Note that the bank management device 300 may have the functions of the image acquisition unit 114 and the extraction unit 115. In this case, the ATM 100a may transmit the video data acquired from the camera 150 to the bank management device 300, and the bank management device 300 may extract the skeletal information. Then, the bank management device 300 may transmit the extracted skeletal information to the server 200.

[0066] (Server 200a) The server 200a differs from the server 200 in that it includes a skeletal information acquisition unit 209 instead of the image acquisition unit 204, the extraction unit 205, and the calculation unit 206. The skeletal information acquisition unit 209 acquires skeletal information from the ATM 100 and supplies the skeletal information to the calculation unit 206.

[0067] <Embodiment 4> Next, Embodiment 4 of the present disclosure will be described. Embodiment 4 is characterized in that a series of event detection processes are executed in the ATM. FIG. 13 is a block diagram showing the configuration of the monitoring system 1b according to Embodiment 4. The monitoring system 1b differs from the monitoring system 1a in that it includes an ATM 100b instead of the ATM 100a and the server 200a.

[0068] ATM100b is different from ATM100a in that it includes a control unit 102b and an operation DB 113 instead of the communication unit 101 and the control unit 102a. The control unit 102b is different from the control unit 102a in that it has a registration information acquisition unit 111, a registration unit 112, a calculation unit 116, a determination unit 117, and an output control unit 118. The registration information acquisition unit 111, the registration unit 112, the calculation unit 116, the determination unit 117, and the output control unit 118 execute the same processes as the registration information acquisition unit 201, the registration unit 202, the calculation unit 206, the determination unit 207, and the output control unit 208, respectively. The operation DB 113 is the same as the operation DB 203. That is, ATM100b is an example of the event detection system 10 described above.

[0069] As described above, according to the fourth embodiment, ATM100b executes a series of event detection processes. Since the exchange of information via the network N is omitted, the communication data volume can be reduced, the processing delay can be avoided, and the security level can be increased.

[0070] Note that the bank management device 300 may have some or all of the functions of the control unit 102b excluding the normal ATM functions.

[0071] <Embodiment 5> Next, a fifth embodiment of the present disclosure will be described. The fifth embodiment is characterized in that parameters (referred to as similarity calculation parameters) used when calculating the similarity of skeletal information are learned during operation. FIG. 14 is a block diagram showing the configuration of a monitoring system 1c according to the fifth embodiment. The monitoring system 1c is different from the monitoring system 1 in that it includes a server 200c instead of the server 200.

[0072] The server 200c includes a calculation unit 206c, a determination history DB 210, and a learning unit 211 instead of the calculation unit 206.

[0073] The calculation unit 206c calculates the similarity between the extracted skeleton information and each registered skeleton information R registered in the motion DB 203 using the similarity calculation parameter.

[0074] The determination history DB 210 is a storage device that associates the skeleton information extracted by the extraction unit 205 in the past with the determination result determined by the determination unit 207 based on the skeleton information and stores it as a determination history.

[0075] Here, an incorrect determination may occur due to an error in any of the skeleton extraction process, the calculation process, and the determination process. For example, an incorrect determination occurs in the following cases. (1) When the skeleton information indicating the call behavior is correctly extracted, but an error occurs in the calculation process or the determination process, and thus it is not determined as a call. (2) When, despite not performing a call behavior, an incorrect skeleton extraction results in a determination as a call. In this way, incorrect determination results and incorrectly extracted skeleton information are stored in the determination history DB 210 as a determination history. When the server 200c executes a learning process using the stored determination history, the accuracy of the model decreases. Therefore, the server 200c may correct the determination result and the extracted skeleton information performed by the determination unit 207 and then store the corrected information in the determination history DB 210 as a determination history. Thereby, a decrease in the accuracy of the model can be prevented. For example, in the case of (1), the correction may be to correct the determination result, and it may not be necessary to correct the skeleton information. Also, in the case of (2), the correction may be to correctly correct the skeleton information, correct the determination result, or correct both the determination result and the skeleton information. Note that the correction may be performed based on the user's input, but it may also be performed by other methods.

[0076] The learning unit 211 learns the similarity calculation parameter using the skeleton information and the determination result stored in the determination history DB 210. The learning method is not limited to this, but for example, distance learning can be mentioned. The learning unit 211 updates the similarity calculation parameter used by the calculation unit 206c to the learned similarity calculation parameter.

[0077] FIG. 15 is a diagram for explaining the learning process of the similarity calculation parameter according to Embodiment 5. The black circles in the figure are the plotted feature amounts of the skeleton information determined to have detected an event on a predetermined space. The white circles in the figure are the plotted feature amounts of the skeleton information not determined to have detected an event on a predetermined space. The feature amounts are calculated from the skeleton information using the similarity calculation parameter. As shown in FIG. 15, before parameter learning, black circles and white circles are interspersed. The learning unit 211 learns the similarity calculation parameter so that the distance between the black circles and the white circles increases and the boundary between the clusters becomes clear. Thereby, the similarity calculation accuracy and the determination accuracy of event detection can be improved during operation.

[0078] In the above-described embodiment, it has been described as a hardware configuration, but it is not limited thereto. The present disclosure can also be realized by causing a processor to execute a computer program for any process.

[0079] In the above example, when the program is loaded into a computer, it includes a group of instructions (or software code) for causing the computer to perform one or more functions described in the embodiment. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, a computer-readable medium or a tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted on a temporary computer-readable medium or a communication medium. By way of example and not limitation, a temporary computer-readable medium or a communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0080] Note that the present disclosure is not limited to the above-described embodiments, and can be appropriately modified without departing from the gist thereof. For example, it is also possible to combine Embodiment 3 or 4 with Embodiment 5.

[0081] Some or all of the above-described embodiments can be described as follows in the appended claims, but are not limited thereto. (Appended Claim 1) Calculation means for calculating a similarity between at least a part of skeleton information extracted from a captured image of a user who has visited an ATM, and at least a part of registered skeleton information extracted from a registration image showing a calling operation of a person and registered in an operation database; Determination means for determining that an event related to the ATM has been detected when the similarity is equal to or greater than a predetermined threshold value An event detection system comprising the above. (Appended Claim 2) Image acquisition means for acquiring the captured image; Extraction means for extracting at least a part of the skeleton information of at least a part of the user's body based on the captured image The event detection system according to Appended Claim 1, further comprising the above. (Appended Claim 3) The image acquisition means acquires the captured image in response to detecting a predetermined operation signal to the ATM, or The extraction means starts extraction of the skeleton information in response to detecting the predetermined operation signal to the ATM The event detection system according to Appended Claim 2. (Appended Claim 4) The registration image shows an operation in which the person makes a call and performs an input operation on the ATM The event detection system according to any one of Appended Claims 1 to 3. (Appended Claim 5) The determination means determines that the event has been detected when the similarity is equal to or greater than a predetermined threshold value and the user is determined to have a predetermined attribute The event detection system according to any one of Supplementary Notes 1 to 4. (Supplementary Note 6) The determination means acquires the attribute of the user based on the information read by the ATM. The event detection system according to Supplementary Note 5. (Supplementary Note 7) When the similarity is equal to or greater than a predetermined threshold value and a predetermined operation signal to the ATM is detected, the determination means determines that the event has been detected. The event detection system according to any one of Supplementary Notes 1 to 6. (Supplementary Note 8) The system further includes a learning means for learning a similarity calculation parameter using the skeleton information and the determination result by the determination means. The calculation means calculates the similarity using the similarity calculation parameter. The event detection system according to any one of Supplementary Notes 1 to 7. (Supplementary Note 9) The system includes a registration means for registering the skeleton information extracted from the registration image as registration skeleton information in the operation database. The event detection system according to any one of Supplementary Notes 1 to 8. (Supplementary Note 10) When it is determined that the user has a predetermined attribute, the registration means inquires of the management device whether to register the skeleton information extracted from the captured image in the operation database. The event detection system according to Supplementary Note 9. (Supplementary Note 11) An ATM and An event detection device for detecting an event related to the ATM and includes The event detection device Calculation means for calculating the similarity between at least a part of the skeleton information extracted from a captured image of a user who visited the ATM and at least a part of the registered skeleton information extracted from a registration image showing a person's call operation and registered in the operation database. Determination means for determining that the event has been detected when the similarity is equal to or greater than a predetermined threshold value, and having a monitoring system. (Appendix 12) Calculate the similarity between at least part of the skeletal information, which is the skeletal information extracted from the captured image of the user who visited the ATM, and at least part of the registered skeletal information extracted from the registered image showing the call operation of the person and registered in the operation database. When the similarity is equal to or greater than a predetermined threshold value, it is determined that an event related to the ATM has been detected. Event detection method. (Appendix 13) Calculation processing for calculating the similarity between at least part of the skeletal information, which is the skeletal information extracted from the captured image of the user who visited the ATM, and at least part of the registered skeletal information extracted from the registered image showing the call operation of the person and registered in the operation database, and Determination processing for determining that an event related to the ATM has been detected when the similarity is equal to or greater than a predetermined threshold value. A non-transitory computer-readable medium storing a program for causing a computer to execute the above.

Explanation of reference numerals

[0082] 10 Event detection system 1, 1a, 1b, 1c Monitoring system 16, 116, 206, 206c Calculation unit 17, 117, 207 Determination unit 100, 100a, 100b ATM 101 Communication unit 102, 102a, 102b Control unit 103 Input unit 104 Display unit 111, 201 Registration information acquisition unit 112, 202 Registration unit 113, 203 Operation DB 114, 204 Image acquisition unit 115, 205 Extraction unit 118,208 Output control unit 150 Camera 200, 200a, 200c Server 209 Skeleton information acquisition unit 210 Judgment history DB 211 Learning unit 300 Bank management device 500 Frame image 600, 601, 700 Display screen U User P Mobile phone R Registered skeleton information

Claims

1. Calculation means for executing a calculation process for calculating a similarity between at least a part of skeleton information extracted from a captured image of a user who has visited an ATM (Automatic Teller Machine) and at least a part of registered skeleton information extracted from a registration image showing a calling operation of a person and registered in an operation database; Determination means for executing a determination process for determining that an event related to the ATM has been detected based on the similarity; and The calculation means and the determination means Repeatedly execute an event detection process including the calculation process and the determination process at a plurality of timings, and further Start the event detection process in response to the first detection of a body area of a person different from the previous one from the captured image taken at the first timing, End without executing the event detection process in response to the first non-detection of the body area of the person from the captured image taken at the second timing after the first timing Event detection system.

2. Image acquisition means for acquiring the captured image; Extraction means for extracting at least a part of the skeleton information of at least a part of the user's body based on the captured image The event detection system according to claim 1, further comprising.

3. The image acquisition means acquires the captured image in response to the detection of a predetermined operation signal to the ATM, Or The extraction means starts the extraction of the skeleton information in response to the detection of the predetermined operation signal to the ATM The event detection system according to claim 2.

4. The registration image shows an operation in which the person makes a call and performs an input operation on the ATM The event detection system according to any one of claims 1 to 3.

5. The determination means determines that the event has been detected when the similarity is equal to or greater than a predetermined threshold value and the user is determined to have a predetermined attribute The event detection system according to any one of claims 1 to 4.

6. The determination means acquires the attribute of the user based on the information read by the ATM The event detection system according to claim 5.

7. The determination means determines that the event has been detected when the similarity is equal to or greater than a predetermined threshold value and a predetermined operation signal to the ATM is detected The event detection system according to any one of claims 1 to 6.

8. The apparatus further comprises a learning unit that learns a similarity calculation parameter using the skeleton information and the determination result by the determination unit. The calculation unit calculates the similarity using the similarity calculation parameter. The event detection system according to any one of claims 1 to 7.

9. A computer executes a calculation process for calculating a similarity between at least a part of skeleton information extracted from a captured image of a user who visited an ATM (Automatic Teller Machine) and at least a part of registered skeleton information extracted from a registered image indicating a calling operation of a person and registered in an operation database. executes a determination process for determining that an event related to the ATM is detected based on the similarity, and further repeatedly executes the event detection process including the calculation process and the determination process at a plurality of timings, and further starts the event detection process in response to the first detection of a body area of a person different from the previous one from the captured image taken at the first timing. ends without executing the event detection process in response to the first disappearance of the detection of the body area of the person from the captured image taken at a second timing after the first timing. Event detection method.

10. a calculation process for calculating a similarity between at least a part of skeleton information extracted from a captured image of a user who visited an ATM (Automatic Teller Machine) and at least a part of registered skeleton information extracted from a registered image indicating a calling operation of a person and registered in an operation database; a determination process for determining that an event related to the ATM is detected based on the similarity, and causes a computer to execute the processes; and further repeatedly causes the computer to execute the event detection process including the calculation process and the determination process at a plurality of timings, and further starts the event detection process in response to the first detection of a body area of a person different from the previous one from the captured image taken at the first timing. a process for causing the computer to end without executing the event detection process in response to the first disappearance of the detection of the body area of the person from the captured image taken at a second timing after the first timing. Program.

Images