Information processing device, information processing system, information processing method and program

Abstract

An information processing device (10) comprises: a phone conversation determination unit (14) which determines whether or not a user visiting an ATM is having a phone conversation, on the basis of first data which is any one of a captured image, audio data, and received radio waves data relating to a target area where the ATM serves as a reference; an action control unit (15) which, upon determination that the user is having a phone conversation, starts acquiring second data that indicates the state of the user and that is different in data class from the first data; and an incident determination control unit (16) which, by using the first data and the second data, acquires a determination result as to whether or not there is an incident.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and a non-transitory computer-readable medium, and more particularly to an information processing device, an information processing system, an information processing method, and a non-transitory computer-readable medium for detecting an event related to an ATM. [Background technology]

[0002] In recent years, fraudulent bank transfers, in which a person is instructed to transfer money from an ATM (Automatic Teller Machine) over the phone, have become a problem, and there is a need to prevent such cases or resolve them quickly. Therefore, a method has been proposed to detect people using mobile phones in front of an ATM and to warn them of the possibility of fraudulent bank transfers.

[0003] For example, Patent Document 1 discloses an automated trading system that, upon detecting a human body, starts taking pictures with a camera and collecting sound with a microphone, and alerts the user based on the captured images and sounds. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-079755 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the method described in the above-mentioned Patent Document 1 has the problem that even when a user is conducting a transaction that is not related to a crime, the user's personal information is collected using a camera and microphone, which places a heavy psychological burden on the user.

[0006] In view of the above-mentioned problems, the object of the present disclosure is to provide an information processing device, an information processing system, an information processing method, and a non-transitory computer-readable medium that contribute to preventing incidents or quickly resolving them while minimizing the increase in psychological burden on users. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present disclosure includes: a call determination means for determining whether a user visiting the ATM is making a call based on first data, which is any one of a photographed image, voice data, and received radio wave data, relating to a target area based on the ATM; an operation control means for starting to acquire second data indicating the state of the user, the second data having a data type different from that of the first data, when it is determined that the user is making a call; an incident determination control means for obtaining a determination result as to whether or not a crime has occurred using the first data and the second data; Equipped with.

[0008] An information processing system according to one aspect of the present disclosure includes: ATMs and Information processing device Equipped with The information processing device includes: a call determination means for determining whether a user visiting the ATM is making a call based on first data, which is any one of a photographed image, voice data, and received radio wave data, related to a target area based on the ATM; an operation control means for starting to acquire second data indicating the state of the user, the second data having a data type different from that of the first data, when it is determined that the user is making a call; an incident determination control means for obtaining a determination result as to whether or not a crime has occurred using the first data and the second data; It has.

[0009] An information processing method according to one aspect of the present disclosure includes: determining whether a user visiting the ATM is making a call based on first data, which is any one of a photographed image, voice data, and received radio wave data, relating to a target area based on the ATM; When it is determined that the user is making a call, the acquisition of second data indicating the state of the user, the second data having a data type different from the first data, is started; The first data and the second data are used to obtain a determination result as to whether or not a crime has occurred.

[0010] According to one aspect of the present disclosure, there is provided a non-transitory computer-readable medium, comprising: a step of determining whether a user visiting the ATM is making a call based on first data, which is any one of a photographed image, voice data, and received radio wave data, relating to a target area based on the ATM; When it is determined that the user is making a call, starting to acquire second data indicating the state of the user, the second data having a data type different from the first data; a step of obtaining a determination result of whether or not there is a crime using the first data and the second data; A program for causing a computer to execute the above is stored. [Effects of the Invention]

[0011] The present disclosure makes it possible to provide an information processing device, an information processing system, an information processing method, and a non-transitory computer-readable medium that collect personal information of users as needed, thereby contributing to preventing incidents or quickly resolving them while minimizing the increase in psychological burden on users. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration of an information processing device according to a first embodiment. [Figure 2] 1 is a flowchart showing the flow of an information processing method according to the first embodiment. [Figure 3] FIG. 10 is a diagram for explaining a call operation according to the second embodiment. [Figure 4] FIG. 10 is a block diagram showing the overall configuration of an information processing system according to a second embodiment. [Figure 5] FIG. 10 is a diagram showing skeletal information extracted from a frame image according to the second embodiment. [Figure 6] 10 is a flowchart showing the flow of a method for transmitting video data by an ATM according to a second embodiment. [Figure 7] 10 is a flowchart showing the flow of a registration method in a conversation detection DB by a server according to the second embodiment. [Figure 8] FIG. 10 is a sequence diagram showing an example of the flow of incident determination processing according to the second embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a display on a display unit of an ATM according to a second embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a display on a display unit of an ATM according to a second embodiment. [Figure 11] 10 is a flowchart showing the flow of a call determination process by a server according to the second embodiment. [Figure 12] FIG. 10 is a block diagram showing the overall configuration of an information processing system according to a third embodiment. [Figure 13] FIG. 11 is a diagram showing an example of a data structure of an operation control DB according to the third embodiment. [Figure 14] FIG. 11 is a sequence diagram showing an example of the flow of incident determination processing according to the third embodiment. [Figure 15] FIG. 11 is a diagram showing an example of a display on a display unit of an ATM according to a third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing an example of the flow of incident determination processing according to a first modified example of the third embodiment. [Figure 17] FIG. 10 is a sequence diagram showing an example of the flow of incident determination processing according to a second modified example of the third embodiment. [Figure 18] FIG. 10 is a block diagram showing the configuration of an information processing device according to a fourth embodiment. [Figure 19] 10 is a flowchart showing the flow of an information processing method according to a fourth embodiment. [Figure 20] FIG. 10 is a block diagram showing the overall configuration of an information processing system according to a fifth embodiment. [Figure 21] FIG. 13 is a diagram showing an example of a data structure of an operation control DB according to the fifth embodiment. [Figure 22] FIG. 10 is a sequence diagram showing an example of the flow of incident determination processing according to the fifth embodiment. [Figure 23] FIG. 10 is a diagram showing a decision tree for a primary determination of whether or not a crime has occurred according to a first modified example of the fifth embodiment. [Figure 24] FIG. 13 is a sequence diagram showing an example of the flow of incident determination processing according to a second modified example of the fifth embodiment. [Figure 25] FIG. 20 is a diagram showing an example of a display on a display unit of a related party terminal according to the sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and for clarity of explanation, duplicate explanations will be omitted as necessary.

[0014] <Issues of the embodiment> Here, the problems of the embodiment will be explained again. In recent years, fraudulent bank transfers, in which perpetrators are instructed to transfer money from an ATM over the phone, have become a problem, and there is a need to prevent such cases from occurring or to resolve them quickly. Therefore, a method has been proposed to detect people using mobile phones in front of an ATM and to warn them of the possibility of fraudulent bank transfers.

[0015] However, if a warning is issued to all users who use mobile phones and are conducting transactions that are not related to crime, it will cause discomfort to the users. Therefore, there is a need to reduce the psychological burden on users by not issuing warnings or taking other measures when it is thought that there is no crime involved.

[0016] In order to determine whether or not a crime has occurred, it is necessary to collect detailed information about the individual user. However, even when a user is conducting a transaction that is not a crime, collecting personal information about the user through cameras and microphones poses a significant psychological burden for the user.

[0017] Furthermore, if it takes too long to determine whether or not a crime has occurred, the psychological burden on the user will increase.

[0018] The following embodiments have been made to solve any of these problems.

[0019] <Terminology> In the following, the following terms are defined as follows: "Visiting an ATM" may mean standing or sitting in a target area relative to the ATM. The target area is the area surrounding the ATM, and is the area where a user typically stands or sits when operating the ATM. The target area may be a predetermined area in front of the ATM. The "determination result of whether or not there is a criminal offense" is also called the criminal offense determination result, and is information that directly or indirectly indicates whether or not there is a criminal offense. Information that indirectly indicates whether or not there is a criminal offense may be information that indicates different content depending on whether or not there is a criminal offense. A "frame image" is an example of a captured image.

[0020] <Embodiment 1> Next, a first embodiment of the present disclosure will be described. The first embodiment can be explained as an outline of the second and third embodiments described below. The first embodiment collects personal information of the user as needed, thereby suppressing an increase in the psychological burden on the user.

[0021] FIG. 1 is a block diagram showing the configuration of an information processing device 10 according to a first embodiment. The information processing device 10 is a computer system that detects ATM-related events that may be criminal. Criminality refers to whether the event is related to a crime or the possibility that it is related to a crime. For example, crime may refer to the possibility that a user visiting an ATM related to the event is involved in a crime. Alternatively, crime may refer to the possibility that a user visiting an ATM related to the event is committing a crime. A crime refers to an illegal and culpable act that meets the elements of a crime stipulated in the Penal Code or other penal laws. The crime may be fraud, particularly a bank transfer fraud. The information processing device 10 includes a call determination unit 14, an operation control unit 15, and an incident determination control unit 16.

[0022] The call determination unit 14 is also called a call determination means. The call determination unit 14 determines whether a user visiting an ATM is making a call based on first data related to the target area. The first data is data that directly or indirectly suggests the user's state or the situation in which the user is placed. Hereinafter, at least one of the user's state and the situation in which the user is placed will be simply referred to as a "state." The first data is data that at least suggests the state in which a call is being made using a mobile phone or the like, and is any one of captured images, audio data, and received radio wave data.

[0023] The captured image of the target area is a captured image generated by a camera installed to capture the target area. When a user visits an ATM, the call determination unit 14 detects an image area of ​​at least a part of the user's body from the captured image. In particular, when the user is talking on a mobile phone, the call determination unit 14 detects the user's talking behavior from the captured image through image analysis, thereby determining whether the user is talking on the mobile phone. The voice data relating to the target area is voice data generated by a microphone installed to pick up the voice of the target area. When a user visits an ATM, the call determination unit 14 detects the user's voice or the voice of the other party from the voice data. In particular, when the user is talking on a mobile phone, the call determination unit 14 identifies the phrases uttered by the user and the content of the conversation through voice recognition from the voice data, and determines whether the user is talking on a mobile phone. The received radio wave data for the target area is data of radio waves received by an antenna configured to receive radio waves for the target area. The call determination unit 14 determines whether the user is making a call using the mobile phone based on the duration of reception of radio waves in the mobile phone frequency band.

[0024] The operation control unit 15 is also called an operation control means. When the operation control unit 15 determines that the user is talking, it starts acquiring second data. The second data is data that directly or indirectly suggests the user's state. The second data is a different data type from the first data. Examples of data that indirectly suggests the user's state are images that show the user's appearance or behavior, or audio data that shows what the user is saying. In other words, the operation control unit 15 further collects basic data to grasp the user's state in detail. Note that the operation control unit 15 does not acquire the second data when the user is not talking. For example, if the first data is a captured image of the target area, the second data may be audio data of the target area. Then, when the operation control unit 15 determines that the user is talking based on the captured image, the operation control unit 15 may control activation of the microphone or start receiving the audio data. For example, if the first data is audio data relating to the target area, the second data may be a captured image relating to the target area. Then, when the operation control unit 15 determines that the user is talking based on the audio data, the operation control unit 15 may control the activation of the camera or start receiving the captured image. If the first data is received radio wave data, the second data may be one or both of a captured image of the target area and audio data of the target area. When the operation control unit 15 determines that the user is talking based on the received radio wave data, the operation control unit 15 may control activation of at least one of the camera and microphone, or start receiving at least one of the captured image and audio data.

[0025] The incident determination control unit 16 is also called an incident determination control means. The incident determination control unit 16 obtains a determination result of whether or not a crime has occurred (incident determination result) using the first data and the second data. For example, the incident determination control unit 16 may grasp the user's condition in more detail based on the first data and the second data, and determine whether or not a crime has occurred based on the user's condition. Furthermore, for example, the incident determination control unit 16 may transmit the first data and the second data to a related party terminal used by a related party such as an attendant or security guard, and the related party may determine whether or not a crime has occurred. The incident determination control unit 16 may then receive the incident determination result from the related party terminal.

[0026] 2 is a flowchart showing the flow of the information processing method according to the first embodiment. First, the call determination unit 14 of the information processing device 10 determines whether a call is made by a user who has visited an ATM based on first data relating to a target area based on the ATM (S10). If the user is not making a call (No in S11), the call determination unit 14 repeats the process shown in S10. On the other hand, if the user is making a call (Yes in S11), the operation control unit 15 starts acquiring second data relating to the target area (S12). Then, the incident determination control unit 16 acquires an incident determination result using the first data and the second data (S13).

[0027] As described above, according to the first embodiment, the information processing device 10 collects additional personal information suggesting the user's condition when the user visiting the ATM is on a call, and obtains an incident determination result based on the collected information. This contributes to preventing incidents from occurring or quickly resolving them. When the user is not on a call, no additional personal information of the user is collected, which reduces the user's psychological burden. Furthermore, by having the information processing device 10 determine whether or not an incident has occurred, the information processing device 10 or relevant parties can respond accordingly. Therefore, the user can avoid the unpleasant situation of receiving a warning when performing a transaction that does not involve an incident. This also reduces the user's psychological burden.

[0028] <Embodiment 2> Next, a second embodiment of the present disclosure will be described. FIG. 3 is a diagram for explaining a call operation according to the second embodiment. The ATM system 100 is a computer system including an ATM 110 and its peripheral devices. Specifically, the ATM system 100 includes the ATM 110, a camera 150, and a microphone 160. A user U visits the ATM 110 to use the camera 150 and stands in front of the ATM 110. At this time, the user U is located within a target area in front of the ATM 110. The user U then operates the operation screen of the ATM 110 to issue instructions for withdrawing, depositing, transferring, etc. cash.

[0029] The camera 150 is disposed at a position and angle that allows it to capture an image of at least a part of the body of a user U who visits the ATM 110. For example, the camera 150 is disposed on top of the ATM 110. In the second embodiment, the camera 150 is configured to capture an image of the upper body of the user U, but instead, the camera 150 may be configured to capture an image of only the face area. Here, when a user U visiting the ATM 110 is talking on a phone using a mobile phone P, the camera 150 captures an image of the user U during the call. This image data is an example of the first data described above. This makes it possible to determine whether the user U visiting the ATM 110 is talking on a phone.

[0030] Furthermore, a microphone 160 is disposed at a predetermined position relative to the ATM 110. In this figure, the microphone 160 is disposed on top of the ATM 110. The microphone 160 picks up the voice of a user U who visits the ATM 110 and generates voice data. Note that the installation position of the microphone 160 is not limited to this, and the microphone 160 may be disposed anywhere as long as it can pick up voice in the target area. Here, when a user U visiting the ATM 110 is talking on the mobile phone P, the microphone 160 is activated. The microphone 160 then picks up the sound of the user U's conversation during the call. This sound data is an example of the second data described above. This allows the content of the conversation of the user information U during the call to be understood.

[0031] That is, in the second embodiment, the presence or absence of a crime is determined based on at least the video data and the audio data.

[0032] 4 is a block diagram showing the overall configuration of an information processing system 1 according to embodiment 2. The information processing system 1 is a computer system that determines whether a user U who visits an ATM 110 is involved in an incident and takes measures to prevent the incident from occurring or to quickly resolve it. In addition to the ATM system 100 described above, the information processing system 1 includes a server 200 and a related party terminal 300.

[0033] (ATM system 100) As described above, the ATM system 100 includes the ATM 110, the camera 150, and the microphone 160. The camera 150 and the microphone 160 are connected to the ATM 110.

[0034] The ATM 110 includes a communication unit 111 , a control unit 112 , an input unit 113 , and a display unit 114 .

[0035] The communication unit 111 is a communication interface with the network N. The input unit 113 is an input device that accepts input. The display unit 114 is an example of an output unit and is a display device. The input unit 113 and the display unit 114 may be configured integrally, such as a touch panel.

[0036] The control unit 112 controls the hardware of the ATM 110. The control unit 112 executes normal operations of the ATM 110 (processes such as cash withdrawal, deposit, and transfer) based on input operations from the user U received by the input unit 113. In the second embodiment, the operation mode of the ATM 110 that executes the above-mentioned normal operations may be referred to as a "normal mode."

[0037] Furthermore, the control unit 112 acquires video data from the camera 150 via the communication unit 111. Then, the control unit 112 transmits the video data to the server 200 via the network N at a predetermined timing. For example, the control unit 112 starts transmitting video data to the server 200 in response to the input unit 113 receiving a predetermined first operation. The transmission of video data may be a transmission of a series of video data including a plurality of frame images, or may be a transmission of frame images on a frame-by-frame basis. The first operation may be an operation by the user U (such as activating a screen) to start various services using the ATM 110 (cash withdrawal, deposit, or transfer), or an operation by the user U to start a transfer service (such as selecting "transfer"). The first operation may also be a specific operation to receive various services. As an example, the first operation may be an operation of inserting a cash card or a bankbook, or an operation of entering a PIN number.

[0038] Furthermore, for example, the control unit 112 terminates the transmission of video data to the server 200 in response to the input unit 113 accepting a predetermined second operation. The second operation is an operation different from the first operation. The second operation may be an operation for terminating various services at the ATM 110 (such as selecting "end"), or may be another operation by the user U for terminating the transfer service (such as selecting "start next transaction"). Note that the trigger for terminating the transmission of video data to the server 200 may be, instead of accepting the second operation, the fact that no operation has been accepted for a predetermined period of time, or the fact that the ATM 110 has executed a process to return the cash card or passbook.

[0039] Furthermore, when the control unit 112 receives a control signal for activating the microphone 160 from the server 200, the control unit 112 activates the microphone 160. Then, the control unit 112 acquires audio data from the microphone 160 via the communication unit 111. Then, the control unit 112 transmits the audio data to the server 200 via the network N.

[0040] Furthermore, when the control unit 112 receives a mode switching control signal from the server 200 via the communication unit 111, it switches the operation mode of the ATM 110. Then, the control unit 112 performs display or audio output according to the operation mode. For example, when the control unit 112 receives a control signal to switch from the normal mode to the warning mode, it causes the display unit 114 to display warning information corresponding to the warning mode. At this time, the control unit 112 may cause an audio output unit (not shown) to output the warning information as audio.

[0041] (Server 200) The server 200 is a computer device that is an example of the information processing device 10 described above. The server 200 detects the user U's call behavior based on the video data received from the ATM 110, and has the relevant parties determine whether or not a crime has occurred based on the audio data collected in conjunction with the detection of the call behavior. If the relevant parties determine that a crime has occurred, the server 200 causes the ATM 110 to switch its operation mode.

[0042] The server 200 includes a registration unit 201 , a call detection DB 202 , an information acquisition unit 203 , a call determination unit 204 , an operation control unit 205 , and an incident determination control unit 206 .

[0043] The registration unit 201 is also called a registration means. The registration unit 201 acquires a registration image showing at least a person's call action in response to a registration request from a device (not shown) used by the administrator of the ATM 110 or the person who manages bank information, or in response to an operation by the administrator of the server 200. The registration image may be an image of a person simply making a call, or may be an image showing a person making a call and performing an input operation on the ATM. The registration image may be a still image (one frame image) or a video including a series of multiple frame images. The registration unit 201 then supplies the registration image to the call determination unit 204, which will be described later, and acquires skeleton information extracted from the registration image from the call determination unit 204 as registration skeleton information R. The registration unit 201 then registers the acquired registration skeleton information R in the call detection DB 202 as a call action.

[0044] The call detection DB 202 is a storage device that stores one or more pieces of registration skeleton information R corresponding to a call action.

[0045] The information acquisition unit 203 is also called information acquisition means. The information acquisition unit 203 acquires video data received from the ATM 110. As a result, the information acquisition unit 203 acquires frame images included in the video data. That is, the information acquisition unit 203 acquires frame images in response to the ATM 110 detecting a first operation signal related to a first operation. Here, the first operation signal is a signal indicating that the input unit 113 has accepted a first operation from the user U to the ATM 110. The information acquisition unit 203 supplies the acquired frame images to the call determination unit 204. The information acquisition unit 203 also acquires voice data received from the ATM 110. The information acquisition unit 203 supplies the acquired voice data to the incident determination control unit 206. Furthermore, when the ATM 110 receives another operation from the user U, the information acquisition unit 203 receives a signal indicating the operation content and notifies the incident determination control unit 206 of the operation content.

[0046] The call determination unit 204 is an example of the call determination unit 14 described above. The call determination unit 204 determines whether or not the user U is making a call based on frame images included in the video data. For example, the call determination unit 204 extracts skeletal information from the frame images and compares the extracted skeletal information with registered skeletal information R registered in the call detection DB 202 to determine whether or not a call is being made. The skeletal information is information made up of "key points," which are characteristic points such as joints, and "bones (bone links)," which indicate links between the key points. The call determination unit 204 then supplies the call determination result to the operation control unit 205.

[0047] The operation control unit 205 is an example of the above-mentioned operation control unit 15. When it is determined that the user U is talking, the operation control unit 205 transmits a control signal to activate the microphone 160 to the ATM 110 via the network N. The control signal is also called a microphone-on control signal. When it is determined that the user is talking, the information acquisition unit 203 starts acquiring, via the information acquisition unit 203, voice data indicating the status of the user U's call, particularly the content of the call. The voice data is then supplied to the incident determination control unit 206 via the information acquisition unit 203.

[0048] The incident determination control unit 206 is an example of the above-mentioned incident determination control unit 16. The incident determination control unit 206 acquires an incident determination result using frame images of video data, which is the first data, and audio data, which is the second data. Specifically, in the second embodiment, the incident determination control unit 206 transmits frame images and audio data to the involved party terminal 300 as data suggesting the user's state, thereby prompting the involved party to determine whether or not a crime has occurred. Hereinafter, the data suggesting the user's state transmitted to the involved party terminal 300 is referred to as status-related data. The frame images included in the status-related data may include frame images from a predetermined period before the microphone-on control, or may include frame images from a predetermined period after the microphone-on control. Next, the incident determination control unit 206 receives a result of the crime determination from the involved party terminal 300. The result of the crime determination may be information indicating whether or not a crime has occurred that the involved party inputs into the involved party terminal 300, or it may be a notification sent from the involved party terminal 300 only if a crime has occurred. In the latter case, the incident determination control unit 206 receives a notification indicating a crime from the involved party terminal 300 when a crime has been determined to occur, but does not receive the notification when a crime has been determined to occur. Therefore, the incident determination control unit 206 can ascertain the result of the crime determination based on the presence or absence of a notification.

[0049] Furthermore, the incident determination control unit 206 receives the result of the incident determination from the related party terminal 300, and if it is determined that an incident has occurred, performs control to switch the operation mode of the ATM 110 to a predetermined operation mode. Specifically, the incident determination control unit 206 transmits a mode switching control signal to the ATM 110 to switch to the operation mode. The above operation mode may be called a third operation mode. The third operation mode is an operation mode for performing an operation different from normal operation. As a result, the ATM 110 switches its operation mode to the third operation mode. The incident determination control unit 206 may also supply the result of the incident determination to the operation control unit 205, which then transmits the mode switching control signal.

[0050] Here, the operation in the third operation mode includes at least one of an operation to display warning information, an operation to cancel or suspend the transfer, and an operation to secure time. The operation of suspending the transfer may be to clearly indicate on the display screen that the transfer has been suspended, or to display on the display screen that the transfer has been completed while the internal processing suspends the transfer. In the latter case, the transfer may appear to be completed at first glance, but this is because there is a possibility that user U may have been confused during the transfer operation, and even if a related party or bank staff points out an incident, this may cause further confusion. After ATM 110 suspends the transfer, if a related party or bank staff contacts the registered family and confirms that there is no incident, ATM 110 may complete the transfer. The operation to save time may be an operation to secure time until the relevant parties or the police arrive at ATM 110, or an operation to secure time for user U to calmly confirm whether or not there is any transfer fraud. Specifically, the operation to save time may be to interrupt the normal transfer process and cause ATM 110 to output information other than that for the normal transfer process.

[0051] Alternatively or additionally, the operation of the third operation mode may include an operation to notify the police or an operation to transmit radio waves that jam radio waves from mobile phones in the target area. The operation of transmitting jamming radio waves may be an operation in which the ATM 110 activates a transmitter that transmits jamming radio waves in the target area.

[0052] (Stakeholder terminal 300) The involved party terminal 300 is an information terminal used by the involved party. The involved party terminal 300 receives frame images and audio data from the incident determination control unit 206 of the server 200, and prompts the involved party to determine whether or not the incident is a criminal offense. The involved party terminal 300 then returns information according to the determination result, i.e., the criminal offense determination result, to the incident determination control unit 206.

[0053] The related party terminal 300 includes a communication unit 301 , a control unit 302 , an input unit 303 , and an output unit 304 .

[0054] The communication unit 301 is a communication interface with the network N. The input unit 303 is an input device that accepts input. The output unit 304 may include a display unit that is a display device and an audio output unit that is a speaker. The input unit 303 and the display unit may be integrally configured, such as a touch panel. The control unit 302 controls the hardware of the involved party terminal 300. For example, when the control unit 302 receives a frame image and audio data from the incident determination control unit 206 of the server 200, the control unit 302 causes the frame image to be displayed on the display unit of the output unit 304 and causes the audio data to be output to the audio output unit of the output unit 304. When the control unit 302 receives an input of a criminal incident determination result from a involved party via the input unit 303, the control unit 302 transmits the criminal incident determination result to the server 200 via the communication unit 301.

[0055] Fig. 5 is a diagram showing skeletal information extracted from a frame image according to the second embodiment. A frame image 500 includes an image area of ​​the upper body of a user U who is talking on a mobile phone P. The skeletal information shown in Fig. 5 also includes a plurality of key points and a plurality of bones detected from the upper body. As an example, Fig. 5 shows the following key points: a right eye A11, a left eye A12, a head A2, a neck A3, a right shoulder A41, a left shoulder A42, a right elbow A51, a left elbow A52, a right hand A61, a left hand A62, and a waist A7.

[0056] The call determination unit 204 of the server 200 compares such skeletal information with the registration skeletal information R corresponding to the upper body and determines whether they are similar, thereby detecting the call behavior. It is important to determine whether the hands are positioned near the head when detecting the call behavior. Therefore, the call determination unit 204 may calculate the similarity by weighting the positional relationship between the right hand A61 and the right eye A11 or the head A2, and the positional relationship between the left hand A62 and the left eye A12 or the head A2. Alternatively, the call determination unit 204 may use only the skeletal information related to the right eye A11, the left eye A12, the head A2, the right hand A61, and the left hand A62 from the extracted skeletal information for calculating the similarity. In addition to a form of making a call while holding the mobile phone P in one's hand, there is also a form of making a call while supporting the mobile phone P on one's shoulder and bringing the mobile phone P close to one's ear. In this case, the positional relationship between the right shoulder A41 and the right eye A11 or the head A2, and the positional relationship between the left shoulder A42 and the left eye A12 or the head A2 may be added to the weighting targets. Alternatively, the server 200 may add skeletal information related to the right shoulder A41 and the left shoulder A42 to the skeletal information related to the right eye A11, the left eye A12, the head A2, the right hand A61, and the left hand A62 as skeletal information used in similarity calculation.

[0057] 6 is a flowchart showing the flow of a method for transmitting video data by the ATM 110 according to the second embodiment. First, the control unit 112 of the ATM 110 determines whether or not a first operation signal related to a first operation of the ATM 110 has been detected (S20). If the control unit 112 determines that the first operation signal has been detected (Yes in S20), it starts transmitting video data acquired from the camera 150 to the server 200 (S21). On the other hand, if the control unit 112 does not determine that the first operation signal has been detected (No in S20), it repeats the process shown in S20. Next, the control unit 112 determines whether or not a second operation signal related to a second operation of the ATM 110 has been detected (S22). If the control unit 112 determines that the second operation signal has been detected (Yes in S22), it ends the transmission of the video data acquired from the camera 150 to the server 200 (S23). On the other hand, if the control unit 112 does not determine that the second operation signal has been detected (No in S22), it repeats the process shown in S22.

[0058] In the above flowchart, triggers for starting and ending transmission of video data are operation signals from the ATM 110, but this is not limiting. For example, the control unit 112 may detect triggers for starting and ending transmission of video data by analyzing video data acquired from the camera 150. For example, the control unit 112 may start transmitting video data when a body region of a person different from that detected last time is detected for the first time from the video data. Furthermore, the control unit 112 may end transmitting video data when a body region of a person different from that detected last time is detected for the first time from the video data and then the body region of that person disappears.

[0059] In this way, by limiting the transmission period of video data to the period between a predetermined start trigger and end trigger, the amount of communication data can be minimized. Also, outside of this period, the call determination process in the server 200 can be omitted, thereby saving computational resources.

[0060] FIG. 7 is a flowchart showing the flow of a registration method for the call detection DB 202 by the server 200 according to the second embodiment. First, the registration unit 201 of the server 200 receives a registration request including a registration image from a device (not shown) that manages bank information (S30). Next, the registration unit 201 supplies the registration image to the call determination unit 204. After acquiring the registration image, the call determination unit 204 extracts a body image from the registration image (S31). Next, the call determination unit 204 extracts registration skeletal information R from the body image (S32). At this time, the call determination unit 204 may use all of the skeletal information extracted from the body image as the registration skeletal information R, or may use only part of the skeletal information (for example, skeletal information related to the right eye, left eye, head, right hand, and left hand) as the registration skeletal information R. Next, the registration unit 201 acquires the registration skeletal information R from the call determination unit 204 and registers the registration skeletal information R in the call detection DB 202 (S33).

[0061] 8 is a sequence diagram showing an example of the flow of the incident determination process according to the second embodiment. First, the camera 150 of the ATM system 100 captures an image of the target area (S500). Then, when the ATM 110 determines that it has detected a first operation signal, which is a trigger for starting transmission of video data, it starts transmitting the video data to the server 200 (S501). As a result, the information acquisition unit 203 of the server 200 starts acquiring the video data.

[0062] Next, the call determination unit 204 of the server 200 executes a call determination process (S502). Here, it is assumed that the call determination unit 204 determines that the user U is talking based on the frame images of the video data. In this case, the operation control unit 205 of the server 200 transmits a microphone-on control signal to the ATM 110 (S503). As a result, the ATM 110 activates the microphone 160 and starts acquiring voice data from the microphone 160 (S504). Then, the ATM 110 starts transmitting the voice data to the server 200 (S505). As a result, the information acquisition unit 203 of the server 200 starts acquiring the voice data.

[0063] The server 200 continues to acquire the video data and audio data until the transmission of the video data is completed by the ATM 110. In other words, when the ATM 110 detects a trigger for completing the transmission of the video data, it may also complete the transmission of the audio data in addition to the transmission of the video data.

[0064] Next, the incident determination control unit 206 of the server 200 transmits the status-related data acquired from the ATM system 100 to the related party terminal 300 (S506). In this example, the status-related data is video data and audio data acquired from the ATM system 100.

[0065] Next, the control unit 302 of the involved party terminal 300 outputs the status-related data to the output unit 304 (S507). Specifically, the control unit 302 causes the display unit to display the video data and the audio output unit to output the audio data. This allows the involved party to determine whether or not there is a crime. Next, the input unit 303 of the involved party terminal 300 accepts the determination operation by the involved party (S508). Here, it is assumed that the input unit 303 of the involved party terminal 300 accepts input of the determination result that there is a crime from the involved party. In this case, the control unit 302 of the involved party terminal 300 notifies the server 200 that there is a crime as the crime determination result (S509).

[0066] Upon receiving the notification of the possibility of an incident, the incident determination control unit 206 of the server 200 transmits a mode switching control signal to the ATM 110 of the ATM system 100 to switch to a warning mode (S510). The warning mode is an example of a third operation mode, and is an operation mode in which warning information is displayed. As a result, the control unit 112 of the ATM 110 switches the operation mode from the normal mode to the warning mode, and displays predetermined warning information on the display unit 114 (S511). In addition, in S510, the incident determination control unit 206 may transmit warning information to the ATM 110 in addition to or instead of the mode switching control signal. In this case, too, in S511, the control unit 112 of the ATM 110 causes the display unit 114 to display predetermined warning information.

[0067] FIG. 9 is a diagram showing an example of a display on the display unit 114 of the ATM 110 according to the second embodiment. For example, the display unit 114 may display a message saying, "Beware of bank transfer fraud!". Note that instead of the above message, the display unit 114 may display a message such as, "There has been an increase in bank transfer fraud recently." The display unit 114 may also display an operation area labeled "Continue" for continuing the normal operation of the ATM 110 and an operation area labeled "Exit" for terminating the normal operation of the ATM 110. If the user U views the message and confirms that it is not a bank transfer fraud, the user U selects the operation area labeled "Continue." On the other hand, if the user U determines that there is a possibility of bank transfer fraud, the user U selects the operation area labeled "Exit."

[0068] Such a display allows the user to recognize that he or she has been involved in a fraudulent bank transfer, and makes it possible to prevent such a fraud from occurring.

[0069] FIG. 10 is a diagram showing an example of a display on the display unit 114 of the ATM 110 according to the second embodiment. For example, the display unit 114 displays a message saying, "Please wait until an attendant arrives." At this time, the ATM 110 may suspend normal operation for a predetermined period of time. By providing the user U with a waiting time in this way, the user U can calmly check whether or not there is any remittance fraud. Furthermore, by ensuring a waiting time, an attendant can take action, such as rushing to the ATM 110. This makes it possible to prevent remittance fraud before it occurs.

[0070] For example, the display unit 114 may display an icon I indicating the remaining waiting time. As an example, the icon I indicates the remaining waiting time by the length of a diagonally shaded area on the circumference. The diagonally shaded area may change so that the angle of the circumference becomes smaller as time passes, that is, the length of the diagonally shaded area becomes shorter. Furthermore, the icon I may indicate the elapsed time by the length of a dotted area other than the diagonally shaded area on the circumference. By clearly displaying the remaining waiting time in this way, it is possible to reduce the frustration of the user U.

[0071] 11 is a flowchart showing the flow of the call determination process (the process shown in S502 of FIG. 8) by the server 200 according to the second embodiment. First, when a frame image included in video data is acquired, the call determination unit 204 detects an image area of ​​a person's body (body area) from the frame image and extracts it as a body image (S41). Extracting may mean cutting out.

[0072] Next, the call determination unit 204 extracts skeletal information from the body image (S42). At this time, the call determination unit 204 may use a skeletal estimation technique using machine learning to extract skeletal information of at least a part of the person's body based on features such as the person's joints recognized in the body image. The call determination unit 204 may use a skeletal estimation technique such as OpenPose, for example.

[0073] Next, the call determination unit 204 calculates the similarity between the extracted skeletal information and each piece of registration skeletal information R registered in the call detection DB 202 (S43). Instead of the above-described similarity, the calculation target may be the similarity between a portion of the extracted skeletal information and each piece of registration skeletal information R, or the similarity between the extracted skeletal information and a portion of each piece of registration skeletal information R. The call determination unit 204 may calculate the above-described similarity directly or indirectly using the skeletal information. For example, the call determination unit 204 may calculate the above-described similarity by converting at least a portion of the extracted skeletal information and at least a portion of each piece of registration skeletal information R registered in the call detection DB 202 into a different format and then calculating the similarity between the converted pieces of information. In this case, the above-described similarity may be the similarity between the converted pieces of information itself, or may be a value calculated using the similarity between the converted pieces of information. The conversion method may be to normalize the size of the skeletal information, to convert it into features using the angles between each bone (i.e., the degree of bending of the joints), or to convert it into 3D posture information based on a pre-trained machine learning model.

[0074] Next, the call determination unit 204 determines whether or not there is registration skeleton information R whose similarity is equal to or greater than a predetermined threshold (S44). If there is registration skeleton information R whose similarity is equal to or greater than the predetermined threshold (Yes in S44), the call determination unit 204 determines that a call action has been detected (S45). The server 200 then proceeds to S503 in FIG. 8. On the other hand, if there is no registration skeleton information R whose similarity is equal to or greater than the predetermined threshold (No in S44), the call determination unit 204 does not determine that a call action has been detected and returns the process to S41.

[0075] Note that S44 to S45 may be specifically performed as follows. For example, first, the call determination unit 204 identifies the number of pieces of registration skeleton information R whose calculated similarity is equal to or greater than a predetermined threshold. Then, the call determination unit 204 determines whether or not a call behavior has been detected based on the number of corresponding pieces of registration skeleton information R. In the second embodiment, the call determination unit 204 determines that a call behavior has been detected if there is at least one piece of registration skeleton information R whose similarity is equal to or greater than the predetermined threshold, and does not determine that a call behavior has been detected if there is no piece of registration skeleton information R whose similarity is equal to or greater than the predetermined threshold. However, instead of this, the call determination unit 204 may determine that a call behavior has been detected if there is a predetermined number or more pieces of registration skeleton information R whose similarity is equal to or greater than the predetermined threshold, and may not determine that a call behavior has been detected if there is less than the predetermined number of pieces of registration skeleton information R whose similarity is equal to or greater than the predetermined threshold.

[0076] As described above, according to the second embodiment, when a user U visiting the ATM 110 is on a call, the server 200 additionally collects personal voice data of the user U that suggests the content of the call, and obtains a criminal incident determination result based on the collected information. This contributes to preventing the occurrence of a criminal incident or quickly resolving the incident. When the user U is not on a call, the server 200 does not collect additional personal voice data of the user U, thereby reducing the increase in the psychological burden on the user U. Furthermore, by the server 200 being able to determine whether or not a criminal incident has occurred, the server 200 or relevant parties can take appropriate action depending on whether or not a criminal incident has occurred. Therefore, the user U can avoid the unpleasant situation of receiving a warning when performing a transaction that is not a criminal incident. This also reduces the increase in the psychological burden on the user U.

[0077] The second embodiment can be modified as follows.

[0078] For example, in the above description, the determination of whether or not a crime has occurred is performed by the person involved, but it may also be performed by the server 200. As an example, the crime determination control unit 206 of the server 200 may estimate the content of the call based on the audio data. Voice recognition may be used to estimate the content of the call. The crime determination control unit 206 may then determine whether or not a crime has occurred based on frame images of the video data and the content of the call. As an example, if a call action is detected from the video data and the audio data contains specific words, phrases, or vocabulary, the crime determination control unit 206 may determine that a crime has occurred. In this case, the processes shown in S506 to S509 in FIG. 8 may be omitted. This eliminates the need to assign a person involved to determine whether or not a crime has occurred, thereby reducing personnel costs.

[0079] In the above description, the ATM 110 starts and ends transmission of video data from the camera 150 in response to start and end triggers for the transmission, but the ATM 110 may start and end transmission regardless of such triggers. For example, the ATM 110 may transmit video data from the camera 150 to the server 200 continuously or at predetermined time intervals throughout the day or during business hours.

[0080] In the above description, the operation control unit 205 activates the microphone 160 when it is determined that the user U is talking. However, instead, the microphone 160 may be in an activated state all day or during business hours. In this case, the operation control unit 205 may start receiving the voice data by requesting the ATM 110 to transmit the voice data.

[0081] In the above explanation, the result of the determination of whether or not a crime is committed is described as a determination result of whether or not a crime is committed that is input into the involved party terminal 300, or a notification sent from the involved party terminal 300 if a crime is committed. However, instead of or in addition to this, the result of the determination of whether or not a crime is committed may be an instruction to switch to a third operating mode corresponding to whether or not a crime is committed. Therefore, by receiving the above instruction, the incident determination control unit 206 of the server 200 can recognize that a determination result of whether or not a crime is committed has been made. In this case, the involved party does not need to input whether or not a crime is committed, and the appropriate operating mode can be switched depending on the specific situation inferred from the video data and audio data.

[0082] Furthermore, in S506 of FIG. 8 described above, when the incident determination control unit 206 transmits status-related data to the related party terminal 300, it may also transmit user information about user U to the related party terminal 300. The user information is attribute information about user U. The user information may include, for example, account holder information read from a bankbook or cash card, or the account holder's name, gender, age, past transaction history, or other attribute information linked to the account number read from the bankbook or cash card. This facilitates determining whether or not the related party is involved in a crime. For example, if the related party can clearly recognize a different person by comparing gender and age information with video data, it can determine that there is a high possibility of criminal involvement. For example, if the related party can determine from the user information that the user has been a victim of fraud in the past, it can determine that there is a high possibility of criminal involvement. For example, if the related party can determine from the user information that the user is elderly, it can carefully determine that there is no criminal involvement. This is because elderly people tend to be the most common victims of fraud.

[0083] Also, in S506 of FIG. 8 described above, the incident determination control unit 206 of the server 200 transmits both video data and audio data to the involved party terminal 300 as status-related data. However, if the audio data is sufficient to understand the content of the call, the incident determination control unit 206 may omit the video data from the status-related data. Here, audio data contains private information, namely the content of the call, and therefore may be more confidential than video data. Therefore, even in such a case, highly confidential information is not collected when the call determination is negative, thereby achieving the effect of contributing to the prevention of incidents or their rapid resolution while minimizing the increase in the user's psychological burden.

[0084] <Embodiment 3> Next, a third embodiment of the present disclosure will be described. Fig. 12 is a block diagram showing the overall configuration of an information processing system 1a according to the third embodiment. The information processing system 1a according to the third embodiment is characterized in that the operation mode of the ATM 110 is automatically switched when a call operation is detected.

[0085] The information processing system 1a includes a server 200a instead of the server 200. The server 200a has an operation control unit 205a instead of the operation control unit 205, an incident determination control unit 206a instead of the incident determination control unit 206, and an operation control DB 207.

[0086] When it is determined that the user U is talking, the operation control unit 205a performs control to switch the operation mode of the ATM 110 to a second operation mode, instead of controlling to activate the microphone 160. Specifically, the operation control unit 205a transmits a mode switching control signal to the ATM 110 to switch to the second operation mode. The second operation mode is an operation mode in which an operation different from a normal operation is performed. The second operation mode may perform an operation different from the third operation mode described above.

[0087] Here, the operation in the second operation mode is performed for the purpose of allowing the user U to remain calm or to secure time until the relevant parties or the police arrive at the ATM 110. In addition to or instead of this, the operation in the second operation mode is also performed for the purpose of making it easier to determine whether or not a crime has occurred. Specifically, the operation in the second operation mode may include interrupting the transfer process performed in the normal mode and outputting information on the display unit 114 or the audio output unit of the ATM 110 that is at least partially different from the information output in the normal mode.

[0088] In the third embodiment, as an example, the operation in the second operation mode may include displaying question information related to a transfer on the display unit 114 and receiving an answer via the input unit 113. Specifically, when it is determined that the user U is making a call, the operation control unit 205a first reads at least one piece of question information Q stored in the operation control DB 207. Then, the operation control unit 205a transmits a mode switching control signal including the read question information Q to the ATM 110 via the network N. Upon receiving the mode switching control signal, the ATM 110 switches its operation mode to the second operation mode. Then, in the second operation mode, the ATM 110 displays the question information Q included in the mode switching control signal, receives an answer from the user U, and transmits answer information related to the answer to the server 200a. In this way, in the second operation mode, the operation control unit 205a causes the ATM 110 to output the question information Q previously stored in the operation control DB 207.

[0089] The operation control DB 207 is a storage device that stores one or more pieces of question information Q that are registered in advance.

[0090] The incident determination control unit 206a has basically the same functions as the incident determination control unit 206, but also uses a response to the question information to obtain a result of the incident determination. The response may be at least one of answer information to the question information and video data representing the user U's reaction when the question information is output (i.e., a predetermined period after switching to the second operation mode). The answer information is an example of data that directly suggests the user's state. When the response is answer information, the incident determination control unit 206a uses the answer information as status-related data in addition to the first data (frame images of video data in the second and third embodiments) to determine whether or not there is an incident. Adding the answer information to the basis for the incident determination can facilitate the incident determination. It can also improve the accuracy of the incident determination. When the response is video data representing the user U's reaction, the incident determination control unit 206a may include video data from a predetermined period before switching to the second operation mode in the status-related data, in addition to the video data from a predetermined period after switching to the second operation mode. However, the video data from the predetermined period before switching to the second operation mode may be omitted. Hereinafter, such a predetermined period based on switching to the second operation mode may be referred to as a target period.

[0091] Furthermore, based on the basis of determining whether the incident is a crime, the second data (audio data in the second embodiment) may be omitted.

[0092] 13 is a diagram showing an example of the data structure of the operation control DB 207 according to the third embodiment. Four pieces of question information Q1 to Q4 are stored in the operation control DB 207. Note that the number of pieces of question information Q is not limited to four.

[0093] Question Q1 is a question asking, "Have you ever transferred money to this account before?" Question Q2 is a question asking, "Have you transferred money after receiving a call from an unknown number?" Question Q3 is a question asking, "Are you doing this because you heard that you would receive money now?" Question Q4 is a question asking, "Have you checked or consulted with your family or acquaintances?"

[0094] The operation control unit 205a may select one piece of question information Q from the questions Q1 to Q4 in a predetermined order or randomly, and transmit the selected piece of question information Q to the ATM 110. Then, upon receiving the answer information, the operation control unit 205a may repeat the above selection and transmission a predetermined number of times, or until a predetermined time has elapsed since the transfer process was interrupted. Alternatively, the above selection and transmission do not have to be repeated. Alternatively, the operation control unit 205a may read out a predetermined number or all of the pieces of question information Q contained in the operation control DB 207, and transmit the read pieces of question information Q to the ATM 110. In this case, the ATM 110 may display the received pieces of question information Q in a predetermined order or randomly on the display unit 114. The ATM 110 may then transmit the received answer information to the server 200 in order, or may transmit the received answer information to the server 200 all at once.

[0095] FIG. 14 is a sequence diagram showing an example of the flow of incident determination processing according to the third embodiment. First, the information processing system 1a executes processing similar to S500 to S502 in FIG. 8. Here, it is assumed that the call determination unit 204 determines, from frame images of video data, that the user U is making a call. In this case, the operation control unit 205a of the server 200 selects question information Q from the operation control DB 207 and reads out the selected question information Q (S520). Then, the operation control unit 205a transmits a mode switching control signal including the read-out question information Q to the ATM 110 of the ATM system 100 via the network N (S521).

[0096] Next, the ATM 110 switches the operation mode to the second operation mode and displays the question information Q included in the mode switching control signal on the display unit 114 (S522).The ATM 110 then accepts an answer operation from the user U via the input unit 113 (S523).The ATM 110 then transmits the answer information to the server 200a (S524).

[0097] The incident determination control unit 206a of the server 200a transmits the video data and answer information acquired from the ATM system 100 to the related party terminal 300 (S525). At this time, the incident determination control unit 206a may also transmit question information Q corresponding to the answer information to the related party terminal 300. The information processing system 1a then executes the same processes as S507 to S511. In addition, in S507, the control unit 302 of the related party terminal 300 outputs the video data and answer information to the output unit 304 as status-related data, and the related party determines whether or not there is an incident based on the output data.

[0098] The process from the display of question information by the ATM 110 to the output of status-related data by the involved party terminal 300 (S522 to S524 and S507) may be repeated, and an incident determination operation (S508) may be performed during the repetition. Furthermore, if an incident determination operation is performed during the repetition, the incident determination control unit 206a of the server 200a may request the ATM 110 to terminate the output of the question information Q. For example, if no incident is determined, the incident determination control unit 206a of the server 200a may send a mode switching control signal to the ATM 110 instructing it to return from the second operation mode to the normal mode. Furthermore, if an incident is determined, the incident determination control unit 206a of the server 200a may send a mode switching control signal to the ATM 110 instructing it to transition from the second operation mode to the third operation mode.

[0099] FIG. 15 is a diagram showing an example of the display on the display unit 114 of the ATM 110 according to the third embodiment. This diagram is displayed in S522 of FIG. 14. For example, the display unit 114 may display a message indicating question information Q1, such as "Have you ever transferred money to this account before?". The display unit 114 may also display an operation area showing the options "Yes" and "No." The user U views the message and selects "Yes" or "No." The selected option becomes the answer information.

[0100] As described above, according to the third embodiment, when a call action is detected, the server 200a automatically switches the operation mode of the ATM 110 to interrupt the transfer process and output information other than the transfer process. This allows the user U to remain calm or secure time until the relevant personnel or the police arrive at the ATM 110. Furthermore, when the server 200a causes the ATM 110 to output question information Q, in addition to securing time, it also makes it easier to determine whether or not a crime has occurred based on the answer information. It also prevents erroneous determinations. Therefore, it is possible to prevent an incident from occurring or to contribute to a quick resolution while minimizing the increase in the user's psychological burden.

[0101] The third embodiment can be modified as follows.

[0102] <First Modification of Third Embodiment> For example, in the above explanation, the second data (audio data in the second embodiment) may be omitted from the basis for determining whether a crime has occurred, but the second data may also be included in the basis for determining whether a crime has occurred. Fig. 16 is a sequence diagram showing an example of the flow of incident determination processing according to a first modified example of embodiment 3. The steps in Fig. 16 differ from the steps in Fig. 14 in that steps S503 to S505 and S531 are included instead of S525 in Fig. 14.

[0103] Specifically, when the call determination unit 204 of the server 200a determines from a frame image of the video data that the user U is talking (S502), the operation control unit 205a transmits a microphone-on control signal to the ATM 110 (S503). As a result, the ATM 110 activates the microphone 160 and starts acquiring voice data from the microphone 160 (S504). Then, the ATM 110 starts transmitting the voice data to the server 200 (S505). Then, the information processing system 1a executes the same processes as those of S520 to S524.

[0104] The incident determination control unit 206a of the server 200a transmits the video data for the target period acquired from the ATM system 100, the audio data for a predetermined period after the microphone-on control, and the response information as status-related data to the related party terminal 300 (S531). Then, the information processing system 1a executes the same processes as S507 to S511. In S507, the control unit 302 of the related party terminal 300 outputs the video data, audio data, and response information to the output unit 304, and the related party determines whether or not there is an incident based on the output data.

[0105] This makes it easier to determine whether a crime has occurred and also makes it more likely that an erroneous determination will be made. Therefore, it is possible to prevent an incident from occurring or to quickly resolve the incident while minimizing the psychological burden on the user.

[0106] <Second Modification of Third Embodiment> In the above explanation, either the related party terminal 300 or the server 200a determines whether or not a crime has occurred, but both the related party terminal 300 and the server 200a may determine whether or not a crime has occurred. Specifically, the crime determination control unit 206a of the server 200a performs a primary determination as to whether or not a crime has occurred, and the related party terminal 300 performs a secondary determination as to whether or not a crime has occurred.

[0107] More specifically, the incident determination control unit 206a of the server 200a performs a primary determination based at least on the response to the question information. If the result of the primary determination indicates that a crime has occurred, the incident determination control unit 206a transmits video data for the target period and audio data for a predetermined period after the microphone-on control to the involved party terminal 300 as status-related data. At this time, the incident determination control unit 206a may include answer information, or the answer information and corresponding question information Q, in the status-related data. The incident determination control unit 206a then receives the crime determination result input to the involved party terminal 300 from the involved party terminal 300.

[0108] Fig. 17 is a sequence diagram showing an example of the flow of incident determination processing according to a second modified example of embodiment 3. The steps in Fig. 17 differ from the steps in Fig. 16 in that steps S540 to S543 are included instead of S531 and S507 to S508 in Fig. 16.

[0109] Specifically, when the incident determination control unit 206a of the server 200a acquires answer information from the ATM 110 (S524), it makes a primary determination based at least on the answer information (S540). For example, the incident determination control unit 206a may make a determination that a crime has occurred in the primary determination if the user U has given an answer to the question information Q that is likely to be a crime. The answer that is likely to be a crime may be determined in advance. Also, for example, the incident determination control unit 206a may make a determination that a crime has occurred in the primary determination if the number of answers that are likely to be a crime that have been given is equal to or greater than a predetermined number.

[0110] Furthermore, the importance level for determining whether or not a crime is involved may differ depending on the content of the question information Q. When answer information is obtained for question information Q of low importance, it is necessary to have the ATM 110 output further question information Q to obtain the answer information in order to determine whether or not a crime is involved. However, when answer information is obtained for question information Q of high importance, it is possible to sufficiently determine whether or not a crime is involved using the answer information already obtained. Therefore, for example, the incident determination control unit 206a may make a determination of whether or not a crime is involved in the primary determination when the user information U answers question information Q of high importance with a high or low possibility of a crime.

[0111] If the primary determination determines that there is a criminal offense, the incident determination control unit 206a of the server 200a transmits the acquired video data, audio data, and response information to the related party terminal 300 as status-related data (S541). Note that if the primary determination determines that there is no criminal offense, the incident determination control unit 206a does not need to transmit the status-related data to the related party terminal 300.

[0112] Next, the control unit 302 of the related party terminal 300 outputs the acquired status-related data to the output unit 304 (S542). As a result, the related party makes a secondary judgment as to whether or not there is a crime. Next, the input unit 303 of the related party terminal 300 accepts the secondary judgment operation by the related party (S543). Here, it is assumed that the input unit 303 of the related party terminal 300 accepts input of the secondary judgment result that there is a crime from the related party. In this case, the control unit 302 of the related party terminal 300 notifies the server 200 that there is a crime as the crime judgment result (S509). Then, the information processing system 1a executes the same processes as S510 to S511.

[0113] As a result, if the primary determination is that there is no possibility of a crime, the secondary determination by the relevant parties is omitted, thereby reducing the monitoring burden on the relevant parties. Therefore, it is possible to prevent cases where there is a high possibility of a crime being overlooked. Note that even if the primary determination is that there is no possibility of a crime, the incident determination control unit 206a of the server 200a may send various data to the relevant parties' terminal 300. The relevant parties' terminal 300 may then perform a secondary determination. This also allows the relevant parties to monitor with discretion, taking into account the results of the primary determination, thereby reducing the monitoring burden. Therefore, it is possible to prevent cases where there is a high possibility of a crime being overlooked.

[0114] <Other Modifications of the Third Embodiment> If the operation in the third operation mode includes an operation for the purpose of saving time, the operation may be an operation of ATM 110 slowing down compared to normal operation, or an operation of continuing to output question information.

[0115] In the third embodiment, a frame image of video data is used as the first data, but instead, audio data or received radio wave data indicating that radio waves from a mobile phone have been detected may be used. When the server 200a acquires the second data, the second data may be at least one of a frame image of video data and audio data, and may be a different type of data from the first data.

[0116] <Embodiment 4> Next, a fourth embodiment of the present disclosure will be described. The fourth embodiment can be described as an outline of the fifth to seventh embodiments described below. In the fourth embodiment, the determination of whether or not a crime has occurred is optimized by changing the operation content of the second operation mode depending on the determination result of whether or not a crime has occurred. Optimizing the determination of whether or not a crime has occurred may mean speeding up the determination of whether or not a crime has occurred or improving the accuracy of the determination.

[0117] 18 is a block diagram showing the configuration of an information processing device 20 according to the fourth embodiment. Similar to the information processing device 10, the information processing device 20 is a computer system that detects ATM-related events that may be incidents. The information processing device 20 includes a call determination unit 24, an operation control unit 25, an incident determination control unit 26, and an update unit 28.

[0118] The call determination unit 24 is also called a call determination means. The call determination unit 24 determines whether a user visiting an ATM is making a call. In this case, the call determination unit 24 may make the call determination based on at least any one of signal data of a captured image of the target area, audio data, and received radio wave data. However, the data on which the call determination is based is not limited to these.

[0119] The operation control unit 25 is also called an operation control means. When it is determined that the user is making a call, the operation control unit 25 switches the operation mode of the ATM from the normal operation mode to the second operation mode. The second operation mode is basically the same as the second operation mode described in the above-mentioned embodiment. However, the second operation mode in embodiment 4 may be an operation mode one of the purposes of which is to collect status-related data to facilitate determination of the presence or absence of an incident.

[0120] The incident determination control unit 26 is also called an incident determination control means. The incident determination control unit 26 transmits status-related data suggesting the user's status during the target period to the involved party terminal and acquires the incident determination result from the involved party terminal. The target period is determined based on the switch to the second operating mode. Here, in the fourth embodiment, the target period includes at least a predetermined period after the switch. For example, the target period may be a predetermined period after the switch, or a predetermined period before and after the switch that spans the switch. Furthermore, the status-related data for the target period is data that directly or indirectly suggests the user's status during the target period, and may be at least one of captured images, audio, and ATM operation content.

[0121] The update unit 28 is also referred to as an update means. The update unit 28 changes the operation content of the second operation mode based at least on the result of the incident determination. For example, the update unit 28 analyzes the characteristics of the status-related data that are considered to be highly important to the determination result based at least on the result of the incident determination and the status-related data. "High importance" may mean that the importance is equal to or greater than a predetermined threshold. Then, the update unit 28 changes the operation content of the second operation mode to operation content related to the analyzed status-related data having the highly important characteristics. "Suitable collection" may mean collection in a short time, or collection of status-related data that allows the characteristics to be extracted with high accuracy.

[0122] FIG. 19 is a flowchart showing the flow of the information processing method according to the fourth embodiment. First, the call determination unit 24 of the information processing device 20 determines whether a user is calling the ATM (S50). If the user is not calling (No in S51), the call determination unit 24 repeats the process shown in S50. On the other hand, if the user is calling (Yes in S51), the operation control unit 25 switches the operation mode of the ATM from the normal mode to the second operation mode (S52). Next, the incident determination control unit 26 transmits status-related data regarding the user's status during the target period based on the switch to the second operation mode to the related party terminal (S53). Then, the incident determination control unit 26 acquires a result of the incident determination from the related party terminal (S54). The update unit 28 changes the operation content of the second operation mode based on the result of the incident determination (S55). Then, the information processing device 20 returns the process to S50.

[0123] According to the fourth embodiment, the operation of the ATM for collecting status-related data, which is the basis for determining whether a transaction is likely to be a crime, is changed depending on the result of the crime determination. This allows for faster and more accurate determination of whether a crime is likely to be a crime. This reduces the time a user is required to operate the ATM and avoids the unpleasant situation of receiving a warning when a transaction that is not likely to be a crime is being performed due to a false determination. As a result, this contributes to preventing or quickly resolving crimes while minimizing the psychological burden on users.

[0124] <Embodiment 5> Next, a fifth embodiment of the present disclosure will be described. The fifth embodiment is an embodiment that embodies the fourth embodiment in an information processing system similar to the third embodiment. Fig. 20 is a block diagram showing the overall configuration of an information processing system 1b according to the fifth embodiment.

[0125] The information processing system 1b differs from the information processing system 1a in that it includes a server 200b instead of the server 200a. The server 200b has a call determination unit 204b, an operation control DB 207b, an operation control unit 205b, and an update unit 208 instead of the call determination unit 204, the operation control DB 207, and the operation control unit 205a.

[0126] The call determination unit 204b is an example of the above-mentioned call determination unit 24. The call determination unit 204b determines whether or not a user is making a call based on at least one of frame image data and audio data included in the video data of the target area. As an example, the call determination unit 204b performs call determination in the same manner as the call determination unit 204 of the second and third embodiments.

[0127] The operation control DB 207b stores the operation details of the second operation mode.

[0128] 21 is a diagram showing an example of the data structure of the action control DB 207b according to the fifth embodiment. For example, the action control DB 207b stores question information Q1 to Q4 as action details, similar to the action control DB 207 according to the third embodiment. However, the action control DB 207b differs from the action control DB 207 in that information on the priority of output is stored for each question information Q. The priority is determined based on the importance indicating how much the answer information corresponding to that question information contributes to determining whether or not an incident has occurred.

[0129] Returning to FIG. 20, the description will continue. The operation control unit 205b is an example of the above-mentioned operation control unit 25. When it is determined that the user is making a call, the operation control unit 205b reads out operation details for the second operation mode from the operation control DB 207b, similar to the operation control unit 205a of the third embodiment, and instructs the ATM 110 to operate according to the operation details. This switches the operation mode of the ATM 110 from the normal mode to the second operation mode. In the fifth embodiment, the operation control DB 207b stores question information Q. Then, the operation control unit 205b causes the ATM 110 to output the question information Q stored in the operation control DB 207b in the second operation mode. At this time, the ATM 110 is caused to output the question information Q stored in the operation control DB 207b in descending order of priority from the question information Q with the highest priority.

[0130] In addition, the operation control unit 205b may terminate the output of the question information Q by the ATM 110 and switch to normal mode in response to obtaining a determination result indicating the possibility of a crime. This prevents the user from being unnecessarily tied up and shortens the time required to operate the ATM.

[0131] The incident determination control unit 206b then includes the response to the question information Q in the status-related data and transmits it to the involved party terminal 300. The status-related data may additionally include data of the same data type as the data used for call determination by the call determination unit 204b, for example, frame images included in video data.

[0132] The case determination control unit 206b is an example of the case determination control unit 26 described above, but has the same functions as the case determination control unit 206a of the third embodiment, so a description thereof will be omitted.

[0133] The update unit 208 is an example of the above-mentioned update unit 28. The update unit 208 changes the output priority of at least one piece of question information Q among the pieces of question information Q stored in the operation control DB 207b, based on the output question information Q and the incident determination result.

[0134] Specifically, the update unit 208 first calculates the importance of each piece of question information Q to the determination result based on the incident determination results and the question information Q that have been accumulated up to that point. The update unit 208 then re-prioritizes the pieces of question information Q according to the importance. Re-prioritizing the pieces of question information according to the importance may mean, for example, determining the priorities so that the greater the importance, the higher the priority. The update unit 208 then associates each piece of question information Q with the priority and stores them in the operation control DB 207b. Alternatively, the update unit 208 may rearrange the question information Q stored in the operation control DB 207b in descending order of priority. This function makes it possible to suitably change the output order of the question information Q the next time the second operation mode is executed.

[0135] The calculation of the importance may be based on the number of pieces of question information Q output from the start of outputting the pieces of question information Q until the relevant person determines that the information is a crime and ends the output. As an example, if the number of pieces of question information Q output is less than a predetermined number, the update unit 208 may increase the importance of the output piece of question information Q by a predetermined amount. The question information Q immediately before the determination that the information is a crime is likely to have triggered the determination, so the increase in importance may be greater by a predetermined amount than the other pieces of question information Q output.

[0136] Fig. 22 is a sequence diagram showing an example of the flow of incident determination processing according to embodiment 5. The steps in Fig. 22 differ from the steps in Fig. 14 in that S550 and S551 are included instead of S520 in Fig. 14.

[0137] In S550, the operation control unit 205b of the server 200b reads out question information Q from the operation control DB 207b in accordance with the priority order. Then, the operation control unit 205b transmits a mode switching control signal including the read out question information Q to the ATM 110 of the ATM system 100 via the network N (S521), and causes the ATM 110 to display the question information Q on the display unit 114 (S522).

[0138] In addition, in S509, the control unit 302 of the involved party terminal 300 notifies the server 200 that there is a criminal possibility as a result of the criminal possibility determination, and then S551 is executed. In S551, the update unit 208 changes the priority of the question information Q stored in the operation control DB 207b based on the output question information and the criminal possibility determination result, and stores the priority in the operation control DB 207b. In this way, the update unit 208 updates the operation control DB 207b.

[0139] Thus, according to the fifth embodiment, the output order of the question information for collecting the condition-related data that is the basis for determining whether or not a crime has occurred is optimized based on the result of the crime determination. Therefore, the crime determination can be made more quickly and with improved accuracy. This achieves the same effects as the fourth embodiment.

[0140] The fifth embodiment can be modified as follows.

[0141] <First Modification of Fifth Embodiment> In the above explanation, the person concerned makes the determination as to whether the incident is a crime, but this may also be done by the incident determination control unit 206b of the server 200b, or by both. As an example of the latter, as in the second modified example of the third embodiment, the incident determination control unit 206b of the server 200b may make a primary determination as to whether the incident is a crime, and the person concerned terminal 300 may make a secondary determination as to whether the incident is a crime.

[0142] In this case, the incident determination process is the same as the steps in Figure 17, except that S550 in Figure 22 and S551 in Figure 22 are added instead of S520 in Figure 17. S503 to S505 may be omitted. In this case, the operation control unit 205b may terminate the output of the question information Q in the second operation mode when the primary determination determines that there is an incident based at least on the response to the question information Q registered in the operation control DB 207b. This achieves the effects of embodiment 4, while omitting the secondary determination by the relevant parties when the primary determination determines that there is no incident, thereby reducing the monitoring burden on the relevant parties.

[0143] In this case, the operation control DB 207b may store, for each piece of question information Q, various parameters for constructing a decision tree such as that shown in FIG. 23 in addition to the question content and priority.

[0144] FIG. 23 is a diagram showing a decision tree for a primary determination of whether or not a crime has occurred according to the fifth embodiment. First, the display unit 114 of the ATM 110 displays the question information Q3, which has the highest priority. If the user answers "Yes" to the question information Q3, the crime determination control unit 206b determines that "a crime has occurred." Therefore, the primary determination result is "a crime has occurred," and the crime determination control unit 206b transmits the response information to the concerned party terminal 300. In this case, the operation mode of the ATM 110 may be a second operation mode in which the question information Q is continuously output, or may be switched to an operation mode for saving time.

[0145] On the other hand, if the user answers "No" to question Q3, question Q1, which has the second highest priority, is displayed on the display unit 114 of the ATM 110. If the user answers "Yes" to question Q1, the incident determination control unit 206b determines that there is no incident. Therefore, the primary determination result is "no incident," and question output ends. The operating mode of the ATM system 100 then returns to normal mode.

[0146] On the other hand, if the user answers "No" to question Q1, question Q2, which has the third highest priority, is displayed on the display unit 114 of the ATM 110. If the user answers "No" to question Q2, the incident determination control unit 206b determines that there is no incident. Therefore, the primary determination result is "No incident," and question output ends. The operating mode of the ATM system 100 then returns to normal mode.

[0147] On the other hand, if the user answers "Yes" to question Q2, question Q4, which has the fourth highest priority, is displayed on the display unit 114 of the ATM 110. If the user answers "Yes" to question Q4, the incident determination control unit 206b determines that there is no incident. Therefore, the primary determination result is "no incident," and question output ends. The operating mode of the ATM system 100 then returns to normal mode.

[0148] On the other hand, if the user answers "No" to question information Q4, the incident determination control unit 206b determines that "an incident is suspected." Therefore, the primary determination result is "an incident is suspected," and the incident determination control unit 206b transmits the response information to the involved party terminal 300. In this case, the operation mode of the ATM 110 may be a second operation mode that continues to output question information Q, or may transition to an operation mode for saving time.

[0149] Although the decision tree in FIG. 23 is for primary determination, the above decision tree may also be used when only the incident determination control unit 206b of the server 200b determines whether or not an incident is suspected.

[0150] <Second Modification of the Fifth Embodiment> Furthermore, the first modified example of the third embodiment may be combined with the fifth embodiment. That is, the incident determination control unit 206b of the server 200b may use audio data in addition to the video data and response information for the target period as the basis for determining whether an incident has occurred. The audio data that forms one of the bases for determining whether an incident has occurred may be audio data after the microphone is turned on, and may include at least audio data for the target period.

[0151] FIG. 24 is a sequence diagram showing an example of the flow of the criminality determination process according to a modified example of the fifth embodiment. The steps in FIG. 24 are the steps in FIG. 16, with S550 in FIG. 22 and S551 in FIG. 22 added instead of S520 in FIG. 16. S551 may be performed at any timing after S509 is executed. This makes it easier to determine whether a criminality exists while still achieving the effects of the fourth embodiment. It also makes it possible to better prevent erroneous determinations.

[0152] <Embodiment 6> Next, a sixth embodiment of the present disclosure will be described. The sixth embodiment is characterized in that the update unit 208 of the server 200b updates the call detection DB 202 used for call determination.

[0153] In the sixth embodiment, the call determination unit 204b determines whether or not a user is talking based on at least one of a plurality of frame images included in the video data of the target area acquired from the ATM 110. For example, if there are a predetermined number or more frame images in the video from which it can be determined that the user is talking, the call determination unit 204b may comprehensively determine that the user is talking. When comprehensively determining whether or not a call is possible in this manner, a frame image whose individual call determination result does not match the comprehensive call determination result is likely to be an erroneous determination. Therefore, in this case, the update unit 208 may update the registration skeleton information R in the call detection DB 202 for that frame image. The update of the registration skeleton information R may be a process including at least one of deletion, modification, and addition.

[0154] For example, even if the call determination unit 204b comprehensively determines that the user is making a call, if the person involved determines whether or not a crime is committed and recognizes that the user is not making a call, the call determination is deemed to be incorrect. Therefore, the update unit 208 may receive a notification of an incorrect determination or a request to change the registration skeleton information R of the positive example from the person involved terminal 300. The update unit 208 may then update the registration skeleton information R of the positive example registered in the call detection DB 202. Also, for example, if the person involved determines that a crime is committed, this means that the person involved recognized that the user was making a call. Therefore, when the update unit 208 receives a crime determination result from the person involved terminal 300, it may determine whether to update the skeleton information extracted from each frame image as the registration skeleton information R of the positive example based on the call determination result for that frame image and the crime determination result.

[0155] Furthermore, if a related person or an administrator of the ATM 110 views video data that was not determined to be a call and recognizes that the user is on a call, the call determination is also incorrect. Therefore, the update unit 208 may receive a notification of an incorrect determination or a request to register skeleton information for registration of a negative example from the related person terminal 300. In this case, skeleton information extracted from a frame image included in the video data may be registered as skeleton information for registration of a negative example. The skeleton information for registration of a negative example may be used during call determination. For example, the call determination unit 204b may determine that a frame image having skeleton information whose similarity with the skeleton information for registration of a negative example is equal to or greater than a predetermined threshold is not a call. This improves the accuracy of call determination.

[0156] FIG. 25 is a diagram illustrating an example of a display screen 601 according to the sixth embodiment. The display screen 601 may be displayed on the display unit of the related party terminal 300 or on the display unit of a management terminal used by an administrator of the ATM 110. The display screen 601 includes frame images that have been comprehensively determined to be users talking but have not been individually determined to be users talking. The display screen 601 also includes a display that allows the related party or administrator to determine whether to register skeletal information extracted from the frame image as a positive example. In this figure, the display screen 601 also displays skeletal information extracted from the frame image. A related party or administrator viewing the display screen 601 may consider whether to register the skeletal information as a positive example, and if so, may send a registration request to the server 200b. As a result, the update unit 208 registers the skeletal information as a positive example in the call detection DB 202 to prevent erroneous determination of images similar to the erroneously determined frame image.

[0157] Although the above-described embodiments have been described as hardware configurations, the present disclosure is not limited to such configurations. Any processing in the present disclosure can also be realized by causing a processor to execute a computer program.

[0158] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0159] The present disclosure is not limited to the above-described embodiment, and may be modified as appropriate without departing from the spirit of the present disclosure. For example, the call determination DB 202 or the operation control DB 207 or 207b of the servers 200, 200a, and 200b may be included in an external device communicably connected to the servers 200, 200a, and 200b, instead of the servers 200, 200a, and 200b.

[0160] For example, some or all of the functions of the servers 200, 200a, and 200b may be included in the ATM 110. As an example, the ATM 110 may have the functions of the call determination units 204 and 204b and the operation control units 205, 205a, and 205b of the servers 200, 200a, and 200b. In addition, the ATM 110 may have the functions of the incident determination control units 206, 206a, and 206b of the servers 200, 200a, and 200b.

[0161] The captured image used for call determination may be a frame image captured at a predetermined timing. For example, the frame image may include at least one of a frame image captured when the user U operates the ATM 110 for the first time, a frame image captured when the user inserts a bankbook or cash card, and a frame image captured when the user starts a transfer operation.

[0162] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. (Appendix A1) a call determination means for determining whether a user visiting an ATM is making a call based on first data, which is any one of photographed images, voice data, and received radio wave data, relating to a target area based on the ATM; an operation control means for starting to acquire second data indicating the state of the user, the second data having a data type different from that of the first data, when it is determined that the user is making a call; an incident determination control means for obtaining a determination result as to whether or not a crime has occurred using the first data and the second data; An information processing device comprising: (Appendix A2) the first data is a photographed image of a user visiting the ATM, The second data is voice data picked up by a microphone installed at a predetermined position relative to the ATM. 10. The information processing device according to claim 1, (Appendix A3) The incident determination control means Estimating the content of the call based on the voice data; Determine whether or not there is a crime based on the captured image and the contents of the call 10. The information processing device according to claim 9, wherein the information processing device is a (Appendix A4) The incident determination control means transmitting the first data and the second data to a related party terminal used by the related party; The result of the determination of whether or not the incident is a crime, which has been input to the related party terminal, is received from the related party terminal. An information processing device according to appendix A1 or A2. (Appendix A5) The incident determination control means transmits user information about the user together with the first data and the second data to the involved party terminal. 1. An information processing device according to claim A4. (Appendix A6) The incident determination control means When an instruction to switch the operation mode of the ATM to a third operation mode in which an operation different from a normal operation is received from the terminal of the person involved as a result of determining whether or not there is an incident, the operation mode of the ATM is switched to the third operation mode. An information processing device according to Appendix A4 or A5. (Appendix A7) When it is determined that the user is making a call, the operation control means switches the operation mode of the ATM to a second operation mode in which an operation different from a normal operation is performed. An information processing device according to any one of appendices A1 to A6. (Appendix A8) The operation control means causes the ATM to output question information stored in advance in an operation control DB in the second operation mode. 10. The information processing device according to claim 7, (Appendix A9) The incident determination control means further uses the response to the question information to obtain a determination result as to whether or not there is an incident. 10. The information processing device according to claim 8, (Appendix A10) The incident determination control means Based on the response to the question information, a primary determination is made as to whether or not there is a crime; If the result of the primary determination indicates that a crime has occurred, the first data and the second data are transmitted to a related party terminal used by the related party; The result of the determination of whether or not the incident is a crime, which has been input to the related party terminal, is received from the related party terminal. An information processing device according to appendix A8 or A9. (Appendix A11) ATM (Automatic Teller Machine) and Information processing device Equipped with The information processing device includes: a call determination means for determining whether a user visiting the ATM is making a call based on first data, which is any one of a photographed image, voice data, and received radio wave data, related to a target area based on the ATM; an operation control means for starting to acquire second data indicating the state of the user, the second data having a data type different from that of the first data, when it is determined that the user is making a call; an incident determination control means for obtaining a determination result as to whether or not a crime has occurred using the first data and the second data; have Information processing system. (Appendix A12) determining whether a user visiting an ATM is making a call based on first data, which is any one of a photographed image, voice data, and received radio wave data, in a target area based on the ATM; When it is determined that the user is making a call, the acquisition of second data indicating the state of the user, the second data having a data type different from the first data, is started; Using the first data and the second data, a determination result as to whether or not a crime has occurred is obtained. Information processing methods. (Appendix A13) a step of determining whether a user visiting an ATM is making a call based on first data, which is any one of captured images, voice data, and received radio wave data, relating to a target area based on the ATM; When it is determined that the user is making a call, starting to acquire second data indicating the state of the user, the second data having a data type different from the first data; a step of obtaining a determination result of whether or not there is a crime using the first data and the second data; A non-transitory computer-readable medium storing a program for causing a computer to execute the above. (Appendix B1) a call determination means for determining whether a user visiting an ATM (Automatic Teller Machine) is making a call based on at least one of a photographed image, voice data, and received radio wave data of a target area based on the ATM; an operation control means for switching the operation mode of the ATM from a first operation mode in which normal operation is performed to a second operation mode in which information at least part of which is different from information output in the first operation mode when it is determined that the user is making a call; An incident determination control means for transmitting data indicating the user's status during a target period based on the switching to a related party terminal used by the related party, and obtaining a determination result of whether or not there is an incident from the related party terminal; an update means for changing the operation content of the second operation mode based at least on the result of the determination of whether or not there is an incident; An information processing device comprising: (Appendix B2) the call determination means determines whether the user is making a call based on at least one of the captured image and audio data of the target area; The data suggesting the user's state includes data of the same data type as the data used to determine the call. 10. The information processing device according to claim 8, wherein the information processing device is a device for processing information according to claim 1. (Appendix B3) the operation control means, in the second operation mode, causes the ATM to output question information stored in the operation control DB; The incident determination control means transmits a response to the question information to the involved terminal as at least a part of data indicating the state of the user. An information processing device according to Appendix B1 or B2. (Appendix B4) In the second operation mode, when it is determined that an incident has occurred based at least on a response to question information pre-stored in the operation control DB, the operation control means terminates output of the question information stored in the operation control DB. 2. The information processing device according to claim 1, wherein the information processing device is a device for processing information. (Appendix B5) The update means changes the output priority of at least one piece of question information stored in the operation control DB based on the output question information and the determination result of whether or not the question information is an incident. An information processing device according to Appendix B3 or B4. (Appendix B6) the call determination means determines whether the user is making a call based on a degree of similarity between at least a part of skeletal information extracted from a captured image of the user visiting the ATM and at least a part of registration skeletal information extracted from a registration image that is registered in advance in a call determination DB; The update means updates registration skeleton information registered in the call determination DB based at least on the determination result of whether or not there is a crime. An information processing device according to any one of appendices B1 to B5. (Appendix B7) ATM (Automatic Teller Machine) and Information processing device Equipped with The information processing device includes: a call determination means for determining whether a user visiting the ATM is making a call based on at least one of a photographed image, voice data, and received radio wave data of a target area based on the ATM; an operation control means for switching the operation mode of the ATM from a first operation mode in which normal operation is performed to a second operation mode in which information at least part of which is different from information output in the first operation mode when it is determined that the user is making a call; An incident determination control means for transmitting data indicating the user's status during a target period based on the switching to a related party terminal used by the related party, and obtaining a determination result of whether or not there is an incident from the related party terminal; an update means for changing the operation content of the second operation mode based at least on the result of the determination of whether or not there is an incident; have Information processing system. (Appendix B8) determining whether a user visiting an ATM (Automatic Teller Machine) is making a call based on at least one of a captured image, voice data, and received radio wave data of a target area based on the ATM; when it is determined that the user is making a call, switching the operation mode of the ATM from a first operation mode in which normal operation is performed to a second operation mode in which information at least part of which is different from information output in the first operation mode is output; Transmitting data suggesting the state of the user during a target period based on the switching to a related party terminal used by the related party, and obtaining a determination result of whether or not there is an incident from the related party terminal; and changing the operation content of the second operation mode based at least on the result of the determination of whether or not an incident has occurred. Information processing methods. (Appendix B9) a step of determining whether a user visiting an ATM (Automatic Teller Machine) is making a call based on at least one of a captured image, voice data, and received radio wave data of a target area based on the ATM; a step of switching the operation mode of the ATM from a first operation mode in which normal operation is performed to a second operation mode in which information at least part of which is different from information output in the first operation mode when it is determined that the user is making a call; a step of transmitting data suggesting the state of the user during a target period based on the switching to a related party terminal used by the related party, and obtaining a determination result of whether or not there is an incident from the related party terminal; changing the operation content of the second operation mode based at least on the result of the determination of whether or not there is an incident; A non-transitory computer-readable medium storing a program for causing a computer to execute the above. [Explanation of symbols]

[0163] 1,1a,1b Information Processing System 10, 20, 200, 200a, 200b Information processing device (server) 14,24,204 Call determination section 15, 25, 205, 205a, 205b Operation control unit 16, 26, 206, 206a, 206b Case Judgment Control Unit 100,100c ATM system 110,110c ATM 111 Communications Department 112,112c Control unit 113 Input section 114 Display section 150 cameras 160 Mike 201 Registration Department 202 Call detection database 203 Information Acquisition Department 207,207b Operation control DB 28,208 Update Department 300 related party terminals 301 Communications Department 302 Control Unit 303 Input section 304 Output section 500 frame images U User P mobile phone R Skeleton information for registration I Icon Q Question information

Claims

1. a call determination means for determining whether a user who has visited an ATM is making a call based on a comparison between skeletal information included in first data, which is a photographed image of the user who has visited the ATM, and pre-registered skeletal information; an operation control means for starting to acquire second data indicating the state of the user when it is determined that the user is making a call, the second data having a data type different from the first data; an incident determination control means for obtaining a determination result of whether or not there is a crime using the first data and the second data; an update means for updating the registered skeleton information used to determine whether or not a call is involved based on the result of the determination of whether or not there is an incident; An information processing device comprising:

2. The incident determination control means Estimating the content of the call based on the second data, which is voice data acquired at a predetermined position relative to the ATM; Determine whether or not there is a crime based on the captured image and the contents of the call The information processing device according to claim 1 .

3. The incident determination control means transmitting the first data and the second data to a participant terminal used by the participant; The result of the determination of whether or not the incident is a crime, which has been input to the related party terminal, is received from the related party terminal.

3. The information processing device according to claim 1 or 2.

4. The incident determination control means transmits user information about the user together with the first data and the second data to the involved party terminal. The information processing device according to claim 3 .

5. the call determination means determines whether the user is making a call based on a similarity between the skeletal information relating to a predetermined region of the body and the registered skeletal information relating to the predetermined region; The information processing device according to claim 1 .

6. when it is determined that the user is making a call, the operation control means switches the operation mode of the ATM from a first operation mode in which normal operation is performed to a second operation mode in which information at least part of which is different from information output in the first operation mode is output, The incident determination control means transmits data suggesting the state of the user during a target period based on the switching to a related party terminal used by the related party, and obtains a determination result of whether or not there is an incident from the related party terminal; The device further includes a change unit that changes the operation content of the second operation mode based at least on the result of the determination of whether or not there is an incident. The information processing device according to claim 1 .

7. ATMs (Automatic Teller Machines) and Information processing device Equipped with The information processing device includes: a call determination means for determining whether or not the user who has visited the ATM is making a call based on a comparison between skeletal information included in the first data of the photographed image of the user who has visited the ATM and pre-registered skeletal information; an operation control means for starting to acquire second data indicating the state of the user when it is determined that the user is making a call, the second data having a data type different from the first data; an incident determination control means for obtaining a determination result of whether or not there is a crime using the first data and the second data; an update means for updating the registered skeleton information used to determine whether or not a call is involved based on the result of the determination of whether or not there is an incident; have Information processing system.

8. The computer determining whether or not a user who has visited an ATM is making a call based on a comparison between skeletal information included in first data, which is a photographed image of the user who has visited the ATM, and pre-registered skeletal information; When it is determined that the user is making a call, the acquisition of second data indicating the state of the user, the second data having a data type different from the first data, is started; Using the first data and the second data, obtain a determination result as to whether or not there is a crime; updating the registered skeleton information used to determine whether or not the call is a criminal offense based on the result of the determination of whether or not the call is a criminal offense; Information processing methods.

9. a step of determining whether a user who has visited an ATM is making a call based on a comparison between skeletal information included in first data, which is a photographed image of the user who has visited the ATM, and pre-registered skeletal information; a step of starting to acquire second data indicating a state of the user, the second data having a data type different from that of the first data, when it is determined that the user is making a call; a step of obtaining a determination result of whether or not there is a crime using the first data and the second data; a procedure for updating the registered skeleton information used to determine whether or not a call is involved based on the result of the determination of whether or not a case has occurred; A program that causes a computer to execute the following.

Images