Information processing device, information processing system, and program

Abstract

To properly notify a staff of the occurrence of incidents at POS terminals.SOLUTION: An information processing device according to an embodiment includes an acquisition unit, a detection unit, a processing unit, and an output unit. The acquisition unit acquires a video image of a POS terminal and an operator's operating position of the POS terminal. The detection unit detects a suspected fraudulent behavior based on the video image. When the detection unit detects the suspected fraudulent behavior, the processing unit generates first notification information including event information indicating the type of the suspected fraudulent behavior and an image related to the suspected fraudulent behavior, and cuts out a video image related to the suspected fraudulent behavior from the video image. The output unit outputs the first notification information generated by the processing unit to a display unit, and if the video image related to the suspected fraudulent behavior is generated by the processing unit after the first notification information is output, outputs second notification information related to the video image to the display unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing apparatus, an information processing system, and a program.

Background Art

[0002] Conventionally, there are known self-type POS terminals (Point Of Sales) where a shopper serves as an operator to register and settle for goods, and semi-self-type POS terminals where a shopper serves as an operator to settle after a store clerk registers the goods. In addition, there is known a technology for notifying store staff of incidents such as improper actions such as unregistered goods and financial troubles in a POS terminal.

[0003] For example, the notified store staff can determine whether to take actions such as approaching the shopper. In such a situation, the notification is required to contain content sufficient for the store staff to determine the necessity of response, such as a video cut out from the video of a surveillance camera. On the other hand, if it takes time from the occurrence of an incident to the notification, there is a risk of missing the timing to take actions such as approaching.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the embodiments of the present invention is to appropriately notify the occurrence of an incident in a POS terminal.

Means for Solving the Problems

[0005] The information processing device of the embodiment comprises an acquisition unit, a detection unit, a processing unit, and an output unit. The acquisition unit acquires video footage of the POS terminal and the operator's position on the POS terminal. The detection unit detects suspected fraudulent behavior based on the video footage. When the detection unit detects suspected fraudulent behavior, the processing unit generates first notification information including event information indicating the type of suspected fraudulent behavior and an image related to the suspected fraudulent behavior, and processes the video footage to extract the video related to the suspected fraudulent behavior. The output unit is generated by the processing unit. The first notification information generated Then, The first notification information is output to the display unit. ,before The processing unit generates a video related to the suspected fraudulent activity. Then The system outputs a second notification regarding the video to the display unit. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 shows an example of the configuration of a POS (Point of Sales) system according to an embodiment. [Figure 2] Figure 2 is a perspective view showing an example of the appearance of a self-service POS terminal as seen from the operator's side according to the embodiment. [Figure 3] Figure 3 shows an example of the hardware configuration of a self-service POS terminal according to this embodiment. [Figure 4] Figure 4 shows an example of the hardware configuration of an attendant terminal according to this embodiment. [Figure 5] Figure 5 shows an example of the hardware configuration of a server device according to this embodiment. [Figure 6] Figure 6 is a diagram showing an example of the functional configuration of a self-service POS terminal according to the embodiment of this embodiment. [Figure 7] Figure 7 is a diagram showing an example of the functional configuration of an attendant terminal according to the embodiment of the embodiment. [Figure 8] Figure 8 is a diagram showing an example of the functional configuration of a server device according to the embodiment of this embodiment. [Figure 9] Figure 9 is a sequence diagram showing an example of the flow of information processing performed in the POS system according to the embodiment. [Figure 10] Figure 10 shows an example of a display screen shown in the information processing according to the embodiment. [Modes for carrying out the invention]

[0007] The information processing device, information processing system, and program will be described below with reference to the drawings.

[0008] Figure 1 shows an example of the configuration of a Point of Sales (POS) system 1 according to an embodiment. As shown in Figure 1, the POS system 1 includes a self-service POS terminal 10, an attendant terminal 30, a server device 50, a camera 70, and a storage device 90. In the POS system 1, the self-service POS terminal 10, the attendant terminal 30, the server device 50, the camera 70, and the storage device 90 are connected to each other via a network N, either by wire or wireless means, enabling communication. Here, the POS system 1 according to this embodiment is an example of an information processing system.

[0009] Self-service POS terminal 10 is a self-service type POS terminal in which shoppers act as operators to register products and make payments. Self-service POS terminal 10 is an example of a sales data processing device that performs product registration processing to register products to be sold and accounting processing for products registered through product registration processing, all at the customer's discretion.

[0010] The attendant terminal 30 is a terminal device used by the attendant (store clerk). The attendant (store clerk) manages and monitors a group of self-POS terminals, including at least one self-POS terminal 10, and attends to customers operating these self-POS terminals 10 as needed. The attendant terminal 30 is, for example, a personal computer, tablet terminal, or smartphone. The attendant terminal 30 is connected to, for example, multiple self-POS terminals 10. In addition, any self-POS terminal 10 among the multiple self-POS terminals 10 connected to each other via the network N can be used as the attendant terminal 30. The attendant terminal 30 notifies the store clerk of the detection of suspected fraudulent activity based on notification information from the server device 50. Here, the attendant terminal 30 in this embodiment is an example of a display unit.

[0011] The server device 50 is installed in the back room of the store. The server device 50 is also called the store server. The server device 50 is an example of an image processing device that performs image processing on video footage from a camera 70 that captures images of the self-service POS terminal 10. The server device 50 is an example of an information processing device that performs detection processing to detect suspicious fraudulent activity using the video footage. The server device 50 is an example of an information processing device that performs notification processing to notify the attendant terminal 30 of event information and images as simplified information when suspicious fraudulent activity is detected, and also generates a video to be sent to the attendant terminal 30 as detailed information. The storage unit 52 of the server device 50 (see Figure 5) stores the PLU file that is distributed to the self-service POS terminal 10. Here, the server device 50 in this embodiment is an example of an information processing device.

[0012] Camera 70 is, for example, placed for each self-service POS terminal 10, but may be placed for two or more self-service POS terminals 10, or multiple cameras 70 may be placed for each self-service POS terminal 10. Camera 70 includes an image sensor such as a CMOS (Complementary Metal-Oxide Semiconductor) or a CCD (Charge Coupled Device). Camera 70 outputs the captured video data to the storage device 90.

[0013] The storage device 90 is configured, for example, as network storage. As an example, the storage device 90 has a storage circuit such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory, a control circuit that controls writing to and reading from the storage circuit, and a communication circuit for communicating with the outside. The storage device 90 stores video data captured by the camera 70. The storage device 90 also stores images and videos extracted from the video by the server device 50.

[0014] Figure 2 is a perspective view showing an example of the appearance of a self-service POS terminal as seen from the operator's side according to the embodiment. As shown in Figure 1, the self-service POS terminal 10 has a main body 11, a basket stand 12, and a bagging stand 13.

[0015] The main unit 11 is installed between the basket stand 12 and the bagging stand 13. The basket stand 12 is a stand on which baskets are placed when registering products. The bagging stand 13 is a stand on which shopping bags are hung in an open state. Products that have been registered are placed in the shopping bags. In other words, registered products are placed on the bagging stand 13.

[0016] The bagging table 13 has a pedestal 131, a bag hanging hook 132, and a temporary placement table 133. The bag hanging hook 132 and the temporary placement table 133 are supported by a support column 134 provided on the pedestal 131 and are provided above the pedestal 131. The pedestal 131 is a table on which items not suitable for being put into bags, such as items larger than bags, are placed. The bag hanging hook 132 is a hook for hanging the bag containing the registered product for sale. The temporary placement table 133 is a table for temporarily placing items that require attention when bagging, such as easily breakable items.

[0017] Also, as shown in FIG. 2, the camera 70 is arranged so as to be able to take pictures within the shooting range including the self-POS terminal 10 and its operation position 9. Here, the shooting range including the operation position 9 is a range in which the operator (user) of the self-POS terminal 10 located at the operation position 9 can be photographed. That is, according to the video obtained within the shooting range including the operation position 9, it is possible to determine the operator of the self-POS terminal 10 and the presence or absence thereof. In addition, the shooting range including the self-POS terminal 10 is a range in which at least the basket stand 12 and the bagging table 13 of the self-POS terminal 10 can be photographed. That is, according to the video obtained within the shooting range including the self-POS terminal 10, it is possible to discriminate various products related to product registration processing and accounting processing, such as products placed in the shopping basket registered in the self-POS terminal 10 and registered products.

[0018] FIG. 3 is a diagram showing an example of the hardware configuration of the self-POS terminal 10 according to the embodiment. As shown in FIG. 3, the self-POS terminal 10 has a touch panel display 14, a deposit / withdrawal unit 15, a scanner 16, a printer 17, an indicator light 18, a card reader 19, a control unit 21, a storage unit 22, and a communication interface 23. The touch panel display 14, the deposit / withdrawal unit 15, the scanner 16, the printer 17, the indicator light 18, the card reader 19, the control unit 21, the storage unit 22, and the communication interface 23 are interconnected via a system bus 29 such as a data bus and an address bus.

[0019] As shown in FIG. 1, the touch panel display 14 is provided, for example, above the scanner 16 in the main body 11. The touch panel display 14 displays various screens and accepts operations by customers.

[0020] As shown in FIG. 1, the cash handling unit 15 is provided, for example, in the central portion of the main body 11. The cash handling unit 15 has, for example, a coin insertion unit, a coin ejection unit, a bill insertion unit, and a bill ejection unit. The coin insertion unit is an insertion port that accepts the insertion of coins paid by customers. The coin ejection unit is a tray that receives coins ejected from the cash handling unit 15 such as change. The bill insertion unit is an insertion port that accepts the insertion of bills paid by customers. The bill ejection unit is an ejection port for bills ejected from the cash handling unit 15 such as change. Note that the cash handling unit 15 may not be provided, for example, when the self-POS terminal 10 is configured as a cashless terminal that does not handle cash.

[0021] As shown in FIG. 1, the scanner 16 is provided, for example, above the cash handling unit 15 in the main body 11. The scanner 16 has a reading window 161 as shown in FIG. 1. The scanner 16 is a reading device that reads product information from code symbols such as wireless tags and barcodes attached to products held over the reading window 161. The wireless tag is an electronic tag such as an RF (Radio Frequency) tag. The wireless tag has an IC (Integrated Circuit) chip that stores information and an antenna. The wireless tag transmits the tag information stored in the IC chip in response to radio waves received from a wireless tag reader. For example, product information regarding the product to which the wireless tag is attached is stored in the wireless tag. The product information includes, for example, a product code that can identify the product.

[0022] The scanner 16 may also have a camera configured to capture images of products held over the reading window 161, for example. The camera of the scanner 16 includes an image sensor such as a CMOS or CCD. In this case, the image data of the products captured by the camera of the scanner 16 is used for object recognition processing by the server device 50 or the self-service POS terminal 10, which acts as an image processing device. This identifies and registers the products included in the image.

[0023] The printer 17 is located, for example, inside the casing of the main unit 11. The printer 17 is a printing device that prints receipts showing the details of a transaction, coupons for product discounts, etc., onto receipt paper. As shown in Figure 1, the printer 17 has a receipt output unit 171. The receipt output unit 171 is located, for example, between the deposit / withdrawal unit 15 and the scanner 16 in the main unit 11. The receipt output unit 171 is an output port that ejects receipts, etc., issued by the printer 17 to the outside of the self-service POS terminal 10.

[0024] As shown in Figure 1, the indicator light 18 is located, for example, behind the touch panel display 14 on the main unit 11. The indicator light 18 is a light that notifies store staff of abnormalities such as an error occurring in the self-service POS terminal 10.

[0025] As shown in Figure 1, the card reader 19 is located, for example, on the side of the scanner 16 in the main unit 11. The card reader 19 is a reading device that reads cards such as credit cards.

[0026] The control unit 21 is a computer that controls the overall operation of the self-service POS terminal 10 and realizes the various functions of the self-service POS terminal 10. The control unit 21 has a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). The CPU is a processor that comprehensively controls the operation of the self-service POS terminal 10. For example, ROM is a storage medium that stores various programs and data. RAM is a storage medium that temporarily stores various programs and data. The CPU uses RAM as a work area (work region) to execute various programs stored in ROM or the storage unit 22, etc.

[0027] Furthermore, the processor in the control unit 21 is not limited to a CPU; other processors such as a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array) may also be used.

[0028] The storage unit 22 is a storage device such as an HDD, SSD, or flash memory. The storage unit 22 stores the control program 221. The storage unit 22 may also store a product master.

[0029] The communication interface 23 is an interface for communicating with external devices of the self-service POS terminal 10, such as the attendant terminal 30 and the server device 50, via the network N.

[0030] Figure 4 shows an example of the hardware configuration of an attendant terminal 30 according to this embodiment. As shown in Figure 4, the attendant terminal 30 has a control unit 31, a storage unit 32, a communication interface 33, and a touch panel display 34. The control unit 31, storage unit 32, communication interface 33, and touch panel display 34 are interconnected via a system bus 39 such as a data bus or an address bus.

[0031] The control unit 31 is a computer that controls the overall operation of the attendant terminal 30 and realizes the various functions of the attendant terminal 30. The control unit 31 has a CPU, ROM, and RAM. The CPU is a processor that comprehensively controls the operation of the attendant terminal 30. For example, ROM is a storage medium that stores various programs and data. RAM is a storage medium that temporarily stores various programs and data. The CPU uses RAM as a work area to execute various programs stored in ROM or the storage unit 32, etc.

[0032] Furthermore, the processor for the control unit 31 is not limited to a CPU; other processors such as a GPU, ASIC, or FPGA may also be used.

[0033] The memory unit 32 is a storage device such as an HDD, SSD, or flash memory. The memory unit 32 stores the control program 321.

[0034] The communication interface 33 is an interface for communicating with external devices of the attendant terminal 30, such as the self-service POS terminal 10 and the server device 50, via the network N.

[0035] The touch panel display 34 displays various screens and accepts operations from store staff.

[0036] Figure 5 shows an example of the hardware configuration of a server device 50 according to this embodiment. As shown in Figure 5, the server device 50 has a control unit 51, a storage unit 52, and a communication interface 53. The control unit 51, the storage unit 52, and the communication interface 53 are interconnected via a system bus 59, such as a data bus or an address bus.

[0037] The control unit 51 is a computer that controls the overall operation of the server device 50 and realizes the various functions of the server device 50. The control unit 51 has a CPU, ROM, and RAM. The CPU is a processor that comprehensively controls the operation of the server device 50. For example, ROM is a storage medium that stores various programs and data. RAM is a storage medium that temporarily stores various programs and data. The CPU uses RAM as a work area (work region) to execute various programs stored in ROM or the storage unit 52, etc.

[0038] Furthermore, the processor for the control unit 51 is not limited to a CPU; other processors such as a GPU, ASIC, or FPGA may also be used.

[0039] The memory unit 52 is a storage device such as an HDD, SSD, or flash memory. The memory unit 52 stores the control program 521.

[0040] The communication interface 53 is an interface for communicating with external devices of the server device 50, such as the self-service POS terminal 10, the attendant terminal 30, and the storage device 90, via the network N.

[0041] Figure 6 is a diagram showing an example of the functional configuration of a self-service POS terminal 10 according to an embodiment of the embodiment. Note that some or all of the functions realized by the self-service POS terminal 10 may be realized through cooperation with external devices of the self-service POS terminal 10, or they may be realized by hardware configurations such as dedicated circuits installed in the self-service POS terminal 10.

[0042] The control unit 21 of the self-service POS terminal 10 executes a control program 221 loaded into RAM to realize the functions of a product registration unit 101, an accounting processing unit 102, and a notification unit 103.

[0043] The product registration unit 101 registers products for sale based on product codes read from code symbols such as wireless tags or barcodes attached to the products. Alternatively, the product registration unit 101 may register products for sale by identifying them through object recognition based on image data captured by the scanner 16's camera. Furthermore, the product registration unit 101 may register products identified through user input, not just by reading product information.

[0044] The accounting processing unit 102 controls the accounting process for settling payments for products registered by the product registration unit 101. For example, the accounting processing unit 102 calculates the total price of the products registered for sale by the product registration unit 101. The accounting processing unit 102 also calculates the change by subtracting the total price from the amount deposited into the deposit / disbursement unit 15. The accounting processing unit 102 also causes the deposit / disbursement unit 15 to dispense the change.

[0045] The notification unit 103 issues an alert (notification) in response to a notification from the attendant terminal 30 or the server device 50. For example, the notification unit 103 notifies of the detection of fraud by displaying a screen indicating the detection of fraud on the touch panel display 14, lighting up the indicator light 18, changing the color of the indicator light 18, or blinking the indicator light 18.

[0046] Figure 7 shows an example of the functional configuration of the attendant terminal 30 according to the embodiment. Some or all of the functions realized by the attendant terminal 30 may be realized through cooperation with external devices of the attendant terminal 30, or they may be realized by hardware configurations such as dedicated circuits installed in the attendant terminal 30.

[0047] The control unit 31 of the attendant terminal 30 realizes the functions of a communication unit 301, a display unit 302, and an input unit 303 by executing a control program 321 loaded into RAM.

[0048] The communication unit 301 communicates with external devices of the attendant terminal 30, such as the self-service POS terminal 10 and the server device 50, via the communication interface 33.

[0049] The display unit 302 displays a notification screen (see Figure 10) that notifies the user of suspected fraudulent activity and a playback screen that plays a video for detailed verification via the touch panel display 34.

[0050] The input unit 303 accepts operations from store staff via the touch panel display 34.

[0051] Figure 8 shows an example of the functional configuration of a server device 50 according to the embodiment. Note that some or all of the functions realized by the server device 50 may be realized through cooperation with external devices of the server device 50, or they may be realized by hardware configurations such as dedicated circuits mounted on the server device 50.

[0052] The control unit 51 of the server device 50 performs the functions of a communication unit 501, a detection unit 502, and a processing unit 503 by executing a control program 521 loaded into RAM.

[0053] The communication unit 501 communicates with external devices of the server device 50, such as the self-POS terminal 10 and the attendant terminal 30, via a communication interface 53. For example, the communication unit 501 outputs a request notification to the storage device 90 requesting video data of the self-POS terminal 10 and the operating position 9. The communication unit 501 also acquires the video data from the storage device 90. The communication unit 501 also outputs simplified information and detailed information generated by the processing unit 503 to the attendant terminal 30. As an example, the communication unit 501 outputs simplified information to the attendant terminal 30 when it is generated. As an example, if a video related to suspected fraudulent behavior, i.e., a detailed confirmation video, is generated after the simplified information has been output, the communication unit 501 outputs detailed information to the attendant terminal 30 when it is generated. Here, the communication unit 501 according to this embodiment is an example of an acquisition unit and an output unit. Also, the simplified information is an example of first notification information. Also, the detailed information is an example of second notification information.

[0054] The detection unit 502 detects suspected fraudulent activity based on video footage captured from the self-service POS terminal 10 and the operating position 9. The detection unit 502 performs image processing on the video footage to detect the user's actions and the movement of products included in the video, thereby detecting suspected fraudulent activity. In addition, the detection unit 502 uses product information of registered products from the self-service POS terminal 10 to detect suspected fraudulent activity.

[0055] The processing unit 503 extracts images related to suspected fraudulent activity, i.e., verification images, from the video footage captured of the self-service POS terminal 10 and the operating position 9. The verification images include at least one of the following: a user verification image, which is an image used to verify the operator (user) of the self-service POS terminal 10, and a fraud verification image, which serves as a basis for determining whether a store employee should respond to the suspected fraudulent activity by speaking to the user.

[0056] The following describes an example of the operation of the POS system 1 according to the embodiment, with reference to the drawings. Figure 9 is a sequence diagram showing an example of the flow of information processing performed in the POS system 1 according to the embodiment. The flow in Figure 9 illustrates processing by an arbitrary attendant terminal 30 and any self-POS terminal 10 (hereinafter referred to as the target self-POS terminal 10) from among a plurality of self-POS terminals 10 corresponding to the attendant terminal 30.

[0057] Camera 70 captures images within a shooting range that includes the self-service POS terminal 10 and the operating position 9, and sequentially outputs the image data obtained from the capture to the storage device 90 (S11). In this case, the storage device 90 sequentially stores the image data from camera 70. The display unit 302 of the attendant terminal 30 displays a notification screen on the touch panel display 34 (S12).

[0058] Subsequently, assume that a transaction has started at the target self-service POS terminal 10. In this case, the self-service POS terminal 10 outputs a transaction start notification to the server device 50 (S13). The self-service POS terminal 10 also sequentially transmits registration information to the server device 50, indicating the registered product and the time of registration. Alternatively, if the server device 50 detects suspicious activity from the video, it refers to the product information registered at the self-service POS terminal 10.

[0059] The communication unit 501 of the server device 50 requests video data from the storage device 90 (S14) in response to a transaction start notification from the self-service POS terminal 10, and receives video data from the storage device 90 (S15).

[0060] The processing unit 503 of the server device 50 extracts an image from the received video that includes the operator of the self-POS terminal 10, i.e., the user, and generates it as a user verification image (S16). The data of the generated user verification image is stored in the storage unit 52 of the server device 50.

[0061] Subsequently, the POS system 1 performs fraud detection processing (S17-S28) until a transaction is completed, i.e., until a transaction completion notification is issued by the self-service POS terminal 10 (S29).

[0062] The communication unit 501 of the server device 50 requests video data from the storage device 90 (S17) and receives video data from the storage device 90 (S18). The detection unit 502 of the server device 50 also starts detecting suspicious activity by performing image processing on the received video (S19).

[0063] If no suspicious activity is detected, the server device 50 repeatedly performs the process of acquiring video data by the communication unit 501 and detecting suspicious activity based on the acquired video data.

[0064] On the other hand, if suspicious activity is detected, the detection unit 502 of the server device 50 outputs a detection notification to the self-service POS terminal 10 (S20). The self-service POS terminal 10 provides notification using indicator lights 18 or the like in response to the detection notification (S21).

[0065] Furthermore, if suspicious activity is detected, the server device 50 begins generating both simplified and detailed information. In other words, as shown in Figure 9, the server device 50 generates simplified and detailed information in parallel and outputs them sequentially.

[0066] Specifically, the processing unit 503 of the server device 50 generates an image for fraud verification from the video (S22). The processing unit 503 generates simplified information including event information indicating the detected suspected fraudulent activity, a user verification image, and a fraud verification image. When the simplified information is generated, the processing unit 503 outputs the simplified information to the attendant terminal 30 (S23). The attendant terminal 30 updates its notification screen in response to the receipt of the simplified information and displays a notification screen 400 (see notification 411 in Figure 10) including event information 421, a user verification image 425, and a fraud verification image 427 (S24).

[0067] Furthermore, the processing unit 503 of the server device 50 generates a detailed confirmation video by extracting the portion of the video in which suspicious behavior is detected, as well as the portions before and after it, in parallel with the generation of simplified information (S25). When the detailed confirmation video is generated, the processing unit 503 outputs it to the attendant terminal 30 as detailed information (S26). The attendant terminal 30 updates the notification screen 400 (see notification 413 in Figure 10) in response to the receipt of the detailed information and activates the play button 423 in the target notification 413 to instruct the playback of the video (S27).

[0068] In addition, the generated detailed confirmation video may be stored in the storage device 90, and the attendant terminal 30 may access the storage device 90. In this case, the detailed information may be information indicating that the detailed confirmation video is ready for playback, which is output from the server device 50 or the storage device 90 to the attendant terminal 30 when the generation of the detailed confirmation video and uploading to the storage device 90 are completed.

[0069] The server device 50 may sequentially output the extracted frames as a video for detailed verification. In other words, the video for detailed verification may not only be output from the server device 50 to the attendant terminal 30 as a playable video file, but may also be output from the server device 50 to the attendant terminal 30 as data that can be combined to form a video file. In this case, the detailed information may be information indicating that the video for detailed verification has become playable, which is output when all the data constituting the video for detailed verification has been output.

[0070] When the display unit 302 of the attendant terminal 30 receives an input from the input unit 303, for example, an operation by a store employee on the play button 423 via the touch panel display 34, it displays a video for detailed confirmation on the touch panel display 34 (S28).

[0071] Subsequently, if the transaction is completed at the target self-service POS terminal 10, the self-service POS terminal 10 outputs a transaction completion notification to the server device 50 (S29), and the process shown in Figure 9 ends.

[0072] Figure 10 shows an example of a notification screen 400 displayed in the information processing according to the embodiment. The notification screen 400 illustrated in Figure 10 includes a display 410 of the detected suspected fraudulent activity, a map display 450 showing the self-POS terminal 10 where the suspected fraudulent activity was detected, a display 470 of the history of the detected suspected fraudulent activity, and a menu display 490.

[0073] The display of detected suspected fraudulent activity 410 includes notifications 411 and 413 for each detection of suspected fraudulent activity, i.e., for each incident. Figure 10 includes notification 411 in response to the detection of an unpaid exit and notification 413 in response to the detection of an unregistered product. Each notification 411 and 413 includes event information 421, a play button 423, and confirmation images 425 and 427, respectively.

[0074] Map display 450 shows the self-service POS terminal 10 where suspicious fraudulent activity was detected. The notification screen 400 in Figure 10 illustrates the case where suspicious fraudulent activity is detected at self-service POS terminals 10 "5" and "6".

[0075] The display of the history of suspected fraudulent activity 470 is a list showing the history of suspected fraudulent activity detected within a predetermined time period. The notification screen 400 in Figure 10 illustrates a case where two suspected fraudulent activities of "product not registered" and one suspected fraudulent activity of "exit without payment" were detected within 5 minutes at the self-service POS terminal 10 labeled "5" and "exit without payment" at the self-service POS terminal 10 labeled "6". Note that the events displayed in the history display 470 are not limited to 5 minutes, but can be any events detected within a predetermined time period. In addition, the events displayed in the history display 470 can be scrolled, for example, using a slider 473.

[0076] The menu display 490 expands, for example, when selected. From the expanded tabs, various settings can be configured, or users can transition to other display screens, such as a screen showing video footage of the current self-service POS terminal 10.

[0077] For example, notification 411, which responds to the detection of an unpaid exit, includes event information 421 indicating that a suspected fraudulent act of "unpaid exit" was detected at "11:55:57" in the self-service POS terminal 10 of "6". Furthermore, notification 411 includes a play button 423.

[0078] For example, notification 413 in response to the detection of an unregistered product includes event information 421 indicating that a suspected fraudulent activity of "unregistered product" was detected at "11:55:57" on the self-service POS terminal 10 number "5". The notification 413 also includes a play button 423.

[0079] Notification 411 exemplifies the display state before detailed information is received. In other words, for notification 411, the video for detailed confirmation is unplayable. On the other hand, notification 413 exemplifies the display state after detailed information is received. In other words, for notification 413, the video for detailed confirmation is playable. For example, the play button 423 of notification 411 is grayed out, indicating that the operation is disabled, as illustrated in Figure 10, and it can be determined from the display that video playback is not possible at that point.

[0080] Note that the display of the play button 423 before detailed information is received is not limited to being grayed out. It is sufficient that the display can distinguish between playable and non-playable states, and this can be changed as desired. For example, the display could be highlighted by blinking when playable, or the display color could differ for each state, or the display could be three-dimensional with a shadow when playable.

[0081] In addition, each of the notifications 411 and 413 displays verification images 425 and 427 as images related to suspected misconduct.

[0082] As an example, the image associated with suspected fraudulent activity is at least one of the user verification image 425 and the fraud detection image 427. The notification screen 400 in Figure 10 illustrates a case where both the user verification image 425 and the fraud detection image 427 are displayed.

[0083] The user verification image 425 is, for example, an image taken at the start of a transaction. Alternatively, the user verification image 425 may be generated when suspicious fraudulent activity is detected, by using, for example, a person detection process such as face detection to search through the video and extract the portion containing the user.

[0084] Fraud detection image 427 is an image that allows for the determination of whether suspected fraudulent behavior is indeed fraudulent behavior. Specifically, fraud detection image 427 is an image that serves as a basis for store employees to decide whether to take action, such as speaking to the user, in response to suspected fraudulent behavior. Fraud detection image 427 is, for example, an image corresponding to the type of suspected fraudulent behavior detected.

[0085] For example, image 427, used for fraud detection in response to an event where payment is not made, is an image extracted showing a moment when the user is not at operation position 9.

[0086] For example, image 427, used for fraud detection in response to an event where payment is not made, is an image extracted at the moment the user leaves the operating position 9.

[0087] For example, image 427, used for fraud detection in response to an event where an unpaid exit occurs, is an image extracted from a predetermined time period after the unpaid exit was detected.

[0088] For example, image 427 used for fraud detection in events where products are not registered is an image that includes products that have been moved to the bagging table 13 without being registered.

[0089] As an example, image 427, used for fraud detection in events where products are not registered, is an image taken at the moment when the product was moved to the bagging table 13 without being registered.

[0090] For example, image 427, used for fraud detection in events where products are not registered, is an image cropped to capture the moment when a product that has been moved to the bagging table 13 without registration is removed from the basket or basket stand 12.

[0091] For example, the image 427 used to verify fraud for events where products are not registered is an image of the basket stand 12 or a basket placed on the basket stand.

[0092] As described above, in the POS system 1 according to this embodiment, the notification screen 400 for notifying store staff displays information about suspected fraud events, including the type of fraud, along with related images. This allows store staff to see the fraud detection event (text) information and images and decide whether to take action, such as speaking to the user of the self-service POS terminal 10. Furthermore, since the notification screen 400 can be displayed even while a video for detailed confirmation is being generated, the time from the occurrence of an incident to notification can be reduced, thereby reducing the risk of missing the opportunity to take action, such as speaking to the user.

[0093] Furthermore, in the POS system 1 according to this embodiment, a video for detailed verification of suspected fraudulent activity can be extracted and played back in cases where a judgment cannot be made based solely on text information and images. This video is shared with store staff after a delay from the suspected fraud event. Since it takes some time for the video to become playable, store staff use it when they want to check the details. This allows necessary information to be quickly notified to store staff when suspected fraudulent activity is detected. Moreover, by sharing the video of the incident when fraud is determined to have occurred, when detailed information is needed, it is possible to support accurate judgment by store staff. For example, store staff can check the display status of the LED that lights up when a product is registered by reviewing the detailed verification video.

[0094] In the above embodiment, for example, as illustrated in Figure 1, one attendant terminal 30 corresponding to a group of self-service POS terminals including two self-service POS terminals 10 is illustrated, but the embodiment is not limited to this. There may be three or more self-service POS terminals 10 corresponding to one attendant terminal 30. Furthermore, the POS system 1 may include multiple attendant terminals 30 and a group of self-service POS terminals corresponding to each of the multiple attendant terminals 30. In addition, at least one of the multiple attendant terminals 30 may be a semi-self-service type POS terminal.

[0095] Furthermore, the technology according to this embodiment can also be applied to other sales data processing devices of the self-service POS terminal 10, such as a semi-self-service type POS terminal where the shopper acts as the operator to make a payment after the store clerk registers the product, or a POS terminal where the store clerk acts as the operator to register the product and make a payment.

[0096] For example, when applying the technology according to the embodiment to a semi-self-service type POS terminal, the display processing described later may be performed at the POS terminal where the store clerk registers the product, or it may be performed at the attendant terminal 30 for managing at least one semi-self-service type POS terminal.

[0097] For example, when applying the technology according to the embodiment to a POS terminal operated by a store employee for product registration and payment, the display processing described later may be performed on the POS terminal operated by the store employee, or on a management terminal located in the back room or elsewhere.

[0098] Furthermore, when displaying the notification screen 400 on the POS terminal used by the operator (user) whose suspicious activity is detected, the playback button 423 for the detailed confirmation video, the map display 450, and the display of the history of other POS terminals 470 do not need to be displayed.

[0099] The example given shows that a video for detailed confirmation is played when the play button 423, which is enabled on the notification screen 400, is selected. However, this is not the only example. The video for detailed confirmation may also be played as soon as it becomes playable, i.e., as soon as the detailed information is received.

[0100] Furthermore, in the display of suspected fraudulent activity (410) and the display of history (470), unconfirmed items may be highlighted, and status information such as confirmed, unconfirmed, resolved, and not resolved may also be displayed.

[0101] As described above, according to the embodiment, it is possible to appropriately notify the occurrence of an incident at a POS terminal.

[0102] Furthermore, each control program executed by each device of the POS system 1 in this embodiment is provided pre-loaded into ROM or the like.

[0103] Each control program executed by each device of the POS system 1 in this embodiment may be configured to be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).

[0104] Furthermore, the control programs executed by each device of the POS system 1 of this embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the control programs executed by each device of the POS system 1 of this embodiment may be provided or distributed via a network such as the Internet.

[0105] The control program 221 executed in the self-service POS terminal 10 of this embodiment has a modular configuration that includes the above-mentioned parts (product registration unit 101, accounting processing unit 102, and notification unit 103). The CPU (processor) reads the control program 221 from the storage medium and loads the above-mentioned parts into the main memory. As a result, the product registration unit 101, accounting processing unit 102, and notification unit 103 are created in the main memory.

[0106] The control program 321 executed in the attendant terminal 30 of this embodiment has a modular configuration that includes the above-described parts (communication unit 301, display unit 302, and input unit 303). The CPU (processor) reads the control program 321 from the storage medium and loads the above-described parts into the main memory. As a result, the communication unit 301, display unit 302, and input unit 303 are created in the main memory.

[0107] The control program 321 executed by the server device 50 of this embodiment has a modular configuration including the above-described parts (communication unit 501, detection unit 502, and processing unit 503). The CPU (processor) reads the control program 321 from the storage medium and loads the above-described parts into the main memory. As a result, the communication unit 501, detection unit 502, and processing unit 503 are created in the main memory.

[0108] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0109] 1 POS system 10 Self-service POS terminals 101 Product Registration Department 102 Accounting Processing Department 103 Hochi Department 11 Main unit 12 Basket Stands 13 Bagging Station 131 Pedestal 132 Bag hanging hooks 133 Temporary Stand 134 Post 14 Touch panel display 15. Deposit and Withdrawal Department 16 Scanners 161 Reading window 17 Printers 171 Receipt ejection unit 18 Indicator light 19 Card Reader 21,31,51 Control Unit 22,32,52 Storage part 23,33,53 Communication Interfaces 29, 39, 59 System bus 30 Attendant terminals 301,501 Communications Department 302 Display section 303 Input section 50 Server Devices 502 Detection Unit 503 Processing Unit 70 Cameras 90 Storage device N Network [Prior art documents] [Patent Documents]

[0110] [Patent Document 1] Japanese Patent Publication No. 2019-016225

Claims

1. An acquisition unit that acquires video footage of the POS terminal and the operator's position when operating the POS terminal, A detection unit that detects suspected fraudulent behavior based on the aforementioned video, When the detection unit detects the suspected fraudulent behavior, the processing unit generates first notification information including event information indicating the type of the suspected fraudulent behavior and an image related to the suspected fraudulent behavior, and processes the extraction of a video related to the suspected fraudulent behavior from the video. When the processing unit generates the first notification information, the output unit outputs the first notification information to the display unit, and when the processing unit generates a video relating to the suspected misconduct, the output unit outputs a second notification information relating to the video to the display unit. An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, wherein the image related to the suspected fraudulent conduct includes at least one of an image extracted from the video that includes the operator of the POS terminal and an image capable of determining whether the suspected fraudulent conduct is fraudulent conduct.

3. The information processing apparatus according to claim 2, wherein the image capable of determining whether the suspected fraudulent behavior is fraudulent behavior is an image that includes a shopping cart.

4. An information processing device according to any one of claims 1 to 3, The system comprises a display unit that displays the first notification information, The information processing system for displaying the first notification information includes the event information, an image related to the suspected fraudulent behavior, and a play button that instructs the playback of the video.

5. The information processing system according to claim 4, wherein the display of the play button indicates that the operation is disabled until the second notification information is received.

6. To acquire video footage of the POS terminal and the operator's position when operating the POS terminal, Based on the aforementioned video footage, to detect suspicious behavior, When the suspected fraudulent activity is detected, a first notification is generated, which includes event information indicating the type of the suspected fraudulent activity and an image related to the suspected fraudulent activity, and a process is performed to extract a video related to the suspected fraudulent activity from the video. When the first notification information is generated, the first notification information is output. When a video relating to the suspected misconduct is generated, a second notification regarding the video will be output. A program that causes a computer to execute something.

Images