Information processing program, information processing device, and information processing system

Abstract

To deter fraud at self-checkout counters while reducing the burden on administrators. [Solution] The information processing device acquires video footage of the scanning area of ​​a self-checkout terminal that scans product barcodes, analyzes the acquired video footage to detect the user's actions of grasping multiple products within the scanning area of ​​the self-checkout terminal, and based on the analysis results of the type of overlapping state of the multiple products when the user's actions of grasping multiple products are detected, sets either the self-checkout terminal or a management terminal that monitors the self-checkout terminal as the notification destination terminal, generates alert image information corresponding to the type of overlapping state of the multiple products, and displays the generated alert image information on the display screen of the set notification destination terminal.

Description

Technical Field

[0001] The present invention relates to an information processing program and the like.

Background Art

[0002] In recent years, in the retail field, the introduction of self-checkout systems, where checkout users themselves scan the barcodes of products to be purchased and perform payment, has been progressing.

[0003] On the other hand, self-checkout systems have been pointed out to be a hotbed of fraudulent acts such as shoplifting due to low public visibility, and there is a demand for detecting fraud at self-checkout systems. For example, fraud at self-checkout systems includes acts of pretending to scan a product, and acts of scanning products in an overlapping manner.

[0004] As a conventional technique for detecting fraud at self-checkout systems, there is a technique that captures images of a person, products, and product barcodes with a camera, and notifies an administrator of an abnormality by determining whether a product held by the person matches the product that was barcode-scanned.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of Invention

Problem to be Solved by the Invention

[0006] However, the above-described conventional technique has a problem that it is not possible to take appropriate measures against fraud at self-checkout systems.

[0007] For example, conventional technology assumes that a person will pick up one item at a time. Therefore, if multiple items are picked up, the accuracy of the detection decreases, and it may not be possible to detect fraud at the self-checkout.

[0008] Furthermore, it is difficult to distinguish whether fraud at self-checkouts is a "mistake" or "intentional," and if an anomaly is notified to the administrator every time fraud is suspected, verification work will be required each time a notification is received, increasing the burden on the administrator.

[0009] In one aspect, the present invention aims to provide an information processing program, an information processing method, and an information processing device that can deter fraud at self-checkout counters while reducing the burden on administrators. [Means for solving the problem]

[0010] In the first plan, the computer is instructed to perform the following processes: The computer acquires video footage of the scanning area of ​​a self-checkout terminal that scans product barcodes, analyzes the acquired video footage to detect the user's actions of picking up multiple products within the scanning area of ​​the self-checkout terminal, and based on the analysis results of the type of overlapping state of the multiple products when the user's actions of picking up multiple products are detected, sets either the self-checkout terminal or the management terminal that monitors the self-checkout terminal as the notification destination terminal, generates alert image information according to the type of overlapping state of the multiple products, and displays the generated alert image information on the display screen of the set notification destination terminal. [Effects of the Invention]

[0011] This can deter fraud at self-checkout counters while reducing the burden on administrators. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 is a diagram illustrating the self-checkout system according to this embodiment. [Figure 2] Figure 2 is a diagram illustrating the first pattern. [Figure 3] Figure 3 is a diagram for explaining the second pattern. [Figure 4] Figure 4 is a diagram for explaining the third pattern. [Figure 5] Figure 5 is a diagram for explaining the fourth pattern. [Figure 6] Figure 6 is a diagram for explaining the fifth pattern. [Figure 7] Figure 7 is a diagram showing the configuration of the information processing system according to the present embodiment. [Figure 8] Figure 8 is a diagram (1) showing an example of alert image information. [Figure 9] Figure 9 is a diagram (2) showing an example of alert image information. [Figure 10] Figure 10 is a functional block diagram showing the configuration of the information processing apparatus according to the present embodiment. [Figure 11] Figure 11 is a diagram showing an example of the data structure of a product information table. [Figure 12] Figure 12 is a diagram showing an example of the data structure of a tracking table. [Figure 13] Figure 13 is a diagram showing an example of the scanning area of a self-checkout register. [Figure 14] Figure 14 is a diagram (1) for supplementary explanation of CLIP. [Figure 15] Figure 15 is a diagram (2) for supplementary explanation of CLIP. [Figure 16] Figure 16 is a flowchart showing the procedure of the determination processing of the determination unit. [Figure 17] Figure 17 is a flowchart showing the processing procedure of the information processing apparatus according to the present embodiment. [Figure 18] Figure 18 is a diagram showing an example of the hardware configuration of a computer that implements the same functions as the information processing apparatus of the embodiment. [Mode for Carrying Out the Invention]

[0013] The following describes in detail, with reference to the drawings, embodiments of the information processing program, information processing apparatus, and information processing system disclosed in this application. However, this invention is not limited by these embodiments. [Examples]

[0014] An example of a self-checkout system according to this embodiment will be described. Figure 1 is a diagram illustrating the self-checkout system according to this embodiment. As shown in Figure 1, the self-checkout system 10 has product storage areas 11a and 11b and a cashier counter 12. Product storage area 11a is where products are placed before scanning. Product storage area 11b is where products are placed after scanning.

[0015] A fixed scanner 13 is placed on the cash register counter 12. The scanner 13 is a device that scans barcodes attached to products.

[0016] User 5 scans the product by holding the product's barcode over the scanner 13. The area where User 5 performs the scan is defined as the scan area. For example, when User 5 uses the self-checkout 10, the store requires User 5 to scan each product placed in the product area 11a one by one and then place them in the product area 11b after scanning. The positions of product area 11a and product area 11b may be reversed left and right.

[0017] Cameras 20a and 20b are cameras that capture the user 5 scanning the product 6 from different directions. For example, camera 20a photographs the cash register counter 12 from above. Camera 20b photographs the cash register counter 12 from diagonally above. In this embodiment, cameras 20a and 20b are shown, but other cameras may also be included. In the following description, cameras 20a and 20b will be referred to as "camera 20" unless otherwise specified.

[0018] Next, we will explain the patterns of potentially fraudulent scanning at self-checkout machines. In this example, we will assume the following patterns 1 to 5.

[0019] Figure 2 is a diagram illustrating the first pattern. In the example shown in Figure 2, user 5 places items 6a and 6b (unscanned items) from the product storage area 11a onto the cash register counter 12. Then, user 5 scans items 6a and 6b one by one from the cash register counter 12 and places items 6a and 6b back onto the product storage area 11b. The scan in the first pattern scans items 6a and 6b and can be considered non-fraudulent.

[0020] Figure 3 illustrates the second pattern. In the example shown in Figure 3, user 5 takes products 6a and 6b (unscanned products) from the product storage area 11a and stacks them together. Then, at the cash register counter 12, user 5 scans the stacked products 6a and 6b one by one and places products 6a and 6b back into the product storage area 11b. The scan in the second pattern scans products 6a and 6b and can be considered non-fraudulent.

[0021] Figure 4 illustrates the third pattern. In the example shown in Figure 4, user 5 takes two unscanned items, 6a and 6b, from the product storage area 11a. Then, at the cash register counter 12, user 5 scans the stacked items 6a and 6b only once (for example, scanning only item 6a and not item 6b), and places items 6a and 6b back into the product storage area 11b. In the fourth pattern, part of the stacked items (for example, item 6b) is not scanned, which can be considered fraudulent.

[0022] Figure 5 illustrates the fourth pattern. In the example shown in Figure 5, user 5 places items 6a and 6b (unscanned items) from the product storage area 11a onto the cash register counter 12. Then, user 5 scans item 6a multiple times (for example, twice) and places items 6a and 6b onto the product storage area 11b. The fourth pattern is considered fraudulent because it substitutes the barcode of item 6b with the barcode of item 6a, making it appear as if item 6b has been scanned.

[0023] Figure 6 illustrates the fifth pattern. In the example shown in Figure 6, user 5 takes out the scanned item 6a from the item storage area 11b and then takes out the unscanned item 6b from the item storage area 11b. After that, user 5 rescans item 6a and then places items 6a and 6b back into the item storage area 11b. The fourth pattern is considered fraudulent because the rescanning of item 6a is used to disguise the scanning of item 6b.

[0024] The above explains patterns 1 through 5.

[0025] Next, the configuration of the information processing system according to this embodiment will be described. Figure 7 is a diagram showing the configuration of the information processing system according to this embodiment. As shown in Figure 7, this information processing system includes a self-checkout register 10, a camera 20, a management terminal 60, and an information processing device 100. The self-checkout register 10, the camera 20, the management terminal 60, and the information processing device 100 are interconnected via a network 50.

[0026] The self-checkout terminal 10 is a terminal used by the user 5 to purchase goods. The self-checkout terminal 10 has a display screen 10a for notifying the user 5 of various information. The self-checkout terminal 10 transmits the scan results from the user 5 to the information processing device 100. For example, the scan results include barcode information. Further details regarding the self-checkout terminal 10 are the same as those described in Figure 1. Note that the information processing system according to this embodiment may include multiple self-checkout terminals in addition to the self-checkout terminal 10. The self-checkout terminal 10 corresponds to a self-checkout terminal.

[0027] Camera 20 corresponds to cameras 20a and 20b described in Figure 1. Camera 20 captures images of the self-checkout machine 10 and transmits the captured image data to the information processing device 100. In the following description, the image data transmitted by camera 20 to the information processing device 100 will be referred to as "image information." The image information will include images of the scanned area.

[0028] The video information includes multiple frames in chronological order. Each frame is assigned a frame number in ascending chronological order. A single frame is a still image captured by camera 20 at a specific point in time. Each frame may also be assigned time data.

[0029] The management terminal 60 is a terminal device used by the administrator 7. The management terminal 60 can be a PC (Personal Computer), a notebook PC, a tablet device, etc. The management terminal 60 displays information transmitted from the information processing device 100.

[0030] The information processing device 100 analyzes video information acquired from the camera 20 to detect the actions of user 5 in the scanning area of ​​the self-checkout register 10, in which case user 5 picks up multiple items. Based on the analysis results of the type of overlapping state of the multiple items when the actions of user 5 picking up multiple items are detected, the information processing device 100 sets the notification destination to at least one of the self-checkout register 10 or the management terminal 60. The information processing device 100 generates alert image information according to the type of overlapping state and sends the alert image information to the set notification destination for display.

[0031] For example, if the information processing device 100 determines that the type of overlapping state of multiple products corresponds to either the first pattern or the second pattern, it sets the self-checkout register 10 as the notification destination. The information processing device 100 then displays alert image information corresponding to the first pattern or the second pattern on the display screen 10a of the self-checkout register 10.

[0032] Figure 8 is an example of alert image information (Figure (1)). As shown in Figure 8, the information processing device 100 displays alert image Im1 on the display screen 10a as alert image information corresponding to the first pattern and the second pattern. For example, alert image Im1 displays text such as "There are multiple items, the items are overlapping." User 5 can continue operating the self-checkout 10 even while alert image Im1 is displayed.

[0033] As explained in Figures 2 and 3, the first and second patterns, although they differ from the scanning method required by the store, do not constitute fraud at the self-checkout. Therefore, the user 5 can be asked to change their scanning method. Furthermore, even though it is not fraud at the self-checkout, it is possible to deter fraud notifications from being sent to the management terminal 60, thereby reducing the burden on the administrator 7.

[0034] On the other hand, if the analysis result of the type of overlapping state of multiple products corresponds to one of the third, fourth, or fifth patterns, the information processing device 100 sets the self-checkout register 10 and the management terminal 60 as notification destinations. The information processing device 100 displays alert image information corresponding to the third to fifth patterns on the display screen 10a of the self-checkout register 10 and the display screen of the management terminal 60.

[0035] Figure 9 is Figure (2) showing an example of alert image information. As shown in Figure 9, the information processing device 100 displays alert image Im1 on the display screen 10a of the self-checkout register 10 as alert image information corresponding to the third to fifth patterns. The content of alert image Im1 is the same as alert image Im1 in Figure 8, but the information processing device 100 may control the self-checkout register 10 to lock its operation.

[0036] Furthermore, the information processing device 100 displays an alert image Im2 on the display screen of the management terminal 60 as alert image information corresponding to the third to fifth patterns. For example, the alert image Im2 displays a table that associates the register ID, time, and fraud. The register ID is information that identifies the self-checkout register. The time is the time when the fraud was detected. The fraud is the content of the fraud at the self-checkout register.

[0037] As explained in Figures 4 to 6, patterns 3 to 5 can be described as fraud at the self-checkout. In this case, the information processing device 100 can notify the administrator 7 of the occurrence of fraud by displaying the alert image information on the management terminal 60.

[0038] Next, an example of the configuration of the information processing device 100 according to this embodiment will be described. Figure 10 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Figure 10, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0039] The communication unit 110 performs data communication with the self-checkout 10, camera 20, and management terminal 60 via the network 50. For example, the communication unit 110 receives video information from the camera 20. The communication unit 110 also receives scan results from the self-checkout 10.

[0040] The input unit 120 is an input device that inputs various types of information to the control unit 150 of the information processing device 100. For example, an administrator may operate the input unit 120 to update the product information table 142, which will be described later.

[0041] The display unit 130 is a display device that displays information output from the control unit 150.

[0042] The storage unit 140 includes a video buffer 141, a product information table 142, and a tracking table 143. The storage unit 140 is a memory, etc.

[0043] The video buffer 141 stores video information captured by the camera 20. The video buffer 141 stores video information for each of the cameras 20a and 20b. For example, the video information is assigned a camera ID to identify the camera, and from the camera ID, the position of the camera 20 and the self-checkout 10 that the camera is photographing can be identified.

[0044] The product information table 142 holds various information about the product. Figure 11 shows an example of the data structure of the product information table. As shown in Figure 11, the product information table 142 includes product ID, barcode, product name, price, product image, etc. The product ID is a number that identifies the product. The barcode is the barcode of the product. The product name is the name of the product. The price is the price of the product. The product image is an image of the product.

[0045] The tracking table 143 holds information about the movement of the product generated by the motion detection unit 152, which will be described later. Figure 12 shows an example of the data structure of the tracking table. As shown in Figure 12, the tracking table 143 includes a track ID, a gripping start time, a specific area input time, a specific area output time, a gripping end time, and a group of image data.

[0046] The Track ID is information that identifies each track during tracking. The track is the movement path of the product, etc. The Grasp Start Time is the time when the user grasped the product. The Specific Area Input Time is the moment the grasped product is entered into a specific area. The specific area is the scanning area. The Specific Area Output Time is the moment the grasped product is output from the specific area. The image data set consists of frames from the Specific Area Input Time to the Specific Area Output Time.

[0047] The control unit 150 includes an acquisition unit 151, an motion detection unit 152, a product identification unit 153, a determination unit 154, and a notification processing unit 155. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.

[0048] The acquisition unit 151 acquires video information from the camera 20 and stores the acquired video information in the video buffer 141. For example, the acquisition unit 151 associates the camera ID with the video information and stores it in the video buffer 141.

[0049] The acquisition unit 151 acquires the scan results from the self-checkout register 10 and outputs the acquired scan results to the product identification unit 153.

[0050] The motion detection unit 152 analyzes the video information stored in the video buffer 141 to detect the user's actions of picking up multiple items in the scanning area of ​​the self-checkout register 10, and stores the detected information about the movement of the items in the tracking table 143.

[0051] Figure 13 shows an example of the scanning area of ​​a self-checkout system. As shown in Figure 13, scanning area A10 is set to the area around the checkout counter 12 where barcodes can be scanned. Scanning area A10 may be pre-configured by an administrator or other person.

[0052] For example, the motion detection unit 152 detects when a user grasps an item, tracks the item, and detects the movement path of the same item.

[0053] The motion detection unit 152 uses HOID (Human Object Interaction Detection) technology to detect when a user grasps an object from video information. More specifically, the motion detection unit 152 inputs the video information into a model such as QPIC to detect the bounding box of the person, the bounding box of the object, and the confidence score for the action label "grasp". For example, QPIC is disclosed in the document "M. Morita, et al. "QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information", CVPR 2021.".

[0054] The motion detection unit 152 identifies the time when the confidence score becomes above a threshold as the gripping start time. The motion detection unit 152 tracks the same person or product across multiple frames and identifies the time when the confidence score falls below a threshold after it has become above a threshold as the gripping end time.

[0055] The motion detection unit 152 tracks the product held by the user using Deep SORT (Simple Online and Realtime Tracking). For example, for each frame, the motion detection unit 152 acquires the appearance features of the object image (image within the object's bounding box) output from a model such as QPIC, and tracks the position (coordinates) of the object whose appearance feature similarity exceeds a threshold, thereby tracking the same product. The motion detection unit 152 assigns the same tracking ID to the movement path of the same product.

[0056] The motion detection unit 152 compares the movement path of the product with the scan area A10 described in Figure 13, and identifies the time at which the product's position entered the scan area A10 as the specific area input time. The motion detection unit 152 identifies the time at which the product's position left the scan area A10 as the specific area output time.

[0057] The motion detection unit 152, by executing the above process, identifies the track ID, gripping start time, specific area input time, specific area output time, and gripping end time, and stores each of the identified pieces of information in the tracking table 143. In addition, the motion detection unit 152 stores frames (image data group) from the specific area input time to the specific area output time in the tracking table 143, associating them with the track ID.

[0058] The motion detection unit 152 performs the above tracking on each product held by the person, and stores the track ID, gripping start time, specific area input time, specific area output time, gripping end time, and image data set for each product in the tracking table 143. Although not shown in the diagram, the motion detection unit 152 stores information that associates each position and time of the track with the track ID in the tracking table 143.

[0059] The product identification unit 153 identifies the product information of the scanned product based on the barcode of the scan result and the product information table 142. For example, the product identification unit 153 compares the barcode of the scan result with the product information table 142 shown in Figure 11 and identifies the product name, price, and product image corresponding to the barcode of the scan result as product information. The product identification unit 153 outputs the product information to the determination unit 154.

[0060] The determination unit 154 determines, based on the tracking table 143, whether multiple items are present on the cash register counter 12, and if multiple items are present on the cash register counter 12, it determines whether the multiple items are overlapping. The determination unit 154 outputs the determination result to the notification processing unit 155.

[0061] The process by which the determination unit 154 determines whether or not multiple items are present on the cash register counter 12 will be explained in more detail. The determination unit 154 arranges the tracks of each item in chronological order and determines whether or not the position of each item is included in the area of ​​the cash register counter 12 at each given time. The area of ​​the cash register counter 12 is predetermined. The determination unit 154 counts the number of items that are simultaneously included in the area of ​​the cash register counter 12 at any given time or within a certain period of time, and determines that multiple items are present on the cash register counter 12 if the number of items is two or more.

[0062] The process by which the determination unit 154 determines whether multiple products overlap will be explained in more detail. The determination unit 154 utilizes a pre-trained CLIP (Contrastive Language-Image Pretraining) model. The CLIP model is trained using a large number of image-language pairs, and learns the degree of agreement between images and words (text) by mapping images and text into a common vector space. In the following explanation, the CLIP model may sometimes be simply referred to as CLIP.

[0063] Figures 14 and 15 are diagrams to provide supplementary explanations for CLIP. First, let's explain Figure 14. Figure 14 explains the training of CLIP55. For convenience, the device used to train CLIP55 will be referred to as the training device. The training device calculates the image vectors for images Im2-1, Im2-2, and Im2-3. The training device also calculates the text vectors for the texts "Number of items 0", "Number of items 1", and "Number of items 3". The training device compares the similarity between the image vectors of CLIP55 images Im2-1 to Im2-3 and the text vectors of "Number of items 0", "Number of items 1", and "Number of items 3", and trains CLIP55 to have high similarity for correct pairs and low similarity for others. The correct pairs are the pair of image Im2-1 and the text "Number of items 0", the pair of image Im2-2 and the text "Number of items 1", and the pair of image Im2-3 and the text "Number of items 2".

[0064] Next, Figure 15 will be explained. Figure 15 illustrates the estimation of the number of products using a trained CLIP55. The determination unit 154 inputs the image vector of the frame (image Im3) and the text vector of the text "Number of products 0" into CLIP55 and calculates a score. The determination unit 154 inputs the image vector of the frame (image Im3) and the text vector of the text "Number of products 1" into CLIP55 and calculates a score. The determination unit 154 inputs the image vector of the frame (image Im3) and the text vector of the text "Number of products 2" into CLIP55 and calculates a score. For example, if the score of the image vector of the frame (image Im3) and the text vector of the text "Number of products 2" is greater than the scores of the other items, the determination unit 154 determines that the number of products is 2. In this case, the determination unit 154 determines that multiple products are overlapping.

[0065] Figures 14 and 15 illustrate the case with 0 to 2 products for ease of explanation, but learning and estimation can be performed similarly even when the number of products is 3 or more.

[0066] In the example above, the determination unit 154 used CLIP 55 to determine whether multiple products overlapped, but it is not limited to this, and it may also determine whether multiple products overlapped by matching product images included in the product information.

[0067] Next, if the determination unit 154 determines that multiple products are not overlapping, it determines whether multiple products have been scanned correctly. For example, the determination unit 154 determines that the product corresponding to the track ID has been scanned correctly if the image data group corresponding to the track ID in the tracking table 143 matches the product image in the product information (the scanned product image).

[0068] If the determination unit 154 determines that multiple products are overlapping, it determines whether an unscanned product and a scanned product are overlapping. Whether a product is unscanned or not can be determined using the product image in the product information, in the same manner as described above.

[0069] If the determination unit 154 determines that an unscanned product and a scanned product are not overlapping, it determines whether the overlapping unscanned product was scanned correctly. Whether a product is unscanned or not can be determined using the product image in the product information, in the same manner as described above.

[0070] Figure 16 is a flowchart showing the procedure for the judgment processing of the judgment unit. The judgment result numbers shown in Figure 16 are numbers that uniquely identify the judgment result. As shown in Figure 16, if the judgment unit 154 does not detect multiple products (step S10, No), it sets the judgment result to the number "J1" (step S11).

[0071] If the determination unit 154 detects multiple products (step S10, Yes), it proceeds to step S12. If the determination unit 154 does not detect any overlapping products (step S12, No), it proceeds to step S13. If the determination unit 154 determines that multiple products have not been scanned correctly (step S13, No), it sets the determination result to the number "J2" (step S14). If the determination unit 154 determines that multiple products have been scanned correctly (step S13, Yes), it sets the determination result to the number "J3" (step S15).

[0072] On the other hand, if the determination unit 154 detects overlapping products (step S12, Yes), it proceeds to step S16. If the determination unit 154 finds that an unscanned product and a scanned product are overlapping (step S16, Yes), it sets the determination result to the number "J4" (step S17).

[0073] On the other hand, if the unscanned product and the scanned product are not stacked (step S16, No), the determination unit 154 proceeds to step S18. If the stacked products are not scanned correctly (step S18, No), the determination unit 154 sets the determination result to the number "J5" (step S19). If the stacked products are scanned correctly (step S18, Yes), the determination unit 154 sets the determination result to the number "J6" (step S20).

[0074] As explained in Figure 16, the determination unit 154 sets one of the numbers J1 to J6 as the determination result and outputs this determination result to the notification processing unit 155.

[0075] The notification processing unit 155 sets either the self-checkout 10 or the management terminal 60 as the notification destination terminal based on the determination result from the determination unit 154. The notification processing unit 155 generates alert image information according to the type of overlapping condition of the products and displays the alert image information on the display screen of the set notification destination. The processing of the notification processing unit 155 will be described in more detail below.

[0076] If the determination result is set to the number "J1", the notification processing unit 155 sets the notification destination to "Self-checkout 10". In this case, the notification processing unit 155 does not generate alert image information.

[0077] If the number "J2" is set in the judgment result, the notification processing unit 155 sets the notification destination to "management terminal 60". In this case, the notification processing unit 155 generates alert image information "scan missed" and displays the alert image information on the notification destination's display screen. For example, if the number "J2" is set in the judgment result, it means that the fourth pattern has been detected.

[0078] If the number "J3" is set in the judgment result, the notification processing unit 155 sets "Self-checkout" as the notification destination. In this case, the notification processing unit 155 generates alert image information "Product overlap" and displays the alert image information on the notification destination's display screen. For example, if the number "J3" is set in the judgment result, it means that the first pattern has been detected.

[0079] If the number "J4" is set in the judgment result, the notification processing unit 155 sets the notification destination to "management terminal 60". In this case, the notification processing unit 155 generates alert image information "overlapping scanned products" and displays the alert image information on the notification destination's display screen. For example, if the number "J4" is set in the judgment result, it means that the fifth pattern has been detected.

[0080] If the number "J5" is set in the judgment result, the notification processing unit 155 sets the notification destination to "management terminal 60". In this case, the notification processing unit 155 generates alert image information "scan missed" and displays the alert image information on the notification destination's display screen. For example, if the number "J5" is set in the judgment result, it means that the third pattern has been detected.

[0081] If the number "J6" is set in the judgment result, the notification processing unit 155 sets the notification destination to "Self-checkout". In this case, the notification processing unit 155 generates alert image information "Product overlap" and displays the alert image information on the notification destination's display screen. For example, if the number "J6" is set in the judgment result, it means that the second pattern has been detected.

[0082] Here, if the judgment result number is set to "J3" or "J6", the notification processing unit 155 may allow user 5 to operate the self-checkout register 10, as this does not constitute fraud. Furthermore, if the judgment result with the number "J3" or "J6" occurs a predetermined number of times or more within a predetermined time, the notification processing unit 155 may change the alert image information. For example, the notification processing unit 155 may change the alert image information "Products overlapping" displayed on the display screen of the self-checkout register 10 to the alert screen information "Payment stopped, please wait for ○○".

[0083] Furthermore, while the notification processing unit 155 initially set the notification destination to the management terminal 60 when the judgment result number was set to "J2", "J4", "J5", or "J6", it is not limited to this. For example, the notification processing unit 155 may further set the self-checkout machine 10 as a notification destination, display alert image information as shown in Im1 of Figure 9 to the self-checkout machine 10, and stop the user 5 from using the self-checkout machine 10.

[0084] Next, an example of the processing procedure of the information processing device 100 according to this embodiment will be described. Figure 17 is a flowchart of the processing procedure of the information processing device according to this embodiment. As shown in Figure 17, the acquisition unit 151 of the information processing device 100 acquires video information from the camera 20 (step S101). The acquisition unit 151 acquires the scan result from the self-checkout 10 (step S102).

[0085] The motion detection unit 152 of the information processing device 100 detects the user's actions of grasping multiple products in the scanning area by analyzing the video information (step S103). The product identification unit 153 of the information processing device 100 identifies the product information of the scanned products based on the scan results and the product information table 142 (step S104).

[0086] The determination unit 154 of the information processing device 100 executes a determination process (step S105). The procedure for the determination process in step S105 is the same as the procedure described in Figure 16.

[0087] The notification processing unit 155 of the information processing device 100 sets either the self-checkout 10 or the management terminal 60 as the notification destination based on the determination result (step S106). The notification processing unit 155 generates alert image information according to the type of overlapping state of the products (step S107). The notification processing unit 155 displays the alert image information on the display screen of the set notification destination (step S108).

[0088] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 analyzes video information acquired from the camera 20 to detect the actions of user 5 who grasps multiple items in the scanning area of ​​the self-checkout 10. Based on the analysis results of the type of overlap state of the multiple items when the actions of user 5 who grasps multiple items are detected, the information processing device 100 sets the notification destination to at least one of the self-checkout 10 or the management terminal 60. The information processing device 100 generates alert image information according to the type of overlap state and sends the alert image information to the set notification destination for display. This makes it possible to deter fraud at the self-checkout while reducing the burden on the administrator.

[0089] When the information processing device 100 detects that multiple unscanned items have passed through the scanning area while overlapping, it sets the management terminal 60 as the notification destination. This allows the management terminal 60 to be notified of fraudulent activity at the self-checkout, corresponding to the third pattern.

[0090] When the information processing device 100 detects that scanned items have passed through the scanning area multiple times, it sets the management terminal as the notification destination. This allows the management terminal 60 to be notified of fraudulent activity at the self-checkout register corresponding to the fourth pattern.

[0091] When the information processing device 100 detects that scanned and unscanned items have passed through the scanning area while overlapping, it sets the management terminal as the notification destination. This allows the management terminal 60 to be notified of fraudulent activity at the self-checkout, corresponding to the fifth pattern.

[0092] The information processing device 100 sets the self-checkout register 10 as the notification destination when it detects that a user has picked up multiple items stacked on top of each other and that the items have passed through the scanning area without overlapping. This prevents the management terminal 60 from mistakenly notifying the user's scanning operation corresponding to the first and second patterns as an unauthorized operation.

[0093] The information processing device 100 notifies the administrator terminal when it detects that a user has stacked multiple items together and removed them a predetermined number of times or more. This allows the administrator to be notified of the existence of a user who is repeatedly making incorrect operations on the self-checkout machine 10.

[0094] The information processing device 100 inputs a vector of image (frame) video information and a vector of text related to the number of items into CLIP to determine whether or not items overlap. This allows for accurate determination of item overlap.

[0095] By the way, the processing performed by the information processing device 100 described above is just one example, and the information processing device 100 may perform other processing. For example, the information processing device 100 does not have to immediately notify the management terminal 60 of alert image information when it detects fraud in any of the third to fifth patterns of self-checkout by the processing described above. The information processing device 100 will notify the management terminal 60 of alert image information when any of the following conditions (1), (2), or (3) are met. The administrator may set any of these conditions in the information processing device 100.

[0096] Condition (1) is that the price of unscanned items is equal to or greater than the first threshold. Condition (2) is that the cumulative price of unscanned items is equal to or greater than the second threshold (the second threshold is greater than the first threshold). Condition (3) is that the number of times fraud is detected at the self-checkout using patterns 3 to 5 is equal to or greater than a predetermined number.

[0097] The information processing device 100 may determine the price of unscanned products in any way. For example, when the determination unit 154 of the information processing device 100 determines whether multiple products have been scanned correctly, if there are images of products in the image data group corresponding to the track IDs in the tracking table 143 that do not match the scanned product images, it identifies the products with such images as unscanned products. The determination unit 154 identifies the price of the product with the highest similarity between the image of the unscanned product and the product image in the product information table 142 as the price of the unscanned product.

[0098] Next, we will describe in order an example of a computer hardware configuration that realizes the same functions as the information processing device 100 shown in the above embodiment.

[0099] Figure 18 shows an example of a computer hardware configuration that realizes similar functions to the information processing device in the embodiment. As shown in Figure 18, the computer 200 has a CPU 201 that performs various calculations, an input device 202 that receives data input from the user, and a display 203. The computer 200 also has a communication device 204 and an interface device 205 that exchange data with the self-checkout 10, camera 20, management terminal 60, and other external devices via a wired or wireless network. The computer 200 also has a RAM 206 for temporarily storing various information and a hard disk drive 207. Each of the devices 201 to 207 is connected to a bus 208.

[0100] The hard disk drive 207 includes an acquisition program 207a, an operation detection program 207b, a product identification program 207c, a judgment program 207d, and a notification processing program 207e. The CPU 201 reads each of the programs 207a to 207e and loads them into the RAM 206.

[0101] The acquisition program 207a functions as the acquisition process 206a. The motion detection program 207b functions as the motion detection process 206b. The product identification program 207c functions as the product identification process 206c. The judgment program 207d functions as the judgment process 206d. The notification processing program 207e functions as the notification processing process 206e.

[0102] The processing of acquisition process 206a corresponds to the processing of acquisition unit 151. The processing of motion detection process 206b corresponds to the processing of motion detection unit 152. The processing of product identification process 206c corresponds to the processing of product identification unit 153. The processing of determination process 206d corresponds to the processing of determination unit 154. The processing of notification processing process 206e corresponds to the processing of notification processing unit 155.

[0103] Furthermore, programs 207a to 207e do not necessarily have to be stored on the hard disk drive 207 from the beginning. For example, each program could be stored on a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card inserted into the computer 200. Then, the computer 200 could read and execute each program 207a to 207e.

[0104] With regard to embodiments including each of the above examples, the following additional information is disclosed.

[0105] (Note 1) Obtain video of the scanning area of ​​a self-checkout terminal that scans product barcodes. By analyzing the acquired video footage, the user's actions of grasping multiple items are detected in the scanning area of ​​the self-checkout terminal. Based on the analysis results of the type of overlapping state of the multiple products when the user's action of grasping the multiple products is detected, either the self-checkout terminal or the management terminal that monitors the self-checkout terminal is set as the notification destination terminal. Alert image information is generated according to the type of overlapping state of the aforementioned multiple products. The generated alert image information is displayed on the screen of the designated notification recipient device. An information processing program characterized by having a computer perform the processing.

[0106] (Note 2) The information processing program described in Note 1 is characterized in that, based on the analysis results, when it is detected that multiple unscanned products have passed through the scan area while overlapping, the management terminal is set as the notification destination.

[0107] (Note 3) The information processing program described in Note 1, characterized in that, based on the analysis results, when it is detected that the scanned product has passed through the scan area multiple times, the management terminal is set as the notification destination.

[0108] (Note 4) The information processing program described in Note 1, characterized in that, based on the analysis results, when it is detected that scanned items and unscanned items have passed through the scan area while overlapping, the management terminal is set as the notification destination.

[0109] (Note 5) The information processing program described in Note 1, characterized in that, based on the analysis results, when it is detected that the user has picked up multiple items stacked on top of each other and that the items have passed through the scanning area without overlapping, the self-checkout terminal is set as the notification destination.

[0110] (Note 6) The information processing program described in Note 5, characterized in that, based on the analysis results, when the user is detected to have taken out multiple items stacked on top of each other a predetermined number of times, the management terminal is added to the notification destination.

[0111] (Note 7) The information processing program according to Note 6, characterized in that if the user takes out multiple items stacked on top of each other less than a predetermined number of times, the computer further executes the process of displaying an alert image information indicating overlapping items on the self-checkout terminal, and if the user takes out multiple items stacked on top of each other more than a predetermined number of times, the computer further executes the process of displaying an alert image information requesting the payment to be stopped on the self-checkout terminal.

[0112] (Note 8) The information processing program described in Note 1, characterized in that it inputs the image vectors contained in the video and the text vectors relating to the number of products into a CLIP (Contrastive Language-Image Pretraining) model, thereby causing the computer to further perform a process to determine whether or not the products overlap.

[0113] (Note 9) Obtain video of the scanning area of ​​a self-checkout terminal that scans product barcodes. By analyzing the acquired video footage, the user's actions of grasping multiple items are detected in the scanning area of ​​the self-checkout terminal. Based on the analysis results of the type of overlapping state of the multiple products when the user's action of grasping the multiple products is detected, either the self-checkout terminal or the management terminal that monitors the self-checkout terminal is set as the notification destination terminal. Alert image information is generated according to the type of overlapping state of the aforementioned multiple products. The generated alert image information is displayed on the screen of the designated notification recipient device. An information processing device having a control unit.

[0114] (Note 10) The information processing device according to Note 9, characterized in that the setting process, based on the analysis results, sets the management terminal as the notification destination when it detects that multiple unscanned products have passed through the scan area while overlapping.

[0115] (Note 11) The information processing device according to Note 9, characterized in that the setting process, based on the analysis results, detects that the scanned product has passed through the scan area multiple times, and sets the management terminal as the notification destination.

[0116] (Note 12) The information processing device according to Note 9, characterized in that the setting process, based on the analysis results, sets the management terminal as the notification destination when it detects that scanned items and unscanned items have passed through the scan area in an overlapping state.

[0117] (Note 13) The information processing device according to Note 9, characterized in that the setting process, based on the analysis results, detects that the user has picked up multiple items stacked on top of each other and that the items have passed through the scanning area without overlapping, and sets the self-checkout terminal as the notification destination.

[0118] (Note 14) The information processing device according to Note 13, characterized in that the setting process is, based on the analysis results, when the user is detected to have taken out multiple items stacked on top of each other a predetermined number of times, the management terminal is added to the notification destination.

[0119] (Note 15) The information processing apparatus according to Note 14, characterized in that the control unit further performs the process of displaying an alert image information indicating overlapping products on the self-checkout terminal if the user takes out multiple products stacked on top of each other less than a predetermined number of times, and displaying an alert image information requesting the payment to be stopped on the self-checkout terminal if the user takes out multiple products stacked on top of each other more than a predetermined number of times.

[0120] (Note 16) The information processing apparatus according to Note 9, characterized in that the control unit further performs a process to determine whether or not products overlap by inputting the image vectors included in the video and the text vectors relating to the number of products into a CLIP (Contrastive Language-Image Pretraining) based model.

[0121] (Note 17) An information processing system having a self-checkout terminal, a management terminal, and an information processing device, The aforementioned information processing device is A video image of the scanning area of ​​the self-checkout terminal that scans the barcode of a product is acquired. By analyzing the acquired video footage, the user's actions of grasping multiple items are detected in the scanning area of ​​the self-checkout terminal. Based on the analysis results of the type of overlapping state of the multiple products when the user's action of grasping the multiple products is detected, either the self-checkout terminal or the management terminal that monitors the self-checkout terminal is set as the notification destination terminal. Alert image information is generated according to the type of overlapping state of the aforementioned multiple products. The control unit has a function that displays the generated alert image information on the display screen of the terminal designated as the notification destination. An information processing system characterized by the following:

[0122] (Note 18) The information processing system described in Note 17, characterized in that the setting process, based on the analysis results, sets the management terminal as the notification destination when it detects that multiple unscanned items have passed through the scan area while overlapping.

[0123] (Note 19) The information processing system according to Note 17, characterized in that the setting process, based on the analysis results, detects that the scanned product has passed through the scan area multiple times, and sets the management terminal as the notification destination.

[0124] (Note 20) The information processing system according to Note 17, characterized in that the setting process, based on the analysis results, sets the management terminal as the notification destination when it is detected that scanned items and unscanned items have passed through the scan area while overlapping.

[0125] (Note 21) The information processing system described in Note 17, characterized in that the setting process, based on the analysis results, detects that the user has picked up multiple items stacked on top of each other and that the items have passed through the scanning area without overlapping, and sets the self-checkout terminal as the notification destination.

[0126] (Note 22) The information processing system according to Note 21, characterized in that the setting process is, based on the analysis results, when the user is detected to have taken out multiple items stacked on top of each other a predetermined number of times, the management terminal is added to the notification destination.

[0127] (Note 23) The information processing system according to Note 22, characterized in that the control unit further performs the following: if the user takes out multiple items stacked on top of each other less than a predetermined number of times, it displays an alert image information indicating overlapping items on the self-checkout terminal; and if the user takes out multiple items stacked on top of each other more than a predetermined number of times, it displays an alert image information requesting the payment to be stopped on the self-checkout terminal.

[0128] (Note 24) The information processing system according to Note 17, characterized in that the control unit further performs a process to determine whether or not products overlap by inputting the image vectors included in the video and the text vectors relating to the number of products into a CLIP (Contrastive Language-Image Pretraining) based model. [Explanation of symbols]

[0129] 100 Information Processing Devices 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 Video buffer 142 Product Information Table 143 Tracking Table 150 Control Unit 151 Acquisition Department 152 Motion detection unit 153 Product Specification Department 154 Judgment section 155 Notification Processing Unit

Claims

1. We acquire video footage of the scanning area of ​​a self-checkout terminal that scans product barcodes. By analyzing the acquired video footage, the user's actions of grasping multiple items are detected in the scanning area of ​​the self-checkout terminal. Based on the analysis results of the type of overlapping state of the multiple products when the user's action of grasping the multiple products is detected, either the self-checkout terminal or the management terminal that monitors the self-checkout terminal is set as the notification destination terminal. Alert image information is generated according to the type of overlapping state of the aforementioned multiple products. The generated alert image information is displayed on the screen of the designated notification recipient device. An information processing program characterized by having a computer perform the processing.

2. The information processing program according to claim 1 is characterized in that, based on the analysis results, when it is detected that multiple unscanned products have passed through the scan area while overlapping, the management terminal is set as the notification destination.

3. The information processing program according to claim 1, characterized in that the setting process, based on the analysis results, detects that the scanned product has passed through the scan area multiple times, and sets the management terminal as the notification destination.

4. The information processing program according to claim 1 is characterized in that, based on the analysis results, when it is detected that scanned items and unscanned items have passed through the scan area while overlapping, the management terminal is set as the notification destination.

5. The information processing program according to claim 1 is characterized in that, based on the analysis results, when it is detected that the user has picked up multiple items stacked on top of each other and that the items have passed through the scanning area without overlapping, the self-checkout terminal is set as the notification destination.

6. The information processing program according to claim 5, characterized in that the setting process, based on the analysis results, detects that the user has taken out multiple items stacked on top of each other a predetermined number of times, and adds the management terminal to the notification destination.

7. The information processing program according to claim 6, characterized in that if the user takes out multiple items stacked on top of each other less than a predetermined number of times, the computer is further instructed to display an alert image on the self-checkout terminal indicating that the items are stacked, and if the user takes out multiple items stacked on top of each other more than a predetermined number of times, the computer is further instructed to display an alert image on the self-checkout terminal requesting that payment be stopped.

8. The information processing program according to claim 1, characterized in that it inputs the image vectors contained in the video and the text vectors relating to the number of products into a model based on CLIP (Contrastive Language-Image Pretraining) to cause a computer to further perform a process to determine whether or not the products overlap.

9. We acquire video footage of the scanning area of ​​a self-checkout terminal that scans product barcodes. By analyzing the acquired video footage, the user's actions of grasping multiple items are detected in the scanning area of ​​the self-checkout terminal. Based on the analysis results of the type of overlapping state of the multiple products when the user's action of grasping the multiple products is detected, either the self-checkout terminal or the management terminal that monitors the self-checkout terminal is set as the notification destination terminal. Alert image information is generated according to the type of overlapping state of the aforementioned multiple products. The generated alert image information is displayed on the screen of the designated notification recipient device. An information processing device having a control unit.

10. An information processing system having a self-checkout terminal, a management terminal, and an information processing device, The aforementioned information processing device is A video image of the scanning area of ​​the self-checkout terminal that scans the barcode of a product is acquired. By analyzing the acquired video footage, the user's actions of grasping multiple items are detected in the scanning area of ​​the self-checkout terminal. Based on the analysis results of the type of overlapping state of the multiple products when the user's action of grasping the multiple products is detected, either the self-checkout terminal or the management terminal that monitors the self-checkout terminal is set as the notification destination terminal. Alert image information is generated according to the type of overlapping state of the aforementioned multiple products. The control unit has a function that displays the generated alert image information on the display screen of the terminal designated as the notification destination. An information processing system characterized by the following:

Images