Notification program, notification method, and information processing device

Abstract

To detect an account mistake of an item.SOLUTION: An information processor 100 acquires item information generated by an accounting machine reading the code of the item. The information processor 100 counts a first number representing the number of purchases of the item on the basis of the acquired item information. The information processor 100 acquires an image obtained by capturing an image of a shopping basket arranged in a predetermined area next to the accounting machine. The information processor 100 counts a second number representing the frequency of the item in the basket being taken out of the basket. The information processor 100 issues an alert on the basis of the first number and the second number.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a notification program and the like. [Background technology]

[0002] Self-checkout registers are becoming common in supermarkets, convenience stores, and other stores. A self-checkout register is a Point of Sale (POS) cash register system in which the user himself performs the entire process, from scanning the product's barcode to paying. For example, introducing self-checkout registers can reduce labor costs and prevent payment errors by store clerks.

[0003] On the other hand, at self-checkouts, it is necessary to detect user fraud, such as failure to read barcodes. To address this issue, there is a conventional technology that analyzes image data from a camera to track people in a store and identify the timing at which the tracked person picks up or moves an item. By using this conventional technology, it becomes possible to automatically determine whether a user has performed a barcode reading operation.

[0004] Fig. 17 is a diagram illustrating a conventional technique. In the example shown in Fig. 17, when image data 10 is input, a self-checkout area 10a is detected, and a scan area 10b of the self-checkout is detected. In the conventional technique, an area 10c of a product held by a user is detected, and when the detected product area 10c enters scan area 10b, it is determined that the user has performed a barcode reading operation. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-53019 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the above-mentioned conventional technology has a problem in that it is not possible to detect missed items in the transaction.

[0007] 17, there are users who move a product to be purchased to the scan area 10b of the self-checkout but do not notice that the barcode reading has failed, and users who pretend to scan the barcode in the scan area 10b. For example, if a user moves a barcode to the scan area 10b and then pretends to scan the barcode, the prior art will determine that the barcode reading operation has been performed.

[0008] In one aspect, the present invention aims to provide a notification program, a notification method, and an information processing device that can detect missed checkouts of products. [Means for solving the problem]

[0009] In the first proposal, a computer is caused to perform the following process: The computer acquires product information generated by the cash register reading a product code. The computer counts a first number indicating the number of products purchased based on the acquired product information. The computer acquires images of baskets placed in a predetermined area adjacent to the cash register. The computer counts a second number indicating the number of times products are removed from the basket. The computer issues an alert based on the first number and the second number. [Effects of the Invention]

[0010] It is possible to detect missed accounting of products. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a system according to the present embodiment. [Figure 2] FIG. 2 is a functional block diagram showing the configuration of the information processing device according to this embodiment. [Figure 3]FIG. 3 is a diagram illustrating an example of a data structure of product information. [Figure 4] FIG. 4 is a diagram for explaining the model information. [Figure 5] FIG. 5 is a diagram illustrating an example of the data structure of the data table. [Figure 6] FIG. 6 is a diagram illustrating an example of the data structure of the determination table. [Figure 7] FIG. 7 is a diagram for explaining the processing of the tracking unit. [Figure 8] FIG. 8 is a diagram for explaining the processing of the counting unit. [Figure 9] FIG. 9 is a flowchart (1) showing the processing procedure of the tracking process. [Figure 10] FIG. 10 is a flowchart (2) showing the processing procedure of the tracking process. [Figure 11] FIG. 11 is a flowchart illustrating a processing procedure of the information processing device according to the present embodiment. [Figure 12] FIG. 12 is a flowchart showing the procedure of the removal operation number counting process. [Figure 13] FIG. 13 is a diagram for explaining the other process (1). [Figure 14] FIG. 14 is a diagram for explaining the other process (2). [Figure 15] FIG. 15 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of the hardware configuration of a self-checkout register. [Figure 17] FIG. 17 is a diagram for explaining the prior art. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of a notification program, a notification method, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0013] 1 is a diagram illustrating an example of a system according to the present embodiment. As shown in FIG. 1, the system 5 includes a camera 30, a self-checkout register 50, an administrator terminal 60, and an information processing device 100.

[0014] The information processing device 100 is connected to the camera 30 and the self-checkout register 50. The information processing device 100 is connected to an administrator terminal 60 via a network 3. The camera 30 and the self-checkout register 50 may be connected to the information processing device 100 via the network 3.

[0015] Camera 30 is a camera that captures video of an area including self-checkout 50 and basket 2a. Camera 30 transmits video data to information processing device 100. In the following description, the video data will be referred to as "video data."

[0016] The video data includes multiple image frames in time series. Each image frame is assigned a frame number in ascending chronological order. One image frame is a still image captured by the camera 30 at a certain timing.

[0017] The self-checkout register 50 is a POS register system in which a user 2 purchasing a product performs operations from reading the product's barcode to paying for the purchase. For example, when the user 2 removes the product to be purchased from the basket 2a and moves it to the scanning area of ​​the self-checkout register 50, the self-checkout register 50 scans the product's barcode.

[0018] User 2 repeats the above operations, and when the scanning of the products is complete, he or she operates the touch panel or the like of the self-checkout register 50 to request payment. When the self-checkout register 50 accepts the payment request, it presents the number of products to be purchased, the purchase amount, etc., and executes the payment process. The self-checkout register 50 stores information about the products scanned by user 2 from the time that user 2 starts scanning until the time that user 2 requests payment in a memory unit, and transmits this information to the information processing device 100 as product information.

[0019] The manager terminal 60 is a terminal device used by the manager of the store. The manager terminal 60 receives notifications of alerts and the like from the information processing device 100.

[0020] The information processing device 100 is a device that notifies an alert to the administrator terminal 60 based on the number of times that the user 2, identified from the video data acquired from the camera 30, takes out products contained in the basket 2a and the number of purchased products identified from the product information. In the following description, the number of times that the user 2 takes out products contained in the basket 2a is referred to as the "number of take-out actions."

[0021] For example, since user 2 scans the barcode of a product after removing it from basket 2a, if the number of times the removal action is performed differs from the number of purchases, it can be said that a product has been omitted from the transaction. Therefore, information processing device 100 can detect omitted products by issuing an alert based on the number of removal actions and the number of purchases.

[0022] Next, an example of the configuration of the information processing device 100 shown in Fig. 1 will be described. Fig. 2 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 2, the information processing device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0023] The communication unit 110 executes data communication between the camera 30, the self-checkout register 50, the manager terminal 60, etc. For example, the communication unit 110 receives video data from the camera 30. The communication unit 110 receives product information from the self-checkout register 50.

[0024] The input unit 120 is an input device that inputs various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0025] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, or the like.

[0026] The storage unit 140 has a video buffer 141, product information 142, model information 143, a data table 144, a determination table 145, and removal operation count information 146. The storage unit 140 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0027] Video buffer 141 stores video data captured by camera 30. The video data includes multiple image frames in time series.

[0028] Product information 142 is information obtained from self-checkout 50, and includes information on products scanned by user 2 between the time the user starts scanning and the time the user makes a payment request. Fig. 3 is a diagram showing an example of the data structure of product information. As shown in Fig. 3, product information 142 associates date and time information with product identification information.

[0029] The date and time information indicates the date and time when the self-checkout 50 read the barcode of the product. The product identification information is information that uniquely identifies the product. For example, the first line in Figure 3 indicates that the barcode of the product with product identification information "item101" was scanned at the date and time "September 10, 2021, 10:13:30."

[0030] The model information 143 is a neural network (NN) that outputs information about an interaction between a user (human) and a product (object) when an image frame is input. For example, the model information 143 corresponds to HOID (Human Object Interaction Detection).

[0031] Fig. 4 is a diagram for explaining model information. As shown in Fig. 4, an image frame 31 is input to model information 143, and detection information 32 is output. The detection information 32 includes user area information 32a, product area information 32b, and interaction information 32c.

[0032] The user area information 32a indicates by coordinates (x, y coordinates of the upper left, x, y coordinates of the lower right) the area of ​​the user included in the image frame 31. The product area information 32b indicates by coordinates (x, y coordinates of the upper left, x, y coordinates of the lower right) the area of ​​the product included in the image frame 31. The product area information 32b also includes a class name specific to the product.

[0033] The interaction information 32c includes a probability value of an interaction between a user and a product detected from the image frame 31, and a class name of the interaction. The class name of the interaction is set to a class name such as "grasp (user grasps product)."

[0034] The model information 143 according to this embodiment outputs the detection information 32 only when there is an interaction between the user and the product. For example, when an image frame in which the user is holding a product is input to the model information 143, the detection information 32 is output. On the other hand, when an image frame in which the user is not holding a product is input to the model information 143, the detection information 32 is not output.

[0035] Data table 144 is a data table used when tracking products detected from each image frame. Fig. 5 is a diagram showing an example of the data structure of the data table. As shown in Fig. 5, data table 144 has a detection result table 144a, a tracking object table 144b, and a tracking inactive object table 144c.

[0036] The detection result table 144a is a table that holds the coordinates of the product area output from the model information 143. In the following explanation, the coordinates of the product area will be referred to as "product area coordinates." The product area coordinates are indicated by [first element, second element, third element, fourth element]. The first element indicates the upper left x coordinate of the product area. The second element indicates the upper left y coordinate of the product area. The third element indicates the lower right x coordinate of the product area. The fourth element indicates the lower right y coordinate of the product area.

[0037] The tracked object table 144b is a table that holds information about products being tracked. The tracked object table 144b has an ID (identification), product area coordinates, a lost count, and a stay count. The ID is identification information assigned to the product area coordinates. The product area coordinates indicate the coordinates of the product area.

[0038] The lost count indicates the number of image frames that are counted when a product corresponding to the product area coordinates is not detected, and the stay count indicates the number of image frames that are counted when a product corresponding to the product area coordinates is not moving.

[0039] The tracking pause object table 144c is a table that holds information about products whose tracking has been paused. The tracking pause object table 144c has an ID, product area coordinates, and a flag. The ID is identification information assigned to the product area coordinates. The product area coordinates indicate the coordinates of the product area.

[0040] The flag is information indicating whether or not the ID and product area coordinates of the tracking-paused object table 144c are to be returned to the tracking object table 144b. When the flag is set to "true", it indicates that the ID and product area coordinates of the corresponding record are to be returned to the tracking object table 144b. When the flag is set to "false", it indicates that the ID and product area coordinates of the corresponding record are not to be returned to the tracking object table 144b.

[0041] Returning to the explanation of FIG. 5, the judgment table 145 is a table used when counting the number of take-out operations. In the following explanation, the area of ​​the temporary table on which the basket 2a is placed, which is installed next to the self-checkout register 50, is referred to as the "basket area." In this embodiment, when the product area coordinates identified from the image frame move from inside the basket area to outside the basket area, 1 is added to the number of take-out operations. By using the judgment table 145, the information processing device 100 can ensure that even if the same product enters and leaves the basket area multiple times, only one item is added to the number of take-out operations.

[0042] Fig. 6 is a diagram showing an example of the data structure of the determination table. As shown in Fig. 6, this determination table 145 associates an ID, a previous frame position, and a counted flag. The ID is identification information assigned to the product area coordinates. The previous frame position is information that identifies whether the product area coordinates detected from the previous image frame are outside or inside the basket area.

[0043] Here, if the product area coordinates of the corresponding ID that are detected from the previous image frame are outside the basket area, "OUT" is set to the previous frame position. If the product area coordinates detected from the previous image frame are inside the basket area, "IN" is set to the previous frame position. The counted flag is a flag that identifies whether or not the process of adding 1 to the number of removal operations has been performed for the corresponding ID.

[0044] In this embodiment, the initial value of the counted flag is set to "false." When the previous image frame position of the product area coordinates of the corresponding ID is set to "IN" and the product area coordinates of the corresponding ID detected from the current image frame position become "OUT," the number of take-out operations is incremented by 1. In this case, the counted flag is updated from "false" to "true."

[0045] The take-out operation count information 146 includes information on the take-out operation count.

[0046] Returning to the explanation of Fig. 2, the control unit 150 has an acquisition unit 151, a tracking unit 152, a counting unit 153, and a determination unit 154. The control unit 150 is realized by, for example, a central processing unit (CPU) or a micro processing unit (MPU). The control unit 150 may also be executed by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0047] The acquisition unit 151 acquires video data from the camera 30 and stores the acquired video data in the video buffer 141. The acquisition unit 151 acquires product information 142 from the self-checkout register 50 and stores the acquired product information 142 in the storage unit 140.

[0048] The tracking unit 152 tracks product area coordinates based on the video data (time-series image frames) stored in the video buffer 141. For example, the tracking unit 152 repeatedly executes a process of sequentially inputting image frames into the model information 143 to identify product area coordinates and updating the data table 144. An example of the process performed by the tracking unit 152 will be described below.

[0049] The tracking unit 152 inputs the image frames stored in the video buffer 141 into the model information 143 and acquires the product area coordinates included in the detection information. The tracking unit 152 registers the product area coordinates in the detection result table 144a. In the following description, the product area coordinates in the detection result table 144a will be referred to as "first product area coordinates." The product area coordinates in the currently tracked object table 144b will be referred to as "second product area coordinates." The product area coordinates in the currently tracked object table 144c will be referred to as "third product area coordinates."

[0050] The tracking unit 152 calculates the "similarity" based on the distance between the centers of the coordinates of each product area to be compared. The shorter the distance between the centers of the coordinates of each product area to be compared, the larger the similarity value. The relationship between the distance between the centers and the similarity is assumed to be defined in advance.

[0051] The tracking unit 152 compares the first product area coordinates with each third product area coordinate in the tracking rest object table 144c to determine whether there is a pair of the first product area coordinates and the third product area coordinates whose similarity is equal to or greater than a threshold value Th1. The value of the threshold value Th1 is set in advance.

[0052] If there is a pair of first product area coordinates and third product area coordinates whose similarity is equal to or greater than the threshold Th1, the tracking unit 152 executes the following process on the tracking rest object table 144c. The tracking unit 152 sets the flag of an entry having third product area coordinates whose similarity with the first product area coordinates is equal to or greater than the threshold Th1 to "true." The tracking unit 152 also deletes from the detection result table 144a any entry having first product area coordinates whose similarity with the third product area coordinates is equal to or greater than the threshold Th1.

[0053] The tracking unit 152 compares the first product area coordinates with each second product area coordinate in the tracked object table 144b to identify the maximum value of similarity between the first product area coordinates and the second product area coordinates. If the maximum value of similarity is equal to or greater than threshold Th3, the tracking unit 152 determines that "the corresponding product is not moving." If the maximum value of similarity is less than threshold Th3 but equal to or greater than threshold Th2, the tracking unit 152 determines that "the corresponding product is traceable." If the maximum value of similarity is less than threshold Th2, the tracking unit 152 determines that "the corresponding product is not traceable." The values ​​of thresholds Th2 and Th3 are set in advance. However, the value of threshold Th3 is assumed to be greater than the value of threshold Th2.

[0054] Fig. 7 is a diagram for explaining the processing of the tracking unit. In case 1A of Fig. 7, the area of ​​the product specified by the first product area coordinates is defined as product area 20a, and the area of ​​the product specified by the second product area coordinates is defined as product area 21a. If the distance between product area 20a and product area 21a is less than distance lA (if the similarity based on the distance is equal to or greater than threshold value Th3), the tracking unit 152 determines that "the relevant product is not moving."

[0055] When the tracking unit 152 determines that the "corresponding product is not moving," it adds 1 to the stay count in the entry corresponding to the product area 21a (second product area coordinates) in the tracked object table 144b.

[0056] 7, the area of ​​the product specified by the first product area coordinates is defined as product area 20b, and the area of ​​the product specified by the second product area coordinates is defined as product area 21b. The tracking unit 152 determines that "the relevant product is traceable" when the distance between product area 20b and product area 21b is equal to or greater than distance 1A and less than distance 1B (when the similarity based on the distance is less than threshold value Th3 and equal to or greater than threshold value Th2).

[0057] If the tracking unit 152 determines that the "corresponding product is trackable," it updates the second product area coordinates to the first product area coordinates in the entry corresponding to the product area 21b (second product area coordinates) in the tracking object table 144b. The tracking unit 152 sets the stay count to 0 in the entry corresponding to the product area 21b (second product area coordinates) in the tracking object table 144b.

[0058] 7, the area of ​​the product specified by the first product area coordinates is defined as product area 20c, and the area of ​​the product specified by the second product area coordinates is defined as product area 21c. If the distance between product area 20c and product area 21c is equal to or greater than distance lB (if the similarity based on the distance is less than threshold value Th2), the tracking unit 152 determines that "the relevant product cannot be tracked."

[0059] If the tracking unit 152 determines that the "corresponding product cannot be tracked," it registers a new entry of the first product area coordinates corresponding to the product area 20c in the tracked object table 144b. When registering a new entry, the tracking unit 152 assigns a new ID and sets the stay count and lost count to 0.

[0060] Here, the tracking unit 152 adds 1 to the lost count for any entry in the tracking object table 144b that has second product area coordinates whose similarity with the first product area coordinates is not equal to or greater than the threshold Th2.

[0061] The tracking unit 152 extracts entries in the tracking object table 144b whose lost counters exceed a threshold value Th4. For any extracted entries whose stay counters have values ​​equal to or greater than a threshold value Th5, the tracking unit 152 moves the corresponding entries (ID, second product area coordinates) to the tracking inactive object table 144c and sets the flag to "false."

[0062] The tracking unit 152 deletes any of the extracted entries whose stay counter value is less than the threshold value Th5.

[0063] The tracking unit 152 moves any entry in the tracking pause object table 144c whose flag is "true" to the tracking object table 144b, and sets the stay counter to zero.

[0064] The tracking unit 152 repeatedly executes the above process each time a new entry is registered in the detection result table 144a, and updates the tracking object table 144b and the tracking inactive object table 144c.

[0065] Returning to the explanation of Fig. 2, the counting unit 153 counts the number of times a user has taken out a product contained in the basket 2a, based on the tracked object table 144b of the data table 144. The counting unit 153 registers the number of times the user has taken out a product contained in the basket 2a as take-out operation count information 146 in the storage unit 140. An example of the processing of the counting unit 153 will be described below.

[0066] FIG. 8 is a diagram for explaining the processing of the counting unit. Step S1 in FIG. 8 will be explained. It is assumed that the counting unit 153 holds the coordinates of the basket area 10e in advance. The counting unit 153 refers to the tracked object table 144b, and when an entry for a new ID is added, adds an entry with the same ID as the new ID to the determination table 145. When adding an entry to the determination table 145, the counting unit 153 sets the counted flag to "false." For ease of explanation, the following explanation will be given assuming that the ID added to the determination table 145 is ID "1." The ID assigned to the second product area coordinates corresponding to the product area 10c is ID "1."

[0067] The counting unit 153 compares the second product area coordinates of the entry with ID "1" in the tracked object table 144b with the basket area 10e. If the second product area coordinates are not included in the basket area 10e, the counting unit 153 sets the previous frame position of the entry with ID "1" to be added to the determination table 145 to "OUT." If the second product area coordinates are included in the basket area 10e, the counting unit 153 sets the previous frame position of the entry with ID "1" to be added to the determination table 145 to "IN." In the example shown in step S1 of FIG. 8, the product area 10c corresponding to the second product area coordinates is included in the basket area 10e, so the previous frame position of the entry with ID "1" to be added to the determination table 145 is set to "IN."

[0068] We now move on to an explanation of step S2 in Fig. 8. The counting unit 153 monitors the currently tracked object table 144b, and each time the currently tracked object table 144b is updated, it compares the second product area coordinates corresponding to ID "1" with the basket area 10e. When the second product area coordinates corresponding to ID "1" (product area 10c) move to an area not included in the basket area 10e, the counting unit 153 refers to the entry for ID "1" in the determination table 145, and refers to the previous frame position and the counted flag.

[0069] For the entry with ID "1" in the determination table 145, if the previous frame position is "IN" and the counted flag is "false", the counting unit 153 adds 1 to the number of removal operations. After adding 1 to the number of removal operations, the counting unit 153 updates the previous frame position to "OUT" and the counted flag to "true".

[0070] On the other hand, if the previous frame position is "OUT" or the counted flag is "true", the counting unit 153 skips the process of adding 1 to the number of removal operations.

[0071] The counting unit 153 repeatedly executes the above process every time an entry with a new ID is added to the tracking object table 144b. If the ID of the entry added to the tracking object table 144b is the same as the ID of an entry registered in the determination table 145, the counting unit 153 skips the process of registering the entry corresponding to the new ID in the determination table 145.

[0072] The determination unit 154 notifies the administrator terminal 60 of an alert based on the product information 142 and the removal operation count information 146. An example of the processing of the determination unit 154 will be described below.

[0073] The determining unit 154 acquires the product information 142 and identifies the number of purchases. For example, the determining unit 154 identifies the number of records in the product information 142 with different date and time information as the number of purchases.

[0074] If the number of purchases differs from the number of take-out operations in the take-out operation count information 146, the determination unit 154 sends an alert to the administrator terminal 60. For example, if the number of purchases is less than the number of take-out operations, there is a risk of a missed transaction, so if the number of purchases is less than the number of take-out operations, the determination unit 154 sends an alert to the administrator terminal 60 to notify that a missed transaction has been detected.

[0075] On the other hand, if the purchase quantity and the number of take-out operations in the take-out operation number information 146 match, the determination unit 154 skips the process of notifying an alert.

[0076] Next, an example of tracking processing executed by the tracking unit 152 of the information processing device 100 according to this embodiment will be described. Figures 9 and 10 are flowcharts showing the processing procedure of the tracking processing. As shown in Figure 9, the tracking unit 152 of the information processing device 100 initializes the tracking object table 144b and the tracking paused object table 144c (step S101).

[0077] The tracking unit 152 acquires the detection information by acquiring the image frame from the video buffer 141 and inputting it into the model information 143 (Step S102). The tracking unit 152 registers the first product area coordinates included in the detection information in the detection result table 144a (Step S103).

[0078] The tracking unit 152 determines whether or not there is an entry in which the similarity between the first product area coordinates and the third product area coordinates in the tracking rest object table 144c is equal to the threshold value Th1 (step S104). If there is an entry (step S105, Yes), the tracking unit 152 proceeds to step S106. On the other hand, if there is no entry (step S105, No), the tracking unit 152 proceeds to step S108.

[0079] The tracking unit 152 sets the flag of the corresponding entry in the tracking pause object table 144c to "true" (step S106), and deletes the corresponding entry from the detection result table 144a (step S107).

[0080] The tracking unit 152 determines whether or not there is an entry in which the similarity between the first product area coordinates and the second product area coordinates in the tracked object table 144b is equal to or greater than a threshold value Th2 (step S108). If there is an entry (step S109, Yes), the tracking unit 152 proceeds to step S110. On the other hand, if there is no entry (step S109, No), the tracking unit 152 proceeds to step S115 in FIG. 10.

[0081] The tracking unit 152 updates the second product area coordinates of the corresponding entry in the currently tracked object table 144b to the first product area coordinates (step S110). The tracking unit 152 determines whether there is an entry in which the similarity between the first product area coordinates and the second product area coordinates in the currently tracked object table 144b is equal to or greater than a threshold value Th3 (step S111).

[0082] If an entry exists (Yes at step S112), the tracking unit 152 adds 1 to the stay count of the entry in the tracking object table 144b (step S113), and proceeds to step S115 in FIG.

[0083] On the other hand, if there is no entry (step S112, No), the tracking unit 152 updates the stay count of the corresponding entry in the tracking object table 144b to 0 (step S114), and proceeds to step S115 in FIG.

[0084] 10 will be explained. The tracking unit 152 adds an entry in which a new ID is assigned to the first product area information whose similarity with the second product area coordinates is less than the threshold Th2 to the tracked object table 144b (step S115). The tracking unit 152 sets the stay count of the entry added to the tracked object table 144b to 0 (step S116).

[0085] The tracking unit 152 adds 1 to the lost count of an entry having second product area coordinates whose similarity to the first product area coordinates is less than the threshold value Th2 among the entries in the tracked object table 144b (step S117).

[0086] The tracking unit 152 determines whether or not there is an entry in the tracked object table 144b whose stay counter value is equal to or greater than threshold value Th5 (step S118). If there is an entry (step S119, Yes), the tracking unit 152 proceeds to step S120. On the other hand, if there is no entry (step S119, No), the tracking unit 152 proceeds to step S121.

[0087] The tracking unit 152 moves entries whose stay counter values ​​are equal to or greater than the threshold value Th5 to the tracking paused object table 144c and sets the flags to "false" (step S120). The tracking unit 152 moves entries whose flags are "true" in the tracking paused object table 144c to the tracking in progress object table 144b and sets the stay count to 0 (step S122). The tracking unit 152 also deletes entries whose stay counter values ​​are equal to or greater than the threshold value Th5 (step S121) and proceeds to step S122.

[0088] If the tracking unit 152 continues the process (step S123, Yes), the process proceeds to step S102 in Fig. 9. On the other hand, if the tracking unit 152 does not continue the process (step S123, No), the tracking unit 152 ends the process.

[0089] Next, a processing procedure of the information processing device according to this embodiment will be described. Fig. 11 is a flowchart showing the processing procedure of the information processing device according to this embodiment. As shown in Fig. 11, the acquisition unit 151 of the information processing device 100 acquires product information 142 from the self-checkout register 50 and stores it in the memory unit 140 (step S201).

[0090] The counting unit 153 of the information processing device 100 counts the number of purchases based on the product information (step S202). The counting unit 153 executes a process of counting the number of take-out operations (step S203).

[0091] The determination unit 154 of the information processing device 100 determines whether the purchase quantity and the number of take-out operations match (Step S204). If the purchase quantity and the number of take-out operations match (Step S205, Yes), the determination unit 154 ends the process.

[0092] On the other hand, if the purchase quantity and the number of take-out operations do not match (No at Step S205), the determining unit 154 notifies an alert to the manager terminal 60 (Step S206).

[0093] Next, an example of the processing procedure for counting the number of take-out operations described in step S203 of Fig. 11 will be described. Fig. 12 is a flowchart showing the processing procedure for counting the number of take-out operations. As shown in Fig. 12, the counting unit 153 of the information processing device 100 starts monitoring the tracking object table 144b (step S301).

[0094] If an entry for a new ID has been added to the tracking object table 144b (step S302, Yes), the counting unit 153 proceeds to step S303. If an entry for a new ID has not been added to the tracking object table 144b (step S302, No), the counting unit 153 proceeds to step S305.

[0095] The counting unit 153 identifies the previous frame position based on the second product area coordinates and basket area of ​​the entry for the new ID (step S303). The counting unit 153 adds the entry with the new ID, the previous frame position, and the counted flag set to "false" to the determination table 145 (step S304).

[0096] The counting unit 153 identifies the current frame position based on the second product area coordinates and the basket area corresponding to the ID of each entry in the determination table 145 (step S305). The counting unit 153 selects an unselected entry in the determination table 145 (step S306).

[0097] The counting unit 153 determines whether the conditions are met in which the previous frame position of the selected entry is “IN”, the counted flag is “false”, and the current frame position corresponding to the ID of the selected entry is “OUT” (step S307).

[0098] If the condition is met (step S308, Yes), the counting unit 153 proceeds to step S309. If the condition is not met (step S308, No), the counting unit 153 proceeds to step S311.

[0099] The counter 153 adds 1 to the number of take-out operations (step S309). The counter 153 updates the previous frame position of the selected entry to "OUT" and the counted flag to "true" (step S310).

[0100] If the counting unit 153 has not selected all the entries in the judgment table 145 (step S311, No), the counting unit 153 proceeds to step S306. If the counting unit 153 has selected all the entries in the judgment table 145 (step S311, Yes), the counting unit 153 proceeds to step S312.

[0101] If the counting unit 153 determines to continue the process (Yes at step S312), the process proceeds to step S302. If the counting unit 153 determines not to continue the process (No at step S312), the process of counting the number of removal operations ends.

[0102] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 issues an alert based on the number of purchases identified from product information 142 acquired from the self-checkout register 50 and the number of take-out operations counted by comparing the product area with the basket area. For example, if the number of take-out operations and the number of purchases differ, it can be said that a product has been omitted from the transaction, and the information processing device 100 can detect an omitted product by issuing an alert based on the number of take-out operations and the number of purchases.

[0103] The processing contents of the above-described embodiment are merely examples, and the information processing device 100 may further execute other processes. In the following description, other processes executed by the information processing device 100 will be described.

[0104] Other processing (1) executed by the information processing device 100 will be described. The counting unit 153 of the information processing device 100 executes the processing using a preset basket area, but this is not limited to this. The counting unit 153 may analyze the image frames registered in the video buffer 141, identify a first area corresponding to the basket area and a second area corresponding to the scan area, and count the number of removal operations using the identified first area.

[0105] FIG. 13 is a diagram illustrating another process (1). In the example shown in FIG. 13, a first area 40a and a second area 40b are identified from an image frame 40. The counting unit 153 may identify the first area 40a and the second area 40b using a conventional technique such as pattern matching, or may identify the first area 40a and the second area 40b using a machine learning model that has already been trained. For example, such a machine learning model is a model that executes machine learning using training data that uses an image frame as input and the coordinates of the first area and the second area as correct answer data.

[0106] If self-checkout register 50 moves or camera 30 changes position while counting unit 153 is performing processing, the number of take-out operations cannot be counted accurately if processing is performed using a preset basket area. In contrast, by analyzing the image frames registered in video buffer 141 and identifying a second area corresponding to the basket area, the basket area can be accurately identified, and the number of take-out operations can be counted accurately.

[0107] Other processing (2) executed by the information processing device 100 will be described. In the above-described information processing device 100, the number of purchases is counted based on the product information 142 acquired from the self-checkout 50, but this is not limited to this. When the self-checkout 50 executes the checkout process, the number of purchased products is displayed on the display screen. For this reason, the information processing device 100 may identify the number of purchases by performing image analysis on the image frame of the display screen captured by the camera 30 (or another camera).

[0108] Fig. 14 is a diagram for explaining other process (2). Image frame 41 in Fig. 14 corresponds to the display screen of self-checkout 50. Area 41a of image frame 41 includes area 41a indicating the number of purchased items. Counting unit 153 identifies the number of purchased items by performing image analysis of area 41a.

[0109] As described above, the counting unit 153 analyzes the image frame on the display screen of the self-checkout register 50 to identify the number of purchased items, and the information processing device 100 can compare the number of removal operations with the number of purchases and issue an alert even when not connected to the self-checkout register 50. Note that although the present embodiment has been described using a barcode, a QR (Quick Response) code or the like may also be used.

[0110] Next, an example of the hardware configuration of a computer that realizes the same functions as the information processing device 100 described in the above embodiment will be described. Fig. 15 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0111] 15, computer 200 has CPU 201 that executes various types of arithmetic processing, input device 202 that accepts data input from a user, and display 203. Computer 200 also has communication device 204 that exchanges data with camera 30, external devices, etc. via a wired or wireless network, and interface device 205. Computer 200 also has RAM 206 that temporarily stores various types of information, and hard disk drive 207. Each of devices 201 to 207 is connected to bus 208.

[0112] The hard disk drive 207 has an acquisition program 207a, a tracking program 207b, a counting program 207c, and a determination program 207d. The CPU 201 reads out each of the programs 207a to 207d and loads them into the RAM 206.

[0113] The acquisition program 207a functions as the acquisition process 206a. The tracking program 207b functions as the tracking process 206b. The counting program 207c functions as the counting process 206c. The determination program 207d functions as the determination process 206d.

[0114] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the tracking process 206b corresponds to the processing of the tracking unit 152. The processing of the counting process 206c corresponds to the processing of the counting unit 153. The processing of the determination process 206d corresponds to the processing of the determination unit 154.

[0115] It should be noted that each of the programs 207a to 207d does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207d.

[0116] Fig. 16 is a diagram illustrating an example of the hardware configuration of self-checkout register 50. As shown in Fig. 17, self-checkout register 50 has a communication interface 400a, an HDD 400b, a memory 400c, a processor 400d, an input unit 400e, and an output unit 400f. The units shown in Fig. 17 are connected to each other via a bus or the like.

[0117] The communication interface 400a is a network interface card or the like, and communicates with other information processing devices. The HDD 400b stores programs and data that operate each function of the self-checkout 50.

[0118] Processor 400d is a hardware circuit that reads out from HDD 400b or the like a program that executes the processing of each function of self-checkout 50 and loads it into memory 400c, thereby operating a process that executes each function of self-checkout 50. In other words, this process executes the same functions as each processing unit that self-checkout 50 has.

[0119] In this way, the self-checkout register 50 operates as an information processing device that executes operation control processing by reading and executing a program that executes the processing of each function of the self-checkout register 50. The self-checkout register 50 can also realize each function of the self-checkout register 50 by reading a program from a recording medium using a media reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the self-checkout register 50. For example, this embodiment may also be applied in a similar manner to cases where another computer or server executes the program, or where these execute the program in cooperation with each other.

[0120] The program that executes the processing of each function of self-checkout 50 can be distributed via a network such as the Internet. The program can also be recorded on a computer-readable recording medium such as a hard disk, FD, CD-ROM, MO, or DVD, and can be executed by being read from the recording medium by a computer.

[0121] The input device 400e detects various input operations by the user, such as input operations for a program executed by the processor 400d. The input operations include, for example, touch operations. In the case of touch operations, the self-checkout 50 further includes a display unit, and the input operation detected by the input device 400e may be a touch operation on the display unit. The input device 400e may be, for example, a button, a touch panel, or a proximity sensor. The input device 400e also reads barcodes. For example, the input device 400e is a barcode reader. The barcode reader has a light source and an optical sensor and scans barcodes.

[0122] The output device 400f outputs data output from the program executed by the processor 400d via an external device, such as an external display device, connected to the self-checkout 50. Note that if the self-checkout 50 has a display unit, the self-checkout 50 does not need to have the output device 400f.

[0123] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0124] (Note 1) The cashier reads the product code to obtain the product information, Counting a first number of times indicating the number of purchases of the product based on the acquired product information; Acquire an image of a basket placed in a predetermined area adjacent to the checkout machine; Counting a second number of times indicating the number of times the customer takes out the products contained in the basket; An alert is sent based on the first number of times and the second number of times. A notification program that causes a computer to execute a process.

[0125] (Supplementary Note 2) From the image of the area where the user pays for the product, a first area where a basket adjacent to the payment machine is placed and a second area where the payment machine reads the code of the product held by the user are identified; The notification program described in Appendix 1, wherein the process of counting the second number of times identifies the action of removing an item contained in the basket in the first area.

[0126] (Appendix 3) The notification program described in Appendix 1 or 2, characterized in that the notification process notifies the user by alert that the product has not been paid for, based on the difference between the first number of times and the second number of times.

[0127] (Appendix 4) The notification program described in Appendix 1 is characterized in that the process of acquiring the product information acquires the product information stored in the memory unit of the accounting machine by the accounting machine reading the code of the product.

[0128] (Appendix 5) The notification program described in Appendix 1 is characterized in that the process of acquiring the product information acquires the product information based on image information displayed on the display screen of the accounting machine when the accounting machine reads the code of the product.

[0129] (Appendix 6) The cashier reads the product code to obtain the product information, Counting a first number of times indicating the number of purchases of the product based on the acquired product information; Acquire an image of a basket placed in a predetermined area adjacent to the checkout machine; Counting a second number of times indicating the number of times the customer takes out the products contained in the basket; An alert is sent based on the first number of times and the second number of times. A notification method characterized in that processing is executed by a computer.

[0130] (Appendix 7) From the image of the area where the user pays for the product, a first area where a basket adjacent to the payment machine is placed and a second area where the payment machine reads the code of the product held by the user are identified; The notification method described in Appendix 6, wherein the process of counting the second number of times identifies the action of removing products contained in the basket in the first area.

[0131] (Appendix 8) The notification method described in Appendix 6 or 7, characterized in that the notification process notifies the user by alert that the product has not been paid for, based on the difference between the first number of times and the second number of times.

[0132] (Appendix 9) The notification method described in Appendix 6, characterized in that the process of acquiring the product information involves the cashier reading the product code to acquire the product information stored in the memory unit of the cashier.

[0133] (Appendix 10) The notification method described in Appendix 6 is characterized in that the process of acquiring the product information acquires the product information based on image information displayed on the display screen of the payment machine when the payment machine reads the product code.

[0134] (Appendix 11) An acquisition unit that acquires product information generated by the checkout machine reading the product code; a counting unit that counts a first number of times indicating the number of purchases of products based on the acquired product information, acquires images of baskets placed in a predetermined area adjacent to the checkout machine, and counts a second number of times indicating the number of times products are removed from the baskets; a determination unit that issues an alert based on the first number of times and the second number of times; An information processing device comprising:

[0135] (Appendix 12) The information processing device described in Appendix 11 is characterized in that the counting unit identifies, from an image taken of the area where the user pays for the product, a first area where a basket adjacent to the cash register is placed and a second area where the cash register reads the code of the product held by the user, and the process of counting the second number of times identifies the action of removing the product contained in the basket in the first area.

[0136] (Appendix 13) The information processing device described in Appendix 11 or 12, characterized in that the judgment unit notifies the user by alert that the product has not been paid for based on the difference between the first number of times and the second number of times.

[0137] (Appendix 14) The information processing device according to appendix 11, wherein the acquisition unit acquires the product information stored in the memory unit of the payment machine by the payment machine reading the code of the product.

[0138] (Appendix 15) The information processing device described in Appendix 11, characterized in that the acquisition unit acquires the product information based on image information displayed on the display screen of the payment machine when the payment machine reads the code of the product. [Explanation of symbols]

[0139] 30 Camera 50 Self-checkout 60 Administrator terminal 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 Video Buffer 142 Product information 143 Model Information 144 Data Tables 145 Decision Table 146 Information on the number of times of extraction 150 control section 151 Acquisition Department 152 Tracking Department 153 Counting Department 154 Judgment section

Claims

1. counting a first number of times indicating the number of purchased items based on an image frame captured from a display screen of the cashier that displays the number of purchased items; Acquire an image of a basket placed in a predetermined area adjacent to the cashier; Counting a second number of times indicating the number of times the customer takes out the products contained in the basket; An alert is sent based on the first number of times and the second number of times. A notification program that causes a computer to execute a process.

2. From an image of an area where the user pays for the product, a first area where a basket adjacent to the payment machine is placed and a second area where the payment machine reads the code of the product held by the user are identified; 2. The notification program according to claim 1, wherein the process of counting the second number of times identifies an action of taking out an item contained in the basket in the first area.

3. 3. The notification program according to claim 1, wherein the notification process notifies the user by an alert that the product has not been paid for, based on the difference between the first number of times and the second number of times.

4. 2. The notification program according to claim 1, wherein the computer is caused to execute a process of acquiring product information of the product stored in a memory unit of the payment machine by the payment machine reading the code of the product.

5. counting a first number of times indicating the number of purchased items based on an image frame captured from a display screen of the cashier that displays the number of purchased items; Acquire an image of a basket placed in a predetermined area adjacent to the cashier; Counting a second number of times indicating the number of times the customer takes out the products contained in the basket; An alert is sent based on the first number of times and the second number of times. A notification method characterized in that processing is executed by a computer.

6. a counting unit that counts a first number of times indicating the number of purchased items based on an image frame captured from a display screen of a checkout machine that displays the number of purchased items, and that acquires an image captured from a basket placed in a predetermined area adjacent to the checkout machine and counts a second number of times indicating the number of times an action of removing items contained in the basket occurs; a determination unit that issues an alert based on the first number of times and the second number of times; An information processing device comprising:

7. The cashier reads the product code and obtains the generated product information. Counting a first number of times indicating the number of purchases of the product based on the acquired product information; Acquire an image of a basket placed in a predetermined area adjacent to the cashier; Counting a second number of times indicating the number of times the customer takes out the products contained in the basket; An alert is sent based on the first number of times and the second number of times. Have the computer execute the process, The process of counting the second number of times includes: Each time a motion corresponding to the product is detected, Refer to a determination table that stores identification information for identifying the area coordinates of the product detected from the previous image, which is the previous image, and whether the area coordinates of the product are outside or inside the area of ​​the basket; A notification program characterized by counting the action as the second number of times when the area information of the product associated with the identification information is changed from inside to outside the area of ​​the basket, and suppressing the counting of the action as the second number of times when the area information of the product associated with the identification information is changed other than from inside to outside the area of ​​the basket.

Images