Notification program, notification method, and information processing apparatus

Abstract

To detect an account mistake of an item.SOLUTION: An information processor 100 acquires item information generated by an accounting machine reading the code of the item. The information processor 100 counts a first number representing the number of purchases of the item on the basis of the acquired item information, and specifies an action that a user made to register the item in the accounting machine from an image of the user in front of the accounting machine. The information processor 100 counts a second number representing the frequency of the actions having been done. The information processor 100 issues an alert on the basis of the first number and the second number.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a notification program and the like.

Background Art

[0002] Self-checkout has become widespread in stores such as supermarkets and convenience stores. Self-checkout is a POS (Point Of Sale) register system in which the user who purchases goods performs operations from reading the barcode of the goods to settlement. For example, by introducing self-checkout, it is possible to suppress labor costs and prevent settlement errors by store clerks.

[0003] On the other hand, in self-checkout, it is required to detect user fraud such as not reading the barcode. To address this issue, for example, there is a conventional technique that analyzes image data of a camera to track a person in a store and identifies the timing when the person being tracked picks up or moves a product. By using such a conventional technique, it becomes possible to automatically determine whether the user has performed a barcode reading operation.

[0004] FIG. 17 is a diagram for explaining the conventional technique. In the example shown in FIG. 17, when image data 10 is input, the self-checkout area 10a is detected, and the self-checkout scan area 10b is detected. In the conventional technique, the area 10c of the product held by the user is detected, and when the detected product area 10c enters the scan area 10b, it is determined that the user has performed a barcode reading operation.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the above-described prior art, there is a problem that accounting omissions of products cannot be detected.

[0007] In FIG. 17, even if the product to be purchased is moved to the self-checkout scanning area 10b, there are users who do not notice the failure to read the barcode, or users who pretend to have the barcode read in the scanning area 10b. For example, when a user pretends to have the barcode read after moving the barcode to the scanning area 10b, the prior art determines that the barcode reading operation has been performed.

[0008] In one aspect, an object of the present invention is to provide a notification program, a notification method, and an information processing apparatus capable of detecting accounting omissions of products.

Means for Solving the Problems

[0009] In the first aspect, the computer is caused to execute the following processing. The computer acquires product information generated by the accounting machine reading the product code. The computer counts a first number indicating the number of products purchased based on the acquired product information. The computer identifies, from an image of the user in front of the accounting machine, an operation in which the user registers a product with the accounting machine. The computer counts a second number indicating the number of times the operation has been performed. The computer issues an alert based on the first number and the second number.

Effects of the Invention

[0010] Accounting omissions of products can be detected.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

MODE FOR CARRYING OUT THE INVENTION

[0012] Hereinafter, embodiments of the notification program, notification method, and information processing apparatus disclosed in the present application will be described in detail with reference to the drawings. Note that the present invention is not limited by this embodiment.

EXAMPLE

[0013] FIG. 1 is a diagram showing an example of the system according to this embodiment. As shown in FIG. 1, this system 5 includes a camera 30, a self-checkout 50, an administrator terminal 60, and an information processing apparatus 100.

[0014] The information processing apparatus 100 is connected to the camera 30 and the self-checkout 50. The information processing apparatus 100 is connected to the administrator terminal 60 via the network 3. The camera 30 and the self-checkout 50 may be connected to the information processing apparatus 100 via the network 3.

[0015] The camera 30 is a camera that captures an image of the area including the self-checkout 50. The camera 30 transmits the video data of the image to the information processing apparatus 100. In the following description, the video data of the image is referred to as "video data".

[0016] The video data includes a plurality of image frames in time series. Each image frame is assigned a frame number in ascending order of time series. One image frame is a still image captured by the camera 30 at a certain timing.

[0017] The self-checkout 50 is a POS register system in which the user 2 who purchases a product performs operations from reading the barcode of the product to settlement. For example, when the user 2 moves the product to be purchased to the scan area of the self-checkout 50, the self-checkout 50 scans the barcode of the product.

[0018] The user 2 repeatedly executes the above operation. When the scanning of the product is completed, the user 2 operates the touch panel or the like of the self-checkout 50 to make a settlement request. When the self-checkout 50 receives the settlement request, it presents the number of products to be purchased, the purchase amount, etc., and executes the settlement process. The self-checkout 50 stores the information of the products scanned during the period from when the user 2 starts scanning to when the settlement request is made in the storage unit, and transmits it to the information processing apparatus 100 as product information.

[0019] The manager terminal 60 is a terminal device used by the store manager. The manager terminal 60 receives alerts and the like from the information processing device 100.

[0020] The information processing device 100 is a device that notifies the manager terminal 60 of an alert based on the number of times the user 2 identified from the video data acquired from the camera 30 performs the operation of registering a product at the self-checkout 50 and the number of purchases of the product identified from the product information. In the following description, the number of times the user 2 performs the operation of registering a product at the self-checkout 50 is referred to as the "number of registration operations".

[0021] For example, if the number of registration operations and the number of purchases are different, it can be said that there is an accounting omission of the product. Therefore, by the information processing device 100 notifying an alert based on the number of registration operations and the number of purchases, an accounting omission of the product can be detected.

[0022] Next, an example of the configuration of the information processing device 100 shown in FIG. 1 will be described. FIG. 2 is a functional block diagram showing the configuration of the information processing device according to the present embodiment. As shown in FIG. 2, the information processing device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0023] The communication unit 110 performs data communication among the camera 30, the self-checkout 50, the manager terminal 60, and the like. For example, the communication unit 110 receives video data from the camera 30. The communication unit 110 receives product information from the self-checkout 50.

[0024] The input unit 120 is an input device that inputs various types of information into the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0025] The display unit 130 is a display device that displays the information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, or the like.

[0026] The storage unit 140 has a video buffer 141, product information 142, model information 143, a data table 144, a determination table 145, and registration operation count information 146. The storage unit 140 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0027] The video buffer 141 stores the video data captured by the camera 30. The video data includes a plurality of image frames in time series.

[0028] The product information 142 is information obtained from the self-checkout 50 and has information on the products scanned during the period from when the user 2 starts scanning until a settlement request is made. FIG. 3 is a diagram showing an example of the data structure of the product information. As shown in FIG. 3, the product information 142 associates date and time information with product identification information.

[0029] The date and time information indicates the date and time when the self-checkout 50 reads the barcode of the product. The product identification information is information that uniquely identifies the product. For example, in the first row of FIG. 3, it is shown that the barcode of the product with the product identification information "item101" was scanned at the date and time "10:13:30 on September 10, 2021".

[0030] The model information 143 is an NN (Neural Network) that outputs information regarding the interaction between a user (human) and a product (object) when an image frame is input. For example, the model information 143 corresponds to HOID (Human Object Interaction Detection).

[0031] FIG. 4 is a diagram for explaining the model information. As shown in FIG. 4, by inputting the image frame 31 into the model information 143, detection information 32 is output. The detection information 32 includes user area information 32a, product area information 32b, and interaction information 32c.

[0032] The user area information 32a indicates the area of the user included in the image frame 31 by coordinates (the upper left x and y coordinates, the lower right x and y coordinates). The product area information 32b indicates the area of the product included in the image frame 31 by coordinates (the upper left x and y coordinates, the lower right x and y coordinates). Further, the product area information 32b includes a class name unique to the product.

[0033] The interaction information 32c includes a probability value of the interaction between the user and the product detected from the image frame 31 and a class name of the interaction. As the class name of the interaction, a class name such as "grasp (the user grasps the product)" is set.

[0034] The model information 143 according to this embodiment outputs the detection information 32 only when there is an interaction between the user and the product. For example, when an image frame in a state where the user is grasping the product is input to the model information 143, the detection information 32 is output. On the other hand, when an image frame in a state where the user is not grasping the product is input to the model information 143, the detection information 32 is not output.

[0035] The data table 144 is a data table used when tracking the products detected from each image frame. FIG. 5 is a diagram showing an example of the data structure of the data table. As shown in FIG. 5, the data table 144 has a detection result table 144a, a tracked object table 144b, and a suspended object table 144c.

[0036] The detection result table 144a is a table that holds the coordinates of the product area output from the model information 143. In the following description, the coordinates of the product area are referred to as "product area coordinates". The product area coordinates are indicated by [the first element, the second element, the third element, the fourth element]. The first element indicates the upper left x coordinate of the product area. The second element indicates the upper left y coordinate of the product area. The third element indicates the lower right x coordinate of the product area. The fourth element indicates the lower right y coordinate of the product area.

[0037] The tracked object table 144b is a table that holds information about the products being tracked. The tracked object table 144b has an ID (identification), product area coordinates, a lost count, and a stay count. The ID is identification information assigned to the product area coordinates. The product area coordinates indicate the coordinates of the product area.

[0038] The lost count indicates the number of image frames counted when the product corresponding to the product area coordinates is not detected. The stay count indicates the number of image frames counted when the product corresponding to the product area coordinates is in a non-moving state.

[0039] The tracked paused object table 144c is a table that holds information about the products whose tracking has been paused. The tracked paused object table 144c has an ID, product area coordinates, and a flag. The ID is identification information assigned to the product area coordinates. The product area coordinates indicate the coordinates of the product area.

[0040] The flag is information indicating whether to return the ID and product area coordinates of the tracked paused object table 144c to the tracked object table 144b. When the flag is set to "true", it indicates that the ID and product area coordinates of the corresponding record are to be returned to the tracked object table 144b. When the flag is set to "false", it indicates that the ID and product area coordinates of the corresponding record are not to be returned to the tracked object table 144b.

[0041] Return to the description of FIG. 5. The determination table 145 is a table used when counting the number of registration operations. In this embodiment, when the product area coordinates specified from the image frame move from outside the preset scan area to inside the scan area, 1 is added to the number of registration operations. By using the determination table 145, the information processing apparatus 100 can ensure that even if the same product enters and exits the scan area multiple times, the number of times added to the number of registration operations is 1.

[0042] FIG. 6 is a diagram showing an example of the data structure of the determination table. As shown in FIG. 6, this determination table 145 associates an ID, a previous frame position, and a counted flag. The ID is identification information assigned to the product area coordinates. The previous frame position is information for identifying whether the product area coordinates detected from the previous image frame are outside or inside the scan area.

[0043] Here, when the product area coordinates corresponding to the ID, and the product area coordinates detected from the previous image frame are outside the scan area, "OUT" is set in the previous frame position. When the product area coordinates detected from the previous image frame are inside the scan area, "IN" is set in the previous frame position. The counted flag is a flag for identifying whether or not a process of adding 1 to the number of registration operation times has been performed for the corresponding ID.

[0044] In this embodiment, "false" is set as the initial value of the counted flag. When the product area coordinates of the corresponding ID detected from the current image frame position are "IN" while the previous image frame position of the product area coordinates of the corresponding ID is set to "OUT", 1 is added to the number of registration operation times. In this case, the counted flag is updated from "false" to "true".

[0045] The registration operation count information 146 has information on the number of registration operations.

[0046] Returning to the description of FIG. 2. The control unit 150 includes an acquisition unit 151, a tracking unit 152, a counting unit 153, and a determination unit 154. The control unit 150 is realized by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Further, the control unit 150 may be executed by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0047] The acquisition unit 151 acquires video data from the camera 30 and stores the acquired video data in the video buffer 141. The acquisition unit 151 acquires product information 142 from the self-checkout 50 and stores the acquired product information 142 in the storage unit 140.

[0048] The tracking unit 152 tracks the product area coordinates based on the video data (time-series image frames) stored in the video buffer 141. For example, the tracking unit 152 repeatedly executes a process of inputting the image frames in order into the model information 143 to identify the product area coordinates and updating the data table 144. An example of the process of the tracking unit 152 will be described below.

[0049] The tracking unit 152 inputs the image frames stored in the video buffer 141 into the model information 143 and acquires the product area coordinates included in the detection information. The tracking unit 152 registers the product area coordinates in the detection result table 144a. In the following description, the product area coordinates in the detection result table 144a are referred to as "first product area coordinates". The product area coordinates in the tracking object table 144b are referred to as "second product area coordinates". The product area coordinates in the tracking pause object table 144c are referred to as "third product area coordinates".

[0050] Note that the tracking unit 152 calculates a "similarity" based on the distance between the centers of each product area coordinate to be compared. The similarity becomes a larger value as the distance between the centers of each product area coordinate to be compared is shorter. It is assumed that the relationship between the distance between the centers and the similarity is defined in advance.

[0051] The tracking unit 152 compares the first product area coordinates with each of the third product area coordinates in the tracking pause object table 144c to determine whether there is a pair of the first product area coordinates and the third product area coordinates for which the similarity is equal to or greater than the threshold Th1. The value of the threshold Th1 is set in advance.

[0052] When there exists a pair of the first product area coordinates and the third product area coordinates whose similarity is equal to or greater than the threshold Th1, the tracking unit 152 executes the following process in the tracking pause object table 144c. The tracking unit 152 sets the flag of the entry having the third product area coordinates whose similarity to the first product area coordinates is equal to or greater than the threshold Th1 to "true". Further, the tracking unit 152 deletes the entry having the first product area coordinates whose similarity to the third product area coordinates is equal to or greater than the threshold Th1 from the detection result table 144a.

[0053] The tracking unit 152 compares the first product area coordinates with each of the second product area coordinates in the tracking object table 144b, and identifies the maximum value of the similarity between the first product area coordinates and the second product area coordinates. When the maximum value of the similarity is equal to or greater than the threshold Th3, the tracking unit 152 determines that "the corresponding product is not moving". When the maximum value of the similarity is less than the threshold Th3 and equal to or greater than the threshold Th2, the tracking unit 152 determines that "the corresponding product can be tracked". When it is less than the threshold Th2, the tracking unit 152 determines that "the corresponding product cannot be tracked". The values of the thresholds Th2 and Th3 are set in advance. However, the value of the threshold Th3 is set to be larger than the value of the threshold Th2.

[0054] FIG. 7 is a diagram for explaining the process of the tracking unit. In Case 1A of FIG. 7, the area of the product specified by the first product area coordinates is defined as product area 20a, and the area of the product specified by the second product area coordinates is defined as product area 21a. When the distance between the product area 20a and the product area 21a is less than the distance lA (when the similarity based on the distance is equal to or greater than the threshold Th3), the tracking unit 152 determines that "the corresponding product is not moving".

[0055] When the tracking unit 152 determines that "the corresponding product is not moving", it adds 1 to the stay count in the entry corresponding to the product area 21a (the second product area coordinates) in the tracking object table 144b.

[0056] In Case 1B of FIG. 7, the area of the product specified by the first product area coordinates is defined as product area 20b, and the area of the product specified by the second product area coordinates is defined as product area 21b. When the distance between product area 20b and product area 21b is equal to or greater than distance 1A and less than lB (when the similarity based on the distance is less than threshold Th3 and equal to or greater than threshold Th2), the tracking unit 152 determines that "the corresponding product can be tracked".

[0057] When the tracking unit 152 determines that "the corresponding product can be tracked", in the entry corresponding to product area 21b (second product area coordinates) in the object-being-tracked table 144b, it updates the second product area coordinates to the first product area coordinates. The tracking unit 152 sets the stay count to 0 in the entry corresponding to product area 21b (second product area coordinates) in the object-being-tracked table 144b.

[0058] In Case 1C of FIG. 7, the area of the product specified by the first product area coordinates is defined as product area 20c, and the area of the product specified by the second product area coordinates is defined as product area 21c. When the distance between product area 20c and product area 21c is equal to or greater than distance lB (when the similarity based on the distance is less than threshold Th2), the tracking unit 152 determines that "the corresponding product cannot be tracked".

[0059] When the tracking unit 152 determines that "the corresponding product cannot be tracked", it registers a new entry of the first product area coordinates corresponding to product area 20c in the object-being-tracked table 144b. When registering a new entry, the tracking unit 152 assigns a new ID and sets the stay count to 0 and the lost count to 0.

[0060] Here, for each entry in the object-being-tracked table 144b that has second product area coordinates whose similarity to the first product area coordinates is less than threshold Th2, the tracking unit 152 adds 1 to the lost count.

[0061] The tracking unit 152 extracts, from each entry in the tracked object table 144b, the entry for which the lost counter exceeds the threshold Th4. Among the extracted entries, for the entries where the value of the stay counter is equal to or greater than the threshold Th5, the tracking unit 152 moves the corresponding entries (ID, second product area coordinates) to the tracked paused object table 144c and sets the flag to "false".

[0062] Among the extracted entries, for the entries where the value of the stay counter is less than the threshold Th5, the tracking unit 152 deletes the corresponding entries.

[0063] The tracking unit 152 moves, to the tracked object table 144b, the entries in the tracked paused object table 144c for which the flag is "true", and sets the stay counter to 0.

[0064] Each time a new entry is registered in the detection result table 144a, the tracking unit 152 repeatedly executes the above processing to update the tracked object table 144b and the tracked paused object table 144c.

[0065] Return to the description of FIG. 2. The counting unit 153 identifies the operation in which the user registers a product at the self-checkout 50 based on the tracked object table 144b of the data table 144, and counts the number of registration operations performed. The counting unit 153 registers the number of registration operations in the storage unit 140 as the registration operation count information 146. An example of the processing of the counting unit 153 will be described below.

[0066] FIG. 8 is a diagram for explaining the processing of the counting unit. The step S1 in FIG. 8 will be described. It is assumed that the counting unit 153 holds in advance the coordinates of the scan area 10b. The counting unit 153 refers to the tracked object table 144b, and when a new ID entry is added, it adds an entry with the same ID as the new ID to the determination table 145. When adding an entry to the determination table 145, the counting unit 153 sets the counted flag to "false". In the following description, for convenience of explanation, the ID added to the determination table 145 will be described as ID "1". The ID assigned to the second product area coordinates corresponding to the product area 10c of the product is ID "1".

[0067] The counting unit 153 compares the second product area coordinates of the entry with ID "1" in the tracked object table 144b with the scan area 10b. When the second product area coordinates are not included in the scan area 10b, the counting unit 153 sets the previous frame position of the entry with ID "1" added to the determination table 145 to "OUT". When the second product area coordinates are included in the scan area 10b, the counting unit 153 sets the previous frame position of the entry with ID "1" added to the determination table 145 to "IN". In the example shown in step S1 of FIG. 8, since the product area 10c corresponding to the second product area coordinates is not included in the scan area 10b, the previous frame position of the entry with ID "1" added to the determination table 145 is set to "OUT".

[0068] Proceeding to the description of step S2 in FIG. 8. The counting unit 153 monitors the tracked object table 144b, and each time the tracked object table 144b is updated, it compares the second product area coordinates corresponding to ID "1" with the scan area 10b. When the second product area coordinates (product area 10c) corresponding to ID "1" move to an area included in the scan area 10b, the counting unit 153 refers to the entry with ID "1" in the determination table 145 and refers to the previous frame position and the counted flag.

[0069] When the counting unit 153 is for the entry with ID "1" in the determination table 145, if the previous frame position is "OUT" and the counted flag is "false", it adds 1 to the registered operation count. Also, after adding 1 to the registered operation count, the counting unit 153 updates the previous frame position to "IN" and updates the counted flag to "true".

[0070] On the other hand, when the previous frame position of the counting unit 153 is "IN", or when the counted flag is "true", the counting unit 153 skips the process of adding 1 to the registered operation count.

[0071] The counting unit 153 repeatedly executes the above process each time a new ID entry is added to the tracked object table 144b. When the ID of the entry added to the tracked object table 144b is the same as the ID of the entry registered in the determination table 145, the counting unit 153 skips the process of registering the entry corresponding to the new ID in the determination table 145.

[0072] The determination unit 154 notifies an alert to the administrator terminal 60 based on the product information 142 and the registered operation count information 146. An example of the process of the determination unit 154 will be described below.

[0073] The determination unit 154 acquires the product information 142 and specifies the purchase count. For example, the determination unit 154 specifies the number of records with different date and time information in the product information 142 as the purchase count.

[0074] When the purchase count is different from the registered operation count in the registered operation count information 146, the determination unit 154 sends an alert to the administrator terminal 60. For example, when the purchase count is less than the registered operation count, there is a risk of accounting omission. Therefore, when the purchase count is less than the registered operation count, the determination unit 154 notifies the administrator terminal 60 of an alert indicating that an accounting omission has been detected.

[0075] On the other hand, when the number of purchases matches the number of registration operations in the registration operation count information 146, the determination unit 154 skips the process of notifying an alert.

[0076] Next, an example of the tracking process executed by the tracking unit 152 of the information processing apparatus 100 according to the present embodiment will be described. FIGS. 9 and 10 are flowcharts showing the processing procedure of the tracking process. As shown in FIG. 9, the tracking unit 152 of the information processing apparatus 100 initializes the tracking object table 144b and the tracking suspended object table 144c (step S101).

[0077] The tracking unit 152 acquires an image frame from the video buffer 141 and inputs it to the model information 143 to obtain detection information (step S102). The tracking unit 152 registers the first product area coordinates included in the detection information in the detection result table 144a (step S103).

[0078] The tracking unit 152 determines whether there is an entry in which the similarity between the first product area coordinates and the third product area coordinates in the tracking suspended object table 144c becomes the threshold Th1 (step S104). When there is an entry (step S105, Yes), the tracking unit 152 proceeds to step S106. On the other hand, when there is no entry (step S105, No), the tracking unit 152 proceeds to step S108.

[0079] The tracking unit 152 sets the flag of the corresponding entry in the tracking suspended object table 144c to "true" (step S106). The tracking unit 152 deletes the corresponding entry from the detection result table 144a (step S107).

[0080] The tracking unit 152 determines whether there is an entry in which the similarity between the first product area coordinates and the second product area coordinates in the object table 144b being tracked is equal to or greater than the threshold Th2 (step S108). If there is an entry (step S109, Yes), the tracking unit 152 proceeds to step S110. On the other hand, if there is no entry (step S109, No), the tracking unit 152 proceeds to step S115 in FIG. 10.

[0081] The tracking unit 152 updates the second product area coordinates of the corresponding entry in the object table 144b being tracked to the first product area coordinates (step S110). The tracking unit 152 determines whether there is an entry in which the similarity between the first product area coordinates and the second product area coordinates in the object table 144b being tracked is equal to or greater than the threshold Th3 (step S111).

[0082] If there is an entry (step S112, Yes), the tracking unit 152 adds 1 to the stay count of the corresponding entry in the object table 144b being tracked (step S113) and proceeds to step S115 in FIG. 10.

[0083] On the other hand, if there is no entry (step S112, No), the tracking unit 152 updates the stay count of the corresponding entry in the object table 144b being tracked to 0 (step S114) and proceeds to step S115 in FIG. 10.

[0084] Proceed to the description of FIG. 10. The tracking unit 152 adds an entry in which a new ID is assigned to the first product area information whose similarity to the second product area coordinates is less than the threshold Th2 to the object table 144b being tracked (step S115). The tracking unit 152 sets the stay count of the entry added to the object table 144b being tracked to 0 (step S116).

[0085] The tracking unit 152 adds 1 to the loss count of an entry in the object table 144b being tracked that has second product area coordinates whose similarity to the first product area coordinates is less than the threshold Th2.

[0086] The tracking unit 152 determines whether there is an entry in the tracked object table 144b whose stay counter value is equal to or greater than the threshold Th5 (step S118). If there is an entry (step S119, Yes), the tracking unit 152 proceeds to step S121. On the other hand, if there is no entry (step S119, No), the tracking unit 152 proceeds to step S121.

[0087] The tracking unit 152 moves an entry whose stay counter value is equal to or greater than the threshold Th5 to the tracked paused object table 144c and sets the flag to "false" (step S120). The tracking unit 152 moves an entry in the tracked paused object table 144c whose flag is "true" to the tracked object table 144b and sets the stay count to 0 (step S122). Note that the tracking unit 152 deletes an entry whose stay counter value is equal to or greater than the threshold Th5 (step S121) and proceeds to step S122.

[0088] If the tracking unit 152 continues the process (step S123, Yes), it proceeds to step S102 in FIG. 9. On the other hand, if the tracking unit 152 does not continue the process (step S123, No), it ends the process.

[0089] Next, the processing procedure of the information processing apparatus according to the present embodiment will be described. FIG. 11 is a flowchart showing the processing procedure of the information processing apparatus according to the present embodiment. As shown in FIG. 11, the acquisition unit 151 of the information processing apparatus 100 acquires the product information 142 from the self-checkout 50 and stores it in the storage unit 140 (step S201).

[0090] The counting unit 153 of the information processing apparatus 100 counts the number of purchases based on the product information (step S202). The counting unit 153 executes the registered operation count processing (step S203).

[0091] The determination unit 154 of the information processing apparatus 100 determines whether the number of purchases matches the number of registration operations (step S204). When the determination unit 154 determines that the number of purchases matches the number of registration operations (step S205, Yes), the process ends.

[0092] On the other hand, when the determination unit 154 determines that the number of purchases does not match the number of registration operations (step S205, No), it notifies the administrator terminal 60 of an alert (step S206).

[0093] Next, an example of the processing procedure of the registration operation count processing described in step S203 of FIG. 11 will be described. FIG. 12 is a flowchart showing the processing procedure of the registration operation count processing. As shown in FIG. 12, the counting unit 153 of the information processing apparatus 100 starts monitoring the tracked object table 144b (step S301).

[0094] When a new ID entry is added to the tracked object table 144b (step S302, Yes), the counting unit 153 proceeds to step S303. When a new ID entry is not added to the tracked object table 144b (step S302, No), the counting unit 153 proceeds to step S305.

[0095] The counting unit 153 identifies the previous frame position based on the second product area coordinates of the new ID entry and the scan area (step S303). The counting unit 153 adds an entry with the new ID, the previous frame position, and the counted flag set to "false" to the determination table 145 (step S304).

[0096] The counting unit 153 identifies the current frame position based on the second product area coordinates corresponding to the ID of each entry in the determination table 145 and the scan area (step S305). The counting unit 153 selects an unselected entry in the determination table 145 (step S306).

[0097] The counter 153 determines whether the previous frame position of the selected entry is "OUT", the counted flag is "false", and the current frame position corresponding to the ID of the selected entry corresponding to the ID satisfies the condition of being "IN" (step S307).

[0098] If the counter 153 satisfies the condition (step S308, Yes), it proceeds to step S309. If the counter 153 does not satisfy the condition (step S308, No), it proceeds to step S311.

[0099] The counter 153 adds 1 to the registered operation count (step S309). The counter 153 updates the previous frame position of the selected entry to "IN" and the counted flag to "true" (step S310).

[0100] If the counter 153 has not selected all the entries in the determination table 145 (step S311, No), it proceeds to step S306. If the counter 153 has selected all the entries in the determination table 145 (step S311, Yes), it proceeds to step S312.

[0101] If the counter 153 continues the process (step S312, Yes), it proceeds to step S302. If the counter 153 does not continue the process (step S312, No), it ends the registered operation count processing.

[0102] Next, the effects of the information processing apparatus 100 according to the present embodiment will be described. The information processing apparatus 100 notifies an alert based on the purchase count specified from the product information 142 acquired from the self-checkout 50 and the registered operation count counted by comparing the product area and the scan area. For example, when the registered operation count and the purchase count are different, it can be said that there is an accounting omission of the product, and the information processing apparatus 100 can detect the accounting omission of the product by notifying an alert based on the registered operation count and the purchase count.

[0103] Incidentally, the processing content of the above-described embodiments is an example, and the information processing apparatus 100 may further execute other processing. In the following description, other processing executed by the information processing apparatus 100 will be described.

[0104] Other processing (1) executed by the information processing apparatus 100 will be described. The counting unit 153 of the information processing apparatus 100 has been executing processing using a preset scan area, but is not limited thereto. The counting unit 153 may analyze the image frame registered in the video buffer 141, identify a first area where the shopping basket is placed and a second area corresponding to the scan area, and use the identified second area to count the number of registration operations.

[0105] FIG. 13 is a diagram for explaining other processing (1). In the example shown in FIG. 13, a first area 40a and a second area 40b are identified from the image frame 40. The counting unit 153 may identify the first area 40a and the second area 40b using a conventional technique such as pattern matching, or may identify the first area 40a and the second area 40b using a machine learning model that has been machine-learned. For example, such a machine learning model is a model obtained by performing machine learning using teacher data having an image frame as an input and the coordinates of the first area and the second area as correct data.

[0106] When the self-checkout 50 moves or the position of the camera 30 is changed while the counting unit 153 is executing processing, if processing is executed using a preset scan area, the number of registration operations cannot be accurately counted. On the other hand, by analyzing the image frame registered in the video buffer 141 and identifying the second area corresponding to the scan area, the scan area can be accurately identified and the number of registration operations can be accurately counted.

[0107] Other processing (2) executed by the information processing apparatus 100 will be described. In the above information processing apparatus 100, the number of purchases was counted based on the product information 142 acquired from the self-checkout 50, but it is not limited to this. At the self-checkout 50, when performing the settlement process, the number of purchased products is displayed on the display screen. Therefore, the information processing apparatus 100 may identify the number of purchases by performing image analysis on the image frame of the display screen photographed by the camera 30 (or another camera).

[0108] FIG. 14 is a diagram for explaining other processing (2). The image frame 41 in FIG. 14 corresponds to the display screen of the self-checkout 50. The area 41a in the image frame 41 includes an area 41a indicating the number of purchased products. The counting unit 153 identifies the number of purchases by performing image analysis on the area 41a.

[0109] As described above, by the counting unit 153 analyzing the image frame of the display screen of the self-checkout 50 to identify the number of purchased products, the information processing apparatus 100 can compare the number of registration operation times with the number of purchases and notify an alert even if it is not connected to the self-checkout 50.

[0110] Next, an example of the hardware configuration of a computer that realizes the same functions as the information processing apparatus 100 shown in the above embodiment will be described. FIG. 15 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. In this embodiment, the description has been made using barcodes, but QR (Quick Response) codes or the like may also be used.

[0111] As shown in FIG. 15, the computer 200 includes a CPU 201 that executes various arithmetic processes, an input device 202 that receives input of data from a user, and a display 203. The computer 200 also includes a communication device 204 that exchanges data with the camera 30, an external device, etc. via a wired or wireless network, and an interface device 205. The computer 200 further includes a RAM 206 that temporarily stores various information, and a hard disk device 207. And each device 201 to 207 is connected to a bus 208.

[0112] The hard disk device 207 has an acquisition program 207a, a tracking program 207b, a counting program 207c, and a determination program 207d. The CPU 201 reads out each program 207a to 207d and expands it in the RAM 206.

[0113] The acquisition program 207a functions as an acquisition process 206a. The tracking program 207b functions as a tracking process 206b. The counting program 207c functions as a counting process 206c. The determination program 207d functions as a determination process 206d.

[0114] The process of the acquisition process 206a corresponds to the process of the acquisition unit 151. The process of the tracking process 206b corresponds to the process of the tracking unit 152. The process of the counting process 206c corresponds to the process of the counting unit 153. The process of the determination process 206d corresponds to the process of the determination unit 154.

[0115] Note that for each program 207a to 207d, it is not necessarily stored in the hard disk device 207 from the beginning. For example, each program is stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, IC card, etc. inserted into the computer 200. And the computer 200 may read out and execute each program 207a to 207d.

[0116] FIG. 16 is a diagram for explaining an example of the hardware configuration of the self-checkout 50. As shown in FIG. 16, the self-checkout 50 includes a communication interface 400a, an HDD 400b, a memory 400c, a processor 400d, an input unit 400e, and an output unit 400f. Further, each unit shown in FIG. 17 is interconnected by a bus or the like.

[0117] The communication interface 400a is a network interface card or the like and communicates with other information processing apparatuses. The HDD 400b stores programs and data for operating the respective functions of the self-checkout 50.

[0118] The processor 400d is a hardware circuit that operates a process for executing each function of the self-checkout 50 by reading a program for executing the processing of each function of the self-checkout 50 from the HDD 400b or the like and expanding it in the memory 400c. That is, this process executes the same functions as each processing unit included in the self-checkout 50.

[0119] In this way, the self-checkout 50 operates as an information processing apparatus that executes operation control processing by reading and executing a program for executing the processing of each function of the self-checkout 50. Further, the self-checkout 50 can also realize each function of the self-checkout 50 by reading a program from a recording medium by a medium reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the self-checkout 50. For example, the present embodiment may be similarly applied when another computer or server executes the program, or when these cooperate to execute the program.

[0120] Also, the program for executing the processing of each function of the self-checkout 50 can be distributed via a network such as the Internet. Further, this program is recorded on a computer-readable recording medium such as a hard disk, FD, CD-ROM, MO, DVD, and can be executed by being read from the recording medium by a computer.

[0121] The input device 400e detects various input operations by the user, such as input operations for programs executed by the processor 400d. The input operations include, for example, touch operations. In the case of a touch operation, the self-checkout 50 further includes a display unit, and the input operation detected by the input device 400e may be a touch operation on the display unit. The input device 400e may be, for example, a button, a touch panel, a proximity sensor, etc. Also, the input device 400e reads barcodes. The input device 400e is, for example, a barcode reader. The barcode reader has a light source and a photosensor and scans barcodes.

[0122] The output device 400f outputs data output from the program executed by the processor 400d to an external device connected to the self-checkout 50, such as an external display device, via the external device. When the self-checkout 50 includes a display unit, the self-checkout 50 may not include the output device 400f.

[0123] Regarding the embodiments including the above embodiments, the following additional remarks are further disclosed.

[0124] (Additional Remark 1) A notification program characterized in that a computer is made to execute a process of acquiring product information generated by a cash register reading a product code, counting a first number indicating the number of purchases of a product based on the acquired product information, identifying, from an image of the previous user of the cash register, an operation in which the user registers a product with the cash register, counting a second number indicating the number of times the operation has been performed, and notifying an alert based on the first number and the second number.

[0125] ​(Appendix 2) From the images taken of the area where the user conducts the accounting of the product, identify a first area where a basket adjacent to the accounting machine is placed and a second area where the accounting machine reads the code of the product held by the user. The process of identifying the operation is characterized in that in the second area, the operation of the user registering the product with the accounting machine is identified. The notification program according to Appendix 1.

[0126] (Appendix 3) The process of sending the notification is characterized in that based on the difference between the first number and the second number, an alert is sent to notify the user that the product has not been paid for. The notification program according to Appendix 1 or 2.

[0127] (Appendix 4) The process of obtaining the product information is characterized in that the product information stored in the storage unit of the accounting machine is obtained by the accounting machine reading the code of the product. The notification program according to Appendix 1.

[0128] (Appendix 5) The process of obtaining the product information is characterized in that the product information is obtained based on the image information displayed on the display screen of the accounting machine when the accounting machine reads the code of the product. The notification program according to Appendix 1.

[0129] (Appendix 6) Obtain the product information generated by the accounting machine reading the code of the product, Based on the obtained product information, count a first number indicating the number of products purchased, From the image of the previous user of the accounting machine, identify the operation of the user registering a product with the accounting machine Count a second number indicating the number of times the operation was performed, Based on the first number and the second number, send an alert A notification method characterized in that a computer executes the process.

[0130] (Appendix 7) From the images taken of the area where the user conducts the accounting of the product, identify a first area where a basket adjacent to the accounting machine is placed, and a second area where the accounting machine reads the code of the product held by the user. The process of identifying the operation is the notification method according to Appendix 6, characterized in that in the second area, the operation of the user registering the product with the accounting machine is identified.

[0131] (Appendix 8) The process of the notification is the notification method according to Appendix 6 or 7, characterized in that based on the difference between the first number and the second number, it is alerted and notified that the user has unpaid for the product.

[0132] (Appendix 9) The process of obtaining the product information is the notification method according to Appendix 6, characterized in that the product information stored in the storage unit of the accounting machine is obtained by the accounting machine reading the code of the product.

[0133] (Appendix 10) The process of obtaining the product information is the notification method according to Appendix 6, characterized in that the product information is obtained based on the image information displayed on the display screen of the accounting machine by the accounting machine reading the code of the product.

[0134] (Appendix 11) An acquisition unit that acquires product information generated by the accounting machine reading the code of the product, A counting unit that counts a first number indicating the number of purchased products based on the acquired product information, and counts a second number indicating the number of times the operation of the user registering the product with the accounting machine is identified from the image of the previous user of the accounting machine, A determination unit that notifies an alert based on the first number and the second number An information processing apparatus characterized by having the above.

[0135] (Appendix 12) The counting unit further specifies, from the image of the area where the user settles the merchandise, a first area where a basket adjacent to the settlement machine is placed, and a second area where the settlement machine reads the code of the merchandise held by the user, and in the second area, the information processing apparatus according to Appendix 11, characterized by identifying an operation in which the user registers the merchandise in the settlement machine.

[0136] (Appendix 13) The determination unit alerts and notifies that the user has unpaid merchandise based on the difference between the first count and the second count, and the information processing apparatus according to Appendix 11 or 12.

[0137] (Appendix 14) The acquisition unit acquires the merchandise information stored in the storage unit of the settlement machine when the settlement machine reads the code of the merchandise, and the information processing apparatus according to Appendix 11.

[0138] (Appendix 15) The acquisition unit acquires the merchandise information based on the image information displayed on the display screen of the settlement machine when the settlement machine reads the code of the merchandise, and the information processing apparatus according to Appendix 11.

Explanation of Signs

[0139] 30 Camera 50 Self-checkout 60 Administrator terminal 100 Information processing apparatus 110 Communication unit 120 Input unit 130 Display unit 140 Storage unit 141 Video buffer 142 Merchandise information 143 Model information 144 Data table 145 Determination table 146 Registration operation count information 150 Control unit 151 Acquisition unit 152 Tracking unit 153 Counting unit 154 determination unit

Claims

A notification program, characterized in that it causes a computer to execute a process of counting a first number indicating the number of purchases of a product based on an image frame obtained by imaging a display screen of a cash register that displays the number of purchases of the product, identifying, from an image of a user in front of the cash register, an operation in which the user registers a product with the cash register, counting a second number indicating the number of times the operation has been performed, and notifying an alert based on the first number and the second number.

2. Identifying, from an image of an area where a user settles the product, a first area where a basket adjacent to the cash register is placed and a second area where the cash register reads a code of the product held by the user, The notification program according to claim 1, wherein the process of identifying the operation identifies, in the second area, an operation in which a user registers the product with the cash register.

3. The notification program according to claim 1 or 2, wherein the process of notifying notifies, by an alert, that the user has not paid for the product based on a difference between the first number and the second number. A notification program according to claim 1, characterized in that it causes the computer to execute a process of acquiring product information of the product stored in a storage unit of the cash register by the cash register reading a code of the product.

5. The process of counting the second number is identifying a product area where the product exists in the image, assigning identification information to the identified product area, determining, for each of the product areas to which the identification information is assigned, whether or not the user has performed an operation of registering the product, if, as a result of the determination, the user has performed an operation of registering the product, adding 1 to the second number, and recording, in association with each other, the identification information and a counted flag indicating whether or not 1 has been added to the second number for the product area to which the identification information is assigned. A notification program, characterized in that it counts a first number indicating the number of purchases of a product based on an image frame obtained by imaging a display screen of a cash register that displays the number of purchases of the product, identifies, from an image of a user in front of the cash register, an operation in which the user registers a product with the cash register counts a second number indicating the number of times the operation has been performed, and notifies an alert based on the first number and the second number. ​ A notification method characterized in that a computer executes a process. According to claim 7, a counting unit that counts a first number indicating the number of purchases of the product based on an image frame obtained by imaging a display screen of a cash register that displays the number of purchases of the product, and identifies an operation in which the user registers a product in the cash register from an image of the previous user of the cash register and counts a second number indicating the number of times the operation has been performed. A determination unit that notifies an alert based on the first number and the second number. An information processing apparatus characterized by comprising the above.

Images