Action determination program, action determination method, and action determination device

Abstract

To prevent a commodity and personal possessions from being mixed up.SOLUTION: A behavior determination program causes a computer to execute: acquiring history information of user's operations of an accounting machine, which registers a commodity to be purchased and also performs accounting processing for the registered commodity; identifying, from an image of the user in front of the accounting machine, a behavior in which the user operates the accounting machine while grasping an object; and determining, based on the acquired history information, whether or not the behavior in which the user operates the accounting machine while grasping the object is a behavior of registering the commodity to be purchased with the accounting machine.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to behavior determination technology.

Background Art

[0002] Self-checkout has become widespread in stores such as supermarkets and convenience stores. Self-checkout is a POS (Point Of Sale) register system in which the user who purchases the goods performs operations from reading the barcode of the goods to settlement. For example, by introducing self-checkout, it is possible to suppress labor costs and prevent settlement errors by store clerks.

[0003] On the other hand, in self-checkout, it is required to detect user fraud such as not reading the barcode. To address this issue, for example, there is a conventional technique that analyzes the image data of a camera to track the people in the store and identifies the timing when the person being tracked picks up or moves a product. By using such a conventional technique, it becomes possible to automatically determine whether the user has performed a barcode reading operation.

[0004] FIG. 26 is a diagram for explaining the conventional technique. In the example shown in FIG. 26, when the image data 10 is input, the self-checkout area 10a is detected, and the self-checkout scan area 10b is detected. In the conventional technique, the area 10c of the product held by the user is detected, and when the detected product area 10c enters the scan area 10b, it is determined that the user has performed a barcode reading operation.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the above prior art, there is an aspect that goods may be confused with personal belongings, so-called private goods.

[0007] That is, in a store or the like, a huge variety of goods are sold, and goods with a short life cycle are sold. For this reason, even if existing object recognition technologies such as YOLO are used, it is difficult to generate a model that can identify each individual product, for example, up to the class name. This is one of the reasons why when an object that is not a product used by the user for payment or the like, such as a wallet, a card, a smartphone, a wearable terminal, etc., moves into the self-checkout scan area 10b shown in FIG. 24, it may be misjudged as a barcode reading operation.

[0008] In one aspect, an object of the present invention is to provide an action determination program, an action determination method, and an action determination device that can prevent confusion between goods and personal belongings.

Means for Solving the Problems

[0009] The action determination program according to one aspect acquires history information of a user's operation on an accounting machine that registers a product to be purchased and performs accounting processing on the registered product, and specifies, from an image of the previous user of the accounting machine, an action of the user operating the accounting machine while gripping an object, and determines, based on the acquired history information, whether the action of the user operating the accounting machine while gripping an object is an action of registering a product to be purchased in the accounting machine, and causes a computer to execute the process.

Effects of the Invention

[0010] According to one embodiment, it is possible to prevent confusion between goods and personal belongings.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

BEST MODE FOR CARRYING OUT THE INVENTION

[0012] Hereinafter, embodiments of the action determination program, action determination method, and action determination apparatus according to the present application will be described with reference to the accompanying drawings. Each embodiment merely shows one example or aspect, and numerical values, functional ranges, usage scenes, etc. are not limited by such exemplifications. And each embodiment can be appropriately combined within a range that does not conflict with the processing content.

EXAMPLE

[0013] FIG. 1 is a diagram showing an example of the system according to this embodiment. As shown in FIG. 1, this system 5 includes a camera 30, a self-checkout 50, an administrator terminal 60, and an information processing apparatus 100.

[0014] The information processing apparatus 100 is connected to the camera 30 and the self-checkout 50. The information processing apparatus 100 is connected to the administrator terminal 60 via the network 3. The camera 30 and the self-checkout 50 may be connected to the information processing apparatus 100 via the network 3.

[0015] The camera 30 is a camera that captures an image of the area including the self-checkout 50. The camera 30 transmits the data of the image to the information processing apparatus 100. In the following description, the data of the image is referred to as "image data".

[0016] The video data includes a plurality of image frames in time series. Each image frame is assigned a frame number in ascending order of time series. One image frame is a still image captured at the timing when the camera 30 is present.

[0017] The self-checkout 50 is a POS checkout system in which the user 2 who purchases a product performs operations from reading the barcode of the product to settlement. For example, when the user 2 moves the product to be purchased to the scan area of the self-checkout 50, the self-checkout 50 scans the barcode of the product.

[0018] The user 2 repeatedly executes the above operation. When the scanning of the product is completed, the user 2 operates a touch panel or the like of the self-checkout 50 to make a settlement request. When the self-checkout 50 receives the settlement request, it presents the number of products to be purchased, the purchase amount, etc., and executes a settlement process. The self-checkout 50 stores the history information of the user's operations on the self-checkout 50 in the storage unit and transmits it to the information processing device 100 as product information. For example, the history information may include, as product information, information on the products scanned by the user 2 from the start of scanning to the settlement request.

[0019] The administrator terminal 60 is a terminal device used by the store administrator. The administrator terminal 60 receives an alert notification or the like from the information processing device 100.

[0020] The information processing device 100 is a device that notifies the administrator terminal 60 of an alert based on the number of times the user 2 identified from the video data acquired from the camera 30 has performed an operation of registering a product at the self-checkout 50 and the number of products purchased identified from the product information. In the following description, the number of times the user 2 has performed an operation of registering a product at the self-checkout 50 is referred to as the "number of registration operations".

[0021] For example, if the number of registration operations and the number of purchases are different, it can be said that there is an accounting omission for the product. Therefore, the information processing apparatus 100 can detect an accounting omission for the product by notifying an alert based on the number of registration operations and the number of purchases.

[0022] Next, an example of the configuration of the information processing apparatus 100 shown in FIG. 1 will be described. FIG. 2 is a functional block diagram showing the configuration of the information processing apparatus according to the present embodiment. As shown in FIG. 2, the information processing apparatus 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0023] The communication unit 110 executes data communication among the camera 30, the self-checkout 50, the administrator terminal 60, and the like. For example, the communication unit 110 receives video data from the camera 30. The communication unit 110 receives history information on operations of a user with respect to the self-checkout 50 from the self-checkout 50.

[0024] The input unit 120 is an input device that inputs various types of information into the information processing apparatus 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0025] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, or the like.

[0026] The storage unit 140 includes a video buffer 141, history information 142, model information 143, a data table 144, a determination table 145, and registration operation information 146. The storage unit 140 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0027] The video buffer 141 stores video data captured by the camera 30. The video data includes a plurality of image frames in time series.

[0028] The history information 142 is the history information of the user's operations on the self-checkout 50. For example, the history information 142 may include product information 142A and payment-related information 142B.

[0029] The product information 142A is the information obtained from the self-checkout 50 when registering the product to be purchased on the self-checkout 50. For example, the product information 142A includes the information of the products scanned between when user 2 starts scanning and when the settlement request is made. FIG. 3 is a diagram showing an example of the data structure of the product information. As shown in FIG. 3, the product information 142A associates date and time information with product identification information.

[0030] The date and time information indicates the date and time when the self-checkout 50 reads the barcode of the product. The product identification information is the information for identifying the product. For example, in the first row of FIG. 3, it is shown that the barcode of the product with the product identification information "item101" was scanned at the date and time "10:13:30 on September 10, 2021".

[0031] The payment-related information 142B is the information obtained from the self-checkout 50 during the operations related to the payment of the price of the products registered on the self-checkout 50. Hereinafter, the operations related to payment may sometimes be described as "payment-related operations". For example, the payment-related operations may include a payment operation for paying the price of the product. In addition, the payment-related operations may include payment preparation operations such as an operation for specifying the payment method of the price of the product and operations related to valuable values given according to the purchase amount of the product, such as points.

[0032] FIG. 4 is a diagram showing an example of the data structure of settlement-related information. As shown in FIG. 4, the settlement-related information 142B may be data in which items such as date and time and operation type are associated. Here, the "date and time" refers to the date and time when the settlement-related operation was performed. Also, the "operation classification" refers to a category for classifying settlement-related operations. For example, in the first row of FIG. 4, it means that an operation of selecting the brand of a point card was performed at the date and time of "September 10, 2021, 10:14:00".

[0033] The model information 143 is an NN (Neural Network) that outputs information regarding the interaction between a user (human) and a product (object) when an image frame is input. For example, the model information 143 corresponds to HOID (Human Object Interaction Detection).

[0034] FIG. 5 is a diagram for explaining the model information. As shown in FIG. 5, by inputting the image frame 31 into the model information 143, the detection information 32 is output. The detection information 32 includes user area information 32a, product area information 32b, and interaction information 32c.

[0035] The user area information 32a indicates the area of the user included in the image frame 31 by coordinates (upper left x, y coordinates, lower right x, y coordinates). The product area information 32b indicates the area of the product included in the image frame 31 by coordinates (upper left x, y coordinates, lower right x, y coordinates). Also, the product area information 32b includes the class name unique to the product.

[0036] The interaction information 32c includes the probability value of the interaction between the user and the product detected from the image frame 31 and the class name of the interaction. The class name of the interaction is set to a class name such as "grasp (the user grasps the product)".

[0037] The model information 143 according to this embodiment outputs the detection information 32 only when there is an interaction between the user and the product. For example, when an image frame in a state where the user is holding the product is input to the model information 143, the detection information 32 is output. On the other hand, when an image frame in a state where the user is not holding the product is input to the model information 143, the detection information 32 is not output.

[0038] The data table 144 is a data table used when tracking the products detected from each image frame. FIG. 6 is a diagram showing an example of the data structure of the data table. As shown in FIG. 6, the data table 144 has a detection result table 144a, a tracked object table 144b, and a paused object table 144c.

[0039] The detection result table 144a is a table that holds the coordinates of the product area output from the model information 143. In the following description, the coordinates of the product area are referred to as "product area coordinates". The product area coordinates are indicated by [first element, second element, third element, fourth element]. The first element indicates the upper left x coordinate of the product area. The second element indicates the upper left y coordinate of the product area. The third element indicates the lower right x coordinate of the product area. The fourth element indicates the lower right y coordinate of the product area.

[0040] The tracked object table 144b is a table that holds information about the products being tracked. The tracked object table 144b has an ID (identification), product area coordinates, a lost count, and a stay count. The ID is identification information assigned to the product area coordinates. The product area coordinates indicate the coordinates of the product area.

[0041] The lost count indicates the number of image frames counted when the product corresponding to the product area coordinates is not detected. The stay count indicates the number of image frames counted when the product corresponding to the product area coordinates is in a state of not moving.

[0042] The tracking suspension object table 144c is a table that holds information about products for which tracking has been suspended. The tracking suspension object table 144c has an ID, product area coordinates, and a flag. The ID is identification information assigned to the product area coordinates. The product area coordinates indicate the coordinates of the product area.

[0043] The flag is information indicating whether to return the ID and product area coordinates of the tracking suspension object table 144c to the tracking object table 144b. When the flag is set to "true", it indicates that the ID and product area coordinates of the corresponding record are to be returned to the tracking object table 144b. When the flag is set to "false", it indicates that the ID and product area coordinates of the corresponding record are not to be returned to the tracking object table 144b.

[0044] Return to the description of FIG. 6. The determination table 145 is a table used when counting the number of registration operations. In this embodiment, when the product area coordinates specified from the image frame move from outside the preset scan area to inside the scan area, 1 is added to the number of registration operations. By using the determination table 145, the information processing apparatus 100 can ensure that even if the same product enters and exits the scan area multiple times, the number of times added to the number of registration operations is 1.

[0045] FIG. 7 is a diagram showing an example of the data structure of the determination table. As shown in FIG. 7, this determination table 145 associates an ID, the previous frame position, and a counted flag. The ID is identification information assigned to the product area coordinates. The previous frame position is information for identifying whether the product area coordinates detected from the previous image frame are outside or inside the scan area.

[0046] Here, for the product area coordinates of the corresponding ID, when the product area coordinates detected from the previous image frame are outside the scan area, "OUT" is set at the previous frame position. When the product area coordinates detected from the previous image frame are inside the scan area, "IN" is set at the previous frame position. The counted flag is a flag that identifies whether the process of adding 1 to the number of registration operation times has been performed for the corresponding ID.

[0047] In this embodiment, "false" is set as the initial value of the counted flag. When the product area coordinates of the corresponding ID detected from the current image frame position become "IN" while the previous image frame position of the product area coordinates of the corresponding ID is set to "OUT", 1 is added to the number of registration operation times. In this case, the counted flag is updated from "false" to "true".

[0048] The registration operation information 146 is information regarding the product registration operation. The registration operation information 146 may include the number of registration operation times. Additionally, the registration operation information 146 may include information in which the identification information of the object for which the product registration operation was specified and the date and time when the product registration operation was specified are associated.

[0049] Returning to the description of FIG. 2. The control unit 150 includes an acquisition unit 151, a tracking unit 152, a determination unit 153, a counting unit 154, and an output unit 155. The control unit 150 is realized by a hardware processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), for example. Further, the control unit 150 may be executed by hard-wired logic such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0050] The acquisition unit 151 acquires video data from the camera 30 and stores the acquired video data in the video buffer 141. The acquisition unit 151 acquires the history information 142 from the self-checkout 50 and stores the acquired history information 142 in the storage unit 140.

[0051] The tracking unit 152 tracks the product area coordinates based on the video data (time-series image frames) stored in the video buffer 141. For example, the tracking unit 152 repeatedly executes a process of inputting the image frames in order into the model information 143 to identify the product area coordinates and updating the data table 144. Hereinafter, an example of the process of the tracking unit 152 will be described.

[0052] The tracking unit 152 inputs the image frames stored in the video buffer 141 into the model information 143 and acquires the product area coordinates included in the detection information. The tracking unit 152 registers the product area coordinates in the detection result table 144a. In the following description, the product area coordinates in the detection result table 144a are referred to as "first product area coordinates". The product area coordinates in the tracking object table 144b are referred to as "second product area coordinates". The product area coordinates in the tracking paused object table 144c are referred to as "third product area coordinates".

[0053] Note that the tracking unit 152 calculates the "similarity" based on the distance between the centers of the respective product area coordinates to be compared. The similarity becomes a larger value as the distance between the centers of the respective product area coordinates to be compared is shorter. It is assumed that the relationship between the distance between the centers and the similarity is defined in advance.

[0054] The tracking unit 152 compares the first product area coordinates with each of the third product area coordinates in the tracking paused object table 144c to determine whether there is a pair of the first product area coordinates and the third product area coordinates for which the similarity is equal to or greater than the threshold Th1. The value of the threshold Th1 is set in advance.

[0055] When there is a pair of a first product area coordinate and a third product area coordinate whose similarity is equal to or greater than a threshold Th1, the tracking unit 152 executes the following process in the tracking pause object table 144c. The tracking unit 152 sets the flag of the entry having a third product area coordinate whose similarity to the first product area coordinate is equal to or greater than the threshold Th1 to "true". Further, the tracking unit 152 deletes the entry having a first product area coordinate whose similarity to the third product area coordinate is equal to or greater than the threshold Th1 from the detection result table 144a.

[0056] The tracking unit 152 compares the first product area coordinate with each second product area coordinate in the tracking object table 144b, and specifies the maximum value of the similarity between the first product area coordinate and the second product area coordinate. When the maximum value of the similarity is equal to or greater than a threshold Th3, the tracking unit 152 determines that "the corresponding product is not moving". When the maximum value of the similarity is equal to or greater than a threshold Th2, the tracking unit 152 determines that "the corresponding product can be tracked". When it is less than the threshold Th2, the tracking unit 152 determines that "the corresponding product cannot be tracked". The values of the thresholds Th2 and Th3 are set in advance. However, the value of the threshold Th3 shall be greater than the value of the threshold Th2.

[0057] FIG. 8 is a diagram for explaining the process of the tracking unit. In case 1A of FIG. 8, the area of the product specified by the first product area coordinate is defined as product area 20a, and the area of the product specified by the second product area coordinate is defined as product area 21a. When the distance between the product area 20a and the product area 21a is less than a distance lA (when the similarity based on the distance is equal to or greater than a threshold Th3), the tracking unit 152 determines that "the corresponding product is not moving".

[0058] When the tracking unit 152 determines that "the corresponding product is not moving", it adds 1 to the stay count in the entry corresponding to the product area 21a (second product area coordinate) in the tracking object table 144b.

[0059] In Case 1B of FIG. 8, the area of the product specified by the first product area coordinates is defined as product area 20b, and the area of the product specified by the second product area coordinates is defined as product area 21b. When the distance between product area 20b and product area 21b is less than distance lB (when the similarity based on the distance is equal to or greater than threshold Th2), tracking unit 152 determines that "the corresponding product can be tracked".

[0060] When tracking unit 152 determines that "the corresponding product can be tracked", in the entry corresponding to product area 21b (second product area coordinates) in the object being tracked table 144b, it updates the second product area coordinates to the first product area coordinates. Tracking unit 152 sets the stay count to 0 in the entry corresponding to product area 21b (second product area coordinates) in the object being tracked table 144b.

[0061] In Case 1C of FIG. 8, the area of the product specified by the first product area coordinates is defined as product area 20c, and the area of the product specified by the second product area coordinates is defined as product area 21c. When the distance between product area 20c and product area 21c is equal to or greater than distance lB (when the similarity based on the distance is less than threshold Th2), tracking unit 152 determines that "the corresponding product cannot be tracked".

[0062] When tracking unit 152 determines that "the corresponding product cannot be tracked", it registers a new entry of the first product area coordinates corresponding to product area 20c in the object being tracked table 144b. When registering a new entry, tracking unit 152 assigns a new ID and sets the stay count to 0 and the lost count to 0.

[0063] Here, for each entry in the object being tracked table 144b that has second product area coordinates with a similarity to the first product area coordinates that is less than threshold Th2, tracking unit 152 adds 1 to the lost count.

[0064] The tracking unit 152 extracts, from each entry in the tracked object table 144b, the entry whose lost counter exceeds the threshold Th4. For the entry among the extracted entries whose stay counter value is equal to or greater than the threshold Th5, the tracking unit 152 moves the corresponding entry (ID, second product area coordinates) to the tracked paused object table 144c and sets the flag to "false".

[0065] For the entry among the extracted entries whose stay counter value is less than the threshold Th5, the tracking unit 152 deletes the corresponding entry.

[0066] The tracking unit 152 moves, to the tracked object table 144b, the entry whose flag is "true" among each entry in the tracked paused object table 144c, and sets the stay counter to 0.

[0067] Each time a new entry is registered in the detection result table 144a, the tracking unit 152 repeatedly executes the above processing to update the tracked object table 144b and the tracked paused object table 144c.

[0068] Return to the description of FIG. 2. Based on the history information 142, the determination unit 153 determines whether the action of the user 2 operating the self-checkout 50 while gripping an object is an action of registering the product purchased at the self-checkout 50.

[0069] Here, the user's operation on the self-checkout 50 may include a first mode of registering the product to be purchased at the self-checkout 50 by reading the barcode of the product, and a second mode of settling the product registered at the self-checkout 50.

[0070] For example, in the initial state of the user 2's checkout, for example, when the product is not registered at the self-checkout 50, since there is no product to purchase, the user stays in the first mode. Such an initial state of checkout can be identified as the initial state of checkout when the settlement by the user who used the self-checkout 50 before the user 2 is completed.

[0071] These two modes, the first mode and the second mode, can transition to the other mode under the conditions exemplified below. FIG. 9 is a diagram showing an example of mode transition. As shown in FIG. 9, from the first mode to the second mode, a transition can occur when a settlement preparation operation or a settlement operation is performed. On the other hand, from the second mode to the first mode, a transition can occur when a product registration operation for registering a product to the self-checkout 50 is performed.

[0072] FIG. 10 is a diagram for explaining an example of a settlement preparation operation. FIG. 10 shows, as an example of an image displayed on the display unit of the self-checkout 50, a settlement screen 41 for settling the products registered in the self-checkout 50.

[0073] As shown in FIG. 10, the settlement screen 41 includes a payment method selection area 42 for selecting a method of paying for the product price, a product information display area 43 for displaying information on the products purchased by the user 2, and another function call area 44 for calling other functions related to settlement.

[0074] In the payment method selection area 42, GUI components corresponding to each of various payment methods such as cash, electronic money, credit card, debit card, and settlement using a payment service are arranged. An operation on any of these GUI components corresponds to one of the preliminary operations for settling the price of the products registered in the self-checkout 50, that is, the purchased products, and thus can be regarded as a settlement preparation operation.

[0075] In the product information display area 43, the number of purchased products and the total amount of the prices of all the purchased products are displayed. In addition, when a discount or the like is applied to the purchased products, the discount amount or the like can also be displayed together.

[0076] The other function call area 44 includes a point card selection button 44A for calling a point card selection screen for selecting the brand of a point card that awards points corresponding to the price of the purchased products. When the point card selection button 44A is operated, the self-checkout 50 may display the point card selection screen shown in FIG. 11.

[0077] FIG. 11 is a diagram for explaining another example of the settlement preparation operation. As shown in FIG. 11, on the point card selection screen 45, GUI components corresponding to each of the brands of the point cards are arranged. An operation on any of these GUI components also corresponds to one of the preliminary operations for settling the price of the products registered in the self-checkout 50, that is, the purchased products, and thus can be regarded as a settlement preparation operation.

[0078] As the settlement preparation operations illustrated in FIGS. 10 and 11 correspond to preliminary operations for settling the price of the purchased products, the possibility that a subsequent settlement operation will be performed is higher than the possibility that a product registration operation will be performed. From this, it can be presumed that the period after the settlement preparation operation is the second mode.

[0079] Also, when the history of the settlement operation is acquired, it is highly likely that after an object that is not a product, such as a wallet, a card, a smartphone, or a wearable terminal, has approached the money insertion slot or the scanner of the self-checkout 50. From this, it can be presumed that the period before the settlement operation is the second mode.

[0080] Using such attributes of the settlement preparation operation and the settlement operation, the determination unit 153 determines the section corresponding to the first mode and the section corresponding to the second mode in the accounting period from the initial state of the accounting of the user 2 to the completion of the accounting.

[0081] As an example only, the determination unit 153 monitors the operations of the user on the self-checkout 50. When a new operation history is stored in the history information 142 by the acquisition unit 151, the determination unit 153 determines whether or not the operation history is a settlement operation.

[0082] At this time, when the operation history is a settlement operation, the determination unit 153 searches for a settlement preparation operation from the operation history included in the accounting period corresponding to the settlement operation among the history information 142. Then, when a settlement preparation operation is found, the determination unit 153 further determines whether a product registration operation is included within the section from the settlement preparation operation found by the search to the settlement operation acquired by the acquisition unit 151.

[0083] Here, when a product registration operation is not included, the determination unit 153 determines the period from the settlement preparation operation to the settlement operation as the second mode. On the other hand, when the settlement preparation operation is not found in the search for the operation history within the accounting period, or when a product registration operation is included, the determination unit 153 determines a specific period, for example, a 30 - second period, retroactively from the settlement operation as the second mode. Then, the determination unit 153 determines the section other than the section determined as the second mode within the accounting period as the first mode.

[0084] FIG. 12 is a diagram showing an example of mode determination. In FIG. 12, an example is shown in which mode determination is executed based on the history information 142 including the product information 142A shown in FIG. 3 and the settlement - related information 142B shown in FIG. 4. According to the examples of the product information 142A and the settlement - related information 142B, the operations of user 2 on the self - checkout 50 are performed in the order of the product registration operation of item101, the product registration operation of item102, the product registration operation of item103, the point - card selection operation, and the settlement - card reading operation. In this case, no product registration operation is included from the point - card selection operation performed at 10:14 to the settlement - card reading operation performed at 10:14:20. Therefore, the section from 10:14 to 10:14:20 is determined as the second mode, and the other sections are determined as the first mode.

[0085] After such mode determination is performed, when the determination unit 153 detects an action of user 2 operating the self - checkout 50 while gripping an object at the time when the second mode is executed at the self - checkout 50, the determination unit 153 determines that the object is not an object of the product to be purchased.

[0086] Return to the description of FIG. 2. The counting unit 154 identifies the operation in which the user registers a product at the self-checkout 50 based on the tracked object table 144b of the data table 144, and counts the number of registration operations performed. The counting unit 154 registers the number of registration operations in the storage unit 140 as registration operation information 146. An example of the process of the counting unit 154 will be described below.

[0087] FIG. 13 is a diagram for explaining the process of the counting unit. Step S1 in FIG. 13 will be described. It is assumed that the counting unit 154 holds the coordinates of the scan area 10b in advance. The counting unit 154 refers to the tracked object table 144b, and when a new ID entry is added, it adds an entry with the same ID as the new ID to the determination table 145. When the counting unit 154 adds an entry to the determination table 145, it sets the counted flag to "false". In the following description, for convenience of explanation, the ID added to the determination table 145 will be described as ID "1". The ID assigned to the second product area coordinates corresponding to the product area 10c of the product is ID "1".

[0088] The counting unit 154 compares the second product area coordinates of the entry with ID "1" in the tracked object table 144b with the scan area 10b. When the second product area coordinates are not included in the scan area 10b, the counting unit 154 sets the previous frame position of the entry with ID "1" added to the determination table 145 to "OUT". When the second product area coordinates are included in the scan area 10b, the counting unit 154 sets the previous frame position of the entry with ID "1" added to the determination table 145 to "IN". In the example shown in step S1 of FIG. 13, since the product area 10c corresponding to the second product area coordinates is not included in the scan area 10b, the previous frame position of the entry with ID "1" added to the determination table 145 is set to "OUT".

[0089] Proceed to the description of step S2 in FIG. 13. The counter unit 154 monitors the tracked object table 144b, and each time the tracked object table 144b is updated, it compares the second product area coordinates corresponding to ID "1" with the scan area 10b. When the second product area coordinates (product area 10c) corresponding to ID "1" move to the area included in the scan area 10b, the counter unit 154 refers to the entry of ID "1" in the determination table 145, and refers to the previous frame position and the counted flag.

[0090] For the entry of ID "1" in the determination table 145, when the previous frame position is "OUT" and the counted flag is "false", the counter unit 154 adds 1 to the registration operation count. Also, after adding 1 to the registration operation count, the counter unit 154 updates the previous frame position to "IN" and updates the counted flag to "true". When the registration operation count is incremented in this way, the counter unit 154 adds an entry in which the ID "1" of the object for which the product registration operation is specified and the date and time when the product registration operation for the ID "1" is specified are associated to the registration operation information 146.

[0091] On the other hand, when the previous frame position is "IN" or the counted flag is "true", the counter unit 154 skips the process of adding 1 to the registration operation count.

[0092] Each time a new ID entry is added to the tracked object table 144b, the counter unit 154 repeatedly executes the above process. When the ID of the entry added to the tracked object table 144b is the same as the ID of the entry registered in the determination table 145, the counter unit 154 skips the process of registering the entry corresponding to the new ID in the determination table 145.

[0093] While the counting of the number of registration operations is being performed, the counter unit 154 counts the number of times an object that is not a purchased item, for example, a personal item, approaches the self-checkout 50. For example, the counter unit 154 determines whether the date and time when a product registration operation is specified for each entry in the registration operation information 146 is within the time of the second mode.

[0094] At this time, if the date and time when the product registration operation is specified is within the time of the second mode, 1 is added to the number of personal item approaches. On the other hand, if the date and time when the product registration operation is specified is not within the time of the second mode, that is, if it is within the time of the first mode, the process of adding 1 to the number of personal item approaches is skipped.

[0095] FIG. 14 is a diagram showing an example of the data structure of the registration operation information. In the examples from the first row to the third row of the registration operation information 146 shown in FIG. 14, since the date and time when the product registration operation is specified belongs to the time of the first mode as shown in FIG. 12, the number of personal item approaches is not incremented. On the other hand, in the case of the fourth and fifth rows of the registration operation information 146, since the date and time when the product registration operation is specified belongs to the time of the second mode as shown in FIG. 12, the number of personal item approaches is incremented. As a result, the number of personal item approaches is counted as 2.

[0096] After such counting of the number of personal item approaches, the counter unit 154 subtracts the number of personal item approaches from the number of registration operations.

[0097] The output unit 155 outputs an alert to the administrator terminal 60 based on the history information 142 and the registration operation information 146. An example of the process of the output unit 155 will be described below.

[0098] The output unit 155 acquires the product information 142A and specifies the number of purchases. For example, the output unit 155 specifies the number of records with different date and time information in the product information 142A as the number of purchases.

[0099] When the number of purchases is different from the number of registration operations after subtracting the number of times of approaching personal belongings, the output unit 155 transmits an alert to the administrator terminal 60. For example, when the number of purchases is less than the number of registration operations after subtracting the number of times of approaching personal belongings, since there is a risk of accounting omission, the output unit 155 outputs an alert indicating that accounting omission has been detected to the administrator terminal 60.

[0100] On the other hand, when the number of purchases matches the number of registration operations after subtracting the number of times of approaching personal belongings, the output unit 155 skips the process of outputting an alert.

[0101] Next, an example of the tracking process executed by the tracking unit 152 of the information processing apparatus 100 according to the present embodiment will be described. FIGS. 15 and 16 are flowcharts showing the processing procedures of the tracking process. As shown in FIG. 15, the tracking unit 152 of the information processing apparatus 100 initializes the tracking object table 144b and the tracking suspended object table 144c (step S101).

[0102] The tracking unit 152 acquires an image frame from the video buffer 141 and inputs it to the model information 143 to acquire detection information (step S102). The tracking unit 152 registers the first product area coordinates included in the detection information in the detection result table 144a (step S103).

[0103] The tracking unit 152 determines whether there is an entry whose similarity between the first product area coordinates and the third product area coordinates in the tracking suspended object table 144c is the threshold Th1 (step S104). When there is an entry (step S105, Yes), the tracking unit 152 proceeds to step S106. On the other hand, when there is no entry (step S105, No), the tracking unit 152 proceeds to step S108.

[0104] The tracking unit 152 sets the flag of the corresponding entry in the tracking suspended object table 144c to "true" (step S106). The tracking unit 152 deletes the corresponding entry from the detection result table 144a (step S107).

[0105] The tracking unit 152 determines whether there is an entry in which the similarity between the first product area coordinates and the second product area coordinates in the object table 144b being tracked is equal to or greater than a threshold value Th2 (step S108). When there is an entry (step S109, Yes), the tracking unit 152 proceeds to step S110. On the other hand, when there is no entry (step S109, No), the tracking unit 152 proceeds to step S115 in FIG. 16.

[0106] The tracking unit 152 updates the second product area coordinates of the corresponding entry in the object table 144b being tracked to the first product area coordinates (step S110). The tracking unit 152 determines whether there is an entry in which the similarity between the first product area coordinates and the second product area coordinates in the object table 144b being tracked is equal to or greater than a threshold value Th3 (step S111).

[0107] When there is an entry (step S112, Yes), the tracking unit 152 adds 1 to the stay count of the corresponding entry in the object table 144b being tracked (step S113), and proceeds to step S115 in FIG. 16.

[0108] On the other hand, when there is no entry (step S112, No), the tracking unit 152 updates the stay count of the corresponding entry in the object table 144b being tracked to 0 (step S114), and proceeds to step S115 in FIG. 16.

[0109] Proceeding to the description of FIG. 16. The tracking unit 152 adds an entry, which assigns a new ID to the first product area information in which the similarity with the second product area coordinates is less than the threshold value Th2, to the object table 144b being tracked (step S115). The tracking unit 152 sets the stay count of the entry added to the object table 144b being tracked to 0 (step S116).

[0110] The tracking unit 152 adds 1 to the loss count of an entry in the object table 144b being tracked that has second product area coordinates in which the similarity with the first product area coordinates is less than the threshold value Th2.

[0111] The tracking unit 152 determines whether there is an entry in the tracked object table 144b for which the value of the stay counter is equal to or greater than the threshold Th5 (step S118). If there is an entry (step S119, Yes), the tracking unit 152 proceeds to step S120. On the other hand, if there is no entry (step S119, No), the tracking unit 152 proceeds to step S121.

[0112] The tracking unit 152 moves an entry for which the value of the stay counter is equal to or greater than the threshold Th5 to the tracked paused object table 144c and sets the flag to "false" (step S120). The tracking unit 152 moves an entry in the tracked paused object table 144c for which the flag is "true" to the tracked object table 144b and sets the stay count to 0 (step S122). Note that the tracking unit 152 deletes an entry for which the value of the stay counter is equal to or greater than the threshold Th5 (step S121) and proceeds to step S122.

[0113] If the tracking unit 152 continues the process (step S123, Yes), it proceeds to step S102 in FIG. 15. On the other hand, if the tracking unit 152 does not continue the process (step S123, No), it ends the process.

[0114] Next, an example of the tracking process executed by the determination unit 153 of the information processing apparatus 100 according to the present embodiment will be described. FIG. 17 is a flowchart showing the processing procedure of the mode determination process. As shown in FIG. 17, when a new operation history is stored in the history information 142 by the acquisition unit 151 (step S151, Yes), the determination unit 153 determines whether the operation history is a settlement operation (step S152).

[0115] At this time, if the operation history is a settlement operation (step S152, Yes), the determination unit 153 searches for a settlement preparation operation from the operation histories included in the accounting period corresponding to the settlement operation in the history information 142 (step S153).

[0116] When the settlement preparation operation is hit (step S154, Yes), the determination unit 153 further determines whether a product registration operation is included within the section from the settlement preparation operation hit by the search to the settlement operation acquired by the acquisition unit 151 (step S155).

[0117] Here, when the product registration operation is not included (step S155, No), the determination unit 153 determines the period from the settlement preparation operation to the settlement operation as the second mode (step S156).

[0118] On the other hand, when the settlement preparation operation is not hit in the search for the operation history within the accounting period, or when the product registration operation is included (step S154, No or step S155, Yes), the determination unit 153 executes the following processing. That is, the determination unit 153 determines a specific period, for example, a 30 - second period, retroactively from the settlement operation as the second mode (step S157).

[0119] Then, the determination unit 153 determines the section other than the section determined as the second mode in step S156 or step S157 within the accounting period as the first mode (step S158), and ends the processing.

[0120] Next, the processing procedure of the information processing apparatus according to the present embodiment will be described. FIG. 18 is a flowchart showing the processing procedure of the information processing apparatus according to the present embodiment. As shown in FIG. 18, the acquisition unit 151 of the information processing apparatus 100 acquires product information 142A from the self - checkout 50 and stores it in the storage unit 140 (step S201).

[0121] The counting unit 154 of the information processing apparatus 100 counts the number of purchases based on the product information (step S202). The counting unit 154 executes a registration operation count processing (step S203). The counting unit 154 executes a personal item approach count processing based on the registration operation information (step S204).

[0122] The counting unit 154 of the information processing apparatus 100 subtracts the number of personal item approaches counted in step S204 from the number of registration operations counted in step S203 (step S205). Then, the output unit 155 of the information processing apparatus 100 determines whether the number of purchases matches the number of registration operations after subtracting the number of personal item approaches (step S206).

[0123] When the output unit 155 determines that the number of purchases matches the number of registration operations after subtracting the number of personal item approaches (step S207, Yes), the process ends.

[0124] On the other hand, when the output unit 155 determines that the number of purchases does not match the number of registration operations after subtracting the number of personal item approaches (step S207, No), it outputs an alert to the administrator terminal 60 (step S208).

[0125] Next, an example of the processing procedure of the registration operation count processing described in step S203 of FIG. 18 will be described. FIG. 19 is a flowchart showing the processing procedure of the registration operation count processing. As shown in FIG. 19, the counting unit 154 of the information processing apparatus 100 starts monitoring the tracked object table 144b (step S301).

[0126] When a new ID entry is added to the tracked object table 144b (step S302, Yes), the counting unit 154 proceeds to step S303. When a new ID entry is not added to the tracked object table 144b (step S302, No), the counting unit 154 proceeds to step S305.

[0127] The counting unit 154 identifies the previous frame position based on the second product area coordinates of the new ID entry and the scan area (step S303). The counting unit 154 adds an entry with the new ID, the previous frame position, and the counted flag set to "false" to the determination table 145 (step S304).

[0128] The counter 154 identifies the current frame position based on the second product area coordinates corresponding to the ID of each entry in the determination table 145 and the scan area (step S305). The counter 154 selects an unselected entry in the determination table 145 (step S306).

[0129] The counter 154 determines whether the conditions are met where the previous frame position of the selected entry is "OUT", the counted flag is "false", and the current frame position corresponding to the ID of the selected entry corresponding to the ID is "IN" (step S307).

[0130] If the conditions are met (step S308, Yes), the counter 154 proceeds to step S309. If the conditions are not met (step S308, No), the counter 154 proceeds to step S311.

[0131] The counter 154 adds 1 to the registered operation count (step S309). The counter 154 updates the previous frame position of the selected entry to "IN" and the counted flag to "true" (step S310).

[0132] If not all entries in the determination table 145 have been selected (step S311, No), the counter 154 proceeds to step S306. If all entries in the determination table 145 have been selected (step S311, Yes), the counter 154 proceeds to step S312.

[0133] If the process is to continue (step S312, Yes), the counter 154 proceeds to step S302. If the process is not to continue (step S312, No), the registered operation count counting process ends.

[0134] Next, an example of the processing procedure of the personal item approach count process described in step S204 of FIG. 18 will be described. FIG. 20 is a flowchart showing the processing procedure of the personal item approach count process. As shown in FIG. 20, the counting unit 154 of the information processing apparatus 100 selects one of the entries included in the registration operation information 146 (step S401).

[0135] Subsequently, the counting unit 154 determines whether the date and time when the product registration operation was specified, which is included in the entry selected in step S401, is during the time of the second mode (step S402).

[0136] At this time, if the date and time when the product registration operation was specified is during the time of the second mode (step S402, Yes), the counting unit 154 adds 1 to the personal item approach count (step S403).

[0137] On the other hand, if the date and time when the product registration operation was specified is not during the time of the second mode, that is, if it is during the time of the first mode (step S402, No), the process of adding 1 to the personal item approach count is skipped.

[0138] Then, until all the entries included in the registration operation information 146 are selected (step S404, No), the processes from step S401 to step S403 are repeated. After that, when all the entries included in the registration operation information 146 are selected (step S404, Yes), the process ends.

[0139] Next, the effects of the information processing apparatus 100 according to this embodiment will be described. The information processing apparatus 100 notifies an alert based on the number of purchases specified from the product information 142A acquired from the self-checkout 50 and the number of registration operations counted by comparing the product area and the scan area. For example, when the number of registration operations and the number of purchases are different, it can be said that there is an accounting omission of the product. By the information processing apparatus 100 notifying an alert based on the number of registration operations and the number of purchases, the accounting omission of the product can be detected.

[0140] Furthermore, the information processing apparatus 100 determines an object identified as being held by a user as the user's possession from an image of the previous user of the accounting machine taken during a specific period retroactively from the time when a settlement operation was performed among the history of the user's operations on the accounting machine. Therefore, according to the information processing apparatus 100, it is possible to suppress the personal possessions from being confused with the store's products.

[0141] By the way, the processing content of the above-described embodiment is an example, and the information processing apparatus 100 may further execute other processing. In the following description, other processing executed by the information processing apparatus 100 will be described.

[0142] In the above embodiment, an example was given in which the registration operation count is incremented when the position of the object being tracked moves from the outside to the inside of the scan area 10b from the previous frame to the current frame, but the counting method of the registration operation count is not limited to this.

[0143] For example, the counting unit 154 of the information processing apparatus 100 can also count the number of take-out operations in which the user takes out the products stored in the basket 2a as the registration operation count based on the object being tracked table 144b of the data table 144. The counting unit 154 registers the number of take-out operations as registration operation information 146 in the storage unit 140. An example of the processing of the counting unit 154 will be described below.

[0144] FIG. 21 is a diagram for explaining the processing of the counting unit. FIG. 22 is a diagram showing an example of the data structure of the determination table. The step S1 in FIG. 21 will be described. It is assumed that the counting unit 154 holds the coordinates of the basket area 10e in advance. The counting unit 154 refers to the tracked object table 144b, and when a new ID entry is added, it adds an entry with the same ID as the new ID to the determination table 145 shown in FIG. 22. When adding an entry to the determination table 145, the counting unit 154 sets the counted flag to "false". In the following description, for convenience of explanation, the ID added to the determination table 145 will be described as ID "1". The ID assigned to the second product area coordinates corresponding to the product area 10c of the product is ID "1".

[0145] The counting unit 154 compares the second product area coordinates of the entry with ID "1" in the tracked object table 144b with the basket area 10e. When the second product area coordinates are not included in the basket area 10e, the counting unit 154 sets the previous frame position of the entry with ID "1" added to the determination table 145 to "OUT". When the second product area coordinates are included in the basket area 10e, the counting unit 154 sets the previous frame position of the entry with ID "1" added to the determination table 145 to "IN". In the example shown in step S1 of FIG. 21, since the product area 10c corresponding to the second product area coordinates is included in the basket area 10e, the previous frame position of the entry with ID "1" added to the determination table 145 is set to "IN".

[0146] Moving on to the explanation of step S2 in FIG. 21. The counting unit 154 monitors the tracked object table 144b, and each time the tracked object table 144b is updated, it compares the second product area coordinates corresponding to ID "1" with the basket area 10e. When the second product area coordinates (product area 10c) corresponding to ID "1" move to an area not included in the basket area 10e, the counting unit 154 refers to the entry with ID "1" in the determination table 145 and refers to the previous frame position and the counted flag.

[0147] When the counting unit 154 is for the entry with the ID "1" in the determination table 145, if the previous frame position is "IN" and the counted flag is "false", it adds 1 to the number of retrieval operations. Also, after adding 1 to the number of retrieval operations, the counting unit 154 updates the previous frame position to "OUT" and updates the counted flag to "true". When the number of retrieval operations is incremented in this way, the counting unit 154 adds an entry in which the ID "1" of the object for which the product registration operation was specified and the date and time when the product registration operation for the ID "1" was specified are associated to the registration operation information 146.

[0148] On the other hand, when the previous frame position becomes "OUT" or when the counted flag becomes "true", the counting unit 154 skips the process of adding 1 to the number of retrieval operations.

[0149] Each time a new ID entry is added to the tracked object table 144b, the counting unit 154 repeatedly executes the above process. When the ID that is the same as the ID of the entry added to the tracked object table 144b is the same as the ID of the entry registered in the determination table 145, the counting unit 154 skips the process of registering the entry corresponding to the new ID in the determination table 145.

[0150] Even when counting the number of retrieval operations as the number of registration operations in this way, by subtracting the number of times of approaching personal belongings from the number of retrieval operations and outputting an alert based on whether the number of retrieval operations after subtraction matches the number of purchases, the same effects as in the above embodiments can be obtained.

[0151] Other processes (1) executed by the information processing apparatus 100 will be described. The counting unit 154 of the information processing apparatus 100 has been executing the process using a preset scan area, but is not limited thereto. The counting unit 154 may analyze the image frame registered in the video buffer 141, identify a first area where the shopping basket is placed and a second area corresponding to the scan area, and use the identified second area to count the number of registration operations.

[0152] FIG. 23 is a diagram for explaining other processing (1). In the example shown in FIG. 23, a first area 40a and a second area 40b are specified from the image frame 40. The counting unit 154 may specify the first area 40a and the second area 40b using a conventional technique such as pattern matching, or may specify the first area 40a and the second area 40b using a machine learning model that has been machine-learned. For example, such a machine learning model is a model that performs machine learning using teacher data with an image frame as an input and the coordinates of the first area and the second area as correct answer data.

[0153] When the self-check 50 moves while the counting unit 154 is executing the process, or when the position of the camera 30 is changed, if the process is executed using a preset scan area, the number of registration operations cannot be accurately counted. On the other hand, by analyzing the image frame registered in the video buffer 141 and specifying the second area corresponding to the scan area, the scan area can be accurately specified and the number of registration operations can be accurately counted.

[0154] Other processing (2) executed by the information processing apparatus 100 will be described. In the above information processing apparatus 100, examples of counting the number of purchases or the number of approaches of personal belongings based on the history information 142 acquired from the self-check 50 have been given, but the present invention is not limited thereto. In the self-check 50, when performing the settlement process, the number of purchased items is displayed on the display screen. Therefore, the information processing apparatus 100 may specify the number of purchases by performing image analysis on the image frame of the display screen photographed by the camera 30 (or another camera).

[0155] FIG. 24 is a diagram for explaining other processing (2). The image frame 41 in FIG. 24 corresponds to the settlement screen 41 of the self-checkout 50 shown in FIG. 10. The area 41a of the image frame 41 includes an area 41a indicating the number of purchased items. The counter 154 identifies the number of purchases by performing image analysis on the area 41a. The counter 154 can identify the selection operation of the payment method by performing image analysis on the payment method selection area 42. The counter 154 can identify the operation of the point card selection button 44A by performing image analysis on the other function call area 44. Here, the settlement screen 41 is taken as an example, but also in the point card selection screen 45 shown in FIG. 11, the operation on the GUI component corresponding to each brand of the point card can be identified by image analysis.

[0156] Thereby, the information processing apparatus 100 can realize the acquisition of history information even if it is not connected to the self-checkout 50.

[0157] In the above embodiment, an example of subtracting the number of personal item approaches from the number of registration operations has been described, but it is also possible to control whether to increment the number of registration operations according to whether the object held by the user is a personal item. For example, the above mode determination is executed in real time, and when the condition of step S308 in FIG. 19 is satisfied, it is controlled whether to increment the number of registration operations according to which mode of the first mode or the second mode it is. As an example only, the counter 154 increments the number of registration operations specifically when it is in the second mode.

[0158] Next, an example of the hardware configuration of the information processing apparatus 10 will be described. FIG. 25 is a diagram showing an example of the hardware configuration. As shown in FIG. 25, the information processing apparatus 10 includes a communication device 100a, a HDD (Hard Disk Drive) 100b, a memory 100c, and a processor 100d. Further, each part shown in FIG. 25 is interconnected by a bus or the like.

[0159] The communication device 100a is, for example, a network interface card and communicates with other servers. The HDD 100b stores programs and databases that operate the functions shown in FIG. 2.

[0160] The processor 100d reads out from the HDD 100b or the like a program that executes the same processes as the respective processing units shown in FIG. 2, and expands it in the memory 100c, thereby operating a process that executes each function described in FIG. 2 and the like. For example, this process executes the same functions as the respective processing units of the information processing apparatus 10. Specifically, the processor 100d reads out from the HDD 100b or the like a program having the same functions as the acquisition unit 151, the tracking unit 152, the determination unit 153, the counting unit 154, the output unit 155, and the like. Then, the processor 100d executes a process that executes the same processes as the acquisition unit 151, the tracking unit 152, the determination unit 153, the counting unit 154, the output unit 155, and the like.

[0161] As described above, the information processing apparatus 10 operates as a computer that executes the parameter calculation method by reading out and executing a program. Further, the information processing apparatus 10 can also read out the program from the recording medium by the medium reading device and realize the same functions as those of the above-described embodiment by executing the read program. Note that the program in this other embodiment is not limited to being executed by the information processing apparatus 10. For example, the present invention can be similarly applied when another computer or server executes the program, or when these cooperate to execute the program.

[0162] This program can be distributed via a network such as the Internet. Also, this program can be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a MO (Magneto-Optical disk), a DVD (Digital Versatile Disc), etc., and can be executed by being read from the recording medium by a computer. In this embodiment, a barcode is taken as an example, but the code attached to a product is not limited to a one-dimensional code such as a barcode, and may be any code including, for example, a two-dimensional code such as a QR (Quick Response) code.

[0163] Regarding the embodiments including the above embodiments, the following additional notes are further disclosed.

[0164] (Supplementary Note 1) Obtain the history information of the user's operations on an accounting machine that registers the purchased product and performs the accounting process for the registered product, Identify, from an image of the previous user of the accounting machine, the action of the user operating the accounting machine while holding an object, Based on the obtained history information, determine whether the action of the user operating the accounting machine while holding an object is an action of registering the product to be purchased in the accounting machine. An action determination program characterized by causing a computer to execute the process.

[0165] (Supplementary Note 2) The history information of the user's operations on the accounting machine has a first mode of registering the product to be purchased in the accounting machine by reading the product code, and a second mode of settling the product registered in the accounting machine, The determination process includes a process of determining that the object is not a product to be purchased when it is detected that the user operates the accounting machine while holding an object at the time when the second mode is executed in the accounting machine. The action determination program according to Supplementary Note 1, characterized by the above.

[0166] (Appendix 3) The accounting machine acquires product information generated by reading the product code, Based on the acquired product information, it counts the first number of times indicating the number of purchased products, From an image of the user in front of the accounting machine, it identifies the action of the user registering a product with the accounting machine, It counts the second number of times indicating the action of the identified user registering a product with the accounting machine, From the second number of times, it counts the third number of times obtained by subtracting the number of times determined to be an object not to be purchased in the determination process, Based on the difference between the first number of times and the third number of times, it notifies an alert, The action determination program according to Appendix 2, characterized in that the computer is further caused to execute the process.

[0167] (Appendix 4) The determination process includes a process of determining a specific period as the second mode retroactively from the time when a settlement operation for the accounting machine is performed based on the history information, The action determination program according to Appendix 2, characterized in that.

[0168] (Appendix 5) The determination process includes a process of determining, based on the history information, a section from a settlement preparation operation corresponding to a preliminary operation of settling the price of the product to be purchased to the settlement operation as the second mode, The action determination program according to Appendix 4, characterized in that.

[0169] (Appendix 6) The settlement preparation operation is a selection operation of a payment method for paying the price of the product to be purchased or an operation related to a valuable value given according to the amount of the product to be purchased, The action determination program according to Appendix 5, characterized in that.

[0170] (Appendix 7) It acquires history information of a user's operation on an accounting machine that implements accounting processing for a registered product while the product to be purchased is registered, Identify, from an image of a user in front of the accounting machine, the action of the user operating the accounting machine while holding an object. Based on the acquired history information, determine whether the action of the user operating the accounting machine while holding an object is an action of registering a product to be purchased with the accounting machine. A behavior determination method characterized in that a computer executes the process.

[0171] (Appendix 8) The history information of the user's operation on the accounting machine has a first mode of registering a product to be purchased with the accounting machine by reading a product code, and a second mode of settling the products registered with the accounting machine. The determination process includes, when the second mode is executed on the accounting machine and an action of the user operating the accounting machine while holding an object is detected, determining that the object is not an object of a product to be purchased. The behavior determination method according to Appendix 7, characterized in that.

[0172] (Appendix 9) Obtain product information generated by the accounting machine by reading a product code. Based on the obtained product information, count a first number of times indicating the number of products purchased. Identify, from an image of the user in front of the accounting machine, the action of the user registering a product with the accounting machine. Count a second number of times indicating the action of the identified user registering a product with the accounting machine. From the second number of times, count a third number of times obtained by subtracting the number of times determined to be an object that is not a product to be purchased in the determination process. Notify an alert based on the difference between the first number of times and the third number of times. The behavior determination method according to Appendix 8, characterized in that the computer further executes the process.

[0173] (Appendix 10) The determination process includes, based on the history information, a process of determining a specific period as the second mode retroactively from the time when a settlement operation on the accounting machine is performed. The action determination method according to appended note 8, characterized by the following.

[0174] (Appended note 11) The determination process includes a process of determining, based on the history information, that an interval from a settlement preparation operation corresponding to a preliminary operation of settling the price of the product to be purchased to the settlement operation is the second mode. The action determination method according to appended note 10, characterized by the following.

[0175] (Appended note 12) The settlement preparation operation is a selection operation of a payment method for paying the price of the product to be purchased or an operation related to a valuable value given according to the amount of the product to be purchased. The action determination method according to appended note 11, characterized by the following.

[0176] (Appended note 13) Obtain history information of a user's operation on an accounting machine that registers a product to be purchased and performs accounting processing on the registered product, Identify, from an image of the previous user of the accounting machine, an action of the user operating the accounting machine while gripping an object, Based on the obtained history information, determine whether the action of the user operating the accounting machine while gripping an object is an action of registering a product to be purchased in the accounting machine. An action determination device including a control unit that executes processing.

[0177] (Appended note 14) The history information of the user's operation on the accounting machine has a first mode of registering a product to be purchased in the accounting machine by reading a product code and a second mode of settling the registered product in the accounting machine, The determination process includes a process of determining that the object is not a product to be purchased when it detects an action of the user operating the accounting machine while gripping an object at the time when the second mode is executed in the accounting machine. The action determination device according to appended note 13, characterized by the following.

[0178] (Appended note 15) Obtain product information generated when the accounting machine reads a product code. Based on the acquired product information, count a first number indicating the number of purchases of the product, From the image of the user in front of the cash register, identify the action of the user registering a product on the cash register, Count a second number indicating the action of the identified user registering a product on the cash register, From the second number, count a third number obtained by subtracting the number of times an object determined not to be a product to be purchased in the determination process, Based on the difference between the first number and the third number, notify an alert, The behavior determination device according to appended note 14, characterized in that the control unit further executes the process.

[0179] (Appended note 16) The determination process includes a process of determining a specific period as the second mode retroactively from the time when a settlement operation for the cash register is performed based on the history information. The behavior determination device according to appended note 14, characterized in that.

[0180] (Appended note 17) The determination process includes a process of determining an interval from a settlement preparation operation corresponding to a preliminary operation of settling the price of the product to be purchased to the settlement operation as the second mode based on the history information. The behavior determination device according to appended note 16, characterized in that.

[0181] (Appended note 18) The settlement preparation operation is a selection operation of a payment method for paying the price of the product to be purchased or an operation related to a valuable value given according to the amount of the product to be purchased. The behavior determination device according to appended note 17, characterized in that.

Explanation of symbols

[0182] 30 Camera 50 Self-checkout 60 Administrator terminal 100 Information processing device 110 Communication unit 120 Input unit 130 Display unit 140 Memory unit 141 Image buffer 142 History information 143 Model information 144 Data table 145 Judgment table 146 Registration operation information 150 Control unit 151 Acquisition unit 152 Tracking unit 153 Judgment unit 154 Counting unit 155 Output unit

Claims

Claims 1. Obtaining history information of a user's operation on an accounting machine that registers purchased products and performs accounting processing on the registered products, identifying, from an image of the previous user of the accounting machine, an action of operating the accounting machine while the user is holding an object, determining, based on the obtained history information, whether the action of operating the accounting machine while the user is holding an object is an action of registering a product to be purchased in the accounting machine, causing a computer to execute the process, the history information of the user's operation on the accounting machine has a first mode of registering a product to be purchased in the accounting machine by reading a product code, and a second mode of settling the product registered in the accounting machine, the determining process includes a process of determining that the object is not an object of a product to be purchased when an action of operating the accounting machine while the user is holding an object is detected at a time when the second mode is executed in the accounting machine, An action determination program characterized by the above. Claims 2. Obtaining product information generated by the accounting machine by reading a product code, counting a first number indicating the number of purchased products based on the obtained product information, identifying, from an image of the previous user of the accounting machine, an action of the user registering a product in the accounting machine, counting a second number indicating the action of the identified user registering a product in the accounting machine, counting a third number obtained by subtracting the number of times determined to be an object that is not a product to be purchased in the determining process from the second number, notifying an alert when the first number and the third number do not match, The action determination program according to claim 1, further causing the computer to execute the process. Claims 3. The determining process includes a process of determining a specific period as the second mode retroactively from the time when a settlement operation on the accounting machine is performed based on the history information. The action determination program according to claim 1, characterized by the above. Claims 4. The determining process includes a process of determining a section from a settlement preparation operation corresponding to a preliminary operation of settling the price of the product to be purchased to the settlement operation as the second mode based on the history information. The action determination program according to claim 3, characterized by the above. Claims 5. The payment preparation operation is a selection operation of a payment method for paying the price of the product to be purchased or an operation related to a valuable value given according to the amount of the product to be purchased. The action determination program according to claim 4, characterized in that.

6. When a product to be purchased is registered, acquisition of history information of a user's operation on an accounting machine that performs accounting processing of the registered product is performed. From an image of the previous user of the accounting machine, an action of the user operating the accounting machine while gripping an object is specified. Based on the acquired history information, it is determined whether or not the action of the user operating the accounting machine while gripping an object is an action of registering a product to be purchased in the accounting machine. The computer executes the process. The history information of the user's operation on the accounting machine has a first mode of registering a product to be purchased in the accounting machine by reading a product code, and a second mode of settling the product registered in the accounting machine. The determination process includes a process of determining that the object is not an object of a product to be purchased when an action of the user operating the accounting machine while gripping an object is detected at the time when the second mode is executed in the accounting machine. An action determination method, characterized in that.

7. When a product to be purchased is registered, acquisition of history information of a user's operation on an accounting machine that performs accounting processing of the registered product is performed. From an image of the previous user of the accounting machine, an action of the user operating the accounting machine while gripping an object is specified. Based on the acquired history information, it is determined whether or not the action of the user operating the accounting machine while gripping an object is an action of registering a product to be purchased in the accounting machine. It includes a control unit that executes the process. The history information of the user's operation on the accounting machine has a first mode of registering a product to be purchased in the accounting machine by reading a product code, and a second mode of settling the product registered in the accounting machine. The determination process includes a process of determining that the object is not an object of a product to be purchased when an action of the user operating the accounting machine while gripping an object is detected at the time when the second mode is executed in the accounting machine. An action determination device.

Images