Information processing device and control method

Abstract

This contributes to providing information processing equipment and control methods that can quickly provide notifications regarding fraudulent activities. [Solution] The information processing device includes a detection unit that detects which area an item has moved to during the process from when a customer moves an item placed in a first area to a second area and registers the item using a registration device, until the item is moved from the second area to a third area; and a control unit that, when it is detected that the customer has not registered the first item in the second area, performs control to present the first information to the customer until the completion of the move of the first item to the third area is detected.

Description

【Technical Field】 【0001】 The present disclosure relates to an information processing apparatus and a control method. 【Background Art】 【0002】 In recent years, for the purpose of reducing the labor of store clerks in retail stores and the like, the spread of self-checkout has been rapidly progressing. In self-checkout, since the customer himself / herself scans the barcode of the product, it is important to take measures against fraudulent acts such as shoplifting where the product is taken away without paying for it. 【0003】 For example, in Patent Document 1, when it is determined that a product has been taken out of the shopping basket, image data captured by a recognition camera is acquired and analyzed to determine whether the product has been placed in a storage bag. When it is determined that the product has been placed in the storage bag, between the shopping basket and the conducting wire of the storage bag, it is determined whether the barcode of the product has been read and scanned by a barcode scanner. When it is determined that the product has not been scanned between the shopping basket and the conducting wire of the storage bag, an error warning is notified. 【Prior Art Documents】 【Patent Documents】 【0004】 【Patent Document 1】 Japanese Unexamined Patent Application Publication No. 2011-054038 【Summary of the Invention】 【Problems to be Solved by the Invention】 【0005】 However, when, as in Patent Document 1, after it is determined that the product has been placed in the storage bag, it is determined whether the product has been scanned and an error warning is notified, there is a possibility that the customer may be late in noticing the error warning, so the countermeasure against fraudulent acts may not be sufficient. 【0006】 Non-limiting embodiments of this disclosure contribute to the provision of information processing and control methods that enable rapid notification of fraudulent activity. [Means for solving the problem] 【0007】 An information processing device according to one embodiment of the present disclosure includes: a detection unit that detects which area an item has moved to during the operation in which a customer moves an item placed in a first area to a second area, registers the item using a registration device, and moves the item from the second area to a third area; and a control unit that, when it is detected that the customer has not registered the first item in the second area, performs control to present first information to the customer until the completion of the move of the first item to the third area is detected. 【0008】 In one embodiment of the present disclosure, the control method involves an information processing device detecting how far an item has moved during the process from when a customer moves an item placed in a first area to a second area, registers the item using a registration device, and moves the item from the second area to a third area. If the device detects that the customer has not registered the first item in the second area, it provides control to the customer by presenting the first information until the completion of the first item's movement to the third area is detected. 【0009】 These comprehensive or specific embodiments may be implemented as a system, apparatus, method, integrated circuit, computer program, or recording medium, or as any combination of a system, apparatus, method, integrated circuit, computer program, and recording medium. [Effects of the Invention] 【0010】 According to one embodiment of this disclosure, notification of fraudulent activity can be provided promptly. 【0011】 Further advantages and effects in an embodiment of the present disclosure will be clarified from the specification and the drawings. Such advantages and / or effects are respectively provided by some embodiments and the features described in the specification and the drawings, but not all of them are necessarily provided in order to obtain one or more identical features. 【Brief Description of the Drawings】 【0012】 [Figure 1] Figure showing a configuration example of a self-checkout system [Figure 2] Figure showing an example of a system configuration including a plurality of self-checkouts [Figure 3] Figure showing an example of the processing state of a checkout and its transition [Figure 4] Figure showing a first example of the transition of the display on a display [Figure 5] Figure showing a second example of the transition of the display on a display [Figure 6] Figure showing a third example of the transition of the display on a display [Figure 7A] Figure showing an example of area setting [Figure 7B] Figure showing an example of area setting [Figure 8] Figure showing examples of preliminary determination and final determination [Figure 9] Figure showing an example of the flow of preliminary determination and final determination [Figure 10] Figure showing an example of a display on a display [Figure 11] Figure showing an example of the transition from the display of preliminary determination [Figure 12] Flowchart showing a first example of the flow of processing executed in an information processing apparatus [Figure 13] Figure showing an example of the extension of preliminary determination [Figure 14] Figure showing an example of a display for attracting attention [Figure 15] Flowchart showing a second example of the flow of processing executed in an information processing apparatus [Figure 16] Block diagram showing an example of the configuration of an information processing apparatus 【Best Mode for Carrying Out the Invention】 【0013】 Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings. However, a more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and duplicate descriptions of substantially the same configurations may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate understanding by those skilled in the art. 【0014】 Note that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and it is not intended to limit the subject matter described in the claims thereby. 【0015】 <Embodiment> In the shoplifting determination in self-checkout where the customer himself / herself scans the product barcode, when the customer moves the product taken out from the product placement area where the product is placed before scanning to the product placement area where the product is placed after scanning without scanning, it is determined that a scanning omission has occurred. This scanning omission corresponds to shoplifting behavior. Note that the scanning omission corresponds to the act of moving to the product placement area where the product is placed after scanning without scanning the product barcode. The scanning omission may be replaced with a scanning error, non-scanning, etc. 【0016】 In this shoplifting determination, at the stage when the unscanned product has finished moving to the product placement area (the stage when the movement to the product placement area is completed), it is determined that a scanning omission has occurred, and by displaying a message or the like indicating that a scanning omission has occurred on a display for the customer or the like, the customer is notified that a scanning omission has occurred. 【0017】 However, if a customer is notified of an unscanned item after it has been moved to the display area, the customer may not notice the notification. For example, a customer who has just finished scanning an unscanned item may be looking at the next item to scan, and therefore may not notice the notification. In this case, notifying a customer of an unscanned item after it has been moved to the display area will not be sufficiently effective in preventing shoplifting. 【0018】 Therefore, in this embodiment, the possibility of a scan error is determined before the scanned product has finished moving to the product storage area where the scanned product will be placed. If it is determined that there is a possibility that the product has not been scanned, the customer is notified of this possibility before the unscanned product has finished moving to the product storage area. This reduces the possibility that the customer will not notice the notification, and thus the effectiveness of preventing shoplifting can be fully achieved. 【0019】 In the following, the determination of whether or not there may have been a scanning error for an item before it is moved to the designated area after scanning may be referred to as a preliminary determination. Conversely, the determination of whether or not there has been a scanning error for an item after it has finished moving to the designated area may be referred to as a final determination. The final determination is the final determination made at the time the item is placed in the designated area, and the determination result may be changed by an employee after the final determination has been made. The determination of whether or not there has been a scanning error may also be considered as a determination of whether or not shoplifting has occurred. 【0020】 Furthermore, the provisional judgment may also be referred to as a preliminary judgment, preliminary judgment, first judgment, etc. Similarly, the final judgment may be referred to as the main judgment, second judgment, etc. 【0021】 Furthermore, in the following explanation, "product" may refer to "items" that are different from the actual products sold in stores. In the following explanation, "product" may be replaced with "items" or similar terms as appropriate. 【0022】 (System Configuration) Figure 1 is a diagram showing an example configuration of a self-checkout system. As shown in Figure 1, the self-checkout system includes an information processing device 1, a scanner 2, a camera 3, and a display 4. Figure 1 also shows an information processing device 10, a camera 20, a microphone 30, and a display 40 according to this embodiment. In addition to the self-checkout system, Figure 1 also shows product display stands A1a and A1b. Hereinafter, the self-checkout system may be simply referred to as a "cash register" or a "POS (Point of Sales) register." 【0023】 Product display stand A1a is a stand on which products are placed before being scanned by scanner 2. Products placed on product display stand A1a before scanning may include store shopping baskets or carts containing the products before scanning. 【0024】 Product display stand A1b is a stand on which products that have been scanned by scanner 2 are placed. Scanned products placed on product display stand A1b may include the customer's shopping basket or other container into which the scanned products are placed. 【0025】 The information processing device 1 is, for example, a computer such as a personal computer or a server. In Figure 1, the information processing device 1 is placed on the product display stand A1a, but it may also be placed in the store's office. The information processing device 1 performs, for example, accounting processing for products. 【0026】 Scanner 2 is a fixed-type scanner attached to the product display stand A1a. Although not shown in Figure 1, a handheld scanner may also be provided. Scanner 2 is an example of a registration device used for registering products. 【0027】 Scanner 2 is connected to information processing device 1. Scanner 2 scans the product code attached to the product. Scanner 2 transmits the scanned product code to information processing device 1. The product code is a barcode, such as a JAN (Japanese Article Number) code. Scanner 2 may also be called a barcode reader or reader. The area where Scanner 2 scans the product code attached to the product is sometimes called the "scan area". The customer scans the product code by bringing the product close to the scan area and passing it through. Note that scanning the product code attached to the product may simply be called scanning the product. 【0028】 Camera 3 is connected to the information processing device 1. Camera 3 transmits image data of the captured image to the information processing device 1. 【0029】 Camera 3 is installed to photograph customers. However, Camera 3 is not required. 【0030】 Display 4 is connected to information processing device 1. Display 4 displays, for example, product information of an item scanned by a customer using scanner 2. Product information may include, for example, the name, price, and quantity of the product. 【0031】 Display 4 may have a touch panel on its screen surface. The touch panel accepts customer input. For example, a customer can input information about products that do not have product codes, such as vegetables or fruits, via the touch panel and register them as purchased items. For example, if a customer purchases two apples, which do not have product codes such as barcodes, they can input the product name "apple" and the quantity "2" via the touch panel and register them as purchased items. Note that the device that accepts customer input is not limited to a touch panel. The device that accepts customer input may be, for example, a key input device separate from Display 4. Alternatively, customer input may be accepted using a smartphone or mobile device owned by the customer. 【0032】 In the following, the actions of scanning product codes attached to products, and the actions of manually registering information for products that do not have product codes, may be collectively referred to as "product information registration" or "product registration." 【0033】 Furthermore, in the following, the series of processes at the register from when a customer begins registering product information until they pay for the registered products will be referred to as "accounting processing." The process of registering product information will be referred to as "registration processing," and the process of paying for the registered products will be referred to as "payment processing." 【0034】 Display 4 displays, for example, information corresponding to the processing status of the cash register. For example, the processing status of the cash register may include "waiting for accounting processing," "waiting for registration processing," "registration processing," and "waiting for payment." 【0035】 The "checkout processing waiting" state indicates the cash register is waiting to process an transaction before a customer operates the register to complete the transaction. The "registration processing waiting" state refers to the state where a customer is operating the register to complete the transaction, and the register is waiting for the customer to register the product information. The "registration processing" state refers to the state within a specific time period from the moment the product information is registered. The "payment waiting" state refers to the state where the customer has completed registering the information for the products they wish to purchase, and the register is waiting to process the payment. 【0036】 The transitions between each processing state of the cash register, and examples of displays on display 4 corresponding to the processing state of the cash register, will be described later. 【0037】 The information processing device 1 performs accounting processing for the self-checkout system. For example, the information processing device 1 acquires the results of scanning product codes by the scanner 2 and customer operation information (e.g., the results of product information input) via the touch panel on the surface of the display 4, and performs calculations for the total price of the products registered by the customer and payment processing for the amount paid by the customer. 【0038】 The information processing device 1 acquires the results of scanning product codes by the scanner 2 and customer operation information (for example, the result of inputting product information) via the touch panel on the surface of the display 4, and performs display control to display product information of the products registered by the customer on the display 4. In addition, the information processing device 1 performs display control to display information corresponding to the processing status of the register on the display 4 based on the customer operation information via the touch panel on the surface of the display 4. 【0039】 Camera 20 is positioned to capture the area around where the customer scans the product with scanner 2, and also captures at least a portion of the display area of ​​display 4. For example, camera 20 is positioned above display 4. The field of view of camera 20 is set to include the area between product stands A1a and A1b, that is, the area around where the customer scans the product with scanner 2, and also include at least a portion of the display area of ​​display 4. 【0040】 However, camera 20 does not need to photograph the area where the customer scans the product with scanner 2. In this case, camera 20 is positioned to photograph at least a portion of the display area of ​​display 4. 【0041】 Note that the camera 20 is not limited to a single camera. Multiple cameras 20 may be installed. For example, a camera 20 may be installed at a position that photographs the area around where the customer scans the product with the scanner 2, and at a position that photographs at least a portion of the display area of ​​the display 4. When cameras 20 are installed at these two positions, the camera 20 installed at the position that photographs the area around where the customer scans the product with the scanner 2 may be called an object detection camera 20, and the camera 20 installed at the position that photographs at least a portion of the display area of ​​the display 4 may be called a feature point detection camera 20. The camera image captured by the object detection camera 20 may be called an object detection camera image, and the camera image captured by the feature point detection camera 20 may be called a feature point detection camera image. 【0042】 In addition, multiple cameras 20 may capture at least a portion of the display area of ​​the display 4, or multiple cameras 20 may capture the area around where the customer scans the product with the scanner 2. 【0043】 The information processing device 10 is, for example, a computer such as a personal computer or a server. The information processing device 10 may be located in the office of the store. The information processing device 10 is connected to the camera 20 by wire or wireless and acquires camera images taken by the camera 20. The camera images may be still images, multiple still images in chronological order, or moving images. 【0044】 The information processing device 10 may also acquire sound from a microphone 30 installed around the self-checkout system. The microphone 30 acquires sound emitted from the self-checkout system and outputs the acquired sound to the information processing device 10. The sound emitted from the self-checkout system includes, for example, sounds corresponding to the processing status of the register, or sounds corresponding to the transition of the processing status of the register. 【0045】 The information processing device 10 performs a determination process, for example, to determine whether or not there is a possibility of fraudulent activity (shoplifting) based on the acquired camera images and / or audio. The camera images and / or audio are examples of sensing results of information output by the self-checkout system. 【0046】 For example, the information processing device 10 performs object detection processing based on the acquired camera image. The information processing device 10 determines the presence of an object and whether or not the object is moving through the object detection processing. Note that the object detection processing does not have to be based on the camera image. The information processing device 10 also determines the status of the cash register based on the acquired camera image and / or audio. Then, the information processing device 10 determines whether or not shoplifting has occurred based on the results of the object detection processing and the status of the cash register. 【0047】 The information processing device 10 is connected to the display 40 and controls the display on the display 40. 【0048】 The display 40, under the control of the information processing device 10, displays information based on the determination results in the information processing device 10. For example, the display 40 notifies customers of information regarding fraudulent activity. Notifications regarding fraudulent activity include, for example, a message indicating that a scan error has occurred, a message indicating that a scan error may have occurred, and a cautionary message to prevent scan errors. Notifications regarding fraudulent activity may be replaced with warnings regarding fraudulent activity. Furthermore, fraudulent activity here may refer to shoplifting. In this context, shoplifting may correspond to the act of moving a product that has been missed from scanning to product stand A1b without scanning the product (an example of registration). 【0049】 Although Figure 1 shows an example where the information processing device 10 is provided separately from the information processing device 1 of the self-checkout system, this disclosure is not limited to this. The processing of the information processing device 10 according to this embodiment may be executed in the information processing device 1 of the self-checkout system. Furthermore, the information processing device 1 of the self-checkout system may be connected to an external server by wired or wireless connection, and the processing of the information processing device 10 according to this embodiment, and / or a part of the processing of the information processing device 1, may be executed on the external server. 【0050】 Furthermore, although Figure 1 shows an example where the camera 20, microphone 30, and display 40 are added externally to the self-checkout system, at least one of the camera 20, microphone 30, and display 40 may be built into the self-checkout system. 【0051】 As shown in Figure 1, there are multiple self-checkout systems within the store. Each of these self-checkout systems communicates with a terminal installed for store employees, exchanging information with them. 【0052】 Figure 2 shows an example of a system configuration including multiple self-checkout registers. Figure 2 shows an information processing device 10 and a store terminal 200, each of which is provided at registers #1 to #N (where N is an integer of 2 or more). The information processing device 10 and the store terminal 200 communicate with each other, for example, via Wi-Fi (registered trademark). 【0053】 The store terminal 200 may take the form of a tablet, smartphone, or other device, or it may be a stationary information processing device having a display, keyboard, touch panel, etc. The store terminal 200 is used by store staff to monitor the usage status of each self-checkout machine and to monitor shoplifting activity. 【0054】 The information processing device 10 transmits information to the store terminal 200, such as the processing status of the register, the number of times scans were performed, and the number of times scan omissions were detected. The information processing device 10 also transmits information to the store terminal 200 indicating whether or not there are people around each register. Furthermore, the information processing device 10 transmits camera images captured by the camera 20 to the store terminal 200. 【0055】 The store terminal 200 transmits instruction information to the information processing device 10, which is generated based on instructions from the store staff. The instruction information includes instructions on the content of the information to be transmitted by the information processing device 10 to the store terminal 200, instructions on the frequency of transmission of the information to be transmitted by the information processing device 10 to the store terminal 200, and instructions regarding the settings of the information processing device 10. The information processing device 10 generates information to be transmitted to the information processing device 10, sets the transmission frequency, and changes the settings of the information processing device 10 according to the instruction information. 【0056】 (Object detection processing) The object detection process performed in the information processing device 10 will be described. 【0057】 The information processing device 10 detects (recognizes) products included in the image from the camera 20, for example, using a known image recognition technology (image recognition processing program) that utilizes artificial intelligence. In other words, the information processing device 10 detects products taken out from the product stand A1a through image recognition processing. The image recognition processing of products may be performed by another device, such as a server. Note that "detection" refers to the process of determining the existence of an object such as a product and identifying the location of that object, while "recognition" may refer to the process of identifying what the item included in the image is. However, in this embodiment, it is not necessary to accurately identify the product, and it may be difficult to accurately identify the product itself because the shooting angle and range of the camera 20 are limited. Therefore, in this embodiment, "detection" and "recognition" will not be distinguished unless there is a particular need to distinguish between them. 【0058】 (Register processing status) The processing status of the cash register determined by the information processing device 10 will be explained below. As described above, the display 4 displays, for example, the processing status of the cash register. The processing status of the cash register includes "accounting processing waiting state," "registration processing waiting state," "registration processing state," and "payment waiting state," etc. 【0059】 Figure 3 shows an example of the processing states and transitions between them of a cash register. Figure 3 shows the processing states of five cash registers and an example of the transitions between each state. 【0060】 The accounting processing standby state (S11 in Figure 3) is the state in which the cash register is waiting to process an account before the customer operates the register to make a payment. For example, if, while in the accounting processing standby state, the customer gives an instruction to start the payment process via the touch panel on display 4, the processing state of the cash register transitions from the accounting processing standby state to the registration processing standby state (S12 in Figure 3). 【0061】 When a customer scans the product code of a certain product while the register is in the registration processing waiting state, the register's processing state transitions from the registration processing waiting state to the registration processing state (S14 in Figure 3). Then, after a certain period of time has elapsed in the registration processing state, the register's processing state transitions from the registration processing state to the registration processing waiting state. 【0062】 If, while in the registration processing waiting state, a customer instructs the register via the touch panel on display 4 to manually enter a product code in order to register a product that does not have a product code, the register's processing state transitions from the registration processing waiting state to the manual entry waiting state (S13 in Figure 3). 【0063】 The "manual input waiting" state refers to a situation where a customer is operating the register to complete a transaction, and the register is waiting for the customer to manually enter the product information. 【0064】 If, while in manual input waiting state, a customer manually enters information for a product that does not have a product code, the register's processing state transitions from manual input waiting state to registration processing state. Then, after a certain period of time has elapsed in the registration processing state, the register's processing state transitions from registration processing state to registration processing waiting state. 【0065】 If, while the register is in the registration processing waiting state, the customer instructs the register to pay for the goods via the touch panel on display 4, the register's processing state transitions from the registration processing waiting state to the payment waiting state (S15 in Figure 3). 【0066】 When a customer completes payment for goods while the register is in the payment waiting state, the register's processing status transitions from payment waiting state to accounting waiting state. 【0067】 Furthermore, the registration processing waiting state, registration processing state, and manual input waiting state may be collectively referred to as the registration in progress state. In other words, the registration in progress state is a state in which the register processing state is neither payment waiting state nor accounting processing waiting state. 【0068】 In this embodiment, the information processing device 10 determines the processing status of the cash register, as shown in Figure 3, based on the acquired camera image and / or audio. Exemplarily, in this embodiment, the information processing device 10 determines the processing status of the cash register corresponding to the display on the display 4 by detecting feature points in the camera image captured from the display area of ​​the display 4. 【0069】 Alternatively, instead of determining the register's processing status based on the acquired camera images and / or audio, as shown in Figure 3, the information processing device 10 may acquire information regarding the register's processing status from the self-checkout system. When acquiring information regarding the register's processing status from the self-checkout system, the information processing device 10 does not need to determine the register's processing status based on the acquired camera images and / or audio. 【0070】 The following describes examples of displays on the display 4 corresponding to the processing status of the cash register. Note that the following examples are just examples, and this disclosure is not limited to these examples. Furthermore, the following shows the correspondence between the display examples on the display 4 and the processing status of the cash register shown in Figure 3, but this correspondence is just an example, and this disclosure is not limited to this. For example, the correspondence between the display examples on the display 4 and the processing status of the cash register may differ depending on the self-checkout system. For example, even if the display examples on the display 4 are common, the processing status of the cash register corresponding to the display examples may differ depending on the self-checkout system. Note that the correspondence between the display examples on the display 4 and the processing status of the cash register may be stored in the information processing device 10 in advance. Also, if the display examples change due to an update of the self-checkout system, the correspondence stored in the information processing device 10 may be updated. 【0071】 (First example of a display corresponding to the register's processing status) Figure 4 shows a first example of the display transitions of Display 4. Figure 4 shows Display Example 1 and Display Example 2, which transitions from Display Example 1. The display transitions shown in Figure 4 correspond to an example of the transition from the registration processing waiting state to the registration processing state in Figure 3. 【0072】 Display Example 1 is a display example corresponding to the registration processing waiting state. The display area shown in Display Example 1 includes an area for displaying a list of registered product names and prices, an area for displaying the number of registered products and the total price, an area corresponding to the "Payment button", and an area corresponding to the "Product Selection button". 【0073】 The area displaying the list of registered product names and prices shows a list of registered product names and their corresponding prices. In the example in Figure 4, "Beef," "Eggs," "Milk," and "Tea" are registered products, and the product names and prices of each registered product are displayed in association with each other. 【0074】 The area displaying the number of registered items and the total price will show the number of registered items and the total price. In the example in Figure 4, it will show that 4 items are registered and the total price for these 4 items is 1200 yen. 【0075】 When the area corresponding to the "Payment button" is pressed by the customer, the registration process is completed and the system proceeds to the payment processing screen. 【0076】 When the area corresponding to the "product selection button" is pressed by the customer, the system will transition to a screen for registering products that do not have a product code. 【0077】 If the customer scans the product code for "coffee" while Display Example 1 is displayed, the display will transition from Display Example 1 to Display Example 2. Furthermore, if the customer scans the product code for "coffee" while Display Example 1 is displayed, a scanning sound may be emitted from the register's speaker. 【0078】 Display Example 2 is an example of a display showing the registration processing status. The display area shown in Display Example 2 includes an area that displays the product name and price of the most recently registered product. Display Example 2 is an example that shows that the operation of scanning the product code attached to the product (in the example of Figure 4, "coffee") has been performed, and indicates the product that was recognized by the operation of scanning the product code. 【0079】 The area displaying the product name and price of the most recently registered item will show the product name and price of the most recently registered item. In the example in Figure 4, since "coffee" was scanned most recently, the product name "coffee" and its price "200 yen" will be displayed. 【0080】 Furthermore, if a certain amount of time has elapsed since Display Example 2 was displayed, a display corresponding to the registration processing waiting state, similar to Display Example 1, will be displayed. 【0081】 For example, the information processing device 10 detects feature points in the camera image and, based on these feature points, distinguishes and determines whether the display on the display 4 corresponds to a registration processing waiting state as in Display Example 1, or to a registration processing state as in Display Example 2. For example, feature points may be detected that distinguish whether there are four areas arranged within the display area, as in Display Example 1, or whether there is one area within the display area, as in Display Example 2. Alternatively, feature points may be detected that identify whether or not a "payment button" exists within the display area. In the example in Figure 4, the information processing device 10 detects feature points that can distinguish the difference between Display Example 1 and Display Example 2. 【0082】 When the information processing device 10 detects a transition from a registration processing waiting state, as shown in Display Example 1, to a registration processing state, as shown in Display Example 2, it determines that the scan was performed successfully. 【0083】 Furthermore, if the information processing device 10 detects a scanning sound, it may determine that the register's processing state has transitioned from the registration processing standby state to the registration processing state, and that the scan has been performed successfully. 【0084】 Alternatively, instead of displaying an example like in Display Example 2, if scanning a product that requires age verification is being performed, a display indicating that there are restrictions on purchasing the product will be shown, consisting of an area for age verification and display areas corresponding to the "Yes" and "No" age verification buttons. The information processing device 10 may also detect feature points in the camera image and, based on these feature points, determine whether the display on the display 4 corresponds to the age verification display or the registration processing status. 【0085】 Detecting that the register is in the registration processing state, that is, that a scan has been successfully performed, can be done by combining detection of the display on the display 4 corresponding to the registration processing state, detection of the scan sound, etc., as described above. For example, a scan may be detected as having been successfully performed only when the display on the display 4 corresponding to the registration state is detected and the scan sound is detected. In other words, if at least one of the display on the display 4 corresponding to the registration state and the scan sound is not detected, it may be detected as not having been successfully performed. This way, even in environments where the display on the display 4 or the scan sound of other registers is easily detected, such as when other registers are nearby, it is possible to accurately detect that a scan has been successfully performed. Alternatively, a scan may be detected as having been successfully performed when at least one of the display on the display 4 corresponding to the registration state or the scan sound is detected. This way, for example, even when the display 4 may be obstructed due to congestion, or when it is difficult to detect the scan sound due to ambient noise, it is possible to accurately detect that a scan has been successfully performed. As the environment in which a scan can be accurately detected as having been successfully performed differs depending on the combination, store employees may be allowed to change which combination to use depending on the situation. 【0086】 Furthermore, if a customer manually enters information about products without barcodes into the self-checkout system instead of registering products by scanning barcodes, the register's processing status transitions to a manual input waiting state. In this case, detection of the register's processing status being in the manual input waiting state may correspond to the detection of at least one image feature point on display 4, specifically the "list of product names for products without product codes" and the "screen for entering quantities." It may also be possible to select which of the "list of product names for products without product codes" or the "screen for entering quantities" to detect. 【0087】 (Second example of the display corresponding to the register's processing status) Figure 5 shows a second example of the display transitions of Display 4. Figure 5 shows Display Example 1 and Display Example 3, which transitions from Display Example 1. The display transitions shown in Figure 5 correspond to an example of the transition from the registration processing waiting state to the payment waiting state in Figure 3. Note that Display Example 1 in Figure 5 is the same as Display Example 1 in Figure 4, so its explanation is omitted. 【0088】 When a customer has finished registering the items they wish to purchase and presses the area corresponding to the "payment button" to begin payment for the registered items, the display on display 4 transitions from display example 1 to display example 3. 【0089】 Display Example 3 is an example of a display corresponding to the payment waiting state. The display area shown in Display Example 3 includes an area for displaying buttons for selecting a payment method, and an area for displaying the number of registered items and the total amount. 【0090】 When the information processing device 10 detects a transition from a registration processing waiting state, as shown in Display Example 1, to a payment waiting state, as shown in Display Example 3, it determines that it is waiting for a customer's payment. 【0091】 (Third example of a display corresponding to the register's processing status) Figure 6 shows a third example of the display transitions of Display 4. Figure 6 shows Display Example 4 and Display Example 5, which transitions from Display Example 4. 【0092】 Display Example 4 is an example of a display showing the accounting processing waiting state (S11 in Figure 3). The display area shown in Display Example 4 includes an area that displays a list of available payment methods and an area corresponding to the "start button" that is pressed when accounting processing is initiated. 【0093】 When a customer presses the area corresponding to the "start button" to begin the accounting process, the display on display 4 transitions from display example 4 to display example 5. 【0094】 Display Example 5 is a display that allows customers to choose whether or not to purchase a shopping bag to take home the items they have purchased. Display Example 5 is an example of a display that shows the checkout processing waiting state (S11 in Figure 3). 【0095】 When a customer presses either selection button #1 or selection button #2 to enter information indicating whether or not they want to purchase a shopping bag, the display on display 4 transitions from display example 5 to a different display. 【0096】 When the information processing device 10 detects a waiting state for accounting processing, as shown in Display Example 4 or Display Example 5, it determines that it is waiting for the customer to start accounting processing (start of scanning). 【0097】 (Provisional and final determination in this embodiment) As described above, the information processing device 10 performs two types of judgments, a preliminary judgment and a final judgment, in determining whether or not shoplifting has occurred (whether or not there are any scan errors) based on the results of the object detection process and the processing status of the cash register. 【0098】 In the preliminary determination, the information processing device 10 determines whether there is a possibility of a scan omission before the customer completes the movement of the product that has passed near scanner 2 to product stand A1b. For example, if the product that has passed near scanner 2 has been scanned normally, the information processing device 10 determines that there is no possibility of a scan omission, and if the product that has passed near scanner 2 has not been scanned normally, it determines that there is a possibility of a scan omission. 【0099】 If the preliminary assessment determines that there is a possibility of a scan being missed, information indicating this possibility will be displayed on the display 40. The display on the display 40 when the preliminary assessment determines that there is a possibility of a scan being missed may be referred to as "Preliminary Assessment Display" below. 【0100】 In the final determination, the information processing device 10 determines whether or not a scan error occurred when the customer has completed moving the product that has passed near scanner 2 to product stand A1b. For example, if the product that has passed near scanner 2 has been scanned normally, the information processing device 10 determines that no scan error occurred, and if the product that has passed near scanner 2 has not been scanned normally, it determines that a scan error occurred. 【0101】 If the final determination indicates that a scan error has occurred, information indicating that a scan error has occurred will be displayed on display 4. The display on display 40 when the final determination indicates that a scan error has occurred may be referred to below as "Final Determination Display". 【0102】 The provisional judgment displayed on display 40 and the final judgment displayed are different from each other. For example, the provisional judgment may be displayed in a way that softens the impression on the customer compared to the final judgment. Examples of displays on display 40 will be described later. 【0103】 Furthermore, if the preliminary assessment determines that there is a possibility of a scan error, the information processing device 10 does not need to send the preliminary assessment result to the store terminal 200. By not sending the preliminary assessment result to the store terminal 200, the store terminal 200 will not display the preliminary assessment result. This avoids presenting excessive information to the staff operating the store terminal 200, thereby reducing the staff's workload. 【0104】 Here, the preliminary and final checks detect whether the product has been moved to product stand A1b. This detection is performed in the object detection process described above. In this object detection process, an area is defined for the camera images taken of the cash register and its surroundings. 【0105】 The following is an example of setting an area in a camera image. 【0106】 Figures 7A and 7B show examples of area settings. Figure 7A shows the shooting range from which camera 20 photographs the self-checkout system illustrated in Figure 1 from above, as well as examples of the starting area and ending area set within the shooting range. Figure 7BB also shows the shooting range from which camera 20 photographs the self-checkout system illustrated in Figure 1 from above, as well as examples of the personal belongings area and dead zone set within the shooting range. 【0107】 The starting area is the area where the items to be scanned and the shopping basket containing those items are expected to be placed. The starting area corresponds to the area where the movement of the items to be scanned begins. The starting area includes at least the area of ​​product stand A1a. The starting area is set within the range of the register, including product stand A1a, but may include a margin beyond the register's range. A wider area than product stand A1a may also be included in the starting area. For example, in addition to the range of product stand A1a, the starting area may include an area that extends the range of product stand A1a toward the customer (for example, in the direction of arrow G in Figure 7A). Also, a part of product stand A1b may be included in the starting area. 【0108】 The movement of an item may be determined to have begun when the customer picks up the item located in the starting area. Alternatively, the movement of an item may be determined to have begun when the position or size of the item in the starting area changes in the camera image. 【0109】 The end area is the area where the shopping bags, reusable bags, baskets, etc., that will hold scanned items are expected to be placed. The end area corresponds to the area where the movement of items ends. The end area includes at least the area of ​​product stand A1b. The end area is set within the range of the register, including product stand A1b, but may include a margin beyond the register's range. Furthermore, the end area may include a wider area than product stand A1b. For example, the start area may include the area of ​​product stand A1b plus an area extending from product stand A1b towards the customer (for example, in the direction of arrow H in Figure 7B). Also, a portion of product stand A1a may be included in the end area. 【0110】 The movement of the product to the end area may be determined to be complete when the product held in the customer's hand moves to the end area and the position and size of the product in the camera image do not change. Alternatively, the movement of the product to the end area may be determined to be complete when the product held in the hand moves to the end area and it is detected that the hand has released the product. 【0111】 The personal belongings area is the area where a cashier (e.g., a customer) holding personal belongings is expected to be visible, and / or where a customer's personal belongings are expected to be placed. The personal belongings area is set outside the range of the cashier in the camera image. The personal belongings area corresponds to the area where personal belongings that could be mistakenly identified as merchandise (e.g., a customer's wallet, smartphone, etc.) first appear. Note that the personal belongings area does not have to be set. However, the range of the cashier in the camera image may be included in the personal belongings area. For example, part or all of the end area described above may be included in the personal belongings area. Also, part of the start area described above may be included in the personal belongings area. Note that the personal belongings area may overlap with the start area or end area. Here, the priority given to which area overlaps with the start area or end area may be arbitrarily set. For example, in this case, the personal belongings area may be given priority. 【0112】 The dead zone is an area where it is not expected that a cashier operating a register with personal belongings (e.g., a customer) will be captured, and / or where it is not expected that a customer's personal belongings will be placed. The dead zone is set outside the range of the register in the camera image. Because the dead zone may include images of adjacent registers or customers or store staff moving around, the dead zone corresponds to an area that can become noise. Items, people, etc. that appear in the dead zone are excluded from image processing such as product detection. Note that the dead zone does not have to be set. 【0113】 Each area may be configured manually based on the camera image obtained after a camera 20 is installed at the cash register and the field of view of the camera 20 is set. 【0114】 Although not shown in Figures 7A and 7B, the area scanned by scanner 2 may also be set as the scan area. 【0115】 Examples of preliminary and final judgments when an area is defined as described above will be explained. 【0116】 Figure 8 shows examples of preliminary and final judgments. Figure 8 illustrates an example of a customer moving items in the self-checkout system shown in Figure 1. 【0117】 The information processing device 10 detects the product (hereinafter referred to as product #x) that has started moving from the starting area. For example, the information processing device 10 detects the product held by the customer in the starting area as the product that has started moving from the starting area. Hereafter, the product that has started moving from the starting area may be referred to as product #x. 【0118】 In the camera image, a frame may be superimposed on the image in the area of ​​product #x, as shown in Figure 8. 【0119】 The information processing device 10 detects that product #x has moved from the start area to the end area and that product #x has not been scanned, and makes a preliminary determination that there is a possibility that product #x was not scanned. At this preliminary determination stage, the movement of product #x to the end area is not yet complete. For example, product #x is in the process of moving within the end area, as shown in Figure 8. The information processing device 10 determines that product #x is in the process of moving if, for example, product #x is detected leaving the end area and the completion of the movement has not been detected. The information processing device 10 may also determine that product #x has moved to the end area if the entire range of product #x is within the end area, or if at least a portion of the range of product #x is within the end area, or if more than half of the range of product #x is within the end area. 【0120】 Furthermore, when the information processing device 10 detects that the movement of unscanned product #x to the end area is complete, it makes a final determination that a scanning omission has occurred for product #x. At this final determination stage, the movement of product #x to the end area is complete. For example, detection of the completion of product #x's movement to the end area corresponds to detection that product #x is stationary in the end area, or detection that the customer has released product #x from the end area. 【0121】 The timeline of the preliminary and final decisions mentioned above will now be explained. 【0122】 Figure 9 shows an example of the flow between preliminary and final judgment. Figure 9 also shows an example of the relationship between time and the movement of the product. The horizontal axis in Figure 9 represents the time axis. In the example shown in Figure 9, product #x is stationary in the starting area at a time before time point T1. 【0123】 At time T1, it is detected that product #x has started moving from the starting area (e.g., product stand A1a). At time T2, it is detected that product #x has passed through the scan area set near scanner 2. At time T3, it is detected that product #x has been placed in the ending area (e.g., product stand A1b). In this case, a preliminary judgment is performed between time T2 and time T3. A final judgment is then performed from time T3 onward. If it is not detected that product #x has been scanned, a preliminary judgment is made during the preliminary judgment time that there is a possibility that product #x was not scanned. If it is not detected that product #x has been scanned, a final judgment is made during the final judgment time that product #x was not scanned. 【0124】 As mentioned above, the information notified to the customer will differ depending on whether the preliminary determination is that there is a possibility of a scan omission for product #x, or the final determination is that a scan omission has occurred for product #x. Below are examples of how the preliminary determination result and the final determination result will be displayed. Note that the display for the preliminary determination result, that there is a possibility of a scan omission for product, will simply be described as "Preliminary Determination Display" or "Display at the Time of Preliminary Determination," and the display for the final determination result, that a scan omission for product has occurred, will simply be described as "Final Determination Display" or "Display at the Time of Final Determination." Note that if the determination result is notified by a method other than displaying the determination result on a screen, etc., "Display" may be replaced with other terms such as "Notification" or "Presentation." Note that the preliminary determination result that there is no possibility of a scan omission for product, and the final determination result that there is no scan omission for product, do not need to be notified to the customer. 【0125】 (Example of display of provisional and final judgments) Figure 10 shows examples of displays on the display 40. Figure 10 shows examples of displays on the display 40 during normal operation, preliminary judgment, and final judgment. 【0126】 Under normal operation, no messages or other information should be displayed on display 40. Here, "normal operation" refers to a situation where a preliminary determination has not been made that there is a possibility of a scan failure for product #x, or where a final determination has not been made that a scan failure for product #x has occurred. 【0127】 The display of a preliminary determination, that is, when a preliminary determination is made that there may be a scan error for product #x, includes a message informing the customer that there may be a scan error. 【0128】 The final determination display, i.e., the display when the final determination is that a scan error occurred for product #x, includes a message informing the customer that a scan error occurred. 【0129】 Furthermore, the message displayed in the preliminary judgment is different from the message displayed in the final judgment. For example, the message displayed in the final judgment is emphasized more than the message displayed in the preliminary judgment. For example, at least one aspect such as font size, the size of the area where the message is displayed, font color, or display brightness may differ between the message displayed in the preliminary judgment and the message displayed in the final judgment. Also, the position of the area where the message is displayed within the display area of ​​the display 40 may differ between the preliminary judgment display and the final judgment display. In addition, the content of the message may differ between the preliminary judgment display and the final judgment display. For example, the message displayed in the preliminary judgment may be softer and less likely to leave a strong impression on the customer than the message displayed in the final judgment display. 【0130】 Although the display on the display 40 for the preliminary and final judgments has been described, the method of presenting information to the customer is not limited to display on the display 40. For example, information regarding at least one of the results of the preliminary judgment and the final judgment may be presented by other means such as sound or warning lights. Furthermore, even when using means other than display, the method of presenting information between the results of the preliminary judgment and the final judgment may differ. 【0131】 For example, the notification to the customer via sound output from a speaker (not shown in the diagram) may differ between the preliminary and final decisions. For example, the volume of the sound output from the speaker during the preliminary decision may be lower than that during the final decision. Alternatively, if an audio message is output from the speaker, different audio messages may be output between the final and preliminary decisions. Alternatively, the notification to the customer via sound output from the speaker may be performed only during the final decision and not during the preliminary decision. 【0132】 However, for customers who have received a predetermined number of notifications regarding shoplifting during this single transaction, the same message and sound as at the final judgment may be sent at the preliminary judgment stage. Alternatively, for customers who have received a predetermined number of notifications regarding shoplifting during one or more transactions prior to the current transaction, the same message and sound as at the final judgment may be sent at the preliminary judgment stage. Here, the notifications regarding shoplifting may include both the notifications at the preliminary judgment stage and the notifications at the final judgment stage, or either one or the other. Furthermore, the process of identifying whether the person currently performing the transaction is the same person as a person who has performed a transaction in the past may be performed using existing image processing (e.g., processing using AI technology) based on images of the person. By doing so, it becomes possible to increase the effectiveness of shoplifting prevention, for example, in cases where the customer is a habitual shoplifter. 【0133】 Here, the notification at the time of the preliminary determination is given to the customer being judged, but does not need to be given to the store staff. Store staff may not be notified at the time of the preliminary determination, but only at the time of the final determination. Since the notification at the time of the preliminary determination is given even if no shoplifting has actually occurred, notifying store staff at the time of the preliminary determination may result in a large amount of notification information, especially if there are many self-checkout registers. However, by notifying store staff at the time of the final determination and not at the time of the preliminary determination, the increase in notification information can be suppressed. 【0134】 Furthermore, the information notified to the store staff may be changed between the preliminary and final determination stages. For example, during the preliminary determination, the store staff may be notified that the product passed through the scanning area without scanning its barcode, while during the final determination, in addition to the final determination result that the product passed through the scanning area without scanning its barcode and has moved to the final area, the store staff may also be notified of information such as an image of the product. This allows for a reduction in the amount of information notified during the preliminary determination stage, and enables appropriate notification to the store staff at the final determination stage regarding which products have been missed during scanning. 【0135】 (Withdrawal of provisional decision) A preliminary determination that there may be a scan omission may be withdrawn by the detection of a specific customer action after the preliminary determination. For example, the information processing device 10 may withdraw the preliminary determination that there may be a scan omission if it detects a specific customer action. Three actions that may be subject to withdrawal are given below as examples, but the actions subject to withdrawal in this disclosure are not limited to these. 【0136】 Example of a specific action 1: After a preliminary determination that there is a possibility of a scan error, if the movement of the relevant product to product stand A1b is detected, the preliminary determination is withdrawn and a final determination is made that a scan error has occurred. In this case, the display of the preliminary determination transitions to the display of the final determination. 【0137】 Example of a specific action 2: After a preliminary determination that there is a possibility of a scan error, if an action is detected that would require rescanning the item in question, the preliminary determination is withdrawn, and the display transitions from the preliminary determination display to the display for normal operation. Here, actions that would require rescanning the item in question include, for example, moving the item in question to the scan area near scanner 2, or returning the item in question to the starting area. 【0138】 Example 3 of a specific action: If, after a preliminary determination that there is a possibility of a scan failure, the movement of the item in question to product stand A1b is not detected, the preliminary determination is withdrawn, and the display transitions from the preliminary determination display to the display for normal operation. Cases in which the movement of the item in question to product stand A1b is not detected include cases where movement exceeding a predetermined distance is detected within a predetermined time, or cases where no movement of the item is detected within a predetermined time. Here, a case in which the movement of the item in question to product stand A1b is not detected includes cases where the item detected as the item is personal property, etc. In the case of personal property, the customer stores the personal property in their own bag, etc., so personal property is not placed on product stand A1b. Another example of a case in which the movement of the item in question to product stand A1b is not detected is when the conditions for detecting the movement of the item to product stand A1b are not met. In this case, a determination that the conditions for detecting movement were not met may be made, for example, if the product could not be detected for a predetermined period of time while it was moving, or if the distance difference between the location where the product was detected at a certain point in time and the location where the product was detected at one or more points prior to that point is greater than a predetermined value. In these cases, there is a high possibility that an error occurred in movement detection, and the result of the provisional determination that there is a possibility of a scan failure is unreliable, so the provisional determination is withdrawn. 【0139】 Furthermore, example 3 of the specific actions may also apply to shoplifting, such as directly moving non-personal items into a bag, or intentionally concealing items to avoid detection. Therefore, it may be possible to allow store employees to change whether or not this action is set as a condition for withdrawing a provisional judgment. 【0140】 Furthermore, even if a customer moves an item from the scanning area without scanning it, the display 40 will not show a preliminary determination after it has been detected that the register is in a payment waiting state (see Figures 3 and 5). For example, even if a customer moves an item from the scanning area without scanning it, the display will not show a preliminary determination that there may be a missed scan after it has been detected that it is in a payment waiting state. Alternatively, if a customer moves an item from the scanning area without scanning it, the display may show a preliminary determination that there may be a missed scan, but this preliminary determination will not be shown after it has been detected that it is in a payment waiting state. This is because, when the register is in a payment waiting state, it is highly likely that the customer has already completed moving all items, and the item that was subject to the preliminary determination is likely to be a personal item rather than a product. 【0141】 Furthermore, if the register is detected to be in a payment waiting state, the preliminary and final judgments themselves do not need to be performed. Alternatively, the preliminary and / or final judgments may be performed, but the customer may not be notified of the results. Specifically, if the register is detected to be in a payment waiting state, the information processing device 10 may stop the object detection process, or it may perform the object detection process but stop the preliminary and final judgment processes, or it may perform the preliminary and final judgment processes but stop the display control of the display 40. The information processing device 10 may choose to stop the object detection process, stop the preliminary and final judgment processes, or stop the display control. This selection will be made during the initial setup before actual operation begins. 【0142】 Furthermore, when the register is in a payment waiting state, the register is unable to register products. Therefore, regardless of whether a preliminary judgment is performed or the result of the preliminary judgment, products cannot be registered even if they are scanned. Notifying customers of the result of the preliminary judgment in this state would cause confusion. To avoid this confusion, the information processing device 10 should either withdraw the preliminary judgment or not display the result of the preliminary judgment. 【0143】 Furthermore, even if a customer moves an item out of the scanning area without scanning it, if the register's processing status is detected as being in a waiting state for payment processing (see Figures 3 and 6), the display 40 will not show a preliminary judgment. For example, even if a customer moves an item out of the scanning area without scanning it, if the waiting state for payment processing is detected, the display will not show a preliminary judgment indicating a possible missed scan. Alternatively, if a customer moves an item out of the scanning area without scanning it, the display may show a preliminary judgment indicating a possible missed scan, but if the waiting state for payment processing is detected, the display will not show the preliminary judgment. 【0144】 Furthermore, if the register is detected to be in a state of waiting for accounting processing, the information processing device 10 may stop the object detection process, or it may continue the object detection process but stop the preliminary and final judgment processes, or it may continue the preliminary and final judgment processes but stop the display control of the display 40. The information processing device 10 may choose to stop the object detection process, stop the preliminary and final judgment processes, or stop the display control. 【0145】 Here, we will explain the display transitions when a provisional decision is withdrawn. 【0146】 Figure 11 shows an example of the transition from the provisional judgment display. Similar to Figure 10, Figure 11 shows examples of the display during normal operation, the provisional judgment display, and the final judgment display. 【0147】 As shown in Figure 11, when Example 1 of a specific operation is detected, the display on the display 40 transitions from the provisional judgment display to the final judgment display. Also, as shown in Figure 11, when Example 2 or Example 3 of a specific operation is detected, the display on the display 40 transitions from the provisional judgment display to the normal operation display. 【0148】 (First example of the processing flow in this embodiment) Next, we will describe a first example of the processing flow executed in the information processing device 10. 【0149】 Figure 12 is a flowchart showing a first example of the processing flow executed in the information processing device 10. The processing shown in Figure 12 is initiated, for example, when a customer who is to pay for items to be purchased approaches the register and the display on the display 4 is detected to have changed in the camera image, or when an item is detected in the camera image. Whether or not a customer has approached the register may be determined, for example, based on whether a human body is visible in the camera image, or based on the detection result of a human presence sensor installed at the register. In addition, in the processing shown in Figure 12, the information processing device 10 acquires camera images at predetermined intervals, performs object detection processing, and determines the processing status of the register. The processing shown in Figure 12 is executed based on the results of the object detection processing and the determination of the processing status of the register. 【0150】 The information processing device 10 determines whether or not the scanning of all items to be purchased has been completed (S101). For example, the information processing device 10 determines that the scanning of all items to be purchased has been completed if the processing status of the register indicated by the feature point is in the payment waiting state. Conversely, the information processing device 10 determines that the scanning of all items to be purchased has not been completed if the processing status of the register indicated by the feature point is in a state other than the payment waiting state. 【0151】 If all scans are not complete (NO in S101), that is, if there are items to be purchased that have not been scanned, the information processing device 10 determines whether or not movement of the item to be registered (hereinafter referred to as item #k) from the starting area has been detected (S102). 【0152】 If no movement of product #k to be registered is detected (NO in S102), the flow returns to S101. 【0153】 If the movement of product #k to be registered is detected (YES in S102), the information processing device 10 determines whether or not a scan of the detected product #k has been performed (S103). For example, the information processing device 10 determines whether or not a scan of product #k has been performed based on whether or not the processing state of the register indicated by the feature points is in the registration processing state. 【0154】 If a scan has not been performed (NO in S103), the information processing device 10 determines whether or not it has detected that product #k has finished moving to the end area (S104). For example, if the feature points do not indicate a registration processing state and the feature points do not indicate a manual input waiting state, it is determined that a scan of the product has not been performed. 【0155】 If it is detected that product #k has completed moving to the end area (YES in S104), the information processing device 10 determines that a scan error has occurred (S105). In other words, it determines that shoplifting has occurred. For example, if product #k moves to the end area and remains stationary for a predetermined time, it is determined that product #k has completed moving to the end area. Alternatively, if product #k moves to the end area and the hand holding product #k leaves product #k, it is determined that product #k has completed moving to the end area. Alternatively, if product #k moves to the end area and the hand holding product #k leaves product #k, and product #k remains stationary for a predetermined time, it is determined that product #k has completed moving to the end area. 【0156】 Then, the information processing device 10 controls the display 40 to display the final determination (S106). 【0157】 If it is not detected that product #k has finished moving to the end area (NO in S104), the information processing device 10 makes a preliminary determination that there may be a scan omission and controls the display 40 to display the preliminary determination (S107). 【0158】 The information processing device 10 determines whether a specific operation to withdraw the display of the provisional judgment has occurred (S108). The specific operation to withdraw the display of the provisional judgment is, for example, one of the specific operations described above, such as Example 2 or Example 3. 【0159】 If a specific action occurs that causes the provisional judgment display to be withdrawn (YES in S108), the information processing device 10 withdraws the provisional judgment display and controls the display 40 to display the normal operation display (S109). Then the flow returns to S101. 【0160】 If no specific action to withdraw the provisional judgment display has occurred (NO in S108), the flow returns to S104, and the information processing device 10 determines whether or not it has detected that product #k has finished moving to the end area (S104). In other words, even after displaying the provisional judgment, the movement of product #k is tracked until the shoplifting act is finally confirmed or an action to withdraw the provisional judgment occurs. 【0161】 If a scan is performed (YES in S103), the information processing device 10 controls the display 40 to display the information as it would during normal operation (S110). 【0162】 Next, the information processing device 10 determines whether or not it has been detected that product #k has completed moving to the end area (S111). 【0163】 If it is not detected that product #k has finished moving to the end area (NO in S111), the flow returns to S111 and S111 is repeated until product #k has finished moving to the end area. 【0164】 If it is detected that product #k has finished moving to the end area, the flow returns to S101. 【0165】 If all scans are complete (YES in S101), meaning there are no planned purchase items that have not yet been scanned, the flow shown in Figure 12 will end. 【0166】 As described above, according to this embodiment, if there is a possibility that an item may not have been scanned between the time the item passes through the scanning area and the time the item moves to the end area, the customer can be notified of this possibility, thus enabling prompt notification to the customer regarding fraudulent activity (e.g., shoplifting). 【0167】 (Variation 1 of the embodiment: Extension of provisional determination) As illustrated in Figure 9, the preliminary determination process determines whether there is a possibility of a missed scan between the time the product passes through the scan area and the completion of its movement to the end area. Here, we will explain an example where the preliminary determination is extended to alert the customer even before the product passes through the scan area. 【0168】 Here, the information processing device 10 determines that it will issue a warning to prevent any scanning errors from occurring between the time it detects the start of movement of the product in the starting area and the time it detects that the product has passed through the scanning area. 【0169】 Figure 13 shows an example of an extension of the provisional judgment. Similar to Figure 9, Figure 13 shows an example of the relationship between time and the movement of the product. The horizontal axis in Figure 13 represents the time axis. In the example shown in Figure 13, product #x is stationary in the starting area at time points prior to T1. The times for the provisional and final judgments in Figure 13 are the same as in Figure 9, so their explanation is omitted. 【0170】 In the example in Figure 13, if it is detected at time T1 that product #x has started moving from the starting area (for example, product stand A1a), the information processing device 10 decides to issue a warning. Then, if it is detected at time T2 that product #x has passed through the scan area set near scanner 2, the information processing device 10 withdraws the warning and makes a preliminary determination as in the example in Figure 9 as to whether or not there is a possibility of a scan being missed. 【0171】 If it is determined that a warning should be issued at this point, the information processing device 10 controls the display 40 to display a warning. The warning displayed here may differ from the provisional determination display and the final determination display described above. 【0172】 Figure 14 shows an example of a warning display. Similar to Figures 10 and 11, Figure 14 shows the display during normal operation, the display for the provisional judgment, and the display for the final judgment. Figure 14 also shows an example of a warning display when the provisional judgment is extended. 【0173】 As shown in Figure 14, the warning display shows information prompting a scan. Note that at the warning stage, the possibility of a missed scan is not determined, so information indicating a missed scan does not need to be displayed. 【0174】 (Example of the processing flow for Variation 1) Next, we will explain an example of the processing flow executed in the information processing device 10 when the above-described extension of the provisional determination is applied. 【0175】 Figure 15 is a flowchart showing a second example of the processing flow executed in the information processing device 10. Note that in Figure 15, processes similar to those in Figure 12 are given the same reference numerals, and their explanations may be omitted. 【0176】 In Figure 15, S201 is added compared to Figure 12. As shown in Figure 15, if movement of product #k to be registered is detected (YES in S102), the information processing device 10 controls the display 40 to display a warning (S201). The information processing device 10 then determines whether or not a scan of the detected product #k has been performed (S103). 【0177】 As described above, according to Variation 1 of this embodiment, by alerting the customer to prevent any scanning omissions from occurring between the detection of the start of movement of the product in the starting area and the detection of the product passing through the scanning area, it is possible to quickly notify the customer of any fraudulent activity (e.g., shoplifting). 【0178】 (Example configuration of information processing device 10) Figure 16 is a block diagram showing an example configuration of an information processing device 10. The information processing device 10 shown in Figure 16 includes a receiving unit 101, a control unit 102, and a transmitting unit 103. The control unit 102 includes an object detection unit 104, a feature point detection unit 105, a sound detection unit 106, a state determination unit 107, a provisional determination unit 108, a final determination unit 109, a display information generation unit 110, a display control unit 111, an alarm information generation unit 112, and a transmission control unit 113. The control unit 102 may be composed of a processor, such as a CPU (central processing unit). For example, each component included in the control unit 102 may be classified as either a "detection unit" or a "control unit" depending on its processing content. For example, at least one of the object detection unit 104, the feature point detection unit 105, the sound detection unit 106, the state determination unit 107, the provisional determination unit 108, and the final determination unit 109 may be referred to as a detection unit included in the control unit 102. 【0179】 The receiving unit 101 acquires camera images from, for example, the camera 20. For example, if the camera 20 includes an object detection camera 20 and a feature point detection camera 20, the receiving unit 101 acquires an object detection camera image captured by the object detection camera 20 and a feature point detection camera image captured by the feature point detection camera 20. Note that the object detection camera image and the feature point detection camera image may be the same camera image. In other words, the camera 20 may capture an image that can detect feature points on the screen displayed on the display 4 and can also detect products being moved by the customer. 【0180】 The receiving unit 101 may also acquire audio from the microphone 30. 【0181】 The object detection unit 104 detects objects based on camera images (e.g., camera images for object detection). For example, the object detection unit 104 detects from the camera images what kind of product a customer is holding and / or what kind of product a customer is moving. The object detection unit 104 outputs object detection information about the detected object to the provisional determination unit 108. The object detection information includes information indicating whether or not the object detection unit 104 has detected an object. If the object detection unit 104 has detected an object, the object detection information may also include information indicating what the detected object is. If the object detection unit 104 has detected the start of movement of an object, the object detection information may also include information indicating the timing of the start of movement of the detected object. Furthermore, the object detection information may include the direction of movement of the detected object, the destination, the position where the movement was completed, etc. For example, the object detection unit 104 may detect which area the product has moved to, such as a start area or an end area, and the result of this detection may be included in the object detection information. 【0182】 Furthermore, in the object detection process, the object detection unit 104 may, instead of detecting what the product held by the customer is, detect whether or not the customer is holding an object. Alternatively, instead of detecting what the product the customer is moving is, the object detection unit 104 may detect whether or not the customer is moving an object. In other words, in the object detection process, detection (recognition) of what the object is is not required. In this case, the object detection information may include information indicating whether or not the customer is holding an object, and / or information indicating whether or not the customer is moving an object. Also, the object detection unit 104 may perform the object detection process based on other information instead of camera images (e.g., camera images for object detection). 【0183】 The feature point detection unit 105 detects feature points in the camera image. For example, the feature point detection unit 105 detects feature points in an image that indicates the processing status of the cash register corresponding to the display on the display 4, and outputs image feature point information indicating the detected image feature points to the state determination unit 107. 【0184】 The voice detection unit 106 detects sounds and other noises that occur when a product is scanned, based on the sound picked up by the microphone 30. 【0185】 The state determination unit 107 determines the processing state of the register corresponding to the display on the display 4. For example, the state determination unit 107 determines the processing state of the register corresponding to the display on the display based on image feature point information. Alternatively, the state determination unit 107 may determine that the processing state of the register is the registration processing state if it detects the sound generated when an item is scanned. The processing state of the register to be determined is at least one of the states shown in Figure 3, for example. Note that determination and detection may be interchangeable. 【0186】 The preliminary determination unit 108 makes a preliminary determination of whether or not there is a possibility of scan errors occurring. For example, the preliminary determination unit 108 makes a preliminary determination of whether or not there is a possibility of scan errors occurring based on the processing status of the register determined by the status determination unit 107 and the object detection information obtained from the object detection unit 104. For example, if the object detection information indicates that the product has passed through the peripheral area of ​​scanner 2 and moved to the end area, and the operation status related to payment indicates that the product is not being registered, the preliminary determination unit 108 determines that there is a possibility of scan errors occurring. Here, the peripheral area is described as being the same area as the start area, but the peripheral area and the start area do not have to be exactly the same. For example, a part of the start area that is close to the end area may be considered the peripheral area. Also, for example, if the object detection information indicates that the product has passed through the peripheral area of ​​scanner 2 and moved to the end area, and the operation status related to payment indicates that the product is being registered, the preliminary determination unit 108 determines that there is no possibility of scan errors occurring. 【0187】 If the provisional determination unit 108 has determined that there is a possibility of scan omissions occurring, and then detects a specific operation, it may withdraw its provisional determination that there is a possibility of scan omissions occurring. 【0188】 Furthermore, the provisional determination unit 108 may determine whether or not an action has been taken to warn of a missed scan. For example, if the object detection information indicates that the product has started moving from the starting area but has not yet passed through the area surrounding the scanner 2, the provisional determination unit 108 determines that an action has been taken to warn of a missed scan. In other words, if it is detected that the period is between the time a certain product has started moving from the starting area and the time when the product passes through the scan area to be scanned, the provisional determination unit 108 determines that an action has been taken to warn of a missed scan. 【0189】 The final determination unit 109 makes a final determination as to whether or not a scan error has occurred. This final determination as to whether or not a scan error has occurred corresponds to the determination as to whether or not a "shoplifting act" has occurred. For example, the final determination unit 109 makes a final determination as to whether or not a scan error has occurred based on the register processing status determined by the status determination unit 107 and the object detection information obtained from the object detection unit 104. For example, if the object detection information indicates that the product has completed moving to the end area and the operation status related to payment indicates that the product is not being registered, the final determination unit 109 determines that a scan error has occurred. For example, if the object detection information indicates that the product has completed moving to the end area and the operation status related to payment indicates that the product is being registered, the final determination unit 109 determines that no scan error has occurred. 【0190】 The display information generation unit 110 generates information to be displayed based on the result of the determination in the provisional determination unit 108 and / or the result of the determination in the final determination unit 109. If the provisional determination unit 108 determines that there is a possibility of scan failure, the display information generation unit 110 generates display information for the provisional determination. If the provisional determination unit 108 determines to withdraw the provisional determination that there is a possibility of scan failure, the display information generation unit 110 generates an instruction to withdraw the display information for the provisional determination. If the provisional determination unit 108 determines that an action to warn about scan failure is being taken, the display information generation unit 110 generates warning display information. If the final determination unit 109 determines that a scan failure has occurred, the display information generation unit 110 generates display information for the final determination. 【0191】 Furthermore, if the final determination unit 109 determines that a scan error has occurred, the display information generation unit 110 may extract the product image of the product that was missed from the camera image. 【0192】 Furthermore, the display information generation unit 110 does not generate any information to display if the final determination unit 109 determines that no scan omissions have occurred, or if the provisional determination unit 108 determines that there is no possibility of scan omissions occurring. 【0193】 The display control unit 111 controls the display on the display 40 installed in the cash register. For example, the display control unit 111 causes information to be displayed on the display 40 according to the display information or instructions generated by the display information generation unit 110. 【0194】 The alarm information generation unit 112 generates alarm information indicating a shoplifting warning when the final determination unit 109 determines that a scan failure has occurred. In this case, the alarm information may include product images of the products that were not scanned, which have been extracted by the display information generation unit 110. 【0195】 The transmission control unit 113 generates transmission information including alarm information and transmits the transmission information via the transmission unit 103. The transmission information may be transmitted to, for example, a terminal owned by store staff, security guards, etc., or to a display device that store staff, security guards, etc. can check. Store staff and security guards may check the transmitted alarm information and take appropriate action, such as issuing a warning, to customers who may be shoplifting. The transmission information may also be transmitted to a display device located outside the store, for example, a display device for remotely monitoring shoplifting. 【0196】 In this embodiment, an example is shown in which an information processing device 10 provided in each self-checkout counter performs object detection processing, preliminary determination, and final determination. However, this disclosure is not limited to this. At least one of the object detection processing, preliminary determination, and final determination may be performed by other devices such as a store terminal 200 or a server installed in the store. For example, if the object detection processing is performed by another device, the information processing device 10 transmits camera images, etc., to the other device (e.g., a store terminal 200 or a server installed in the store) and obtains the results of the object detection processing from the other device. 【0197】 Furthermore, in the above embodiment, the information processing device 10 (for example, the object detection unit 104) detected the movement of the product using camera images. However, the movement of the product may be detected by other means. For example, the information processing device 10 may detect the movement of the product based on changes in weight detected by weight sensors provided on product stands A1a and A1b. Alternatively, for example, if the product is fitted with a wireless tag such as an RFID, the information processing device 10 may detect the movement of the product by detecting the movement of the wireless tag. In addition, the information processing device 10 may detect the movement of the product by using both the camera image for object detection and the detection results from the weight sensors or wireless tags. In this case, the information processing device 10 may detect which area the product has moved to based on the detection results from the weight sensors or wireless tags and determine whether the movement of the product has been completed. For example, if the weight detected from product stand A1a decreases, it may be determined that the movement of the product has started, and if the weight detected from product stand A1b increases, it may be determined that the movement of the product has been completed. Alternatively, the location of the product detected by the wireless tag may be used to determine which area the product has moved to. If the information processing device 10 is configured to acquire information indicating that a scan has been performed when the scanner 2 scans a product, the same operation as in the above embodiment can be achieved without using camera images. 【0198】 Furthermore, in the above embodiment, the information processing device 10 determined the processing status of the cash register corresponding to the display on the display 4 by detecting feature points of the image. However, the information processing device 10 may also determine the processing status of the cash register using other methods, such as recognizing characters displayed on the display 4 or using pattern matching of the entire image displayed on the display 4. 【0199】 The object detection unit 104 (an example of a detection unit) of the control unit 102 of the information processing device 10 described above detects which area the product has moved to during the operation in which the customer moves a product (an example of an item) placed in the starting area (an example of a first area) to the scanning area (an example of a second area) to scan the product, and then moves the product from the scanning area to the ending area. The display control unit 110 (an example of a control unit) detects that the customer has not scanned the product in the scanning area, and performs display control to display a provisional judgment (an example of presenting information to the customer) until the completion of the product's movement to the ending area is detected. 【0200】 In addition, the "...part" in the above embodiment may be a "...circuitry", a "...device", a "...unit", or a "...module". 【0201】 This disclosure can be implemented using software, hardware, or software integrated with hardware. 【0202】 Each functional block used in the description of the above embodiments may be implemented partially or entirely as an integrated circuit (LSI), and each process described in the above embodiments may be controlled partially or entirely by a single LSI or a combination of LSIs. An LSI may consist of individual chips, or it may consist of a single chip that includes some or all of the functional blocks. An LSI may have data inputs and outputs. Depending on the degree of integration, LSIs may be referred to as ICs, system LSIs, super LSIs, or ultra LSIs. 【0203】 The method of integration is not limited to LSIs; it may also be implemented using dedicated circuits, general-purpose processors, or dedicated processors. Furthermore, FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI manufacturing, or reconfigurable processors that allow for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used. This disclosure may be implemented as digital or analog processing. 【0204】 Furthermore, if advancements in semiconductor technology or related technologies lead to the emergence of integrated circuit technologies that replace LSIs, then naturally, these technologies can be used to integrate functional blocks. The application of biotechnology, for example, is a possible possibility. 【0205】 This disclosure is applicable to all types of devices, systems, and equipment with communication capabilities (collectively referred to as communication devices). Non-exclusive examples of communication devices include telephones (mobile phones, smartphones, etc.), tablets, personal computers (PCs) (laptops, desktops, notebooks, etc.), cameras (digital still / video cameras, etc.), digital players (digital audio / video players, etc.), wearable devices (wearable cameras, smartwatches, tracking devices, etc.), game consoles, digital book readers, telehealth and telemedicine devices, vehicles or mobile transport with communication capabilities (automobiles, airplanes, ships, etc.), and combinations of the above-mentioned devices. 【0206】 Communication devices are not limited to portable or movable devices, but also include all kinds of non-portable or fixed devices, devices, and systems, such as smart home devices (appliances, lighting equipment, smart meters or measuring instruments, control panels, etc.), vending machines, and any other "things" that may exist on an IoT (Internet of Things) network. 【0207】 Communication includes data communication via cellular systems, wireless LAN systems, and communication satellite systems, as well as data communication using combinations of these. 【0208】 Furthermore, the communication device also includes devices such as controllers and sensors that are connected to or linked to a communication device that performs the communication functions described in this disclosure. For example, this includes controllers and sensors that generate control signals and data signals used by the communication device that performs the communication functions of the communication device. 【0209】 Furthermore, communication equipment includes infrastructure facilities such as base stations, access points, and any other devices, devices, and systems that communicate with or control the aforementioned non-limited types of equipment. 【0210】 Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of this disclosure. Furthermore, the components in the above embodiments may be combined in any way without departing from the spirit of the disclosure. 【0211】 The specific examples of this disclosure have been described in detail above, but these are merely illustrative and do not limit the scope of the claims. The technologies described in the claims include various modifications and changes to the specific examples described above. [Industrial applicability] 【0212】 This disclosure is useful for detecting fraudulent activity in self-checkout systems. [Explanation of Symbols] 【0213】 1.10 Information Processing Device 2 Scanners 3.20 Camera 30 microphones 4, 40 displays A1a, A1b Product display stand 101 Receiving Unit 102 Control Unit 103 Transmitter 200 store terminals

Claims

[Claim 1] The system includes a detection unit that detects which area the item has moved to during the process from when the customer moves an item placed in the first area to the second area and registers the item using a registration device, until the customer moves the item from the second area to the third area. If the system detects that the customer has not registered the first item in the second area, the control unit performs control to present the first information to the customer until the completion of the movement of the first item to the third area is detected. An information processing device equipped with the following features. [Claim 2] The information processing device acquires an image of the customer's actions, The detection unit detects in the image whether or not the customer has registered the first item. The information processing apparatus according to claim 1. [Claim 3] If the control unit detects that the customer has not registered the first item in the second area, and if it detects that the first item has completed moving to the third area, it performs control to present second information that is different from the first information. The information processing apparatus according to claim 1. [Claim 4] When the control unit detects that the customer has started moving the first item from the first area, it performs control to present a third piece of information different from the first and second pieces of information until it detects that the customer has not registered the first item in the second area, or has registered the first item. The information processing apparatus according to claim 3. [Claim 5] If the control unit detects that the customer has registered the first item in the second area, it will not present the first information to the customer. The information processing apparatus according to claim 1. [Claim 6] If the control unit detects that the customer has not registered the first item in the second area, and if it detects a specific action before the first item has completed moving to the third area, it stops presenting the first information to the customer. The information processing apparatus according to claim 1. [Claim 7] The aforementioned specific action is at least one of the following actions: the customer moving the first item to the first area or the second area, and the customer moving the first item to an area different from the first area, the second area, and the third area. The information processing apparatus according to claim 6. [Claim 8] If the control unit detects that the registration device is unable to register the item, it will not output the first information, even if it detects that the customer has not registered the first item in the second area. The information processing apparatus according to claim 1. [Claim 9] Information processing device, During the process in which a customer moves an item placed in the first area to the second area and registers the item using a registration device, and then moves the item from the second area to the third area, the system detects which area the item has moved to. If it is detected that the customer has not registered the first item in the second area, the system will provide the customer with the first information until the completion of the first item's movement to the third area is detected. Control method.

Images