Information processing device and determination method

Abstract

To provide an information processing device and a determination method for appropriately detecting a fraudulent activity.SOLUTION: The information processing device includes: an acquisition unit that acquires a camera image obtained by capturing an image of at least a part of a display screen of a register for registering information about a commodity to be purchased by a customer; a first detection unit that detects the presence / absence of movement of the commodity; a second detection unit that based on the display screen of the register included in the camera image and / or a sound issued from the register, detects the presence / absence of registration of the information about the commodity; and a determination unit that determines whether or not to execute measures against commodity shoplifting based on the presence / absence of movement of the commodity and the presence / absence of registration of the commodity.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and a determination method. [Background technology]

[0002] In recent years, the spread of self-checkout systems has been rapidly increasing in order to reduce the man-hours of store staff in retail stores. However, since customers scan the barcodes of products themselves, it is important to take measures to prevent fraudulent activities such as shoplifting.

[0003] There are self-checkout registers that use weight sensors to detect shoplifting. For example, a device with a self-scanning function disclosed in Patent Document 1 has a storage unit that stores products whose product codes have been read, and the storage unit has a weight sensor. The device in Patent Document 1 issues an alarm if the weight of the storage unit increases without a product being scanned. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 7-141569 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in self-checkout registers that use weight sensors such as those described in Patent Document 1 to detect fraudulent behavior by customers, it is necessary to link the sensing of the weight sensor with the reading of product codes, but it is difficult to link this in real time, and there are cases where fraudulent behavior cannot be detected properly.

[0006] Non-limiting examples of the present disclosure contribute to providing an information processing device and a determination method that can appropriately detect fraudulent activity. [Means for solving the problem]

[0007] An information processing device according to one embodiment of the present disclosure includes an acquisition unit that acquires a camera image of at least a portion of a display screen of a cash register on which information about a product purchased by a customer is registered; a first detection unit that detects whether the product has been moved; a second detection unit that detects whether the product information has been registered based on the display screen of the cash register and / or a sound emitted from the cash register that is included in the camera image; and a determination unit that determines whether to implement anti-shoplifting measures for the product based on whether the product has been moved and whether the product has been registered.

[0008] In a determination method according to one embodiment of the present disclosure, an information processing device acquires a camera image of at least a portion of a display screen of a cash register on which information about a product purchased by a customer is registered, detects whether the product has been moved, detects whether the information about the product has been registered based on the display screen of the cash register and / or a sound emitted from the cash register contained in the camera image, and determines whether to implement anti-shoplifting measures for the product based on whether the product has been moved and whether the product has been registered.

[0009] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0010] According to an embodiment of the present disclosure, fraudulent activity can be appropriately detected.

[0011] Further advantages and benefits of an embodiment of the present disclosure will become apparent from the specification and drawings. Such advantages and / or benefits may be provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of them may be provided to obtain one or more identical features. [Brief explanation of the drawings]

[0012] [Figure 1] Diagram showing an example of the self-checkout system configuration [Figure 2] FIG. 1 is a diagram illustrating an example of object detection processing. [Figure 3] A diagram showing an example of the cash register processing state and its transition [Figure 4] FIG. 10 is a diagram showing a first example of a transition of a display on a display; [Figure 5] FIG. 10 is a diagram showing a second example of a transition of the display on the display; [Figure 6] FIG. 10 is a diagram showing a third example of a transition of the display on the display; [Figure 7] 1 is a flowchart showing a first example of a flow of processing executed in an information processing device. [Figure 8] 10 is a flowchart showing a second example of the flow of processing executed in the information processing device. [Figure 9] 10 is a flowchart showing a third example of the flow of processing executed in the information processing device. [Figure 10] 10 is a flowchart showing a fourth example of the flow of processing executed in the information processing device. [Figure 11] FIG. 10 is a diagram showing a fourth example of a transition of the display on the display; [Figure 12] FIG. 5 is a diagram showing a fifth example of transition of display on the display; [Figure 13] FIG. 6 is a diagram showing a sixth example of transition of display on the display; [Figure 14] 10 is a flowchart showing a fifth example of the flow of processing executed in the information processing device. [Figure 15] A block diagram showing an example of the configuration of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or redundant explanation of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art.

[0014] The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0015] <Embodiment> (System Configuration) FIG. 1 is a diagram showing an example of the configuration of a self-checkout system. As shown in FIG. 1, the self-checkout system has an information processing device 1, scanners 2a and 2b, a camera 3, weight sensors 4a and 4b, and a display 5. FIG. 1 also shows an information processing device 10, a camera 20, and a microphone 30 according to this embodiment. In addition to the self-checkout system, FIG. 1 also shows product stands A1a and A1b and a stand A2. Hereinafter, the self-checkout system may be simply referred to as a "cash register" or a "POS (Point of Sales) cash register."

[0016] The product stand A1a is a stand on which products are placed before being scanned by the scanners 2a and 2b. The products placed on the product stand A1a before being scanned may include shopping baskets or carts from a store containing products before being scanned.

[0017] The product stand A1b is a stand on which products are placed after being scanned by the scanners 2a and 2b. The scanned products placed on the product stand A1b may include a customer's shopping basket into which the scanned products are placed.

[0018] The information processing device 1 is, for example, a computer such as a personal computer or a server. In Fig. 1, the information processing device 1 is placed outside the counter A2, but it may be placed inside the counter A2 or in the store office. The information processing device 1 performs, for example, a checkout process for products.

[0019] The scanner 2a is a handheld scanner. The scanner 2b is a fixed scanner fixed to the stand A2. Hereinafter, when there is no need to distinguish between the scanners 2a and 2b, they will be simply referred to as the scanner 2.

[0020] The scanner 2 is connected to the information processing device 1. The scanner 2 scans the product code attached to the product. The scanner 2 transmits the scanned product code to the information processing device 1. The product code is, for example, a barcode such as a JAN (Japanese Article Number) code. The scanner 2 may also be referred to as a barcode reader or a reader.

[0021] The camera 3 is connected to the information processing device 1. The camera 3 transmits image data of a captured image to the information processing device 1.

[0022] Camera 3 is positioned to photograph customers.

[0023] The weight sensor 4a is connected to the information processing device 1. The weight sensor 4a is installed, for example, inside the product stand A1a, and transmits to the information processing device 1 a signal corresponding to the weight of the product placed on the product stand A1a (or a shopping basket or the like containing the product).

[0024] The weight sensor 4b is connected to the information processing device 1. The weight sensor 4b is installed, for example, inside the product stand A1b, and transmits to the information processing device 1 a signal corresponding to the weight of the product placed on the product stand A1b (or a customer's shopping basket containing the product, etc.).

[0025] The display 5 is connected to the information processing device 1. The display 5 displays, for example, product information of a product scanned by a customer using the scanner 2. The product information may be, for example, the name, price, and quantity of the product.

[0026] The display 5 may be provided with a touch panel on its screen surface. The touch panel accepts customer operations. A customer inputs information about a product without a product code, such as a vegetable or fruit, via the touch panel and registers the product as a purchased item. For example, when a customer purchases two apples without a barcode, the customer inputs the product name "apple" and the purchase quantity "2" via the touch panel and registers the purchased item. Note that the device that accepts customer operations is not limited to a touch panel. The device that accepts customer operations may be, for example, a key input device separate from the display 5. Alternatively, customer operations may be accepted using a smartphone or mobile terminal owned by the customer.

[0027] In the following, the operation of scanning the product code attached to the product and the operation of manually registering information about a product that does not have a product code attached may be collectively referred to as "registering product information" or "registering a product."

[0028] In the following, the series of processes at the cash register from when a customer starts registering product information to when they pay for the registered products will be referred to as the “checkout process.” The process of registering product information will be referred to as the “registration process,” and the process of paying for the registered products will be referred to as the “payment process.”

[0029] The display 5 displays, for example, a display corresponding to the processing status of the cash register. For example, the processing status of the cash register includes "checkout processing waiting state," "registration processing waiting state," "registration processing state," and "payment waiting state."

[0030] The transaction waiting state indicates the state in which the register is waiting to process the transaction before the customer operates the register to complete the transaction. The registration waiting state indicates the state in which the register is waiting for the customer to register product information while the customer is operating the register to complete the transaction. The registration processing state indicates the state within a specific period of time from the point in time when product information is registered. The payment waiting state indicates the state in which the customer has completed registering information about the product they wish to purchase and the register is waiting to process the payment.

[0031] The transitions between the processing states of the cash register and examples of displays on the display 5 corresponding to the processing states of the cash register will be described later.

[0032] The information processing device 1 performs accounting processing for the self-checkout system. For example, the information processing device 1 acquires the results of scanning product codes with the scanner 2 and customer operation information (e.g., product information input results) via the touch panel on the screen surface of the display 5, and performs calculations of the total price of the products registered by the customer and processing of the payment made by the customer.

[0033] The information processing device 1 acquires the result of scanning the product code by the scanner 2 and the customer's operation information (for example, the result of inputting product information) via the touch panel on the screen surface of the display 5, and performs display control to display the product information of the product registered by the customer on the display 5. The information processing device 1 also performs display control to display information corresponding to the processing status of the cash register on the display 5 based on the customer's operation information via the touch panel on the screen surface of the display 5, etc.

[0034] Camera 20 is installed in a position where it can capture an image of the area around where a customer scans an item with scanner 2 (see dotted frame A3 in FIG. 1 ) and at least a portion of the display area of ​​display 5. For example, camera 20 is installed above display 5. The angle of view of camera 20 is set to be between product stand A1a and product stand A1b, that is, an angle of view that includes the area around where a customer scans an item with scanner 2 and at least a portion of the display area of ​​display 5.

[0035] However, the camera 20 does not have to capture an image of the area where the customer scans the product with the scanner 2. In this case, the camera 20 is installed in a position where it captures an image of at least a part of the display area of ​​the display 5.

[0036] The number of cameras 20 is not limited to one in this example. Multiple cameras 20 may be installed. For example, a camera 20 may be installed at a position where it captures an area around where a customer scans a product with the scanner 2 (see dotted frame A3 in FIG. 1 ) and a position where it captures at least a portion of the display area of ​​the display 5. When cameras 20 are installed at these two positions, the camera 20 installed at a position where it captures an area around where a customer scans a product with the scanner 2 may be referred to as an object detection camera 20, and the camera 20 installed at a position where it captures at least a portion of the display area of ​​the display 5 may be referred to as a feature point detection camera 20. A camera image captured by the object detection camera 20 may be referred to as an object detection camera image, and a camera image captured by the feature point detection camera 20 may be referred to as a feature point detection camera image.

[0037] Furthermore, multiple cameras 20 may capture at least a portion of the display area of ​​the display 5, or multiple cameras 20 may capture the area around where the customer scans the product with the scanner 2.

[0038] The information processing device 10 is, for example, a computer such as a personal computer or a server. The information processing device 10 may be placed in a store office. The information processing device 10 is connected to a camera 20 via a wired or wireless connection and acquires camera images captured by the camera 20. The camera images may be still images, multiple still images that are consecutive in time, or moving images.

[0039] Furthermore, the information processing device 10 may acquire audio from a microphone 30 installed in the vicinity of the self-checkout system. The microphone 30 acquires audio emitted from the self-checkout system and outputs the acquired audio to the information processing device 10. The audio emitted from the self-checkout system includes, for example, a sound corresponding to the processing state of the checkout or a sound corresponding to a transition of the processing state of the checkout.

[0040] The information processing device 10 performs a process of determining whether or not there is a possibility of fraudulent activity (shoplifting) based on the acquired camera images and / or audio. The camera images and / or audio are an example of the sensing results of information output in a self-checkout system. Note that a case where there is a possibility of fraudulent activity (shoplifting) may correspond to a case where it is necessary to take measures against shoplifting. In other words, determining whether or not there is a possibility of fraudulent activity (shoplifting) corresponds to determining whether or not it is necessary to take measures against shoplifting.

[0041] For example, the information processing device 10 performs an object detection process based on the acquired camera images. The information processing device 10 determines the presence of an object and whether the object is moving through the object detection process. Note that the object detection process does not have to be a method based on camera images. The information processing device 10 also determines the processing status of the cash register based on the acquired camera images and / or audio. Then, the information processing device 10 determines whether or not shoplifting countermeasures are necessary based on the results of the object detection process and the processing status of the cash register.

[0042] 1 shows an example in which the information processing device 10 is provided separately from the information processing device 1 of the self-checkout system, but the present disclosure is not limited to this. The processing of the information processing device 10 according to the present embodiment may be executed in the information processing device 1 of the self-checkout system. Furthermore, the information processing device 1 of the self-checkout system may be connected to an external server via a wired or wireless connection, and the processing of the information processing device 10 according to the present embodiment and / or part of the processing of the information processing device 1 may be executed in the external server.

[0043] (Object detection processing) The object detection process executed in the information processing device 10 will be described.

[0044] FIG. 2 is a diagram illustrating an example of object detection processing. The information processing device 10 detects (recognizes) merchandise included in an image captured by the camera 20, for example, using a known image recognition technology (image recognition processing program) employing artificial intelligence. In other words, the information processing device 10 detects merchandise removed from the merchandise stand A1a through image recognition processing. The merchandise image recognition processing may be performed by a separate device, such as a server. Note that "detection" refers to the process of determining the presence of an object, such as a merchandise, and identifying the location of that object, while "recognition" may refer to the process of identifying the item included in the image. However, in this embodiment, accurate merchandise identification is not required, and accurate merchandise identification itself may be difficult due to the limited angle and range of imaging available to the camera 20. Therefore, in this embodiment, "detection" and "recognition" will not be distinguished from each other unless a distinction is particularly necessary.

[0045] The information processing device 10 calculates a recognition score for each candidate product (candidate product) detected by image recognition processing. The recognition score indicates the degree of accuracy of the detection result of the detected candidate product. In the example of FIG. 2, the information processing device 10 detects five candidate products as products included in the image of the camera 20, but this is not limited to this. The number of candidate products detected by the information processing device 1 may be, for example, one or six or more.

[0046] The information processing device 10 may determine (finally detect) the candidate product with the highest recognition score as the product included in the image of the camera 20. For example, in the example of FIG. 2, steak beef B may be finally detected as the product included in the image of the camera 3.

[0047] Alternatively, if the candidate product has the highest recognition score and the recognition score of the candidate product is equal to or greater than a predetermined threshold (first threshold), the information processing device 10 may finally detect the candidate product with the highest recognition score as the product included in the image of the camera 20. For example, the first threshold is set to 90%. In this case, if the recognition score of the candidate product with the highest recognition score is 90% or greater, the information processing device 10 may finally detect the candidate product with the highest recognition score as the product included in the image of the camera 20.

[0048] (Cashier processing status) The following describes the cash register processing status determined by the information processing device 10. As described above, the display 5 displays, for example, a display corresponding to the cash register processing status. The cash register processing status includes "checkout processing standby status," "registration processing standby status," "registration processing status," and "payment standby status."

[0049] 3 is a diagram showing an example of the cash register processing states and their transitions. In FIG. 3, five cash register processing states and examples of the transitions between the states are shown.

[0050] The transaction waiting state (S11 in FIG. 3) is the state in which the cash register is waiting to process a transaction before the customer operates the cash register to complete the transaction. For example, if a customer in the transaction waiting state issues a transaction start instruction via the touch panel of display 5, the cash register's processing state transitions from the transaction waiting state to the registration waiting state (S12 in FIG. 3).

[0051] In the registration processing standby state, when a customer scans the product code of a certain product, the processing state of the register transitions from the registration processing standby state to the registration processing state (S14 in FIG. 3). Then, after a certain period of time has passed in the registration processing state, the processing state of the register transitions from the registration processing state to the registration processing standby state.

[0052] In the registration processing standby state, if a customer instructs the cash register via the touch panel of the display 5 to manually input information in order to register a product that does not have a product code, the processing state of the cash register transitions from the registration processing standby state to the manual input standby state (S13 in Figure 3).

[0053] The manual input standby state refers to a state in which a customer is operating the cash register to make a transaction, and the cash register is waiting for the customer to manually input product information.

[0054] In the manual input standby state, if a customer manually inputs information about a product that does not have a product code, the cash register's processing state transitions from the manual input standby state to the registration processing state. Then, after a certain period of time has passed in the registration processing state, the cash register's processing state transitions from the registration processing state to the registration processing standby state.

[0055] In the registration processing standby state, if the customer instructs via the touch panel of the display 5 to pay for the product, the processing state of the cash register transitions from the registration processing standby state to the payment standby state (S15 in Figure 3).

[0056] When the customer completes payment for the merchandise in the payment waiting state, the cash register processing state transitions from the payment waiting state to the transaction processing waiting state.

[0057] The registration waiting state, the registration processing state, and the manual input waiting state may be collectively referred to as a "registration in progress state." In other words, the registration in progress state is a state in which the cash register is neither in a payment waiting state nor in a transaction waiting state.

[0058] In this embodiment, the information processing device 10 determines the processing state of the cash register based on the acquired camera image and / or audio, as shown in Fig. 3. For example, in this embodiment, the information processing device 10 determines the processing state of the cash register corresponding to the display on the display 5 by detecting feature points of the image from a camera image taken of the display area of ​​the display 5.

[0059] Below, display examples of the display 5 corresponding to the cash register processing status are described. Note that the display examples shown below are merely examples, and the present disclosure is not limited to these display examples. Also, while the correspondence between the display examples of the display 5 and the cash register processing statuses shown in FIG. 3 is shown below, the correspondence shown below is merely an example, and the present disclosure is not limited to this. For example, the correspondence between the display examples of the display 5 and the cash register processing statuses may differ depending on the self-checkout system. For example, even if the display examples of the display 5 are the same, the cash register processing statuses corresponding to the display examples may differ depending on the self-checkout system. Note that the correspondence between the display examples of the display 5 and the cash register processing statuses may be stored in advance in the information processing device 10. Also, if the display examples change due to an update to the self-checkout system, the correspondence stored in the information processing device 10 may also be updated.

[0060] (First example of display corresponding to the cash register processing status) Fig. 4 is a diagram showing a first example of a transition of the display on the display 5. Fig. 4 shows a display example 1 and a display example 2 which transitions from the display example 1. The transition of the display examples shown in Fig. 4 corresponds to an example of a transition from a registration processing standby state to a registration processing state in Fig. 3.

[0061] Display example 1 is a display example corresponding to the registration process standby state. The display area shown in display example 1 includes an area displaying a list of registered product names and prices, an area displaying the number of registered products and the total price, an area corresponding to the "Payment button," and an area corresponding to the "Product selection button."

[0062] In the area displaying a list of registered product names and prices, the product names of the registered products and the corresponding prices are displayed in a list. In the example of Fig. 4, "beef," "eggs," "milk," and "tea" are registered products, and the product names and prices of each registered product are displayed in association with each other.

[0063] The area displaying the number of registered items and the total price displays the number of registered items and the total price. In the example of Figure 4, four items have been registered, and the total price of the four items is displayed as 1,200 yen.

[0064] When the customer presses the area corresponding to the "Payment button," the registration process is completed and the display shifts to the payment process screen.

[0065] When the customer presses the area corresponding to the "product selection button," the display shifts to a screen for registering a product that does not have a product code.

[0066] When a customer scans the product code for "coffee" while Display Example 1 is displayed, the display changes from Display Example 1 to Display Example 2. Note that when a customer scans the product code for "coffee" while Display Example 1 is displayed, a scanning sound may be emitted from the speaker at the cash register.

[0067] Display example 2 is a display example showing the registration processing status. The display area shown in display example 2 includes an area showing the product name and price of the most recently registered product. Display example 2 is an example that displays that an action has been taken to scan the product code attached to the product ("coffee" in the example of FIG. 4), and shows the product that was recognized by the action of scanning the product code.

[0068] The area that displays the product name and price of the most recently registered product displays the product name and price of the most recently registered product. In the example in Figure 4, since "coffee" was the most recently scanned item, the product name "coffee" and its price "200 yen" are displayed.

[0069] If a certain time has elapsed after the display example 2 is displayed, a display corresponding to the registration process standby state similar to the display example 1 is displayed.

[0070] For example, the information processing device 10 detects feature points in the camera image and, based on the feature points, distinguishes between whether the display on the display 5 is a display corresponding to a registration processing standby state as in Display Example 1 and whether the display is in a registration processing state as in Display Example 2. For example, a feature point that distinguishes whether four areas are lined up in the display area as in Display Example 1 or whether one area is present in the display area as in Display Example 2 may be detected. Also, for example, a feature point that specifies whether a "payment button" is present in the display area may be detected. In the example of FIG. 4, the information processing device 10 detects feature points that can distinguish between Display Example 1 and Display Example 2.

[0071] When the information processing device 10 detects the scanning sound, it may determine that the processing state of the cash register has transitioned from the registration processing standby state to the registration processing state.

[0072] The information processing device 10 may adjust the content (for example, position, size, etc.) of the feature points to be detected according to the display content of the display 5.

[0073] (Second example of display corresponding to the cash register processing status) FIG. 5 is a diagram showing a second example of the transition of the display on the display 5. FIG. 5 shows display example 1 and display example 3, which transitions from display example 1. FIG. 5 shows an example in which the product to be scanned is "beer," which corresponds to a product that requires age verification. The transition of the display examples shown in FIG. 5 corresponds to another example of the transition from the registration processing standby state to the registration processing state in FIG. 3. Note that display example 1 in FIG. 5 is the same as display example 1 in FIG. 4, and therefore a description thereof will be omitted.

[0074] In the example of FIG. 5, when a customer scans the product code for "beer" while display example 1 is being displayed, the display transitions from display example 1 to display example 3.

[0075] Display example 3 is a display example showing a state in which a product requiring age verification has recently been scanned. In other words, display example 3 is a display indicating that there are restrictions on the purchase of a product recognized by the action of scanning a product code attached to the product ("beer" in the example of FIG. 5). The state in which a product requiring age verification is being scanned may correspond to the display example of the registration processing state. The display area shown in display example 3 displays an area for displaying age verification and a display area corresponding to the "Yes" and "No" age verification buttons.

[0076] The customer checks the message displayed in the age verification display area and presses the area corresponding to either "Yes" or "No." In the example of FIG. 5, if the customer presses the area corresponding to "Yes," the product name "Beer" and its price are displayed, as in display example 2 of FIG. 4. Note that in the example of FIG. 5, if the customer presses the area corresponding to "No," a warning is issued to prevent the purchase of the product. Display example 3 is not limited to beer, and can be applied to any product with an age restriction for purchase. For example, display example 3 can also be applied to shochu, sake, etc.

[0077] For example, the information processing device 10 detects feature points in the camera image and, based on the feature points, distinguishes between whether the display on the display 5 corresponds to a registration processing standby state, as in Display Example 1, and whether the display on the display 5 corresponds to a registration processing state, as in Display Example 3. For example, feature points that distinguish whether four areas are lined up in the display area, as in Display Example 1, or whether three areas are present in the display area, as in Display Example 3, may be detected. Furthermore, for example, feature points that identify whether a "payment button" is present in the display area may be detected. In the example of FIG. 5, the information processing device 10 detects feature points that can distinguish between Display Example 1 and Display Example 3. Note that, in the above example, a configuration for detecting the registration processing state was described using a display example for a product with an age restriction for purchase. However, the registration processing state can be detected in a similar manner for products with other restrictions besides age restrictions for purchase. For example, if confirmation of a certain qualification (e.g., membership qualification held by a customer eligible to purchase) or input of a reservation number is required for purchase, the registration processing state may be detected using a display example that indicates this.

[0078] Note that the state in which a product requiring age verification is being scanned corresponds to the display example of the registration processing state, so the processing state of the cash register may be determined to be the registration processing state when the display on the display 5 is a display example of the state in which a product requiring age verification is being scanned, as in Display Example 3, or a display example showing the registration processing state, as in Display Example 2. Which of these two cases is used for the determination may be selected arbitrarily.

[0079] (Third example of display corresponding to the cash register processing status) Fig. 6 is a diagram showing a third example of the transition of the display on the display 5. Fig. 6 shows display example 1, display example 4, which is a transition from display example 1, and display example 5, which is a transition from display example 4. Fig. 6 shows an example in which a customer registers a product that does not have a product code attached via the touch panel. Note that display example 1 in Fig. 6 is the same as display example 1 in Fig. 4, and therefore a description thereof will be omitted.

[0080] In the example of Figure 6, when Display Example 1 is displayed and a customer presses the area corresponding to the "Product Selection Button" to register Product B, which is a product that does not have a product code, the display transitions from Display Example 1 to Display Example 4.

[0081] Display example 4 is a display example showing the state in which a product without a barcode is being registered. Display example 4 may be considered to be a display example showing the manual input standby state (S13 in FIG. 3). The display area shown in display example 4 displays a list of product names of products sold in the store that do not have product codes. In display example 4, products A to F are displayed as the product names of products that do not have product codes. The areas for products A to F are areas that correspond to the selection buttons for manually input products. When display example 4 is displayed, the customer presses the selection button for the product name of the product to be registered.

[0082] When Display Example 4 is displayed and the customer presses the selection button for the product name of the product to be registered, the display transitions from Display Example 4 to Display Example 5. In the example of FIG. 6, for example, the selection button for "Product B" is pressed.

[0083] Display example 5 is a display example showing a state in which the quantity of the product to be purchased of the product name selected in display example 4 (product B in the example of FIG. 6) is registered. Display example 4 may be considered a display example showing a manual input standby state (S13 in FIG. 3). The display area shown in display example 5 displays an area corresponding to buttons for the numbers "0" to "9," an area corresponding to a "Cancel" button, and an area corresponding to a "Register" button.

[0084] When display example 5 is displayed, the customer presses the area corresponding to the number button to input the number of product B to be purchased. After inputting the number of product B to be purchased, the customer presses the "Register" button to complete the registration of product B. If the customer inputs the wrong number of products to be purchased, they may press the "Cancel" button to cancel the information they entered.

[0085] In addition, when display example 5 is displayed, after the customer presses the area corresponding to the number button to input the number of product B to be purchased, the product name "product B" and its price may be displayed on display 5, as in display example 2 of Figure 4.

[0086] For example, the information processing device 10 detects feature points in the camera image and, based on the feature points, distinguishes between whether the display on the display 5 is a display corresponding to a registration processing standby state, as in Display Example 1, and whether the display is a manual input standby state, as in Display Example 4 or Display Example 5. For example, feature points that distinguish whether four areas are lined up in the display area, as in Display Example 1, whether many product names are displayed in a list, as in Display Example 4, or whether numeric buttons are displayed in a list, as in Display Example 5, may be detected. Furthermore, feature points that specify whether a "payment button" is present in the display area may be detected. In the example of FIG. 6, the information processing device 10 detects feature points that can distinguish between Display Example 1 and Display Example 4 or Display Example 5.

[0087] Since both Display Example 4 and Display Example 5 are displays corresponding to the manual input standby state, the cash register processing state may be determined to be the manual input standby state when a screen displaying a list of products is displayed as in Display Example 4, or when a screen for inputting the quantity is displayed as in Display Example 5. It may be arbitrarily selected which of these two cases to make the determination in.

[0088] (First example of processing flow in this embodiment) Next, a description will be given of a first example of the flow of processing executed in the information processing device 10. In the first example, the registration processing state and the manual input standby state are determined based on feature points of a camera image.

[0089] Fig. 7 is a flowchart showing a first example of the flow of processing executed by information processing device 10. The processing shown in Fig. 7 is started, for example, when a customer who is paying for a product they plan to purchase approaches the cash register and a change in the display on display 5 is detected in the camera image, or when a product is detected in the camera image. Note that whether or not a customer has approached the cash register may be determined, for example, based on whether a human body is captured in the camera image, or based on the detection result of a human presence sensor installed in the cash register.

[0090] The information processing device 10 acquires a camera image from the camera 20 (S101). The information processing device 10 performs object detection and / or feature point detection based on the camera image.

[0091] The information processing device 10 determines whether or not the registration of all the products to be purchased has been completed (S102). For example, if the processing status of the cash register indicated by the feature point is a payment waiting state, the information processing device 10 determines that the registration of all the products to be purchased has been completed. Furthermore, if the processing status of the cash register indicated by the feature point is a state other than a payment waiting state, the information processing device 10 determines that the registration of all the products to be purchased has not been completed.

[0092] If all registrations have not been completed (NO in S102), that is, if there are products to be purchased that have not been registered, the information processing device 10 determines whether object detection has been performed for the products to be registered (S103).

[0093] If object detection has not been performed for the product to be registered (NO in S103), the flow returns to S101.

[0094] If object detection of the product to be registered is performed (YES in S103), the information processing device 10 determines whether the processing state of the cash register indicated by the feature point is a registration processing state (S104).

[0095] If the feature point indicates the registration processing state (YES in S104), the flow returns to S101.

[0096] If the feature point does not indicate the registration processing state (NO in S104), the information processing device 10 determines whether the processing state of the cash register indicated by the feature point is a manual input standby state (S105).

[0097] If the feature point indicates a manual input standby state (YES in S105), the flow returns to S101.

[0098] If the feature points do not indicate a manual input standby state (NO in S105), the information processing device 10 executes anti-shoplifting measures (S106), and the flow returns to S101.

[0099] Note that the shoplifting countermeasures in the present disclosure are not particularly limited. For example, as a shoplifting countermeasure, the information processing device 10 may transmit alarm information to a terminal owned by the store staff, security guard, or the like to warn the store staff, security guard, or the like. Alternatively, the information processing device 10 may display alarm information, such as information indicating that a registration process such as a barcode scan has not been performed, on a display for customers installed near the cash register. The information processing device 10 may also warn the customer by outputting sound from a speaker installed near the cash register. Alternatively, a light or the like may be installed at the cash register, and the information processing device 10 may perform an operation such as turning on the light when an action that may be shoplifting is performed.

[0100] If all registrations have been completed (YES in S102), the flow shown in FIG. 7 ends.

[0101] Through the process shown in FIG. 7, the information processing device 10 detects an action in which, despite object detection of a product to be registered, the product code of the product to be registered is not scanned and product information of the product to be registered is not manually entered. This action corresponds to a customer moving the product but not registering the product information, and corresponds to an action that may be shoplifting. The information processing device 10 can detect such actions that may be shoplifting in real time using camera images, and therefore can appropriately detect fraudulent activity. Furthermore, the information processing device 10 can detect actions that may be shoplifting in real time without using information from sensors (cameras, weight sensors) included in the self-checkout system, and therefore can appropriately detect fraudulent activity with a simple configuration.

[0102] (Second example of processing flow in this embodiment) Next, a description will be given of a second example of the flow of processing executed in the information processing device 10. In the second example, the registration processing state is determined based on voice, and the manual input standby state is determined based on feature points of a camera image.

[0103] Fig. 8 is a flowchart showing a second example of the flow of processing executed in the information processing device 10. In Fig. 8, the same processes as those in Fig. 7 are denoted by the same reference numerals, and the description thereof may be omitted.

[0104] The difference between FIG. 7 and FIG. 8 is that S104 in FIG. 7 is replaced with S201 in FIG.

[0105] 8, when object detection of the product to be registered is performed (YES in S103), the information processing device 10 determines whether the processing state of the cash register indicated by the sound is the registration processing state (S201). For example, when the information processing device 10 detects a scanning sound generated from a speaker or the like of the cash register, it determines that the processing state of the cash register indicated by the sound is the registration processing state. The sound generated when the product is scanned may be referred to as the scanning sound.

[0106] If the voice indicates the registration processing state (YES in S201), the flow returns to S101.

[0107] If the voice does not indicate the registration processing state (NO in S201), the information processing device 10 determines whether the processing state of the cash register indicated by the feature point is a manual input standby state (S105).

[0108] If the feature point indicates a manual input standby state (YES in S105), the flow returns to S101.

[0109] If the feature points do not indicate a manual input standby state (NO in S105), the information processing device 10 executes anti-shoplifting measures (S106), and the flow returns to S101.

[0110] Through the process shown in FIG. 8, the information processing device 10 detects an action in which, despite object detection of a product to be registered, the product code of the product to be registered is not scanned and product information of the product to be registered is not manually entered. This action corresponds to a customer moving the product but not registering the product information, and corresponds to an action that may be shoplifting. The information processing device 10 can detect such actions that may be shoplifting in real time using camera images, and therefore can appropriately detect fraudulent activity. Furthermore, the information processing device 10 can detect actions that may be shoplifting in real time without using information from sensors (cameras, weight sensors) included in the self-checkout system, and therefore can appropriately detect fraudulent activity with a simple configuration.

[0111] (Third example of processing flow in this embodiment) Next, a description will be given of a third example of the flow of processing executed in the information processing device 10. In the third example, the registration processing state is determined based on both the feature points of the camera image and the audio, and the manual input standby state is determined based on the feature points of the camera image.

[0112] Fig. 9 is a flowchart showing a third example of the flow of processing executed in the information processing device 10. In Fig. 9, the same processes as those in Figs. 7 and 8 are denoted by the same reference numerals, and descriptions thereof may be omitted.

[0113] In FIG. 9, when object detection of the product to be registered is performed (YES in S103), the information processing device 10 determines whether the processing state of the cash register indicated by the feature point is a registration processing state (S104).

[0114] If the feature point indicates the registration processing state (YES in S104), the flow returns to S101.

[0115] If the feature point does not indicate the registration processing state (NO in S104), the information processing device 10 determines whether the processing state of the cash register indicated by the voice is the registration processing state (S201).

[0116] If the voice indicates the registration processing state (YES in S201), the flow returns to S101.

[0117] Here, the determination time of the feature point detection process and the determination time of the voice detection process may differ. In such a case, the earlier determination process may be performed first, followed by the later determination process. Alternatively, the determination processes may be performed after the timings of the feature point detection process and the voice detection process are synchronized. In FIG. 9 , the determination time of the feature point detection process is, for example, the timing of the determination in S104, and the determination time of the voice detection process is, for example, the timing of the determination in S201. The determination times of the feature point detection process and the voice detection process may differ when, for example, the timing at which the display on the display 5 switches to the registration process state after a barcode scan is performed differs from the timing at which voice is output after the barcode scan by a predetermined time or more. Alternatively, the determination times of the feature point detection process and the voice detection process may differ when the processing time from when an image is acquired from the camera 20 to when the feature point detection process is completed differs from the processing time from when voice is acquired from the microphone 30 to when the voice detection process is completed by a predetermined time or more.

[0118] If the voice does not indicate the registration processing state (NO in S201), the information processing device 10 determines whether the processing state of the cash register indicated by the feature point is a manual input standby state (S105).

[0119] If the feature point indicates a manual input standby state (YES in S105), the flow returns to S101.

[0120] If the feature points do not represent the image at the time of manual input (NO in S105), the information processing device 10 executes anti-shoplifting measures (S106), and the flow returns to S101.

[0121] Through the process shown in FIG. 9 , the information processing device 10 detects an action in which, despite object detection of a product to be registered, the product code of the product to be registered is not scanned and product information of the product to be registered is not manually entered. This action corresponds to a customer moving the product but not registering the product information, which may be shoplifting. The information processing device 10 can detect such actions that may be shoplifting in real time using camera images, thereby appropriately detecting fraudulent activity. Furthermore, the information processing device 10 can detect actions that may be shoplifting in real time without using information from sensors (cameras, weight sensors) included in the self-checkout system, thereby appropriately detecting fraudulent activity with a simple configuration. In other words, even if an existing self-checkout system does not have sufficient interfaces for exchanging information with other devices, a function for detecting actions that may be shoplifting can be added to the self-checkout system.

[0122] (Variation 1 of this embodiment) As described above, the operation of registering product information includes a scanning operation of scanning a product code and a manual input operation of manually inputting product information. In the scanning operation, the customer scans the product code of the product to be scanned by bringing it close to the scanner 2. Therefore, the scanning process of the product to be scanned is performed from the start of movement of the product to be scanned to the completion of movement. Note that the start of movement here may be, for example, the timing when the product begins to move from the product stand A1a. Furthermore, the completion of movement may be the timing when the product has completed its movement to the product stand A1b.

[0123] On the other hand, in manual input operations, the customer may perform manual input before moving the product to be manually input from the product stand A1a, may perform manual input after the product to be manually input has been moved to the product stand A1b, or may perform manual input during the movement.

[0124] As described above, in a manual input operation, the timing of manually registering product information may vary compared to a scanning operation. Therefore, in a variation of this embodiment, a time interval for determining the registration process status is set, thereby suppressing erroneous determinations caused by variations in the timing of manual registration of product information.

[0125] For example, two time periods, Period 1 and Period 2, are set. The registration process status is determined within Period 1 when the registration process is performed by a scanning operation, and the registration process status is determined within Period 2 when the registration process is performed by a manual input operation. By setting Period 2 as a longer time period than Period 1, the valid period for determining the registration process status when product information is registered by manual input is extended, thereby suppressing erroneous determinations caused by variations in the timing of registration when product information is registered by manual input.

[0126] There are no particular limitations on the method for setting Period 1 and Period 2. For example, in object detection, the periods may be set based on the timing at which the product to be registered is detected, the timing at which a product registered before the product to be registered is detected, and offsets added to or subtracted from these timings.

[0127] For example, period 1 may be the period from the time when the product to be registered is first detected to the time when it is last detected in object detection.

[0128] For example, period 2 may be the period from the time when a product registered immediately before the product to be registered was last detected in object detection to the time when the product to be registered was first detected. Alternatively, period 2 may be the period from the time when a product registered immediately before the product to be registered was last detected in object detection to the time when a product scheduled to be registered after the product to be registered was first detected. Alternatively, period 2 may be the period from the time when a product registered immediately before the product to be registered was last detected in object detection to the time when a specific offset time was added to the time when the product to be registered was last detected. Alternatively, period 2 may be the period starting from the time when a product registered immediately before the product to be registered was last detected in object detection to the time when the product to be registered is registered.

[0129] (Processing flow in variation 1 of this embodiment) The flow of the process executed in the information processing device 10 in the above-described variation 1 of this embodiment will be described.

[0130] Fig. 10 is a flowchart showing a fourth example of the flow of processing executed in the information processing device 10. Note that in Fig. 10, the same processes as those in Figs. 7, 8, and 9 are denoted by the same reference numerals, and descriptions thereof may be omitted.

[0131] 10, when an object detection of a product to be registered is performed (YES in S103), the information processing device 10 determines whether the processing state of the register indicated by the sound within period 1 is a registration processing state (S401). As described above, period 1 is a period set for determining the registration processing state when performing registration processing by a scanning operation.

[0132] If the voice indicates the registration processing state within the period 1 (YES in S401), the flow returns to S101.

[0133] If the voice does not indicate a registration processing state within period 1 (NO in S401), the information processing device 10 determines whether the processing state of the cash register indicated by the feature point is a manual input standby state within period 2 (S402). Period 2 is a period provided for determining the registration processing state when performing registration processing by manual input operation. Period 2 may be a longer time interval than period 1.

[0134] If the feature point indicates a manual input standby state within period 2 (YES in S402), the flow returns to S101.

[0135] If the feature point does not indicate a manual input standby state within period 2 (NO in S402), the information processing device 10 executes anti-shoplifting measures (S106), and the flow then returns to S101.

[0136] Through the process shown in FIG. 10, the information processing device 10 detects an action in which, despite object detection of a product to be registered, the product code of the product to be registered is not scanned and product information of the product to be registered is not manually entered. This action corresponds to a customer moving the product but not registering the product information, and corresponds to an action that may be shoplifting. The information processing device 10 can detect such actions that may be shoplifting in real time using camera images, and therefore can appropriately detect fraudulent activity. Furthermore, the information processing device 10 can detect actions that may be shoplifting in real time without using information from sensors (cameras, weight sensors) included in the self-checkout system, and therefore can appropriately detect fraudulent activity with a simple configuration.

[0137] In addition, the process shown in Figure 10 extends the validity period for determining the registration process status when product information is registered by manual input, thereby reducing erroneous determinations caused by variations in the timing of registration when product information is registered by manual input.

[0138] Note that even when the information processing device 10 is capable of acquiring information indicating that a product has been scanned and information indicating that product information has been manually entered from the self-checkout system and is capable of determining the processing status of the register based on the information acquired from the self-checkout system, different periods may be set for each of the two types of registration operations described above. For example, if the information processing device 10 acquires information indicating that a product has been scanned from the self-checkout system within the above-described period 1, it may determine that the processing status of the register is a registration processing state. Furthermore, if the information processing device 10 acquires information indicating that product information has been manually entered from the self-checkout system within the above-described period 2, it may determine that the processing status of the register is a manual input standby state.

[0139] (Variation 2 of this embodiment) Before a customer starts registering an item at the cash register, or when the customer completes registering an item and is about to pay for the registered item, the customer takes out belongings such as a wallet, cards, or smartphone from a bag, pocket, or the like. During object detection, the information processing device 10 may detect items other than items sold in the store, such as wallets, cards, or smartphones. If an item other than items sold in the store is detected, there is a possibility that shoplifting may be erroneously determined to have occurred, even though it is not.

[0140] Therefore, in variation 2 of this embodiment, the state before the registration process is performed and the state after the registration process is completed and payment process is performed are detected, and the state before the registration process and the state after the registration process is completed and payment process is performed are detected, and the need to implement shoplifting countermeasures is determined in states excluding these states.

[0141] (Fourth example of display corresponding to the cash register processing status) 11 is a diagram showing a fourth example of the transition of the display on the display 5. In FIG. 11, display example 6 and display example 7, which is a transition from display example 6, are shown.

[0142] Display example 6 is an example of a display showing the transaction waiting state (S11 in FIG. 3). The display area shown in display example 6 includes an area that displays a list of available payment methods and an area corresponding to the "Start button" that is pressed to start the transaction.

[0143] When the customer presses the area corresponding to the "Start button" to start the transaction, the display on the display 5 transitions from Display Example 6 to Display Example 7.

[0144] Display example 7 is a display example showing a registration process waiting state (S12 in FIG. 3) similar to display example 1. However, unlike display example 1, display example 7, which is transitioned from display example 6, does not have any products registered.

[0145] For example, information processing device 10 detects feature points in a camera image and, based on the feature points, distinguishes between whether the display on display 5 is in a transaction processing standby state, as in display example 6, and whether the display on display 5 is in a registration processing standby state, as in display example 7. In the example of FIG. 11, information processing device 10 detects feature points that can distinguish between display example 6 and display example 7.

[0146] (Fifth example of display corresponding to the cash register processing status) Fig. 12 is a diagram showing a fifth example of the transition of display on display 5. Fig. 12 shows display example 6 and display example 8, which transitions from display example 6. Note that display example 6 in Fig. 12 is similar to display example 6 in Fig. 11, and therefore a description thereof will be omitted.

[0147] When the customer presses the area corresponding to the "Start button" to start the transaction, the display on the display 5 may transition from display example 6 to display example 8.

[0148] Display example 8 is a display that allows the customer to select whether or not to purchase a shopping bag to carry home the items they have purchased. Display example 8 is a display example that shows another variation of the checkout waiting state (S11 in FIG. 3).

[0149] When a customer presses either selection button #1 or selection button #2 to input information on whether or not they would like to purchase a shopping bag, the display on display 5 transitions from display example 8 to display example 7 shown in Figure 11.

[0150] In the case of the transition of the display on the display 5 shown in FIG. 11 , the information processing device 10 may start anti-shoplifting operations when the screen of Display Example 7 is detected, or when the screen of Display Example 6 is no longer detected after it has been detected. The anti-shoplifting operations may be, for example, at least a part of the operations shown in the flow of FIG. 14 , which will be described later. The timing when Display Example 7 is displayed and the timing when the screen of Display Example 6 is no longer detected are the timings when a customer starts registering a product. Therefore, after these timings, an object detected by the object detection process is likely to be a product sold in the store. Therefore, by taking anti-shoplifting measures after these timings, it is possible to reduce the possibility of erroneously determining a customer's behavior toward an item that is not sold in the store (for example, the behavior of removing a wallet from a bag) as shoplifting and erroneously implementing anti-shoplifting measures.

[0151] In addition, in the case of the transition of the display on the display 5 shown in FIG. 12, when the screen of display example 8 is detected, the information processing device 10 may start an anti-shoplifting operation. Alternatively, when the screen of display example 7 shown in FIG. 11 is detected, the information processing device 10 may start an anti-shoplifting operation. In the case of the transition of the display on the display 5 shown in FIG. 12, when the screen of display example 8 is detected and then is no longer detected, the information processing device 10 may start an anti-shoplifting operation. The timing when display example 8 is displayed, the timing when display example 7 is displayed, and the timing when the screen of display example 8 is no longer detected are timings before the customer starts registering the product. Therefore, after these timings, an object detected by the object detection process is likely to be a product sold in the store. Therefore, by implementing anti-shoplifting measures after these timings, it is possible to reduce the possibility of erroneously determining a customer's behavior toward an item that is not sold in the store (for example, the behavior of removing a wallet from a bag) as shoplifting and erroneously implementing anti-shoplifting measures.

[0152] (Sixth example of display corresponding to the cash register processing status) Fig. 13 is a diagram showing a sixth example of the transition of the display on the display 5. Fig. 13 shows display example 1 and display example 9, which transitions from display example 1. The transition of the display examples shown in Fig. 13 corresponds to an example of the transition from the registration processing standby state to the payment standby state in Fig. 2. Note that display example 1 in Fig. 13 is similar to display example 1 in Fig. 4, and therefore a description thereof will be omitted.

[0153] When a customer has completed the registration of the product to be purchased and presses the area corresponding to the "Payment button" to begin paying for the registered product, the display on display 5 transitions from Display Example 1 to Display Example 9.

[0154] Display example 9 is a display example corresponding to the payment waiting state. The display area shown in display example 9 includes an area for displaying a button for selecting a payment method, and an area for displaying the number of registered products and the total amount.

[0155] Here, if the screen of display example 9 shown in FIG. 13 is detected, the information processing device 10 may terminate the anti-shoplifting operation. Alternatively, if the screen of display example 9 is detected and then no longer detected, the information processing device 10 may terminate the anti-shoplifting operation. The anti-shoplifting operation may be, for example, at least a part of the operation of the flow in FIG. 14 described below. The timing when display example 9 is displayed and the timing when the screen of display example 9 is no longer detected are the timings when the product to be purchased is confirmed. Therefore, after these timings, the registered product is unlikely to change. Therefore, by implementing anti-shoplifting measures after these timings, it is possible to reduce the possibility of erroneously determining a customer's behavior toward an item that is not sold in the store (for example, the behavior of removing a wallet from a bag) as shoplifting and erroneously implementing anti-shoplifting measures.

[0156] When the customer presses the button corresponding to the desired payment method, the display changes to a screen requesting payment using the desired payment method (omitted in Figure 13). When the customer has completed payment, the display changes to a screen indicating that the transaction is waiting to be processed (for example, display example 6 in Figure 11).

[0157] (Processing flow in variation 2 of this embodiment) The flow of the process executed in the information processing device 10 in the above-described variation 2 of the present embodiment will be described.

[0158] Fig. 14 is a flowchart showing a fifth example of the flow of processing executed in the information processing device 10. Note that in Fig. 14, the same processes as those in Fig. 7, Fig. 8, and Fig. 9 are denoted by the same reference numerals, and descriptions thereof may be omitted.

[0159] 14, if the feature points do not represent an image at the time of manual input (NO in S105), information processing device 10 determines whether the processing state of the cash register is in the middle of registration (S501). For example, if the processing state of the cash register is neither a transaction processing standby state nor a payment standby state, information processing device 10 determines that the processing state of the cash register is in the middle of registration.

[0160] If the processing status of the cash register is not in the registration process state (NO in S501), the flow returns to S101.

[0161] If the processing status of the cash register is in the middle of registration (YES in S501), information processing device 10 executes anti-shoplifting measures (S106), and the flow then returns to S101.

[0162] Note that if the processing status of the register is not in the middle of registration (NO in S501), the flow may return to S101 and the object detection process may be stopped instead. Alternatively, if the register is not in the middle of registration (NO in S501), the object detection process may be performed, but the process of determining whether to implement shoplifting countermeasures may be stopped. Alternatively, if the register is not in the middle of registration (NO in S501), the object detection process and the process of determining whether to implement shoplifting countermeasures may be performed, but the process of issuing an alarm may be stopped. Furthermore, although the process of issuing an alarm is stopped, the information processing device 10 may leave an information history regarding the result of the object detection process and / or the result of the process of determining whether to implement shoplifting countermeasures.

[0163] Through the process shown in FIG. 14, the information processing device 10 detects an action in which, while the register is in the middle of registering, object detection of a product to be registered is performed, but the product code of the product to be registered is not scanned, and product information of the product to be registered is not manually entered. This action corresponds to a customer moving the product but not registering the product information, and corresponds to an action that may be shoplifting. The information processing device 10 can detect such actions that may be shoplifting in real time using camera images, and therefore can appropriately detect fraudulent activity. Furthermore, the information processing device 10 can detect actions that may be shoplifting in real time without using information from sensors (cameras, weight sensors), etc., included in the self-checkout system, and therefore can appropriately detect fraudulent activity with a simple configuration.

[0164] In addition, the processing shown in Figure 14 allows the information processing device 10 to exclude states in which a customer may move objects other than merchandise, such as when the cash register is waiting for payment or waiting for transaction processing, thereby enabling more appropriate detection of fraudulent activity.

[0165] (Configuration example of information processing device 10) Fig. 15 is a block diagram showing an example configuration of information processing device 10. Information processing device 10 shown in Fig. 15 has a receiving unit 101, a control unit 102, and a transmitting unit 103. The control unit 102 includes an object detection unit 104, a feature point detection unit 105, a sound detection unit 106, a registration determination unit 107, a shoplifting determination unit 108, an alarm information generation unit 109, and a transmission control unit 110. The control unit 102 may be configured by a processor such as a CPU (central processing unit), for example.

[0166] The receiving unit 101 acquires camera images from, for example, the camera 20. For example, if the camera 20 includes an object detection camera 20 and a feature point detection camera 20, the receiving unit 101 acquires an object detection camera image captured by the object detection camera 20 and a feature point detection camera image captured by the feature point detection camera 20. Note that the object detection camera image and the feature point detection camera image may be the same camera image. In other words, the camera 20 may capture an image that is capable of detecting feature points on the screen displayed on the display 5 and is capable of detecting a product being moved by a customer.

[0167] The receiving unit 101 may also acquire audio from the microphone 30 .

[0168] The object detection unit 104 detects an object based on a camera image (e.g., an image from an object detection camera). For example, the object detection unit 104 detects from the camera image what product the customer is holding and / or what product the customer is carrying and moving. The object detection unit 104 outputs object detection information regarding the detected object to the shoplifting determination unit 108. The object detection information includes information indicating whether the object detection unit 104 has detected an object. If the object detection unit 104 detects an object, the object detection information may include information indicating what the detected object is. If the object detection unit 104 detects that an object has started to move, the object detection information may include information indicating the timing at which the detected object started to move.

[0169] In the object detection process, the object detection unit 104 may detect whether or not the customer is holding an object, instead of detecting the product the customer is holding. Alternatively, the object detection unit 104 may detect whether or not the customer is moving while holding an object, instead of detecting the product the customer is holding and moving. In other words, the object detection process does not need to detect (recognize) the object. In this case, the object detection information may include information indicating whether or not the customer is holding an object and / or information indicating whether or not the customer is moving while holding an object. Furthermore, the object detection unit 104 may perform the object detection process based on other information, instead of a camera image (e.g., an image from a camera for object detection).

[0170] The feature point detection unit 105 detects feature points of the camera image. For example, the feature point detection unit 105 detects feature points of an image showing the processing status of the cash register corresponding to the display on the display 5, and outputs image feature point information showing the detected image feature points to the state determination unit 107.

[0171] The sound detection unit 106 detects, based on the sound picked up by the microphone 30, the sound generated when the product is scanned.

[0172] The state determination unit 107 determines the processing state of the cash register corresponding to the display on the display 5. For example, the state determination unit 107 determines the processing state of the cash register corresponding to the display on the display based on image feature point information. Alternatively, the state determination unit 107 may determine that the processing state of the cash register is a registration processing state when a sound generated when a product is scanned is detected. The determined processing state of the cash register is, for example, at least one of the states shown in FIG. 3 etc. Note that the terms "determine" and "detect" may be interchangeable.

[0173] Shoplifting determination unit 108 determines whether to implement shoplifting countermeasures. For example, shoplifting determination unit 108 determines whether to implement shoplifting countermeasures based on the cash register processing status determined by status determination unit 107 and object detection information acquired from object detection unit 104. For example, if the object detection information indicates that a product has been moved and the operation status related to payment indicates that the product is not being registered, shoplifting determination unit 108 determines that shoplifting countermeasures should be implemented.

[0174] When the shoplifting determination unit 108 determines that a shoplifting countermeasure should be implemented, the alarm information generation unit 109 generates alarm information indicating an alarm that issues a shoplifting warning.

[0175] The transmission control unit 110 generates transmission information including the alarm information and transmits the transmission information via the transmission unit 103. The transmission information may be transmitted, for example, to a terminal owned by a store staff member, a security guard, or the like, or may be transmitted to a display device that the store staff member, security guard, or the like checks. The store staff member or security guard may check the transmitted alarm information and take action such as issuing a warning to a customer who may be shoplifting. The transmission information may also be transmitted to a display device located outside the store, such as a display device for remotely monitoring shoplifting.

[0176] Furthermore, it goes without saying that the present invention is applicable even when the display 5 displays other than the display examples shown in the above-described embodiment.

[0177] (Other variations) In the above embodiment, the self-checkout system includes weight sensors 4a and 4b, but weight sensors 4a and 4b may be omitted. Also, information processing device 10 according to the present embodiment may include weight sensors 4a and 4b, or may acquire detection results from weight sensors 4a and 4b.

[0178] In the above embodiment, the information processing device 10 detects the movement of a product using an image captured by an object detection camera. However, the movement of a product may be detected by other means. For example, the information processing device 10 may detect the movement of a product based on a change in weight detected by the weight sensors 4a and 4b. Alternatively, for example, a wireless tag such as an RFID tag may be attached to the product, and the information processing device 10 may detect the movement of the product by detecting the movement of the wireless tag. Alternatively, the information processing device 10 may detect the movement of a product by using both the image captured by an object detection camera and the detection results of the weight sensor or wireless tag.

[0179] In the above embodiment, the information processing device 10 detects feature points of the image to determine the processing status of the cash register corresponding to the display on the display 5. However, the information processing device 10 may determine the processing status of the cash register using other methods, such as recognizing characters displayed on the display 5 or using pattern matching of the entire image displayed on the display 5.

[0180] In addition, in the above embodiment, the information processing device 10 may also perform product recognition. In this case, if the attributes of a product recognized from the object detection camera image do not match the cash register processing status corresponding to the display on the display 5 or the sound emitted by the cash register, anti-shoplifting measures may be implemented. Product attributes include, for example, whether the product has a barcode or whether there are age restrictions, etc., for purchasing the product. Alternatively, product attributes may include attributes related to differences in the cash register operation method for registering the product. For example, if the recognized product does not have a barcode but the cash register processing status corresponds to a screen for products with barcodes, the information processing device 10 may implement anti-shoplifting measures. Furthermore, if the recognized product has age restrictions, etc., for purchase, but no age verification display, etc., is displayed, the information processing device 10 may implement anti-shoplifting measures. Note that, depending on the range or angle of the object detection camera image, it may be difficult to accurately recognize the product. However, the concept of this modification can be applied even if the product cannot be accurately recognized as long as attributes, such as the presence or absence of a barcode, can be recognized.

[0181] The receiving unit 101 (an example of an acquisition unit) of the information processing device 10 described above acquires camera images. The camera images are acquired by capturing at least a portion of the display screen of a cash register on which information about products purchased by customers is registered. The object detection unit 104 (an example of a first detection unit) detects whether or not a product has been moved. The status determination unit 107 (an example of a second detection unit) detects the processing status of the cash register based on the display screen of the cash register included in the camera image and / or the sound emitted from the cash register. The processing status of the cash register indicates, for example, whether or not a product has been registered. The shoplifting determination unit 108 (an example of a determination unit) determines whether or not to implement anti-shoplifting measures for the product based on whether or not a product has been moved and the processing status of the cash register.

[0182] It should be noted that the "... section" in the above embodiments may be a "... circuitry," a "... device," a "... unit," or a "... module."

[0183] The present disclosure can be realized in software, hardware, or software in conjunction with hardware.

[0184] Each functional block used in the description of the above embodiments may be partially or entirely realized as an LSI, which is an integrated circuit, and each process described in the above embodiments may be partially or entirely controlled by a single LSI or a combination of LSIs. The LSI may be composed of individual chips, or may be composed of a single chip that includes some or all of the functional blocks. The LSI may have data input and output. Depending on the degree of integration, the LSI may be called an IC, system LSI, super LSI, or ultra LSI.

[0185] The integrated circuit method is not limited to LSI, but may be realized by a dedicated circuit, a general-purpose processor, or a dedicated processor. Also, a field programmable gate array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI, may be used. The present disclosure may be realized as digital processing or analog processing.

[0186] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, it is natural that such technology may be used to integrate functional blocks. The application of biotechnology, etc. is also a possibility.

[0187] The present disclosure may be implemented in any type of apparatus, device, or system with communications capabilities (collectively referred to as communications apparatus), including, but not limited to, telephones (e.g., cell phones, smartphones), tablets, personal computers (PCs) (e.g., laptops, desktops, notebooks), cameras (e.g., digital still / video cameras), digital players (e.g., digital audio / video players), wearable devices (e.g., wearable cameras, smartwatches, tracking devices), game consoles, digital book readers, telehealth / telemedicine devices, communications-enabled vehicles or mobile transportation (e.g., cars, airplanes, ships), and combinations of the above.

[0188] Communications equipment is not limited to portable or mobile equipment, but also includes non-portable or fixed equipment, devices, and systems of any kind, such as smart home devices (such as appliances, lighting equipment, smart meters or metering devices, control panels, etc.), vending machines, and any other "things" that may exist on an IoT (Internet of Things) network.

[0189] Communications include data communications via cellular systems, wireless LAN systems, communications satellite systems, etc., as well as data communications via combinations of these.

[0190] A communications apparatus also includes devices such as controllers and sensors connected or coupled to a communications device that performs the communications functions described in this disclosure, such as controllers and sensors that generate control and data signals used by the communications device to perform the communications functions of the communications apparatus.

[0191] The communication apparatus also includes infrastructure facilities, such as base stations, access points, and any other apparatus, device, or system that communicates with or controls the various apparatuses listed above, but are not limited to these.

[0192] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure. Furthermore, the components of the above-described embodiments may be combined in any manner without departing from the spirit of the disclosure.

[0193] Although specific examples of the present disclosure have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and alterations of the specific examples exemplified above. [Industrial Applicability]

[0194] The present disclosure is useful for detecting fraudulent activity at self-checkout registers. [Explanation of symbols]

[0195] 1, 10 Information processing device 2,2a,2b scanner 3, 20 cameras 4a, 4b Weight sensor 5. Display A1a, A1b Product stand 30. Mike 101 Receiving unit 102 Control section 103 Transmitter

Claims

1. an acquisition unit that acquires a camera image of at least a part of a display screen of a cash register on which information of products purchased by customers is registered; a first detection unit that detects whether the product has moved; a second detection unit that detects whether or not information about the product has been registered based on the display screen of the cash register and / or a sound emitted from the cash register included in the camera image; a determination unit that determines whether to implement a shoplifting prevention measure for the product based on whether the product has been moved and whether the product has been registered; An information processing device comprising:

2. the determination unit determines that an anti-shoplifting measure should be taken for the product when the product determined to have been moved by the first detection unit has not been registered. The information processing device according to claim 1 .

3. The registration is performed by at least one of a first operation in which the customer scans a product code attached to the product, and a second operation in which the customer inputs information about the product via an operation unit. The information processing device according to claim 2 .

4. the second detection unit detects whether the product is registered using a different method between the first action and the second action. The information processing device according to claim 3 .

5. when the first action is performed, the display screen of the cash register displays a message indicating that the first action has been performed; the determination unit determines that the product has been registered when the second detection unit detects an indication that the first action has been performed. The information processing device according to claim 4 .

6. When the first action is performed, the cash register outputs a sound indicating that the first action has been performed; The determination unit determines that the product has been registered when the second detection unit detects the sound. The information processing device according to claim 4 .

7. When the second operation is being awaited, the display screen of the cash register displays a message indicating that the second operation is being awaited; the determination unit determines that the product has not been registered when the second detection unit has not detected an indication that the first action has been performed and an indication that the second action is waiting.

7. The information processing device according to claim 5.

8. the indication indicating that the first action has been performed is an indication indicating a product recognized by the first action; The information processing device according to claim 5 .

9. the indication indicating that the first action has been performed is an indication indicating that there is a restriction on the purchase of the product recognized by the first action; The information processing device according to claim 5 .

10. the second detection unit detects whether the product is registered within different validity periods between the first action and the second action. The information processing device according to claim 3 .

11. The camera image is taken by an external camera that is not included in the system including the cash register. The information processing device according to claim 1 .

12. the determination unit determines whether to implement the shoplifting countermeasure during a period excluding a period before the registration of the product is started and / or a period after the registration of the product is completed and payment processing for the product is started. The information processing device according to claim 1 .

13. The information processing device acquiring a camera image capturing at least a part of a display screen of a cash register on which information of a product to be purchased by a customer is registered; Detecting whether the product has been moved or not; detecting whether or not the product information has been registered based on the display screen of the cash register included in the camera image and / or a sound emitted from the cash register; determining whether to implement a shoplifting prevention measure for the product based on whether the product has been moved and whether the product has been registered; Judgment method.

Images