Information processor, and fraudulent behavior determination method

Abstract

To provide an information processor capable of properly determining fraudulent acts using manual input of merchandise information.SOLUTION: The information processor includes: a determination unit that refers to the product code presence / absence information that associates the product with information indicating the presence or absence of the product code, and determines whether the detected product is a product with a product code or a product without a product code based on camera images; and a determination unit that is configured so as to, when merchandise information of a product is manually inputted into an input device, determine whether there was any fraudulent act that used the manual input based on the determination result whether the product has a product code or does not have a product code.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and a method for determining improper conduct.

Background Art

[0002] In recent years, due to the reduction of labor costs for store employees in retail stores and the like, the popularity of self-checkout has been rapidly progressing. In self-checkout, since the customer himself / herself scans the barcodes of the products, measures against improper conduct such as shoplifting become important.

[0003] There is a self-checkout that uses a weight sensor to detect shoplifting. For example, the apparatus having a self-scanning function disclosed in Patent Document 1 includes a storage unit for storing products whose product codes have been read, and the storage unit has a weight sensor. When the weight of the storage unit increases without the product being scanned, the apparatus of Patent Document 1 generates an alarm.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in a self-checkout that uses a weight sensor as in Patent Document 1 to detect improper conduct by customers, there are cases where improper conduct using manual input of products without barcodes (barcode-less products) such as vegetables and fruits cannot be appropriately detected.

[0006] For example, a customer may manually enter a product without a barcode (barcode-free product) that is cheaper than a product with a barcode (barcode-attached product) without scanning the barcode-attached product with a scanner. For example, a customer may manually enter a potato (cheaper than meat) without a barcode instead of scanning the meat with a barcode and move the meat to the storage section. A device that detects fraud using a weight sensor may not be able to detect fraud using the above-described manual entry.

[0007] Non-limiting examples of the present disclosure contribute to providing an information processing apparatus and a fraud determination method that can appropriately detect fraud using manual entry of product information.

Means for Solving the Problem

[0008] An information processing apparatus according to an embodiment of the present disclosure refers to product code presence / absence information associating a product with information indicating the presence or absence of a product code, and determines whether the product detected based on a camera image is a product with a product code or a product without a product code. When product information of a product is manually entered in an input device, based on the result of determining whether the product is a product with a product code or a product without a product code, a determination unit that determines whether there has been fraud using the manual entry.

[0009] A fraud determination method according to an embodiment of the present disclosure refers to product code presence / absence information associating a product with information indicating the presence or absence of a product code, determines whether the product detected based on a camera image is a product with a product code or a product without a product code, and when product information of a product is manually entered in an input device, determines whether there has been fraud using the manual entry based on the result of determining whether the product is a product with a product code or a product without a product code.

[0010] Note that these general or specific aspects may be implemented in a system, apparatus, method, integrated circuit, computer program, or recording medium, or may be implemented in any combination of a system, apparatus, method, integrated circuit, computer program, and recording medium.

Advantages of the Invention

[0011] According to an embodiment of the present disclosure, it is possible to appropriately detect fraud using manual input of product information.

[0012] Further advantages and effects in an embodiment of the present disclosure will be clarified from the specification and drawings. Such advantages and / or effects are provided by some embodiments and the features described in the specification and drawings, respectively, but not necessarily all are provided in order to obtain one or more identical features.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings. However, a more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configuration may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art.

[0015] Note that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and it is not intended to limit the subject matter described in the claims thereby.

[0016] <First Embodiment> (System Configuration) FIG. 1 is a diagram showing a configuration example of a self-checkout system including an information processing apparatus 1 according to the first embodiment. As shown in FIG. 1, the self-checkout system includes an information processing apparatus 1, scanners 2a and 2b, a camera 3, weight sensors 4a and 4b, and a display 5. In FIG. 1, in addition to the self-checkout system, product stands A1a and A1b and a stand A2 are shown.

[0017] Product stand A1a is a stand on which products before being scanned by scanners 2a and 2b are placed. The products before being scanned placed on product stand A1a may also include shopping baskets or carts of the store in which the products before being scanned are placed.

[0018] Product stand A1b is a stand on which products after being scanned by scanners 2a and 2b are placed. The products after being scanned placed on product stand A1b may also include shopping baskets of customers in which the products after being scanned are placed.

[0019] The information processing device 1 is a computer such as a personal computer or a server. In FIG. 1, the information processing device 1 is placed outside the stand A2, but it may also be placed inside the stand A2 or in the store office. The information processing device 1 performs detection processing for illegal acts such as settlement processing of goods and illegal input by customers.

[0020] The scanner 2a is a handy type scanner. The scanner 2b is a fixed type scanner fixed to the stand A2. Hereinafter, when the scanners 2a and 2b are not distinguished, they are simply referred to as the scanner 2.

[0021] The scanner 2 is connected to the information processing device 1. The scanner 2 scans the product code attached to the product. The scanner 2 transmits the scanned product code to the information processing device 1. The product code is a barcode such as a JAN (Japanese Article Number) code, for example. The scanner 2 may also be referred to as a barcode reader or a reader.

[0022] The camera 3 is connected to the information processing device 1. The camera 3 transmits the image data of the captured image to the information processing device 1.

[0023] The camera 3 is installed to capture the vicinity where the customer scans the product with the scanner 2 (see the dotted frame A3 in FIG. 1). For example, the camera 3 is installed on the upper part of the display 5 or the upper part of the stand A2, and the viewing angle is set to include the area between the product placement table A1a and the product placement table A1b, that is, the vicinity where the customer scans the product with the scanner 2.

[0024] The weight sensor 4a is connected to the information processing device 1. The weight sensor 4a is installed inside the product placement table A1a, for example, and transmits a signal corresponding to the weight of the product placed on the product placement table A1a (or a shopping basket in the store containing the product) to the information processing device 1.

[0025] The weight sensor 4b is connected to the information processing device 1. The weight sensor 4b is installed, for example, inside the product placement table A1b, and transmits a signal corresponding to the weight of the product placed on the product placement table A1b (or the customer's shopping basket containing the product, etc.) to the information processing device 1.

[0026] The display 5 is connected to the information processing device 1. The display 5 displays, for example, the product information of the product scanned by the customer using the scanner 2. The product information may be, for example, the name, price, and quantity of the product.

[0027] The display 5 may be provided with a touch panel on the screen surface. The touch panel receives the customer's operations. The customer manually enters, for example, the information of a product without a product code such as vegetables and fruits via the touch panel and registers it as a purchased product. For example, when the customer purchases an apple without a barcode, the customer manually enters the product name "apple" and the purchased quantity "X" via the touch panel and registers it as a purchased product. Note that the device for receiving the customer's operations is not limited to the touch panel. The device for receiving the customer's operations may be, for example, a key input device separate from the display 5. Alternatively, the customer's operations may be received using a smartphone or a mobile terminal owned by the customer.

[0028] Based on the control of the information processing device 1, the display 5 displays a screen related to detecting fraud using manual input. For example, when the customer manually enters a product with a barcode as a product without a barcode that is cheaper than the product, the display 5 displays a screen indicating that fraud using manual input has been committed. More specifically, when the customer does not scan the meat with a barcode using the scanner 2 and manually enters potatoes (cheaper than meat) without a barcode and moves the meat to the product placement table A1b. In this case, based on the control of the information processing device 1, the display 5 displays a screen indicating that fraud using manual input has been committed.

[0029] (Product Detection Process) FIG. 2 is a diagram for explaining an example of product detection processing. The information processing apparatus 1 detects (recognizes) a product included in the image of the camera 3 using, for example, a known image recognition technique (a program for image recognition processing) using artificial intelligence. In other words, the information processing apparatus 1 detects the product taken out from the product placement table A1a by image recognition processing. The image recognition processing of the product may be executed by another apparatus such as a server.

[0030] The information processing apparatus 1 calculates a recognition score for each of the product candidates (candidate products) detected by the image recognition processing. The recognition score indicates the degree of certainty of the detection result of the detected candidate product. In the example of FIG. 2, the information processing apparatus 1 detects five candidate products as the products included in the image of the camera 3, but it is not limited to this. The number of candidate products detected by the information processing apparatus 1 may be, for example, one or six or more.

[0031] The information processing apparatus 1 may determine (final detection) the candidate product with the highest recognition score as the product included in the image of the camera 3. For example, in the example of FIG. 2, the beef steak B may be finally detected as the product included in the image of the camera 3.

[0032] Alternatively, when the candidate product with the highest recognition score and the recognition score of the candidate product are equal to or higher than a predetermined threshold value (first threshold value), the information processing apparatus 1 may finally detect the candidate product with the highest recognition score as the product included in the image of the camera 3. For example, the first threshold value is set to 90%. In this case, when the recognition score of the candidate product with the highest recognition score is 90% or more, the information processing apparatus 1 may finally detect the candidate product with the highest recognition score as the product included in the image of the camera 3. On the other hand, when the recognition score of the candidate product with the highest recognition score is less than 90%, the information processing apparatus 1 may display an image indicating that the product could not be detected on the display 5. The final detection method of the product included in the image of the camera 3 is not limited to the above example.

[0033] (Barcode presence / absence information) FIG. 3 is a diagram showing an example of bar code presence / absence information. As shown in FIG. 3, bar code presence / absence information for each product is stored in the storage unit of the information processing apparatus 1.

[0034] In the example of FIG. 3, products "onion" and "potato" are products without bar codes. Note that since products "onion" and "potato" are products without bar codes, they are products that are manually input in an input device such as the touch panel of the display 5.

[0035] In the example of FIG. 3, products "Coke A", "Beef Steak B", "Beer C", and "Sake D" are products with bar codes. Note that since products "Coke A", "Beef Steak B", "Beer C", and "Sake D" are products with bar codes, they are products that are scanned by the scanner 2.

[0036] (Candidate Product List) FIG. 4 is a diagram showing an example of a candidate product list. The information processing apparatus 1 generates a candidate product list in which information indicating whether each candidate product (see FIG. 2) detected by image recognition processing is a product with a bar code or a product without a bar code is attached. The information processing apparatus 1 refers to the bar code presence / absence information (see FIG. 3) stored in the storage unit, and attaches information indicating whether each candidate product detected by image recognition processing is a product with a bar code or a product without a bar code.

[0037] For example, "Beef Steak B" in the candidate product list shown in FIG. 4 is a product with a bar code as shown in FIG. 3. Therefore, the information processing apparatus 1 attaches information "with bar code" indicating that the candidate product "Beef Steak B" detected by image recognition is a product with a bar code. The information processing apparatus 1 also attaches information indicating whether other candidate products detected by image recognition processing are products with bar codes or products without bar codes, and generates a candidate product list as shown in FIG. 4.

[0038] (Example of Operation for Detecting Improper Acts) For simplicity of explanation, here, the customer takes out the barcoded "Steak Beef B" from the product stand A1a and moves it to the product stand A1b. The customer manually enters the non-barcode product "Potato" (cheaper than Steak Beef B) in the input device without scanning the barcoded "Steak Beef B" with the scanner 2. That is, the customer manually enters the barcoded "Steak Beef B" as a "Potato" that is cheaper than "Steak Beef B" and makes an illegal purchase.

[0039] The information processing device 1 detects products (candidate products) that may be included in the image of the camera 3 by image recognition processing (see Fig. 2). The information processing device 1 refers to the barcode presence / absence information (see Fig. 3) and generates a candidate product list in which information indicating whether the detected candidate product is a barcoded product or a non-barcoded product is added. Here, the information processing device 1 generates the candidate product list shown in Fig. 4.

[0040] Since the customer has taken out the barcoded "Steak Beef B" from the product stand A1a, in the candidate product list shown in Fig. 4, the recognition score of the candidate product "Steak Beef B" is the highest. Also, the candidate product "Steak Beef B" is a "barcode present" product.

[0041] The information processing device 1 determines whether there is a manual input in the input device. Here, since the customer is manually entering "Potato", the information processing device 1 determines that there is a manual input in the input device.

[0042] When there is a manual input of a non-barcoded product, the information processing device 1 determines whether a non-barcoded product is included among the candidate products included in the generated candidate product list and having a recognition score equal to or higher than a predetermined threshold (second threshold). For example, it determines whether a non-barcoded product is included among the candidate products having a recognition score of 90% or higher. In the example of Fig. 4, since no non-barcoded product is included among the candidate products having a recognition score of 90% or higher, the information processing device 1 determines that no non-barcoded product is included among the candidate products having a recognition score equal to or higher than the second threshold.

[0043] When the information processing apparatus 1 determines that the barcode-less item is not included in the candidate items having a recognition score equal to or higher than the second threshold even though the barcode-less item has been manually entered, it detects that there has been an improper act. That is, when the barcode-less item has been manually entered but the barcode-less item is not included in the candidate items (candidate items having a recognition score equal to or higher than the second threshold) that can be finally detected based on the image of the camera 3, the information processing apparatus 1 detects that there has been an improper act. Note that the second threshold may be the same as or different from the first threshold.

[0044] (Block diagram) FIG. 5 is a diagram showing an example of the block configuration of the information processing apparatus 1. As shown in FIG. 5, the information processing apparatus 1 includes a control unit 11, a storage unit 12, and a communication unit 13.

[0045] The control unit 11 controls the entire information processing apparatus 1. The control unit 11 may be configured by a processor such as a CPU (central processing unit), for example.

[0046] The control unit 11 includes a detection unit 11a, an assignment unit 11b, an improper act determination unit 11c, and an alarm generation unit 11d. The control unit 11 may implement the functions of the foregoing units according to a program stored in the storage unit 12.

[0047] The detection unit 11a detects (extracts) candidate items of products included in the image of the camera 3 and calculates the recognition score of each of the detected candidate items (see FIG. 2).

[0048] The assignment unit 11b refers to the barcode presence / absence information stored in the storage unit 12, assigns information indicating whether the candidate item detected by the detection unit 11a is a barcode-present item or a barcode-absent item, and generates a candidate item list (see FIG. 4).

[0049] When product information of a product without a barcode is manually input in the input device, the improper behavior determination unit 11c determines whether there is any improper behavior in the manual input based on information on whether the candidate product in the candidate product list is a product with a barcode or a product without a barcode.

[0050] When the improper behavior determination unit 11c determines the occurrence of an improper behavior, the alarm generation unit 11d generates alarm information. For example, the alarm generation unit 11d generates image data of an image including characters such as "Please scan the barcode of the product". The image data is transmitted to the display 5 via the communication unit 13.

[0051] Also, the alarm generation unit 11d generates, for example, an alarm signal. The alarm signal is transmitted to the mobile terminal held by the store clerk via the communication unit 13. The mobile terminal that receives the alarm signal notifies the store clerk of the occurrence of improper manual input, for example, by sound.

[0052] The storage unit 12 stores an OS (operating system) program and an application program to be executed by the control unit 11. For example, the storage unit 12 stores an application program that performs settlement processing of self-checkout and monitoring processing of shoplifting. Also, the storage unit 12 stores various data necessary for the processing by the control unit 11. Also, the storage unit 12 stores the barcode presence / absence information described in FIG. 3. The storage unit 12 may be, for example, an SSD (solid state drive), a RAM (Random Access Memory), a flash memory, a ROM (Read Only Memory), and / or an HDD (hard disk drive).

[0053] The communication unit 13 communicates with the scanner 2, the camera 3, the weight sensors 4a, 4b, and the display 5 by wire or wirelessly. Also, the communication unit 13 communicates with the mobile terminal held by the store clerk wirelessly.

[0054] (Operation Flow) FIG. 6 is a flowchart showing an operation example in the determination of unauthorized acts of the information processing apparatus 1.

[0055] The information processing apparatus 1 determines whether or not all the products placed on the product stand A1a by the customer have been scanned by the scanner 2 (S1). The information processing apparatus 1 may determine whether or not all the products placed on the product stand A1a have been scanned by the scanner 2 based on, for example, the signal of the weight sensor 4a.

[0056] If the information processing apparatus 1 determines that not all the products placed on the product stand A1a by the customer have been scanned by the scanner 2 (No in S1), it performs product detection (candidate product detection) based on the image of the camera 3 (S2). For example, the information processing apparatus 1 detects candidate products included in the image of the camera 3 by known image recognition processing using artificial intelligence, and calculates the recognition score of the detected candidate products (see FIG. 2).

[0057] The information processing apparatus 1 refers to the barcode presence / absence information (see FIG. 3) stored in the storage unit 12, and assigns information indicating whether the candidate products detected in step S2 are products with barcodes or products without barcodes, and generates a candidate product list (S3). For example, the information processing apparatus 1 generates the candidate product list shown in FIG. 4.

[0058] The information processing apparatus 1 determines whether or not a product without a barcode has been manually input at the input device (S4).

[0059] If the information processing apparatus 1 determines that a product without a barcode has not been manually input at the input device (No in S4), the process proceeds to step S1. For example, when a product with a barcode is scanned by the scanner 2, the information processing apparatus 1 proceeds to step S1.

[0060] On the other hand, when the information processing apparatus 1 determines that a barcode-less product has been manually input at the input device (Yes in S4), it determines whether the barcode-less product is included in the candidate products that can be finally detected among the candidate products included in the candidate product list generated in S3 (S5). For example, the information processing apparatus 1 determines whether the barcode-less product is included in the candidate products having a recognition score equal to or higher than a second threshold value such as 90%.

[0061] When the information processing apparatus 1 determines that the barcode-less product is included in the candidate products that can be finally detected among the candidate products included in the candidate product list generated in S3 (Yes in S5), the process proceeds to S1. That is, when there is a manual input of a barcode-less product and the barcode-less product may be included in the image of the camera 3, the information processing apparatus 1 does not detect any irregularities and proceeds to S1.

[0062] On the other hand, when the information processing apparatus 1 determines that the barcode-less product is not included in the candidate products that can be finally detected among the candidate products included in the candidate product list generated in S3 (No in S5), it detects an irregularity (S6). That is, when there is a manual input of a barcode-less product but a barcode-bearing product may be included in the image of the camera 3, the information processing apparatus 1 detects an irregularity. When the information processing apparatus 1 detects an irregularity in S6, it generates alarm information and an alarm signal and proceeds to S1.

[0063] When the information processing apparatus 1 determines that the customer has scanned all the products placed on the product stand A1a with the scanner 2 in S1 (Yes in S1), it executes accounting processing (S7). For example, the information processing apparatus 1 displays the total amount of the products purchased by the customer on the display 5 and executes payment processing.

[0064] (Summary of the First Embodiment) As described above, the information processing apparatus 1 refers to the barcode presence / absence information in which products are associated with information indicating the presence or absence of barcodes, and assigns information indicating whether the product detected based on the image of the camera 3 is a product with a product code or a product without a product code. When the product information of a product is manually input in the input device, the information processing apparatus 1 determines whether there is any fraud in the manual input based on the information indicating whether the product is a product with a product code or a product without a product code assigned to the product. By this operation, the information processing apparatus 1 can appropriately detect fraud using the manual input of product information.

[0065] (Modification Example 1) The information processing apparatus 1 detects fraud when the candidate products with a recognition score equal to or higher than the second threshold do not include products without barcodes, but is not limited to this. The information processing apparatus 1 may detect fraud based on the ratio of products with barcodes among the candidate products with a recognition score equal to or higher than the second threshold.

[0066] FIG. 7 is a diagram for explaining Modification Example 1 of the first embodiment. A candidate product list is shown in FIG. 7. In the description of FIG. 7, the second threshold is set to 90%. When the ratio of products with barcodes among the candidate products with a recognition score equal to or higher than the second threshold is 75% or more, the information processing apparatus 1 detects fraud.

[0067] As shown in FIG. 7, assume that there are four candidate products with a recognition score of 90% or more. In this case, the information processing apparatus 1 detects fraud based on the ratio of products with barcodes among the four candidate products with a recognition score of 90% or more. In the example of FIG. 7, since the ratio of products with barcodes is 3 / 4 (75%), the information processing apparatus 1 detects fraud. That is, even if there is a manual input of a product without a barcode, the information processing apparatus 1 detects fraud when the ratio of candidate products with barcodes is high.

[0068] As described above, the information processing apparatus 1 detects fraud based on the ratio of the products with barcodes among the candidate products having a recognition score equal to or higher than the second threshold value. Thereby, the information processing apparatus 1 can appropriately detect fraud using manual input of product information.

[0069] Note that the information processing apparatus 1 may detect fraud based on the ratio of the products without barcodes among the candidate products having a recognition score equal to or higher than the second threshold value. For example, the information processing apparatus 1 may detect fraud when the ratio of the products without barcodes is smaller than a predetermined threshold value.

[0070] (Modification Example 2) The information processing apparatus 1 detects fraud when the candidate products having a recognition score equal to or higher than the second threshold value do not include products without barcodes, but is not limited thereto. The information processing apparatus 1 may detect fraud when the product finally detected as the product included in the image of the camera 3 is not a product without a barcode. Note that the case where the product is not a product without a barcode includes the case where the product has a barcode and the case where it is impossible to determine whether the product has a barcode or not.

[0071] For example, the information processing apparatus 1 finally detects the candidate product with the highest recognition score as the product included in the image of the camera 3. When there is a manual input of a product without a barcode, the information processing apparatus 1 detects fraud when the product finally detected by the above method is not a product without a barcode.

[0072] Alternatively, when the candidate product with the highest recognition score has a recognition score equal to or higher than the first threshold value, the information processing apparatus 1 finally detects the candidate product with the highest recognition score as the product included in the image of the camera 3. When there is a manual input of a product without a barcode, the information processing apparatus 1 detects fraud when the product finally detected by the above method is not a product without a barcode.

[0073] As described above, when the product finally detected as a product included in the image of the camera 3 is not a product without a barcode, the information processing apparatus 1 detects fraud. Thereby, the information processing apparatus 1 can appropriately detect fraud using manual input of product information.

[0074] (Modification 3) The information processing apparatus 1 detects fraud when the candidate products having a recognition score equal to or higher than the second threshold value do not include products without barcodes, but is not limited thereto. When the candidate products having a recognition score equal to or higher than the second threshold value include products with barcodes, the information processing apparatus 1 does not have to detect fraud. Thereby, the information processing apparatus 1 can appropriately detect fraud using manual input of product information.

[0075] <Second Embodiment> In the second embodiment, when a product without a barcode is manually input, the information processing apparatus 1 detects fraud based on the probability that the type of the manually input product without a barcode matches the type of the object detected by the image recognition process.

[0076] Note that the system configuration in the second embodiment is the same as that in FIG. 1, and the description thereof is omitted. Also, the block configuration in the second embodiment is the same as that in FIG. 5, but the function of the control unit 11 is different. For example, the control unit 11 determines whether there is fraud in the manual input based on the probability that the object detected from the image of the camera 3 matches the product manually input at the input device.

[0077] (Operation Flow) FIG. 8 is a flowchart showing an operation example in fraud determination of the information processing apparatus 1 according to the second embodiment.

[0078] The information processing apparatus 1 determines whether or not all the products placed on the product stand A1a have been scanned by the scanner 2 (S11). The information processing apparatus 1 may determine whether or not all the products placed on the product stand A1a have been scanned by the scanner 2 based on, for example, the signal of the weight sensor 4a.

[0079] When the information processing device 1 determines that the customer has not scanned all the products placed on the product stand A1a with the scanner 2 (No in S11), it performs object detection based on the image of the camera 3 (S12).

[0080] For example, the information processing device 1 detects (recognizes) the objects included in the image of the camera 3 by using a known image recognition process using artificial intelligence, and calculates the recognition score of the detected objects. The information processing device 1 finally detects the objects included in the image of the camera 3 based on the calculated recognition score. For example, the information processing device 1 finally detects the object with the highest recognition score as the object included in the image of the camera 3.

[0081] The information processing device 1 determines whether an item without a barcode has been manually input in the input device (S13).

[0082] When the information processing device 1 determines that an item without a barcode has not been manually input in the input device (No in S13), the process proceeds to step S11.

[0083] On the other hand, when the information processing device 1 determines that an item without a barcode has been manually input in the input device (Yes in S13), it calculates the probability that the object detected in S12 matches the manually input item without a barcode (S14).

[0084] Note that when the type of the object finally detected in S12 matches the type of the manually input item without a barcode, the information processing device 1 may use the recognition score of the object finally detected in S12 as the probability that the object detected in S12 matches the manually input item without a barcode. For example, assume that the object finally detected in S12 is "potato", and the manually input item without a barcode is also "potato". In this case, the information processing device 1 may use the recognition score of the "potato" finally detected in S12 as the probability that the "potato" finally detected in S12 matches the "potato" of the manually input item without a barcode.

[0085] On the other hand, when the type of the object finally detected in S12 does not match the type of the manually inputted barcode-less item, the information processing apparatus 1 may set the probability that the object detected in S12 matches the manually inputted barcode-less item to, for example, "0". For example, when the object finally detected in S12 is "sweet potato" and the manually inputted barcode-less item is "potato", the information processing apparatus 1 may set the probability that the object detected in S12 matches the manually inputted barcode-less item to "0".

[0086] The information processing apparatus 1 determines whether the probability calculated in S14 is less than a predetermined threshold value (third threshold value) (S15).

[0087] When the probability calculated in S14 by the information processing apparatus 1 is not less than the third threshold value (No in S15), the process proceeds to S11. That is, when there is a high possibility that the object detected by object detection from the camera image matches the manually inputted item, the information processing apparatus 1 proceeds to S11.

[0088] On the other hand, when the probability calculated in S14 by the information processing apparatus 1 is less than the third threshold value (Yes in S15), it detects fraud (S16). That is, when there is a low possibility that the object detected by object detection from the camera image matches the manually inputted item, the information processing apparatus 1 detects fraud.

[0089] When the information processing apparatus 1 determines in S11 that the customer has scanned all the items placed on the product stand A1a with the scanner 2 (Yes in S11), it executes accounting processing (S17). For example, the information processing apparatus 1 displays the total amount of the items purchased by the customer on the display 5 and executes payment processing.

[0090] In this embodiment, the reason for using the probability that the type of the manually inputted barcode-less item matches the type of the object detected by image recognition processing is as follows.

[0091] First, products without barcodes are often products with individual differences such as agricultural products, and the possibility of the products being exactly the same is low. Also, for example, when handling products such as "Make-in" and "Baron potatoes", both are "potatoes" in terms of type, and it may be difficult to accurately distinguish them from the image.

[0092] The image recognition process in this embodiment is performed for the purpose of determining whether the product has a barcode or not. That is, if it does not affect the result of this determination, it is not necessary to accurately recognize individual products. Here, for products of the same type, it is often the case that whether they have a barcode or not is the same. Therefore, from the perspective of detecting fraud, it is often sufficient to evaluate the probability that the types of products match.

[0093] In addition, when the product is a standardized product with the same appearance, etc., or when the presence or absence of a barcode is different even for products of the same type, fraud may be detected based on the probability that the product without a manually entered barcode matches the object detected by the image recognition process. Unless there is a particular need to distinguish, in this specification, the probability that the type of the product without a manually entered barcode matches the type of the object detected by the image recognition process is treated as an aspect of the probability that the product without a manually entered barcode matches the object detected by the image recognition process.

[0094] (Summary of the Second Embodiment) As described above, the information processing apparatus 1 determines whether there is fraud in the manual input based on the probability that the object detected by the image of the camera 3 matches the product for which product information is manually entered in the input device. By this operation, the information processing apparatus 1 can appropriately detect fraud using the manual input of product information.

[0095] <The Third Embodiment> The third embodiment is a combination of the first embodiment and the second embodiment.

[0096] (Operation Flow) FIG. 9 is a flowchart showing an operation example in the determination of unauthorized acts of the information processing apparatus 1 according to the third embodiment.

[0097] The information processing apparatus 1 determines whether all the products placed on the product stand A1a have been scanned by the scanner 2 (S21). The information processing apparatus 1 may determine whether all the products placed on the product stand A1a have been scanned by the scanner 2 based on, for example, the signal of the weight sensor 4a.

[0098] If the information processing apparatus 1 determines that not all the products placed on the product stand A1a have been scanned by the scanner 2 (No in S21), it performs product detection (candidate product detection) based on the image of the camera 3 and object detection based on the image of the camera 3 (S22).

[0099] The information processing apparatus 1 refers to the barcode presence / absence information (see FIG. 3) stored in the storage unit 12, and assigns information indicating whether the candidate product detected in step S22 is a product with a barcode or a product without a barcode, and generates a candidate product list (S23). For example, the information processing apparatus 1 generates the candidate product list shown in FIG. 4.

[0100] The information processing apparatus 1 determines whether the product without a barcode has been manually input at the input device (S24).

[0101] If the information processing apparatus 1 determines that the product without a barcode has not been manually input at the input device (No in S24), it transfers the process to step S21. For example, when a product with a barcode is scanned by the scanner 2, the information processing apparatus 1 transfers the process to step S21.

[0102] On the other hand, if the information processing apparatus 1 determines that the product without a barcode has been manually input at the input device (Yes in S24), it determines whether the product without a barcode is included in the finally detectable candidate products among the candidate products included in the candidate product list generated in S23 (S25).

[0103] When the candidate products that can be finally detected among the candidate products included in the candidate product list generated in S23 do not include products without barcodes (No in S25), the information processing apparatus 1 shifts the process to S28.

[0104] On the other hand, when the candidate products that can be finally detected among the candidate products included in the candidate product list generated in S23 include products without barcodes (Yes in S25), the information processing apparatus 1 calculates the probability that the object detected in S22 matches the manually input product without a barcode (S26).

[0105] The information processing apparatus 1 determines whether the probability calculated in S26 is less than a predetermined threshold (third threshold) (S27).

[0106] When the probability calculated in S26 is not less than the third threshold (No in S27), the information processing apparatus 1 shifts the process to S21. That is, when there is a manual input of a product without a barcode (Yes in S24), when it is highly likely that the image of the product without a barcode is included in the image of the camera 3 (Yes in S25), and when it is highly likely that the object detected by object detection using the camera image matches the manually input product (No in S27), the information processing apparatus 1 shifts the process to S21.

[0107] On the other hand, when the probability calculated in S26 is less than the third threshold (Yes in S27), the information processing apparatus 1 detects an illegal act (S28). That is, when there is a manual input of a product without a barcode (Yes in S24), when it is unlikely that the image of the product without a barcode is included in the image of the camera 3 (No in S25), or when it is unlikely that the object detected by object detection using the camera image matches the manually input product (Yes in S27), the information processing apparatus 1 detects an illegal act.

[0108] When the information processing apparatus 1 determines in S21 that all the products placed on the product stand A1a by the customer have been scanned by the scanner 2 (Yes in S21), it executes accounting processing (S29). For example, the information processing apparatus 1 displays the total amount of the products purchased by the customer on the display 5 and executes payment processing.

[0109] (Summary of the Third Embodiment) As described above, the information processing apparatus 1 refers to the barcode presence / absence information in which products are associated with information indicating the presence / absence of barcodes, and assigns information indicating whether the product detected based on the image of the camera 3 is a product with a product code or a product without a product code. When the product information of a product is manually input at the input device, the information processing apparatus 1 determines whether there is any fraud in the manual input based on the information indicating whether the product to which the product code is assigned is a product with a product code or a product without a product code. Further, the information processing apparatus 1 determines whether there is any fraud in the manual input based on the probability that the object detected by the image of the camera 3 matches the product for which the product information is manually input at the input device. By this operation, the information processing apparatus 1 can appropriately detect fraud using the manual input of product information.

[0110] <Fourth Embodiment> In the fourth embodiment, in addition to detecting fraud in the manual input of products without barcodes described in each of the above embodiments, fraud in the scanning of products with barcodes is detected. In the fourth embodiment, "barcode present" in the barcode presence / absence information (see FIG. 3) is further classified by the number of types of barcodes.

[0111] Note that the system configuration in the fourth embodiment is the same as that in FIG. 1, and the description thereof is omitted. Also, the block configuration in the fourth embodiment is the same as that in FIG. 5, but the function of the control unit 11 is different. For example, the control unit 11 detects whether there is any fraud in the manual input based on the number of types of product codes of the product scanned by the scanner and the number of types of product codes of the product detected by the image of the camera 3.

[0112] (Barcode presence / absence information) FIG. 10 is a diagram showing an example of barcode presence / absence information. In FIG. 10, parts different from the barcode presence / absence information described in FIG. 3 will be described.

[0113] There are products that can visually recognize multiple types of barcodes. For example, a 6-can pack of beer products has a barcode indicating that it is a 6-can pack of beer products on the outer packaging that combines 6 cans of beer. Also, in the 6-can pack of beer products, a part of the can of beer is exposed from the outer packaging, and the barcodes attached to each can of beer can be exposed. That is, the 6-can pack of canned beer products can visually recognize two types of barcodes.

[0114] The barcode presence / absence information stored in the storage unit 12 of the information processing apparatus 1 includes information on the presence / absence of barcodes for each major item and minor item of products, and further includes information indicating whether the number of types of barcodes attached to a product is one or more. For example, in FIG. 10, for "C Beer", information "Barcode present: multiple" indicating that it is a product with a barcode and there are multiple (for example, two) types of barcodes is associated. In this case, one type of barcode is assigned to one can of C Beer, and one type of barcode is assigned to the 6-can pack, and a total of two types of barcodes are assigned to C Beer.

[0115] In barcode-present products having multiple types of barcodes, there may be cases of improper behavior. For example, in a 6-can pack of beer, there may be a case where the barcode indicating that it is a 6-can pack of canned beer products attached to the outer packaging is not scanned, and the barcode attached to the inner can of beer is scanned. In this case, the payment is not made at the price of the 6-can pack, but at the price of one can of beer. The information processing apparatus 1 detects such improper behavior.

[0116] (Candidate product list) FIG. 11 is a diagram showing an example of a candidate product list. The information processing apparatus 1 generates a candidate product list in which information indicating whether each candidate product in the major items and minor items detected by the image recognition process is a product with a barcode or a product without a barcode, and information indicating the number of types of barcodes is attached to each candidate product. The information processing apparatus 1 refers to the barcode presence / absence information (see FIG. 10) stored in the storage unit 12, and attaches information indicating whether each candidate product detected by the image recognition process is a product with a barcode or a product without a barcode, and information indicating the number of types of barcodes to each candidate product.

[0117] For example, "C beer" in the candidate product list shown in FIG. 11 is a "product with a barcode: multiple" product as shown in FIG. 10. Therefore, the information processing apparatus 1 attaches information "product with a barcode: multiple" indicating that the candidate product "C beer" detected by the image recognition is a product with a barcode and has multiple types of barcodes. For example, there are a total of two types of barcodes, one barcode for one bottle of C beer and one barcode for a 6-can pack of C beer. The information processing apparatus 1 also attaches information to other candidate products detected by the image recognition process, and generates a candidate product list as shown in FIG. 11.

[0118] (Operation flow) FIG. 12 is a flowchart showing an operation example in the fraud determination of the information processing apparatus 1 according to the fourth embodiment.

[0119] The information processing apparatus 1 determines whether all the products placed on the product stand A1a have been scanned by the scanner 2 (S31). The information processing apparatus 1 may determine whether all the products placed on the product stand A1a have been scanned by the scanner 2 based on, for example, the signal of the weight sensor 4a.

[0120] When the information processing apparatus 1 determines that the customer has not scanned all the products placed on the product stand A1a with the scanner 2 (No in S31), it performs product detection (candidate product detection) based on the image of the camera 3 (S32). For example, the information processing apparatus 1 detects candidate products included in the image of the camera 3 by known image recognition processing using artificial intelligence, and calculates the recognition score of the detected candidate products.

[0121] The information processing apparatus 1 refers to the barcode presence / absence information (see FIG. 10) stored in the storage unit 12, and assigns to the candidate products detected in step S32 information indicating whether they are products with barcodes or products without barcodes, and information indicating the number of types of barcodes, and generates a candidate product list (S33). For example, the information processing apparatus 1 generates the candidate product list shown in FIG. 11.

[0122] The information processing apparatus 1 determines whether a product with a barcode has been scanned by the scanner 2 or a product without a barcode has been manually input at the input device (S34).

[0123] When the information processing apparatus 1 determines that a product without a barcode has been manually input at the input device (manual input in S34), it performs fraud detection processing for the manual input (S35). For example, the information processing apparatus 1 performs the fraud detection processing for manual input described in the first to third embodiments. After the fraud detection processing, the information processing apparatus 1 shifts the processing to S31.

[0124] On the one hand, when the information processing apparatus 1 determines that a product with a barcode has been scanned by the scanner 2 (scan in S34), it refers to the barcode presence / absence information (see FIG. 10) based on the scan information scanned by the scanner 2, and acquires the number of types of barcodes of the product in the major item (whether it is one or multiple). When the number of types of barcodes acquired by the information processing apparatus 1 is multiple, it determines whether the barcode of the candidate product in the smallest item that can be finally detected among the candidate product lists generated in S33 has been scanned by the scanner 2. That is, the information processing apparatus 1 determines whether an appropriate barcode has been scanned by the scanner 2 in a product having multiple types of barcodes (S36).

[0125] For example, based on the barcode scanned by the scanner 2, the information processing apparatus 1 identifies "C Beer" in the major item of FIG. 10 and acquires that "C Beer" has multiple types of barcodes (the barcode for one can of C Beer and the barcode for a 6-can pack). When the information processing apparatus 1 acquires that the product has multiple types of barcodes, it determines whether the barcode of the candidate product in the smallest item that can be finally detected among the candidate product lists generated in S33 has been scanned by the scanner 2.

[0126] When the barcode of the product scanned by the scanner 2 matches the barcode of the candidate product in the smallest item that can be finally detected among the candidate product lists generated in S33 (Yes in S36), the information processing apparatus 1 transfers the process to S31. That is, when an appropriate barcode among the multiple types of barcodes of the product acquired by detecting the product using the camera image in S32 has been scanned by the scanner 2, the information processing apparatus 1 transfers the process to S31. For example, when a 6-can pack of beer (having two types of barcodes) is detected by image recognition processing and the barcode attached to the outer package of the 6-can pack among the two types of barcodes has been scanned, the information processing apparatus 1 transfers the process to S31.

[0127] On the other hand, if the barcode of the product scanned by the scanner 2 does not match the barcode of the candidate product in the smallest item that can be finally detected among the candidate product lists generated in S33 (No in S36), the information processing apparatus 1 detects fraud (S37). That is, if an appropriate barcode among the multiple barcodes of the product obtained by detecting the product using the camera image in S32 is not scanned by the scanner 2, the information processing apparatus 1 detects fraud. For example, even though a 6-can pack of beer (having two types of barcodes) is detected by image recognition processing, if the barcode attached to the beer inside the 6-can pack among the two types of barcodes is scanned, the information processing apparatus 1 detects fraud. After detecting fraud, the information processing apparatus 1 shifts the process to S31.

[0128] If the information processing apparatus 1 determines in S31 that the customer has scanned all the products placed on the product stand A1a with the scanner 2 (Yes in S31), it executes the accounting process (S38). For example, the information processing apparatus 1 displays the total amount of the products purchased by the customer on the display 5 and executes the payment process.

[0129] In this embodiment, the case of two types of canned beer, one can and a 6-can pack, is shown. However, the present invention is not limited to this, and it goes without saying that it is also applicable to cases such as a 12-can pack or a 24-can pack. Needless to say, it is also applicable when there are three or more types of the number of canned beers sold. For example, the present invention is also applicable to the case of three types of canned beer, one can, a 6-can pack, and a 12-can pack. In this case, the number of barcodes is three.

[0130] Also, in this embodiment, canned beer is shown as a product to which multiple types of barcodes are assigned. However, it goes without saying that the present disclosure is not limited to this and is applicable to any product to which multiple types of barcodes are assigned.

[0131] (Summary of the Fourth Embodiment) As described above, the information processing apparatus 1 determines whether there is any improper behavior in manual input based on the number of types of product codes of the product scanned by the scanner 2 and the number of types of product codes of the product detected from the image of the camera 3. By this operation, the information processing apparatus 1 can appropriately detect improper behavior using manual input of product information.

[0132] As described above, the embodiments have been described with reference to the drawings, but the present disclosure is not limited to such examples. It is obvious that those skilled in the art can come up with various modification examples or correction examples within the scope described in the claims. Such modification examples or correction examples are also understood to belong to the technical scope of the present disclosure. Also, within the scope not departing from the gist of the present disclosure, the components in the embodiments may be arbitrarily combined. The embodiments may be arbitrarily combined. The modification examples may be arbitrarily combined.

[0133] In the above-described embodiment, information indicating whether the candidate product detected from the image of the camera 3 is a product with a barcode or a product without a barcode is given. However, the purpose of giving this information is to associate the product detected from the image of the camera 3 with the result of determining whether the product has a barcode or not. Therefore, this association may be performed in a manner other than giving information to the candidate product.

[0134] In the above-described embodiment, the notation “... unit” used for each component may be replaced with other notations such as “... circuitry”, “... assembly”, “... device”, “... unit”, or “... module”.

[0135] This disclosure can be implemented by software, hardware, or software in cooperation with hardware. Each functional block used in the description of the above embodiments is realized, partially or entirely, as an LSI which is an integrated circuit, and each process described in the above embodiments may be controlled, partially or entirely, by one LSI or a combination of LSIs. The LSI may be composed of individual chips, or may be composed of one chip so as to include part or all of the functional blocks. The LSI may be provided with data input and output. Depending on the degree of integration, the LSI may also be referred to as an IC, a system LSI, a super LSI, or an ultra LSI.

[0136] The method of integrating into a circuit is not limited to LSI, and it may be realized by a dedicated circuit, a general-purpose processor, or a dedicated processor. Further, after manufacturing the LSI, an FPGA (Field Programmable Gate Array) that can be programmed, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used. This disclosure may be realized as digital processing or analog processing.

[0137] Furthermore, if a circuit integration technology that replaces the LSI appears due to the progress of semiconductor technology or other derived technologies, naturally, the technology may be used to integrate the functional blocks. The application of biotechnology and the like are possible.

Industrial Applicability

[0138] This disclosure is useful for detecting fraudulent manual input in self-checkout.

Explanation of Signs

[0139] 1 Information processing device 2, 2a, 2b Scanner 3 Camera 4a, 4b Weight sensor 5 Display A1a, A1b Product placement stand

Claims

1. a determination unit that refers to product code presence / absence information that associates a product with information indicating the presence or absence of a product code and determines whether a product detected based on a camera image is a product with a product code or a product without a product code; a determination unit that, when product information of a product is manually inputted into an input device, determines whether or not a fraudulent act utilizing the manual input has occurred based on a result of determining whether the product has a product code or does not have a product code; An information processing device having the above configuration.

2. The product detected based on the camera image has a score indicating the degree of certainty of the detection result, the determination unit determines whether or not a fraudulent act using the manual input has occurred for a product having a score equal to or higher than a predetermined threshold based on a result of determining whether the product has a product code or does not have a product code. The information processing device according to claim 1 .

3. The product detected based on the camera image has a score indicating the degree of certainty of the detection result, the determination unit determines whether or not a fraudulent act using the manual input has occurred based on a ratio of products with product codes having a score equal to or higher than a predetermined threshold to products having a score equal to or higher than the predetermined threshold, or a ratio of products without product codes having a score equal to or higher than the predetermined threshold to products having a score equal to or higher than the predetermined threshold. The information processing device according to claim 1 .

4. The determination unit further determines whether or not a fraudulent act using the manual input has occurred based on a probability that an object detected by the camera image matches a product whose product information has been manually input into the input device. The information processing device according to claim 1 .

5. The determination unit further determines whether or not a fraudulent act using the manual input has occurred based on the number of types of product codes of the products scanned by the scanner and the number of types of product codes of the products detected by the camera image. The information processing device according to claim 1 .

6. Refer to product code presence / absence information in which the product is associated with information indicating the presence or absence of a product code, and determine whether the product detected based on the camera image has a product code or does not have a product code; When product information of a product is manually inputted into an input device, a determination is made as to whether or not a fraudulent act utilizing the manual input has occurred based on a result of a determination as to whether the product is a product with a product code or a product without a product code. Method of determining fraud.

Images