Information processing device and fraud determination method

Abstract

This information processing device comprises: a determination unit that refers to commodity code presence / absence information in which commodities and information indicating the presence or absence of commodity codes are associated, and determines whether a commodity detected on the basis of a camera image is a commodity with a commodity code or a commodity without a commodity code; and a determination unit that, when the commodity information about the commodity is manually input in an input device, determines whether there is fraud using the manual input on the basis of the result of determining whether the commodity is a commodity with a commodity code or a commodity without a commodity code.

Description

Information processing device and fraudulent activity determination method

[0001] The present disclosure relates to an information processing device and a method for determining fraudulent activity.

[0002] In recent years, the spread of self-checkout systems has been rapidly increasing in order to reduce the man-hours of store staff in retail stores. However, since customers scan the barcodes of products themselves, it is important to take measures to prevent fraudulent activities such as shoplifting.

[0003] There are self-checkout registers that use weight sensors to detect shoplifting. For example, a device with a self-scanning function disclosed in Patent Document 1 includes a storage unit that stores products whose product codes have been read, and the storage unit has a weight sensor. The device in Patent Document 1 issues an alarm if the weight of the storage unit increases without a product being scanned.

[0004] Japanese Patent Application Publication No. 7-141569

[0005] However, self-checkout registers that use weight sensors such as those in Patent Document 1 to detect fraudulent customer behavior may not be able to properly detect fraudulent behavior involving manual input of products that do not have barcodes, such as vegetables and fruits (non-barcoded products).

[0006] For example, a customer may manually enter a cheaper product without a barcode without scanning the product with a barcode (product with a barcode) with a scanner. For example, a customer may manually enter a cheaper product without a barcode (potatoes) without a barcode (which are cheaper than the meat) without scanning the meat with a scanner, and then move the meat to the storage section. Devices that use weight sensors to detect fraudulent activity may not be able to detect the above-mentioned fraudulent activity that uses manual entry.

[0007] Non-limiting examples of the present disclosure contribute to providing an information processing device and a fraud determination method that can appropriately detect fraudulent activities involving manual input of product information.

[0008] An information processing device according to one embodiment of the present disclosure includes a determination unit that references product code presence / absence information that associates products with information indicating the presence or absence of a product code, and determines whether a product detected based on a camera image is a product with a product code or a product without a product code, and a determination unit that, when product information for a product is manually entered into an input device, determines whether or not fraudulent activity has occurred using the manual input, based on the result of determining whether the product is a product with a product code or a product without a product code.

[0009] A fraudulent activity determination method according to one embodiment of the present disclosure refers to product code presence / absence information that associates a product with information indicating whether or not it has a product code, determines whether a product detected based on a camera image has a product code or does not have a product code, and, if product information for a product is manually entered into an input device, determines whether or not fraudulent activity has occurred using the manual input based on the result of determining whether the product has a product code or does not have a product code.

[0010] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0011] According to an embodiment of the present disclosure, fraudulent activity involving manual input of product information can be appropriately detected.

[0012] Further advantages and benefits of an embodiment of the present disclosure will become apparent from the specification and drawings. Such advantages and / or benefits may be provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of them may be provided to obtain one or more identical features.

[0013] FIG. 1 shows an example of the configuration of a self-checkout system including an information processing device according to a first embodiment. FIG. 2 shows an example of a diagram product detection process. FIG. 3 shows an example of a diagram candidate product list showing an example of barcode presence / absence information. FIG. 4 shows an example of the block configuration of an information processing device. Flowchart showing an example of the operation of an information processing device in determining fraudulent activity. FIG. 5 explains variant 1 of the first embodiment. Flowchart showing an example of the operation of an information processing device in determining fraudulent activity. Flowchart showing an example of the operation of an information processing device in determining fraudulent activity according to a third embodiment. FIG. 6 shows an example of a diagram candidate product list showing an example of barcode presence / absence information. Flowchart showing an example of the operation of an information processing device in determining fraudulent activity according to a fourth embodiment.

[0014] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or redundant explanation of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art.

[0015] The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0016] <First embodiment> (System configuration) Fig. 1 is a diagram showing an example of the configuration of a self-checkout system including an information processing device 1 according to a first embodiment. As shown in Fig. 1, the self-checkout system has an information processing device 1, scanners 2a and 2b, a camera 3, weight sensors 4a and 4b, and a display 5. In addition to the self-checkout system, Fig. 1 also shows product stands A1a and A1b and a stand A2.

[0017] The product stand A1a is a stand on which products are placed before being scanned by the scanners 2a and 2b. The products placed on the product stand A1a before being scanned may include shopping baskets or carts from a store containing products before being scanned.

[0018] The product stand A1b is a stand on which products are placed after being scanned by the scanners 2a and 2b. The scanned products placed on the product stand A1b may include a customer's shopping basket into which the scanned products are placed.

[0019] The information processing device 1 is, for example, a computer such as a personal computer or a server. In Fig. 1, the information processing device 1 is placed outside the table A2, but it may be placed inside the table A2 or in the store's office. The information processing device 1 performs, for example, a payment process for products and a process for detecting fraudulent activities such as fraudulent manual input by customers.

[0020] The scanner 2a is a handheld scanner, and the scanner 2b is a fixed scanner fixed to a stand A2. Hereinafter, when there is no need to distinguish between the scanners 2a and 2b, they will be simply referred to as the scanner 2.

[0021] The scanner 2 is connected to the information processing device 1. The scanner 2 scans product codes attached to products. The scanner 2 transmits the scanned product codes to the information processing device 1. The product codes are, for example, bar codes such as JAN (Japanese Article Number) codes. The scanner 2 may also be referred to as a barcode reader or a reader.

[0022] The camera 3 is connected to the information processing device 1. The camera 3 transmits image data of a captured image to the information processing device 1.

[0023] Camera 3 is installed so as to capture the area around where customers scan products with scanner 2 (see dotted frame A3 in FIG. 1 ). For example, camera 3 is installed above display 5 or above stand A2, and the angle of view is set so as to include the area between product stand A1a and product stand A1b, i.e., the area around where customers scan products with scanner 2.

[0024] The weight sensor 4a is connected to the information processing device 1. The weight sensor 4a is installed, for example, inside the product stand A1a, and transmits a signal to the information processing device 1 according to the weight of the product placed on the product stand A1a (or a shopping basket or the like containing the product).

[0025] The weight sensor 4b is connected to the information processing device 1. The weight sensor 4b is installed, for example, inside the product stand A1b, and transmits to the information processing device 1 a signal corresponding to the weight of the product placed on the product stand A1b (or a customer's shopping basket containing the product, etc.).

[0026] The display 5 is connected to the information processing device 1. The display 5 displays, for example, product information of a product scanned by a customer using the scanner 2. The product information may be, for example, the name, price, and quantity of the product.

[0027] The display 5 may be provided with a touch panel on its screen surface. The touch panel accepts customer operations. A customer manually inputs information about a product without a product code, such as vegetables or fruits, via the touch panel and registers the product as a purchased product. For example, when a customer purchases apples, which do not have a barcode, the customer manually inputs the product name "apples" and the purchase quantity "X" via the touch panel and registers the product as a purchased product. Note that the device that accepts customer operations is not limited to a touch panel. The device that accepts customer operations may be, for example, a key input device separate from the display 5. Alternatively, customer operations may be accepted using a smartphone or mobile terminal owned by the customer.

[0028] Display 5 displays a screen related to the detection of fraudulent activity using manual input under the control of information processing device 1. For example, if a customer manually inputs a product with a barcode as a product without a barcode that is cheaper than the product with a barcode, display 5 displays a screen indicating that fraudulent activity using manual input has occurred. More specifically, the customer does not scan meat, a product with a barcode, with scanner 2, but manually inputs potatoes, a product without a barcode (which is cheaper than the meat), and moves the meat to product stand A1b. In this case, display 5 displays a screen indicating that fraudulent activity using manual input has occurred under the control of information processing device 1.

[0029] (Product Detection Processing) Fig. 2 is a diagram illustrating an example of product detection processing. The information processing device 1 detects (recognizes) products included in the image captured by the camera 3 using, for example, a known image recognition technology (image recognition processing program) that uses artificial intelligence. In other words, the information processing device 1 detects products removed from the product stand A1a through image recognition processing. The product image recognition processing may be performed by a separate device such as a server.

[0030] The information processing device 1 calculates a recognition score for each candidate product (candidate product) detected by image recognition processing. The recognition score indicates the degree of accuracy of the detection result of the detected candidate product. In the example of FIG. 2, the information processing device 1 detects five candidate products as products included in the image of the camera 3, but this is not limited to this. The number of candidate products detected by the information processing device 1 may be, for example, one or six or more.

[0031] The information processing device 1 may determine (finally detect) the candidate product with the highest recognition score as the product included in the image captured by the camera 3. For example, in the example of FIG. 2 , steak beef B may be finally detected as the product included in the image captured by the camera 3.

[0032] Alternatively, if the candidate product has the highest recognition score and the recognition score of that candidate product is equal to or greater than a predetermined threshold (first threshold), the information processing device 1 may finally detect the candidate product with the highest recognition score as a product included in the image captured by the camera 3. For example, the first threshold is 90%. In this case, if the recognition score of the candidate product with the highest recognition score is 90% or greater, the information processing device 1 may finally detect the candidate product with the highest recognition score as a product included in the image captured by the camera 3. On the other hand, if the recognition score of the candidate product with the highest recognition score is less than 90%, the information processing device 1 may display an image on the display 5 indicating that the product could not be detected. The method for finally detecting a product included in the image captured by the camera 3 is not limited to the above example.

[0033] (Barcode Presence Information) Fig. 3 is a diagram showing an example of barcode presence information. As shown in Fig. 3, the storage unit of the information processing device 1 stores barcode presence information for each product.

[0034] 3, the products "onion" and "potato" are products without barcodes. Note that, since the products "onion" and "potato" are products without barcodes, they are products that are manually input using an input device such as a touch panel on the display 5.

[0035] 3, the products "Cola A," "Beef Steak B," "Beer C," and "Sake D" are barcoded products. Note that since the products "Cola A," "Beef Steak B," "Beer C," and "Sake D" are barcoded products, they are scanned by the scanner 2.

[0036] (Candidate Product List) Figure 4 is a diagram showing an example of a candidate product list. The information processing device 1 generates a candidate product list in which each candidate product (see Figure 2) detected by image recognition processing is assigned information indicating whether it is a product with a barcode or a product without a barcode. The information processing device 1 refers to barcode presence / absence information (see Figure 3) stored in the memory unit, and assigns information indicating whether it is a product with a barcode or a product without a barcode to each candidate product detected by image recognition processing.

[0037] For example, "Beef Steak B" in the candidate product list shown in Fig. 4 is a product with a barcode, as shown in Fig. 3. Therefore, the information processing device 1 assigns information "with barcode" indicating that the candidate product "Beef Steak B" detected by image recognition is a product with a barcode. The information processing device 1 also assigns information as to whether the other candidate products detected by image recognition processing are products with barcodes or products without barcodes, and generates a candidate product list as shown in Fig. 4.

[0038] (Example of fraud detection operation) For simplicity of explanation, in this example, a customer removes the barcoded product "Steak Beef B" from product stand A1a and moves it to product stand A1b. The customer does not scan the barcoded product "Steak Beef B" with scanner 2, but manually inputs the non-barcoded product "Potatoes" (cheaper than Steak Beef B) into the input device. In other words, the customer manually inputs the barcoded product "Steak Beef B" as "Potatoes" which are cheaper than "Steak Beef B," thereby making a fraudulent purchase.

[0039] The information processing device 1 uses image recognition processing to detect products (candidate products) that may be included in the image captured by the camera 3 (see FIG. 2). The information processing device 1 references barcode presence / absence information (see FIG. 3) and generates a candidate product list in which the detected candidate products are assigned information indicating whether they have a barcode or do not have a barcode. Here, the information processing device 1 generates the candidate product list shown in FIG. 4.

[0040] Since the customer has taken out the barcoded product "Steak Beef B" from the product stand A1a, the candidate product "Steak Beef B" has the highest recognition score in the candidate product list shown in Figure 4. Furthermore, the candidate product "Steak Beef B" is a product that "has a barcode."

[0041] The information processing device 1 determines whether or not manual input has been made to the input device. In this case, the customer has manually input "potato," so the information processing device 1 determines that manual input has been made to the input device.

[0042] When a product without a barcode is manually input, the information processing device 1 determines whether any of the candidate products included in the generated candidate product list have a recognition score equal to or greater than a predetermined threshold (second threshold). For example, the information processing device 1 determines whether any of the candidate products have a recognition score equal to or greater than 90%. In the example of Figure 4, since no product without a barcode is included in the candidate products with a recognition score equal to or greater than 90%, the information processing device 1 determines that no product without a barcode is included in the candidate products with a recognition score equal to or greater than the second threshold.

[0043] The information processing device 1 detects fraudulent activity when it determines that no barcoded products are included among the candidate products having a recognition score equal to or higher than the second threshold, even though no barcoded products have been manually input. In other words, the information processing device 1 detects fraudulent activity when no barcoded products are included among the candidate products that can ultimately be detected based on the image captured by the camera 3 (candidate products having a recognition score equal to or higher than the second threshold), even though no barcoded products have been manually input. The second threshold may be the same as or different from the first threshold.

[0044] (Block Diagram) Fig. 5 is a diagram showing an example of a block configuration of the information processing device 1. As shown in Fig. 5, the information processing device 1 has a control unit 11, a storage unit 12, and a communication unit 13.

[0045] The control unit 11 controls the entire information processing device 1. The control unit 11 may be configured by a processor such as a CPU (central processing unit), for example.

[0046] The control unit 11 includes a detection unit 11 a, an assignment unit 11 b, a fraud determination unit 11 c, and an alarm generation unit 11 d. The control unit 11 may realize the functions of the above-mentioned units in accordance with a program stored in the storage unit 12.

[0047] The detection unit 11a detects (extracts) product candidates included in the image captured by the camera 3, and calculates a recognition score for each of the detected candidate products (see FIG. 2).

[0048] The assigning unit 11b refers to the barcode presence / absence information stored in the memory unit 12, assigns information indicating whether the candidate product detected by the detection unit 11a is a product with a barcode or a product without a barcode, and generates a candidate product list (see Figure 4).

[0049] When product information for a product without a barcode is manually entered into an input device, the fraudulent activity determination unit 11c determines whether or not there has been fraudulent activity in the manual input based on the information assigned to the candidate product in the candidate product list as to whether the product is a product with a barcode or a product without a barcode.

[0050] When the fraudulent activity determination unit 11c determines that fraudulent activity has occurred, the alarm generation unit 11d generates alarm information. For example, the alarm generation unit 11d generates image data of an image including text such as "Please scan the barcode of the product." The image data is transmitted to the display 5 via the communication unit 13.

[0051] The alarm generation unit 11d also generates, for example, an alarm signal. The alarm signal is transmitted to a mobile terminal carried by the store clerk via the communication unit 13. The mobile terminal that receives the alarm signal notifies the store clerk of the occurrence of unauthorized manual input, for example, by sound.

[0052] The storage unit 12 stores an operating system (OS) program and application programs to be executed by the control unit 11. For example, the storage unit 12 stores an application program that processes a self-checkout transaction and monitors shoplifting. The storage unit 12 also stores various data necessary for processing by the control unit 11. The storage unit 12 also stores the barcode presence / absence information described in FIG. 3. The storage unit 12 may be, for example, a solid state drive (SSD), random access memory (RAM), flash memory, read only memory (ROM), and / or a hard disk drive (HDD).

[0053] The communication unit 13 communicates, by wire or wirelessly, with the scanner 2, camera 3, weight sensors 4a and 4b, and display 5. The communication unit 13 also communicates wirelessly with a mobile terminal carried by a store clerk.

[0054] (Operation Flow) FIG. 6 is a flowchart showing an example of the operation of the information processing device 1 in determining whether or not there is any fraudulent activity.

[0055] The information processing device 1 determines whether or not the customer has scanned all of the products placed on the product stand A1a with the scanner 2 (S1). The information processing device 1 may determine whether or not all of the products placed on the product stand A1a have been scanned with the scanner 2, for example, based on a signal from the weight sensor 4a.

[0056] If the information processing device 1 determines that the customer has not scanned all of the products placed on the product stand A1a with the scanner 2 (No in S1), it performs product detection (candidate product detection) based on the image from the camera 3 (S2). For example, the information processing device 1 detects candidate products included in the image from the camera 3 by a known image recognition process using artificial intelligence, and calculates recognition scores for the detected candidate products (see FIG. 2).

[0057] The information processing device 1 refers to the barcode presence / absence information (see FIG. 3) stored in the storage unit 12, and assigns information indicating whether the candidate product detected in step S2 is a barcode-containing product or a barcode-free product, thereby generating a candidate product list (S3). For example, the information processing device 1 generates the candidate product list shown in FIG.

[0058] The information processing device 1 determines whether or not a product without a barcode has been manually input into the input device (S4).

[0059] If the information processing device 1 determines that the product without a barcode has not been manually input into the input device (No in S4), the information processing device 1 proceeds to step S1. For example, if the product with a barcode has been scanned by the scanner 2, the information processing device 1 proceeds to step S1.

[0060] On the other hand, if the information processing device 1 determines that a product without a barcode has been manually input via the input device (Yes in S4), it determines whether the product without a barcode is included in the candidate products that can ultimately be detected among the candidate products included in the candidate product list generated in S3 (S5). For example, the information processing device 1 determines whether the candidate products having a recognition score equal to or greater than a second threshold, such as 90%, include a product without a barcode.

[0061] If the information processing device 1 determines that a product without a barcode is included in the final detectable candidate products included in the candidate product list generated in S3 (Yes in S5), the information processing device 1 proceeds to S1. In other words, if a product without a barcode is manually entered and the image from camera 3 indicates that a product without a barcode may be included, the information processing device 1 does not detect fraud and proceeds to S1.

[0062] On the other hand, if the final detectable candidate products among the candidate products included in the candidate product list generated in S3 do not include a product without a barcode (No in S5), the information processing device 1 detects fraudulent activity (S6). That is, the information processing device 1 detects fraudulent activity if a product with a barcode may be included in the image from camera 3 despite the manual input of a product without a barcode. Note that if the information processing device 1 detects fraudulent activity in S6, it generates alarm information and an alarm signal and transitions to S1.

[0063] If the information processing device 1 determines in S1 that the customer has scanned all of the products placed on the product stand A1a with the scanner 2 (Yes in S1), it executes the checkout process (S7). For example, the information processing device 1 displays the total price of the products purchased by the customer on the display 5 and executes the payment process.

[0064] (Summary of the first embodiment) As described above, the information processing device 1 refers to barcode presence / absence information that associates a product with information indicating the presence or absence of a barcode, and assigns information indicating whether the product has a product code or does not have a product code to a product detected based on an image from the camera 3. When product information about a product is manually entered into an input device, the information processing device 1 determines whether there has been any fraudulent activity in the manual input based on the information assigned to the product indicating whether the product has a product code or does not have a product code. This operation allows the information processing device 1 to appropriately detect fraudulent activity involving the manual input of product information.

[0065] (Variation 1) The information processing device 1 detects fraudulent activity when no products without barcodes are included among the candidate products having a recognition score equal to or greater than the second threshold, but this is not limited to this. The information processing device 1 may also detect fraudulent activity based on the proportion of products with barcodes among the candidate products having a recognition score equal to or greater than the second threshold.

[0066] FIG. 7 is a diagram illustrating a first modification of the first embodiment. FIG. 7 shows a candidate product list. In the explanation of FIG. 7, the second threshold is set to 90%. The information processing device 1 detects fraudulent activity when the proportion of barcode-bearing products among candidate products equal to or greater than the second threshold is 75% or greater.

[0067] As shown in Figure 7, assume there are four candidate products with recognition scores of 90% or higher. In this case, the information processing device 1 detects fraudulent activity based on the proportion of products with barcodes among the four candidate products with recognition scores of 90% or higher. In the example of Figure 7, the proportion of products with barcodes is 3 / 4 (75%), so the information processing device 1 detects fraudulent activity. In other words, the information processing device 1 detects fraudulent activity when a high proportion of candidate products with barcodes is present despite the manual input of products without barcodes.

[0068] As described above, the information processing device 1 detects fraudulent activity based on the proportion of barcode-equipped products among the number of candidate products having a recognition score equal to or greater than the second threshold. This allows the information processing device 1 to appropriately detect fraudulent activity involving manual input of product information.

[0069] The information processing device 1 may detect fraudulent activity based on the proportion of products without barcodes among candidate products having a recognition score equal to or greater than a second threshold. For example, the information processing device 1 may detect fraudulent activity if the proportion of products without barcodes is less than a predetermined threshold.

[0070] (Variation 2) The information processing device 1 detects fraudulent activity when no product without a barcode is included among the candidate products having a recognition score equal to or greater than the second threshold, but this is not limited to this. The information processing device 1 may also detect fraudulent activity when the product last detected as a product included in the image of the camera 3 is not a product without a barcode. Note that cases where the product is not a product without a barcode include cases where the product has a barcode and cases where it is not possible to determine whether the product has a barcode or does not have a barcode.

[0071] For example, the information processing device 1 finally detects the candidate product with the highest recognition score as the product included in the image of the camera 3. When a product without a barcode is manually input, the information processing device 1 detects fraudulent activity if the product finally detected by the above-described method is not a product without a barcode.

[0072] Alternatively, if the candidate product has the highest recognition score and the recognition score of that candidate product is equal to or greater than a first threshold, the information processing device 1 finally detects the candidate product with the highest recognition score as a product included in the image of the camera 3. If a product without a barcode has been manually input, the information processing device 1 detects fraudulent activity if the product finally detected by the above-described method is not a product without a barcode.

[0073] As described above, the information processing device 1 detects fraudulent activity if the last detected product included in the image captured by the camera 3 is not a product without a barcode. This allows the information processing device 1 to appropriately detect fraudulent activity involving manual input of product information.

[0074] (Variation 3) The information processing device 1 detects fraudulent activity when no products without barcodes are included among the candidate products having a recognition score equal to or higher than the second threshold, but this is not limited to this. The information processing device 1 does not need to detect fraudulent activity when products with barcodes are included among the candidate products having a recognition score equal to or higher than the second threshold. This allows the information processing device 1 to appropriately detect fraudulent activity involving manual input of product information.

[0075] <Second embodiment> In the second embodiment, when a product without a barcode is manually input, the information processing device 1 detects fraudulent activity based on the probability that the type of the manually input product without a barcode matches the type of object detected by image recognition processing.

[0076] The system configuration in the second embodiment is the same as that in Fig. 1, and therefore a description thereof will be omitted. The block configuration in the second embodiment is also the same as that in Fig. 5, but the function of the control unit 11 is different. For example, the control unit 11 determines whether or not there has been any fraudulent activity in the manual input based on the probability that an object detected in an image from the camera 3 matches a product manually input into the input device.

[0077] (Operation Flow) FIG. 8 is a flowchart showing an example of the operation of the information processing device 1 according to the second embodiment in determining whether or not there is any fraudulent activity.

[0078] The information processing device 1 determines whether the customer has scanned all of the products placed on the product stand A1a with the scanner 2 (S11). The information processing device 1 may determine whether all of the products placed on the product stand A1a have been scanned with the scanner 2, for example, based on a signal from the weight sensor 4a.

[0079] If the information processing device 1 determines that the customer has not scanned all of the products placed on the product stand A1a with the scanner 2 (No in S11), it performs object detection based on the image from the camera 3 (S12).

[0080] For example, the information processing device 1 detects (recognizes) an object included in the image captured by the camera 3 by a known image recognition process using artificial intelligence, and calculates a recognition score for the detected object. Based on the calculated recognition score, the information processing device 1 finally detects the object included in the image captured by the camera 3. For example, the information processing device 1 finally detects the object with the highest recognition score as the object included in the image captured by the camera 3.

[0081] The information processing device 1 determines whether or not a product without a barcode has been manually input into the input device (S13).

[0082] If the information processing device 1 determines that the non-barcoded product has not been manually input into the input device (No in S13), the process proceeds to step S11.

[0083] On the other hand, if the information processing device 1 determines that a barcode-less product has been manually entered into the input device (Yes in S13), it calculates the probability that the object detected in S12 matches the manually entered barcode-less product (S14).

[0084] Note that, if the type of object finally detected in S12 matches the type of manually inputted product without a barcode, the information processing device 1 may use the recognition score of the object finally detected in S12 as the probability that the object detected in S12 matches the manually inputted product without a barcode. For example, the object finally detected in S12 may be a "potato." The manually inputted product without a barcode may be a "potato." In this case, the information processing device 1 may use the recognition score of the "potato" finally detected in S12 as the probability that the "potato" finally detected in S12 matches the "potato" of the manually inputted product without a barcode.

[0085] In contrast, if the type of object finally detected in S12 does not match the type of manually inputted product without a barcode, the information processing device 1 may set the probability that the object detected in S12 matches the manually inputted product without a barcode, for example, to "0." For example, if the object finally detected in S12 is a "sweet potato" and the manually inputted product without a barcode is a "potato," the information processing device may set the probability that the object detected in S12 matches the manually inputted product without a barcode to "0."

[0086] The information processing device 1 determines whether the probability calculated in S14 is smaller than a predetermined threshold value (third threshold value) (S15).

[0087] If the probability calculated in S14 is not smaller than the third threshold value (No in S15), the information processing device 1 proceeds to S11. That is, if there is a high possibility that the object detected by the camera image matches the manually input product, the information processing device 1 proceeds to S11.

[0088] On the other hand, if the probability calculated in S14 is smaller than the third threshold (Yes in S15), the information processing device 1 detects fraudulent activity (S16). That is, the information processing device 1 detects fraudulent activity when there is a low possibility that an object detected by object detection using a camera image matches a manually input product.

[0089] If the information processing device 1 determines in S11 that the customer has scanned all of the products placed on the product stand A1a with the scanner 2 (Yes in S11), it executes a checkout process (S17). For example, the information processing device 1 displays the total price of the products purchased by the customer on the display 5 and executes a payment process.

[0090] In this embodiment, the reason why the probability that the manually input type of a barcode-less product matches the type of an object detected by image recognition processing is used is as follows.

[0091] First, products without barcodes are often agricultural products and other products with individual differences, making it unlikely that the products will match exactly. Also, for example, there are cases where products such as "May Queen" and "Danshaku Imo" are both types of "potatoes," and are difficult to accurately distinguish from images, are sold together.

[0092] The image recognition process in this embodiment is performed for the purpose of determining whether a product has a barcode or does not have a barcode. In other words, there is no need to accurately recognize individual products unless it affects the result of this determination. Here, since there are many cases where products of the same type match whether they have a barcode or do not have a barcode, from the perspective of fraud detection, it is often sufficient to evaluate the probability that the product types match.

[0093] Note that in cases where products are standardized products with the same appearance, or where products of the same type differ in whether they have a barcode or not, fraud may be detected based on the probability that a manually entered product without a barcode matches an object detected by image recognition processing, rather than the probability that the types match. Unless there is a particular need to distinguish, in this specification, the probability that a manually entered product without a barcode matches the type of an object detected by image recognition processing is treated as one aspect of the probability that a manually entered product without a barcode matches an object detected by image recognition processing.

[0094] (Summary of the Second Embodiment) As described above, the information processing device 1 determines whether or not there has been any fraudulent activity in the manual input based on the probability that an object detected in an image captured by the camera 3 matches a product whose product information has been manually input into the input device. This operation enables the information processing device 1 to appropriately detect any fraudulent activity involving the manual input of product information.

[0095] Third Embodiment The third embodiment is a combination of the first and second embodiments.

[0096] (Operation Flow) FIG. 9 is a flowchart showing an example of the operation of the information processing device 1 according to the third embodiment in determining whether or not there is any fraudulent activity.

[0097] The information processing device 1 determines whether the customer has scanned all of the products placed on the product stand A1a with the scanner 2 (S21). The information processing device 1 may determine whether all of the products placed on the product stand A1a have been scanned with the scanner 2, for example, based on a signal from the weight sensor 4a.

[0098] If the information processing device 1 determines that the customer has not scanned all of the products placed on the product stand A1a with the scanner 2 (No in S21), it performs product detection (candidate product detection) based on the image from the camera 3 and object detection based on the image from the camera 3 (S22).

[0099] The information processing device 1 refers to the barcode presence / absence information (see FIG. 3) stored in the storage unit 12, and assigns information indicating whether the candidate product detected in step S22 is a barcode-containing product or a barcode-free product to generate a candidate product list (S23). For example, the information processing device 1 generates the candidate product list shown in FIG.

[0100] The information processing device 1 determines whether or not a product without a barcode has been manually input into the input device (S24).

[0101] If the information processing device 1 determines that the product without a barcode has not been manually input into the input device (No in S24), the information processing device 1 proceeds to step S21. For example, if the product with a barcode has been scanned by the scanner 2, the information processing device 1 proceeds to step S21.

[0102] On the other hand, if the information processing device 1 determines that a product without a barcode has been manually entered into the input device (Yes in S24), it determines whether the product without a barcode is included in the candidate products included in the candidate product list generated in S23 that can be finally detected (S25).

[0103] If the information processing device 1 does not include a barcode-less product among the candidate products included in the candidate product list generated in S23 that can be finally detected (No in S25), the information processing device 1 proceeds to S28.

[0104] On the other hand, if the candidate products included in the candidate product list generated in S23 include a product without a barcode among the candidate products that can ultimately be detected (Yes in S25), the information processing device 1 calculates the probability that the object detected in S22 matches the manually entered product without a barcode (S26).

[0105] The information processing device 1 determines whether the probability calculated in S26 is smaller than a predetermined threshold value (third threshold value) (S27).

[0106] If the probability calculated in S26 is not less than the third threshold value (No in S27), the information processing device 1 proceeds to S21. That is, if a product without a barcode has been manually input (Yes in S24), if there is a high probability that the image of the product without a barcode is included in the image from camera 3 (Yes in S25), and if there is a high probability that the object detected by the camera image matches the manually input product (No in S27), the information processing device 1 proceeds to S21.

[0107] On the other hand, if the probability calculated in S26 is smaller than the third threshold (Yes in S27), the information processing device 1 detects fraudulent activity (S28). That is, the information processing device 1 detects fraudulent activity when, despite the manual input of a product without a barcode (Yes in S24), it is unlikely that the image from camera 3 contains an image of the product without a barcode (No in S25), or when it is unlikely that the object detected by object detection using the camera image matches the manually input product (Yes in S27).

[0108] If the information processing device 1 determines in S21 that the customer has scanned all of the products placed on the product stand A1a with the scanner 2 (Yes in S21), it executes the checkout process (S29). For example, the information processing device 1 displays the total price of the products purchased by the customer on the display 5 and executes the payment process.

[0109] (Summary of the Third Embodiment) As described above, the information processing device 1 refers to barcode presence / absence information that associates a product with information indicating the presence or absence of a barcode, and assigns information indicating whether the product has a product code or does not have a product code to a product detected based on an image from the camera 3. When product information about a product is manually entered into an input device, the information processing device 1 determines whether there has been any fraudulent activity in the manual input based on the information assigned to the product indicating whether the product has a product code or does not have a product code. Furthermore, the information processing device 1 determines whether there has been any fraudulent activity in the manual input based on the probability that an object detected by an image from the camera 3 matches the product whose product information was manually entered into the input device. This operation allows the information processing device 1 to appropriately detect any fraudulent activity involving the manual input of product information.

[0110] <Fourth embodiment> In the fourth embodiment, in addition to detecting fraudulent acts when manually entering products without barcodes as described in the above embodiments, fraudulent acts when scanning products with barcodes are detected. In the fourth embodiment, the "barcode presence" in the barcode presence / absence information (see FIG. 3) is further classified by the number of barcode types.

[0111] The system configuration in the fourth embodiment is the same as that in Fig. 1, and therefore a description thereof will be omitted. The block configuration in the fourth embodiment is also the same as that in Fig. 5, but the function of the control unit 11 is different. For example, the control unit 11 detects whether or not there has been any fraudulent manual input based on the number of different product codes possessed by the product scanned by the scanner and the number of different product codes possessed by the product detected in the image from the camera 3.

[0112] (Barcode Presence Information) Fig. 10 is a diagram showing an example of barcode presence information. In Fig. 10, differences from the barcode presence information explained in Fig. 3 will be explained.

[0113] Some products have multiple types of visible barcodes. For example, a six-can pack of beer has a barcode on the packaging that holds the six cans together, indicating that it is a six-can pack of beer. In addition, a six-can pack of beer has portions of the cans exposed from the packaging, exposing the barcodes on each can. In other words, a six-can pack of canned beer has two types of visible barcodes.

[0114] The barcode presence / absence information stored in the storage unit 12 of the information processing device 1 includes information on the presence or absence of a barcode for each product in the major and minor categories, and further includes information indicating whether the product has one or more types of barcodes. For example, in Figure 10, "Beer C" is associated with information "Barcode Present: Multiple," which indicates that the product has a barcode and that there are multiple types of barcodes (e.g., two types). In this case, one type of barcode is assigned to one bottle of Beer C, and one type of barcode is assigned to a six-can pack, meaning that a total of two types of barcodes are assigned to Beer C.

[0115] Fraudulent activity may occur with barcode-bearing products that have multiple types of barcodes. For example, in the case of a six-can pack of beer, the barcode on the packaging, which indicates that the product is a six-can pack of canned beer, may not be scanned, but the barcode on the can itself may be scanned. In this case, the payment is not based on the price of the six-can pack, but on the price of a single can of beer. The information processing device 1 detects such fraudulent activity.

[0116] (Candidate Product List) Figure 11 is a diagram showing an example of a candidate product list. The information processing device 1 generates a candidate product list in which each candidate product in the major and minor categories detected by the image recognition process is assigned information indicating whether the product has a barcode or not, and information indicating the number of barcode types. The information processing device 1 refers to the barcode presence / absence information stored in the memory unit 12 (see Figure 10), and assigns each candidate product detected by the image recognition process information indicating whether the product has a barcode or not, and information indicating the number of barcode types.

[0117] For example, "Beer C" in the candidate product list shown in FIG. 11 is a "Barcode present: Multiple" product as shown in FIG. 10. Therefore, the information processing device 1 assigns information "Barcode present: Multiple" to the candidate product "Beer C" detected by image recognition, indicating that it is a barcoded product and that it has multiple types of barcodes. For example, there are two types of barcodes in total: one barcode for each bottle of Beer C and one barcode for each six-can pack of Beer C. The information processing device 1 also assigns information to other candidate products detected by image recognition processing, and generates a candidate product list such as that shown in FIG. 11.

[0118] (Operation Flow) FIG. 12 is a flowchart showing an example of the operation of the information processing device 1 according to the fourth embodiment in determining whether or not there is any fraudulent activity.

[0119] The information processing device 1 determines whether the customer has scanned all of the products placed on the product stand A1a with the scanner 2 (S31). The information processing device 1 may determine whether all of the products placed on the product stand A1a have been scanned with the scanner 2, for example, based on a signal from the weight sensor 4a.

[0120] If the information processing device 1 determines that the customer has not scanned all of the products placed on the product stand A1a with the scanner 2 (No in S31), it performs product detection (candidate product detection) based on the image from the camera 3 (S32). For example, the information processing device 1 detects candidate products included in the image from the camera 3 by a known image recognition process using artificial intelligence, and calculates recognition scores for the detected candidate products.

[0121] The information processing device 1 refers to the barcode presence / absence information (see FIG. 10) stored in the storage unit 12, and assigns information indicating whether the candidate product detected in step S32 is a barcode-containing product or a barcode-free product, and information indicating the number of barcode types, to generate a candidate product list (S33). For example, the information processing device 1 generates the candidate product list shown in FIG. 11.

[0122] The information processing device 1 determines whether a product with a barcode has been scanned by the scanner 2 or whether a product without a barcode has been manually input into the input device (S34).

[0123] If the information processing device 1 determines that a product without a barcode has been manually input into the input device (manual input in S34), the information processing device 1 performs fraud detection processing on the manual input (S35). For example, the information processing device 1 performs the fraud detection processing on the manual input described in the first to third embodiments. After the fraud detection processing, the information processing device 1 proceeds to S31.

[0124] On the other hand, if the information processing device 1 determines that a barcoded product has been scanned by the scanner 2 (scanning in S34), it references the barcode presence / absence information (see FIG. 10 ) based on the scan information from the scanner 2 and obtains the number of barcode types (one or multiple) for the product in the major category. If the number of barcode types obtained is multiple, the information processing device 1 determines whether the barcode of the candidate product in the final detectable minor category in the candidate product list generated in S33 has been scanned by the scanner 2. In other words, the information processing device 1 determines whether the appropriate barcode has been scanned by the scanner 2 for a product with multiple types of barcodes (S36).

[0125] 10 based on the barcode scanned by the scanner 2, and acquires that "Beer C" has multiple types of barcodes (a barcode for one bottle of Beer C and a barcode for a six-can pack). When the information processing device 1 acquires that a product has multiple types of barcodes, it determines whether the barcode of the candidate product in the final detectable sub-item in the candidate product list generated in S33 has been scanned by the scanner 2.

[0126] If the barcode of the product scanned by the scanner 2 matches the barcode of the candidate product in the final detectable subitem in the candidate product list generated in S33 (Yes in S36), the information processing device 1 proceeds to S31. That is, if an appropriate barcode is scanned by the scanner 2 from among multiple types of barcodes of the product acquired by product detection using camera images in S32, the information processing device 1 proceeds to S31. For example, if a six-can pack of beer (having two types of barcodes) is detected by the image recognition process and one of the two types of barcodes, the barcode attached to the exterior of the six-can pack, is scanned, the information processing device 1 proceeds to S31.

[0127] On the other hand, if the barcode of the product scanned by the scanner 2 does not match the barcode of the candidate product in the final detectable subitem in the candidate product list generated in S33 (No in S36), the information processing device 1 detects fraudulent activity (S37). That is, if the scanner 2 does not scan the appropriate barcode among the multiple types of barcodes of the product acquired by product detection using camera images in S32, the information processing device 1 detects fraudulent activity. For example, if a six-can pack of beer (having two types of barcodes) is detected by image recognition processing, but one of the two types of barcodes attached to the beer inside the six-can pack is scanned, the information processing device 1 detects fraudulent activity. After detecting fraudulent activity, the information processing device 1 transitions to S31.

[0128] If the information processing device 1 determines in S31 that the customer has scanned all of the products placed on the product stand A1a with the scanner 2 (Yes in S31), it executes a checkout process (S38). For example, the information processing device 1 displays the total price of the products purchased by the customer on the display 5 and executes a payment process.

[0129] In this embodiment, two types of canned beer are shown, one can and a six-can pack, but the present invention is not limited to this and can be applied to, for example, a 12-can pack or a 24-can pack. It goes without saying that the present invention can also be applied to cases where the number of canned beer sold is three or more. For example, the present invention can also be applied to cases where there are three types of canned beer sold, one can, a six-can pack, and a 12-can pack. In this case, the number of barcodes is three.

[0130] Furthermore, in this embodiment, canned beer is shown as an example of a product to which multiple types of barcodes are assigned, but it goes without saying that the present disclosure is not limited to this and can be applied to any product to which multiple types of barcodes are assigned.

[0131] (Summary of the Fourth Embodiment) As described above, the information processing device 1 determines whether or not there has been any fraudulent activity in the manual input based on the number of types of product codes possessed by the products scanned by the scanner 2 and the number of types of product codes possessed by the products detected in the images captured by the camera 3. This operation enables the information processing device 1 to appropriately detect any fraudulent activity involving the manual input of product information.

[0132] Although the embodiments have been described above with reference to the drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims. It is understood that such modifications or alterations also fall within the technical scope of the present disclosure. Furthermore, the components in the embodiments may be combined in any manner without departing from the spirit of the present disclosure. The embodiments may be combined in any manner. The modified examples may be combined in any manner.

[0133] In the above-described embodiment, information indicating whether a candidate product has a barcode or does not have a barcode is assigned to the candidate product detected from the image captured by the camera 3. However, the purpose of assigning this information is to associate the product detected from the image captured by the camera 3 with the determination result of whether the product has a barcode or does not have a barcode. Therefore, this association may be performed in a manner other than by assigning information to the candidate product.

[0134] In the above-described embodiments, the notation "... part" used for each component may be replaced with other notations such as "... circuit," "... assembly," "... device," "... unit," or "... module."

[0135] The present disclosure can be realized by software, hardware, or software in conjunction with hardware. Each functional block used in the description of the above embodiments may be partially or entirely realized as an LSI, which is an integrated circuit, and each process described in the above embodiments may be partially or entirely controlled by a single LSI or a combination of LSIs. The LSI may be composed of individual chips, or may be composed of a single chip that includes some or all of the functional blocks. The LSI may have data input and output. Depending on the degree of integration, the LSI may also be called an IC, system LSI, super LSI, or ultra LSI.

[0136] The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit, a general-purpose processor, or a dedicated processor. Also, a field programmable gate array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI, may be used. The present disclosure may be realized as digital processing or analog processing.

[0137] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, it is natural that such technology may be used to integrate functional blocks. The application of biotechnology, etc. is also a possibility.

[0138] The disclosures of the specifications, drawings and abstracts contained in Japanese patent application No. 2023-198193 filed on November 22, 2023 and Japanese patent application No. 2024-030386 filed on February 29, 2024 are incorporated herein by reference in their entirety.

[0139] The present disclosure is useful for detecting manual input fraud at self-checkout registers.

[0140] REFERENCE SIGNS LIST 1 Information processing device 2, 2a, 2b Scanner 3 Camera 4a, 4b Weight sensor 5 Display A1a, A1b Product stand

Claims

1. An information processing device having: a determination unit that references product code presence / absence information that associates products with information indicating the presence or absence of a product code, and determines whether a product detected based on a camera image is a product with a product code or a product without a product code; and a determination unit that, when product information of a product is manually entered into an input device, determines whether or not fraudulent activity has occurred using the manual input, based on the result of the determination as to whether the product is a product with a product code or a product without a product code.

2. The information processing device of claim 1, wherein the products detected based on the camera images have a score indicating the degree of accuracy of the detection result, and the judgment unit judges whether or not fraudulent activity has occurred using the manual input for products having a score above a predetermined threshold based on the result of the judgment as to whether the product has a product code or does not have a product code.

3. The information processing device of claim 1, wherein the products detected based on the camera images have a score indicating the degree of accuracy of the detection result, and the determination unit determines whether or not fraudulent activity using the manual input has occurred based on the proportion of products with product codes having a score above a predetermined threshold among products having a score above the predetermined threshold, or the proportion of products without product codes having a score above the predetermined threshold among products having a score above the predetermined threshold.

4. The information processing device of claim 1, wherein the determination unit further determines whether or not fraudulent activity has occurred using the manual input based on the probability that an object detected by the camera image matches a product whose product information has been manually entered into the input device.

5. The information processing device of claim 1, wherein the determination unit further determines whether or not fraudulent activity using manual input has occurred based on the number of different product codes held by the product scanned by the scanner and the number of different product codes held by the product detected by the camera image.

6. A fraudulent activity determination method which refers to product code presence / absence information that associates a product with information indicating the presence or absence of a product code, and determines whether a product detected based on a camera image has a product code or does not have a product code, and when product information for a product has been manually entered into an input device, determines whether or not fraudulent activity has occurred using the manual input based on the result of determining whether the product has a product code or does not have a product code.

Images