Information processing program, information processing device, and information processing system

Abstract

Determine the product categories best suited for fraud detection. [Solution] The information processing program causes a computer to execute the following process: Based on a plurality of appearance images stored in a storage area that stores the appearance images and prices of each of the plurality of products 110, 120 to be judged in association with each other, the plurality of products are divided into one or more categories, based on the plurality of prices stored in the storage area, a statistical amount of the prices of the plurality of products within each of the one or more divided categories is calculated, the division pattern is changed and the division process and calculation process are executed so that the calculated one or more statistical amounts satisfy predetermined conditions, and category information indicating the category of each of the plurality of products when the predetermined conditions are satisfied.

Description

[Technical Field]

[0001] This invention relates to an information processing program, an information processing device, and an information processing system. [Background technology]

[0002] Fraudulent activity, such as the theft of goods, is one of the major causes of losses at self-checkout counters. To detect fraudulent activity, methods are sometimes used that utilize AI (Artificial Intelligence) technology, which employs image recognition models that perform image recognition processing using captured images of products, to recognize the products.

[0003] For example, a known method involves outputting an alert to a POS (Point of Sales) system based on a comparison between the number and category of products recognized by AI technology and the number and category of products registered in the POS system through product scanning. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-039930 [Patent Document 2] Japanese Patent Publication No. 2020-077275 [Patent Document 3] U.S. Patent Application Publication No. 2023 / 0368625 [Patent Document 4] U.S. Patent Application Publication No. 2020 / 0151692 [Patent Document 5] Japanese Patent Publication No. 2021-197106 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] In image recognition processing, it may be difficult to accurately distinguish between different products with similar appearances, such as wines with different brands, apples with different varieties, etc. Therefore, when classifying and comparing products into categories using the above-described method, there is a possibility that different products with similar appearances within the same category may not be distinguished. If the detection of fraud fails due to this, the loss amount in self-checkout can increase.

[0006] The fraud to be detected for which the above-mentioned inconveniences may occur is not limited to the fraud of stealing products. Fraud may include, for example, in addition to the act of stealing products, the failure to register products in the POS system due to mistakes, forgetfulness, misunderstandings, etc. of customers (purchasers).

[0007] In one aspect, an object of the present invention is to determine a product category suitable for fraud detection.

Means for Solving the Problems

[0008] In one embodiment, an information processing program may cause a computer to execute the following processing. The processing may divide a plurality of products into one or more categories based on a plurality of the appearance images stored in a storage area that stores the appearance images and prices of each of the plurality of products to be determined in association with each other. Further, the processing may calculate a statistic of the prices of the plurality of products within each of the divided one or more categories based on the plurality of the prices stored in the storage area. Furthermore, the processing may execute the processing of dividing and the processing of calculating by changing the division pattern so that one or more of the calculated statistics satisfy a predetermined condition. Also, the processing may output category information indicating the category of each of the plurality of products when the predetermined condition is satisfied.

Effects of the Invention

[0009] In one aspect, it is possible to determine a product category suitable for fraud detection.

Brief Description of the Drawings

[0010] [Figure 1] This is a diagram illustrating an example of a label switch in a self-checkout system. [Figure 2] This is a diagram illustrating an example of the banana trick at a self-checkout counter. [Figure 3] This is a block diagram showing the functional configuration of a fraud detection device related to a comparative example. [Figure 4] This figure shows examples of categories. [Figure 5] This figure shows an example of a category definition based on appearance (shape). [Figure 6] This figure shows an example of the system configuration according to one embodiment. [Figure 7] This figure shows an example of the system configuration according to one embodiment. [Figure 8] This is a block diagram showing an example of a computer hardware configuration according to one embodiment. [Figure 9] Block diagram showing an example of the functional configuration of a system according to one embodiment. [Figure 10] This figure illustrates an example of clustering in an image feature space. [Figure 11] This figure shows an example of the relationship between the number of categories, the amount of undetected losses, and the false alarm rate. [Figure 12] This figure shows an example of product information. [Figure 13] This is a flowchart illustrating an example of how the category generation process works. [Figure 14] This is a flowchart illustrating the operation of the first example of the evaluation metric acquisition process. [Figure 15] This is a flowchart illustrating the operation of the second example of the evaluation metric acquisition process. [Figure 16] This is a flowchart illustrating an example of how machine learning processing works. [Figure 17] This is a flowchart illustrating an example of how fraud detection processing works. [Figure 18] This is a diagram illustrating the first example of fraud detection processing. [Figure 19] This figure shows an example of an alert output in the first example of fraud detection processing. [Figure 20] This is a diagram illustrating a second example of fraud detection processing. [Figure 21] This figure shows an example of an alert output in the second example of fraud detection processing. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will now be described with reference to the drawings. However, the embodiments described below are merely illustrative and are not intended to exclude various modifications or applications of techniques not explicitly stated below. For example, these embodiments can be implemented with various modifications without departing from their spirit. In the drawings used in the following description, parts denoted by the same reference numerals represent the same or similar parts unless otherwise specified.

[0012] [A] Comparative example [A-1] Regarding fraudulent activities at self-checkout counters Examples of fraudulent activities committed by customers to steal goods at self-checkout counters installed in retail stores include label switching and the banana trick.

[0013] Label switching is the act of scanning (checking out) the label of a relatively inexpensive product instead of a relatively expensive product. A label is a label used to identify a product by machine reading, and may be, for example, a barcode label with a barcode printed on it. The banana trick is the act of selecting a relatively inexpensive product on the self-checkout screen instead of a relatively expensive product, such as produce, for products without labels, and underreporting the quantity, or both.

[0014] Figure 1 illustrates an example of a label switch in a self-checkout register 100. The self-checkout register 100 is an example of a POS (Point of Sale) device. Figure 1 shows how the label of a confectionery item 120 (labeled "snack B") is displayed in the label scanning area 101 of the self-checkout register 100, obscuring the label of a fine wine item 110 (labeled "fine wine A"). In this case, the self-checkout register 100 registers the price of the relatively inexpensive confectionery item 120 instead of the relatively expensive fine wine item 110, so the difference between the prices of the fine wine item 110 and the confectionery item 120 represents the retailer's loss.

[0015] Figure 2 illustrates an example of the banana trick at a self-checkout register 100. Figure 2 shows that bananas 140 are selected on the self-checkout register 100's screen (POS screen) 102 instead of Shine Muscat grapes 130. In this case, the self-checkout register 100 registers the price of the relatively inexpensive bananas 140 instead of the relatively expensive Shine Muscat grapes 130, so the difference in price between Shine Muscat grapes 130 and bananas 140 becomes the retailer's loss.

[0016] [A-2] Explanation of the comparative example To detect the aforementioned fraudulent activities involving product counterfeiting, it is crucial to accurately recognize products from images of their appearance. However, accurately distinguishing between similar-looking but distinct products can be difficult, and there are limitations to recognizing individual products based on images in self-checkout systems like Self-Checkout 100.

[0017] Therefore, as a comparative example, let's consider estimating the product category instead of identifying the product based on an image.

[0018] Figure 3 is a block diagram showing the functional configuration of the fraud detection device 200 according to the comparative example. The fraud detection device 200 according to the comparative example determines whether or not fraudulent activity has occurred by comparing images of products captured by a camera (camera images) with POS information of products registered in the self-checkout register 100.

[0019] The camera may be, for example, various imaging devices (not shown in Figures 1 and 2) installed in the store with a field of view that overlooks the self-checkout 100. The POS information may include, for example, information that uniquely identifies the product registered in the self-checkout 100 by scanning a label or selecting it on the screen 102 (e.g., product name).

[0020] The fraud detection device 200 includes a category recognition unit 210 and a determination unit 220. The category recognition unit 210 recognizes the category of a product based on a camera image, and may be, for example, an image recognition model (machine learning model) trained to classify product appearance images into predetermined categories. The determination unit 220 converts POS information into a category based on pre-associated information between products and categories, and outputs an alert to the self-checkout register 100 if the converted category does not match the category recognized by the category recognition unit 210.

[0021] This allows for the detection of fraudulent activity and the output of an alert, even if multiple products have similar appearances, as long as they belong to different categories.

[0022] In the comparative example method, the user pre-defines a category for each product. The category recognition unit 210 is trained to output a pre-defined category for a given product image when an image of the product's appearance is input.

[0023] However, in the methods used for comparative examples, it can sometimes be difficult to subjectively define categories.

[0024] Figure 4 shows an example of a category, and Figure 5 shows an example of a category definition based on appearance (shape).

[0025] As illustrated in Figure 4, if a category is created that includes many products such as daily necessities, the products within that category will not be distinguished, potentially leading to significant losses due to fraud. Thus, it can be difficult to determine the appropriate level of granularity for categories.

[0026] Furthermore, for example, if we try to define categories based on appearance (shape) to group similar-looking products, there may be many products that are difficult to categorize. As shown in Figure 5, if we define categories such as Box (box, case) and Bottle (bottle) as examples of shapes and try to associate products with these categories, there may be many products whose shapes are difficult to describe, making it difficult for users to associate products with categories.

[0027] Therefore, in one embodiment, a method for determining product categories suitable for fraud detection is described. In another embodiment, a method for generating appropriate training data for an image recognition model that performs fraud detection is described. Furthermore, in yet another embodiment, a method for performing fraud detection using an image recognition model trained with said training data is described.

[0028] [B] Description of one embodiment Figures 6 and 7 show an example configuration of System 1 according to one embodiment. System 1 is an example of an information processing system or a fraud detection system, and may be applied to a system including a self-checkout counter installed in a store such as a retail store.

[0029] As shown in Figure 6, System 1 may, in example, include one or more cameras 2, one or more fraud detection devices 3, one or more servers 4, and one or more POS devices 5.

[0030] POS device 5 is an example of an information processing device or computer, and is an automated checkout device, such as a self-checkout register, that allows customers to register products. Product registration may include scanning product labels, selecting products on the screen of POS device 5, etc. POS device 5 may output POS information, including information that uniquely identifies the products registered by the customer (for example, the product name), as event information to the fraud detection device 3. In addition to product registration, POS device 5 may also be equipped with various functions for processing customer payments (payment, checkout).

[0031] In addition to the product name, various identifiers such as product codes and POS system codes can be used to uniquely identify a product. For convenience, in the following explanation, the information that uniquely identifies a product will be referred to as the "product name."

[0032] POS device 5 may be included in a POS system that manages POS information between multiple POS devices 5, between multiple stores, etc., and may be connected to other devices within the POS system via a network (not shown) so as to be able to communicate with each other. In this case, POS device 5 may output various information such as POS information and payment information to the POS system.

[0033] Camera 2 may be any type of imaging device installed in the store with a field of view that overlooks the POS device 5 (for example, the field of view shown in Figure 1). The video captured by Camera 2 (camera video) may include multiple images (multiple frames) in which the POS device 5 and the appearance of products, such as the products that a customer intends to register with the POS device 5, are included in the imaging range. The captured images may also include the customer. The camera video captured by Camera 2 is output to the fraud detection device 3.

[0034] Camera 2 may be permanently installed within the store so that its imaging range, such as the field of view and resolution, remains constant across multiple frames of the camera footage. Alternatively, Camera 2 may be installed within the store so that its imaging range changes over time, provided that its field of view and resolution are such that it can capture a certain image area that includes at least the POS device 5 and the appearance of the products across multiple frames.

[0035] Alternatively, one camera 2 may be installed to photograph multiple POS devices 5. In this case, the camera 2 only needs to have a field of view (e.g., wide-angle) and resolution that allows it to capture multiple image areas across multiple frames, each containing at least the appearance of the POS device 5 and the products.

[0036] The fraud detection device 3 is an example of an information processing device or computer, and performs fraud detection processing to detect fraudulent activity based on camera images acquired from camera 2 and event information acquired from POS device 5. The fraud detection processing may be implemented using, for example, an image recognition model (machine learning model) trained by a method such as DL (Deep Learning). The fraud detection device 3 may be, for example, a computer such as an edge terminal installed near or inside POS device 5, or a computer such as a server installed in the back room of a store or in a remote location via a network. Examples of remote locations include other stores, the headquarters or branch offices of retailers, and data centers.

[0037] In one embodiment, the fraudulent activity to be detected by the fraud detection device 3 will be explained by focusing on the fraudulent activity illustrated in Figures 1 and 2. However, the fraudulent activity to be detected is not limited to fraudulent activity involving the fraudulent acquisition of goods. For example, the method according to one embodiment can also detect failures in registering goods in the POS system (e.g., POS device 5) due to customer error, forgetfulness, misunderstanding, or other negligence.

[0038] Server 4 is an example of an information processing device or computer, and generates category information used for training (machine learning processing) an image recognition model that implements fraud detection processing by fraud detection device 3. The category information is information that indicates the category (optimal category) of each of the multiple products to be processed (to be judged), and is an example of training data used to train the image recognition model. The training data may associate an image of the appearance of the product (appearance image) with the product category indicated by the category information. The multiple products to be processed may be, for example, multiple (e.g., all) products sold at the store where the POS device 5 is installed.

[0039] The image recognition model may be trained in the fraud detection device 3, which receives training data from server 4, or it may be trained in server 4. If training is performed in server 4, server 4 may output (provide) the trained image recognition model to the fraud detection device 3. The trained image recognition model may also be retrained in server 4 or the fraud detection device 3 in response to changes in the installation environment of either or both of the camera 2 and the POS device 5, changes in the products being processed, etc.

[0040] As described above, the machine learning processing (training and retraining) of the image recognition model may be performed on either or both of the server 4 and the fraud detection device 3. The inference processing using the image recognition model may be performed on the fraud detection device 3 as at least part of the fraud detection processing.

[0041] Furthermore, the functions of server 4 and fraud detection device 3 may be integrated and implemented in a single server 4, as illustrated in Figure 7. In this case, server 4 may have the function of fraud detection device 3 in addition to the function of generating category information. The following description assumes that system 1 has the configuration shown in Figure 6.

[0042] [B-1] Hardware Configuration Example The fraud detection device 3, server 4, and POS device 5 of System 1 may each have similar hardware configurations.

[0043] In one embodiment, the fraud detection device 3 may be a virtual server (VM) or a physical server. Furthermore, the functions of the fraud detection device 3 may be implemented by one computer or by two or more computers. Moreover, at least a portion of the functions of the fraud detection device 3 may be implemented using hardware (HW) resources and network (NW) resources provided by a cloud environment.

[0044] Furthermore, the server 4 in one embodiment may be a virtual server (VM) or a physical server. Also, the functions of server 4 may be implemented by one computer or by two or more computers. Moreover, at least a portion of the functions of server 4 may be implemented using hardware and network resources provided by a cloud environment.

[0045] The following describes hardware configuration examples for multiple computers that implement the functions of the fraud detection device 3, server 4, and POS device 5, using computer 10 shown in Figure 8 as an example to represent these computers.

[0046] Figure 8 is a block diagram showing an example of the hardware configuration of a computer 10 according to one embodiment. When multiple computers are used as HW resources to realize the functions of the fraud detection device 3, server 4, or POS device 5, each computer may have the HW configuration illustrated in Figure 8.

[0047] As shown in Figure 8, the computer 10 may, as an example of its hardware configuration, include a processor 10a, a graphics processing unit 10b, a memory 10c, a storage unit 10d, an IF (Interface) unit 10e, an IO (Input / Output) unit 10f, and a read unit 10g.

[0048] Processor 10a is an example of an arithmetic processing unit that performs various control and calculations. Processor 10a may be connected to each block in the computer 10 via bus 10j so as to be able to communicate with each other. Processor 10a may be a multiprocessor containing multiple processors, a multicore processor having multiple processor cores, or a configuration having multiple multicore processors.

[0049] Examples of processor 10a include integrated circuits (ICs) such as CPUs, MPUs, APUs, DSPs, ASICs, and FPGAs. Note that two or more combinations of these integrated circuits may be used as processor 10a. CPU stands for Central Processing Unit, MPU for Micro Processing Unit, APU for Accelerated Processing Unit, DSP for Digital Signal Processor, ASIC for Application Specific IC, and FPGA for Field-Programmable Gate Array.

[0050] The graphics processing unit 10b controls the screen display to output devices such as monitors, which are part of the I / O unit 10f. The graphics processing unit 10b may also be configured as an accelerator that performs machine learning processing and inference processing using machine learning models. Examples of graphics processing units 10b include various arithmetic processing units, such as integrated circuits (ICs) like GPUs (Graphics Processing Units), APUs, DSPs, ASICs, or FPGAs.

[0051] Each of the memory 10c and storage unit 10d stores various data, programs, and other information. Examples of memory 10c include volatile memory such as DRAM (Dynamic Random Access Memory) and non-volatile memory such as PM (Persistent Memory), or both. Examples of storage unit 10d include magnetic disk devices such as HDD (Hard Disk Drive), semiconductor drive devices such as SSD (Solid State Drive), and various storage devices such as non-volatile memory. Examples of non-volatile memory include flash memory, SCM (Storage Class Memory), and ROM (Read Only Memory).

[0052] The memory unit 10d may store a program 10h (information processing program) that implements all or part of the various functions of the computer 10. For example, the processor 10a of the fraud detection device 3 can implement the functions of the control unit 36 ​​(see Figure 9), described later, by loading the program 10h (e.g., machine learning program, fraud detection program, etc.) stored in the memory unit 10d into memory 10c and executing it. Also, for example, the processor 10a of the server 4 can implement the functions of the control unit 46 (see Figure 9), described later, by loading the program 10h (e.g., category generation program, training data generation program, machine learning program, etc.) stored in the memory unit 10d into memory 10c and executing it. Furthermore, for example, the processor 10a of the POS device 5 can implement the functions of the POS device 5, described later, by loading the program 10h (e.g., POS program, etc.) stored in the memory unit 10d into memory 10c and executing it.

[0053] The IF unit 10e is an example of a communication interface that controls connections and communications between the computer 10 and the camera 2, between computers 10, and between the computer 10 and other computers. For example, the IF unit 10e may include an adapter compliant with telecommunications such as Ethernet (registered trademark) (e.g., LAN (Local Area Network)) or optical communication such as FC (Fibre Channel). The adapter may support wireless, wired, or both communication methods. The program 10h may be downloaded from the network to the computer 10 via the communication interface and stored in the storage unit 10d.

[0054] The I / O unit 10f may include one or both of an input device and an output device. Examples of input devices include a keyboard, mouse, and touch panel. Examples of output devices include a monitor, projector, and printer. The output device may be connected to the graphics processing unit 10b. The I / O unit 10f may also include a touch panel that integrates the input and output devices. For example, the I / O unit 10f of the POS device 5 may be equipped with a touch panel as a customer user interface.

[0055] The reading unit 10g is an example of a reader that reads data and program information recorded on the recording medium 10i. The reading unit 10g may include a connection terminal or device to which the recording medium 10i can be connected or inserted. Examples of the reading unit 10g include an adapter compliant with USB (Universal Serial Bus), a drive device for accessing a recording disk, and a card reader for accessing flash memory such as an SD card. The recording medium 10i may store a program 10h, and the reading unit 10g may read the program 10h from the recording medium 10i and store it in the storage unit 10d.

[0056] Examples of recording media 10i include non-temporary computer-readable recording media such as magnetic / optical discs and flash memory. Examples of magnetic / optical discs include flexible discs, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray discs, and HVDs (Holographic Versatile Discs). Examples of flash memory include semiconductor memory such as USB memory and SD cards.

[0057] The hardware configuration of computer 10 described above is illustrative. Therefore, the addition or deletion of hardware within computer 10 (for example, adding or deleting arbitrary blocks), division, integration in any combination, or addition or deletion of buses may be performed as appropriate.

[0058] For example, the POS device 5 may have hardware configurations specific to self-checkout systems in addition to the hardware configuration shown in Figure 8. Examples of self-checkout specific hardware configurations include a scanner device for reading labels (e.g., barcode labels) and various payment devices for processing payments (settlement) using cash, credit cards, electronic money, etc. The scanner device may include either or both a scanner formed integrally with the POS device 5, or a handheld scanner that can be held by the customer. The scanner device may also include a wireless communication device that reads (recognizes) information recorded on IC tags (labels) such as RFID (Radio Frequency Identification) tags attached to products, either in place of or in addition to barcode labels.

[0059] [B-2] Example of Functional Configuration Figure 9 is a block diagram showing an example of the functional configuration of System 1 according to one embodiment. For convenience, in the following description, each functional configuration of System 1 will be assumed to be provided in only one of the fraud detection device 3 and the server 4, but it is not limited to this. Each functional configuration may be provided redundantly in both the fraud detection device 3 and the server 4, or it may be provided in a distributed or divided manner.

[0060] The fraud detection device 3 may, as an example of its functional configuration, include a training unit 32, an inference unit 33, a determination unit 34, and an output unit 35. The fraud detection device 3 may also include a storage area capable of storing at least one type of information, such as a category recognition model 31a, verification information 31b, and image information 31c. The inference unit 33 and the category recognition model 31a are examples of the category recognition unit 30. The training unit 32, the inference unit 33 (category recognition unit 30), the determination unit 34, and the output unit 35 are examples of the control unit 36.

[0061] Server 4 may, as an example of its functional configuration, include a division unit 42, an undetected loss amount calculation unit 43, a false alarm rate calculation unit 44, and a determination unit 45. Server 4 may also include a storage area capable of storing at least one type of information, namely product information 41a and corresponding information 41b. The division unit 42, undetected loss amount calculation unit 43, false alarm rate calculation unit 44, and determination unit 45 are examples of the control unit 46.

[0062] The respective functions of the control unit 36 ​​of the fraud detection device 3 and the control unit 46 of the server 4 may be realized by the processor 10a shown in Figure 8 executing the program 10h on the memory 10c. Furthermore, the respective functions of the storage area of ​​the fraud detection device 3 and the storage area of ​​the server 4 may be realized by the storage areas of either or both of the memory 10c and the storage unit 10d shown in Figure 8.

[0063] When a product label is scanned by a scanner device such as a scanner or wireless communication device, the POS device 5 outputs the product name recognized by the scan to the fraud detection device 3 (determination unit 34) along with the scan result. The scan result is an example of POS information or event information.

[0064] Furthermore, when the POS device 5 receives an alert from the fraud detection device 3 (output unit 35) indicating that fraudulent activity has been detected in the scan results, it may perform alert processing. Alert processing may include, for example, displaying a screen corresponding to the content of the alert on the IO unit 10f (e.g., touch panel) of the POS device 5, or outputting a notification corresponding to the content of the alert to the retailer's (store's) employees, or both. In addition, during alert processing, the POS device 5 may notify the POS system of the content of the alert.

[0065] (Server 4 explanation) First, let's describe a server 4 according to one embodiment. Server 4 may output a category, which is an example of training data for the category recognition model 31a, by executing the following processes (i) to (iii).

[0066] (i) Server 4 divides multiple products into one or more categories based on multiple appearance images stored in a memory area that stores the appearance image and price of each of the multiple products to be processed in association with each other.

[0067] (ii) Server 4 calculates statistical data on the prices of multiple products within each of the one or more divided categories, based on the multiple prices stored in the memory area.

[0068] (iii) Server 4 modifies the division pattern so that one or more calculated statistics satisfy the predetermined conditions, and then executes the division process and the calculation process described above.

[0069] As described above, the server 4 according to one embodiment generates and outputs a category (optimal category) that satisfies predetermined conditions based on objective criteria. The predetermined conditions include at least one of the following conditions (a) and (b). Note that the processing in (i) to (iii) above is an example of processing when the predetermined condition is condition (a) below.

[0070] (a) The amount of undetected losses is small. The undetected loss amount is the amount of loss incurred when fraudulent activity is missed (not detected) during the fraud detection process by the category recognition model 31a, which is trained based on the categories generated by server 4. By including condition (a) in the predetermined conditions, the amount of loss when fraudulent activity is missed can be kept relatively small.

[0071] (b) Low false alarm rate. A false alarm occurs when the fraud detection process by the category recognition model 31a, trained based on categories generated by server 4, incorrectly identifies a non-fraudulent activity as fraudulent (false positive), resulting in the output of an alert. The false alarm rate represents the percentage of cases that are not truly fraudulent but result in a false alarm. Frequent false alarms, such as a high false alarm rate, have negative impacts on both retailers and customers. The false alarm rate is one of the important KPIs (Key Performance Indicators). The KPI can be improved by including condition (b) in the predetermined conditions.

[0072] The following explanation uses, but is not limited to, the case where the given conditions include both condition (a) and condition (b). The given conditions may include only one of condition (a) or condition (b) (for example, only condition (a) or only condition (b)).

[0073] Each of the one or more categories generated by Server 4 is a group of one or more products. Since these categories are generated to satisfy predetermined conditions, they do not necessarily need to be understandable (acceptable) to "people" such as administrators, employees, and customers of System 1, and may be called pseudo-categories or artificial categories, for example.

[0074] Server 4 may, for example, define a predetermined condition as an objective function and generate a category that optimizes the objective function. For example, Server 4 may calculate the value of the objective function using a dataset in which the product name, image, and price of each of a plurality of products to be processed are paired, and search for a category in which the calculated value becomes smaller.

[0075] As an example, Server 4 identifies one or more categories so as to optimize the objective functions of condition (a) and condition (b) respectively, with condition (a) and condition (b) as the objective functions. As a process for optimizing a plurality of objective functions, various methods such as mathematical optimization processing and multi-objective optimization processing that consider a plurality of objective functions may be used. Examples of multi-objective optimization processing include various methods such as heuristic processing and multi-objective optimization (evolutionary multi-objective optimization) processing using evolutionary computation.

[0076] Hereinafter, as a process for optimizing a plurality of objective functions, mathematical optimization processing will be described as an example.

[0077] (An example of mathematical optimization processing) Server 4, as exemplified by the following formulas (1) to (3), a set G of product names, a candidate (category set candidate) ω of a set of categories obtained by dividing elements of the set G of product names i , category set candidate ω i and a category c k i contained therein are each determined. G = {g1, g2, ···, g N} ···(1) ω i = {c1 i , c2 i , ···, c K i} ···(2) c k i = {g j} ···(3)

[0078] In equation (1) above, g represents a single product name, and N represents the total number of products to be processed. In equation (2) above, K represents the number of categories. K may be a predetermined constant, for example, or a variable that changes sequentially or stepwise during the mathematical optimization process. i is one or more variables that represent the category division pattern.

[0079] In equations (2) and (3) above, c k i This indicates the k-th category in the i-th (i-th) partition pattern. k is a variable used to identify the category, and is an integer between 1 and K. In equation (3) above, j is the category c k i g is one or more variables that represent the products that are associated with (classified) the product. j is, c k i These are the product names of one or more products that are associated (classified) with the specified category. Note that this includes all possible category set candidates ω. i The Omega G Therefore, |Ω G |=B N (The number of bells) is true.

[0080] As illustrated in equations (4) and (5) below, a certain candidate set of categories ω i In this case, the amount of undetected loss is L(ω i ) and the false alarm rate is E(ω i When given this, the optimal set of categories ω^ can be obtained, for example, by equation (6) below.

number

[0081] In the above equation (4), mc k This is category c k This is a statistical measure of the prices of the goods included in (belonging to) a category. For example, mc k This is category c k You may indicate the maximum price difference for all combinations of products included in the set. In this case, the undetected loss amount L(ωi ) is a candidate category set ω i All categories c1~c K The average of the price difference (statistic) calculated for each is shown.

[0082] Note that the amount of undetected loss L(ω i ) is not limited to formula (4) above. k Instead of the largest price difference, for example, category c k The statistics may also be the variance, range, standard deviation, etc., of the prices of the products included in (belonging to) category c. k This may include representative values ​​such as the maximum, minimum, median, and average prices of the products included.

[0083] In equation (5) above, Acc is the category set candidate ω of the category recognition model 31a. i This shows the recognition rate (Accuracy: correct recognition rate) when recognizing the category from the product image according to the following. False positive rate E(ω i This value is obtained by subtracting the recognition rate from 1, and is, for example, the probability of a false positive and a false negative.

[0084] The false alarm rate is E(ω i The false alarm rate E(ω i ) is a way of using 1-recognition rate instead of, for example, category recognition model 31a is a candidate for the set of categories ω i The probability of a false positive when recognizing a product category from an image is also acceptable.

[0085] In equation (6) above, the set of optimal categories ω^ is the amount of undetected loss L(ω) when the product is divided into categories using the i-th division pattern. i ) and false alarm rate E(ω i Candidate category sets ω that minimize ) i This shows that argmin is the amount of undetected loss L(ω i ) and false alarm rate E(ω iThis is a function for simultaneously optimizing each of the values ​​of ). In equation (6) above, the undetected loss amount L(ω i ) and false alarm rate E(ω i The transpose matrix, which is the vector representation of each of the elements, is used as an argument to argmin.

[0086] As described above, Server 4, for example, randomly divides N products into K categories (see equations (1) to (3) above) and calculates the undetected loss amount L and false alarm rate E at that time (see equations (4) and (5) above). Server 4 sequentially divides N products into K categories while changing the combination (divide) pattern of the correspondence relationship between products and categories, and identifies the division pattern that minimizes the undetected loss amount L and false alarm rate E using argmin (see equation (6) above). Then, Server 4 outputs the identified division pattern, in other words, the set of optimal categories ω^, as correspondence information 41b that shows the correspondence relationship between product names and categories.

[0087] (An example of multi-objective optimization processing) The method described above results in an enormous number of combinations when dividing N products into K categories, potentially increasing processing time. Therefore, as an example of multi-objective optimization, we will explain a method for finding the optimal (or near-optimal) solution using heuristic processing.

[0088] In this process, the fraud detection device 3 determines the category to which the captured image of the product belongs by image recognition of the captured image. Therefore, the server 4 may group products with similar appearances into the same category.

[0089] For example, Server 4 may cluster multiple products based on image features extracted from product images. Examples of image features include feature vectors (real-valued vectors) extracted from the intermediate layers of a convolutional neural network (NN) that receives an image as input. For instance, if the image size (resolution) is 512 x 512 pixels, the image features may be 256-dimensional real-valued vectors obtained from the intermediate layers of the convolutional NN that receives the image as input. Examples of convolutional NNs include feature extractors similar to those used in image recognition models. Note that image features are not limited to such feature vectors; various types of information capable of representing the similarity between images may be used.

[0090] Figure 10 illustrates an example of clustering in an image feature space. Each plot (point) in the graphs in the two-dimensional space indicated by labels A1 to A3 in Figure 10 represents an image feature of a single product. Server 4 may generate candidate category sets by merging image features that are close together in the two-dimensional space using hierarchical clustering, based on image feature clustering methods. Note that while Figure 10 shows image features in a two-dimensional space for convenience, in reality, close image features in a multi-dimensional space (e.g., multi-dimensional) may be merged.

[0091] Server 4 may, for example, calculate the undetected loss amount L and the false alarm rate E for a candidate category set ω, and repeatedly generate candidate category set ω until predetermined conditions are met.

[0092] Figure 11 shows an example of the relationship between the number of categories, the amount of undetected losses L, and the false alarm rate E. In Figure 11, the horizontal axis represents the number of categories (number of candidate categories), the left vertical axis represents the amount of undetected losses L (yen), and the right vertical axis represents the false alarm rate E (%). The dashed line graph indicated by the symbol B1 shows the change in the amount of undetected losses L according to the number of categories, and the solid line graph indicated by the symbol B2 shows the change in the false alarm rate E according to the number of categories.

[0093] As shown in Figure 11, as the number of categories increases, the undetected loss amount L decreases, but the false alarm rate E increases. When the number of categories is maximized (matching the number of products), the undetected loss amount L is minimized, but the false alarm rate E is maximized. On the other hand, as the number of categories decreases, the false alarm rate E decreases, but the undetected loss amount L increases. When the number of categories is minimized (all products belong to one category), the false alarm rate E is minimized, but the undetected loss amount L is maximized.

[0094] Thus, the undetected loss amount L and the false alarm rate E are in a trade-off relationship with respect to the number of categories. Therefore, in server 4, the predetermined conditions may include a predetermined threshold of the upper limit amount (yen) of the undetected loss amount L for condition (a), and a predetermined threshold of the upper limit percentage (%) of the false alarm rate E for condition (b). Server 4 may optimize the objective function so that the predetermined conditions, including these thresholds, are satisfied.

[0095] As an example, as shown by symbol A1 in Figure 10, Server 4 generates category candidates that include multiple image features that are close in distance in a two-dimensional space, for example, those whose distance is below a predetermined threshold, by performing hierarchical clustering on 14 image features (images). In symbol A1, 7 category candidates are generated. Server 4 calculates the undetected loss amount L and the false alarm rate E when classifying the 14 products into the 7 category candidates, and if at least one of them exceeds the threshold, it regenerates the category candidates, for example, by further merging the category candidates or image features.

[0096] As shown by symbol A2 in Figure 10, Server 4 generates category candidates that are close in distance in a two-dimensional space or that contain image features, based on seven category candidates. In symbol A2, four category candidates are generated. Server 4 calculates the undetected loss amount L and false alarm rate E when classifying 14 products into four category candidates, and if at least one of them exceeds a threshold, it regenerates the category candidates, for example, by further merging the category candidates or image features.

[0097] As shown by symbol A3 in Figure 10, Server 4 generates category candidates that are close in distance in a two-dimensional space or that contain image features, based on four category candidates. In symbol A3, three category candidates are generated. Server 4 calculates the undetected loss amount L and false alarm rate E when classifying 14 products into three category candidates, and outputs the category candidate as the optimal category if predetermined conditions are met, including both being below a threshold.

[0098] Figure 10 shows an example where the number of category candidates decreases as clustering progresses, but this is not the only method; methods that increase the number of category candidates (sequentially dividing the category candidates) may also be used.

[0099] Furthermore, in the mathematical optimization process described above, server 4 may apply the hierarchical clustering method described above in heuristic processing to the process of equations (1) to (3) above, which randomly divides N products into K categories.

[0100] (Explanation of an example of the functional configuration of Server 4) Next, with reference to Figure 9, an example of the functional configuration of Server 4 for realizing the above-described process will be explained.

[0101] The division unit 42 divides the multiple products to be processed into one or more categories based on the product information 41a stored in the memory area.

[0102] Figure 12 shows an example of product information 41a. Product information 41a is information that associates the appearance image and price of each of the multiple products to be processed. As shown in Figure 10, product information 41a may include a product name, which is an example of information that uniquely identifies a product, an appearance image of the product, and the price of the product.

[0103] The exterior image is an image representing the appearance of the product. For example, it may be an image captured by an imaging device such as camera 2, or it may be various images such as images or illustrations provided by the product manufacturer. Furthermore, the exterior image may be an image that has undergone various processing such as correction, transformation, or cropping. The same applies to the following explanation.

[0104] The price is the price of the product in question, and may, for example, be the selling price of the product at the store where the POS device 5 is installed, or it may be a price determined by various methods, such as the list price of the product, the average price across multiple stores, the highest or lowest price, etc.

[0105] The division unit 42 divides multiple products into one or more categories based on multiple appearance images contained in the product information 41a. For example, when multi-objective optimization processing is performed, the division unit 42 extracts image features from each of the multiple appearance images and divides (classifies) multiple products into one or more categories based on the similarity between the image features. The division results may be managed, for example, as correspondence information between product names and categories (category information).

[0106] The undetected loss amount calculation unit 43 calculates statistical data for the prices of multiple products within each of the one or more divided categories based on the multiple prices included in the product information 41a. The undetected loss amount calculation unit 43 then calculates the above-mentioned undetected loss amount L by calculating the average value of the statistical data for each category.

[0107] For example, the undetected loss amount calculation unit 43 may obtain the price of a product from the product information 41a for products associated with the same category in the division results by the division unit 42, and calculate a statistical amount based on the obtained price. The maximum price difference may be obtained, for example, by subtracting the minimum price from the maximum price among products in the same category.

[0108] The false alarm rate calculation unit 44 calculates the false alarm rate E of the category recognition model 31a when the category recognition model 31a has been trained based on one or more divided categories.

[0109] As an example, the false alarm rate calculation unit 44 may use the category recognition model 31a to perform a process for acquiring evaluation indicators for the false alarm rate each time a division result is output from the division unit 42. The false alarm rate calculation unit 44 may calculate the false alarm rate E described above based on evaluation indicators such as the recognition rate and false positive rate acquired through the evaluation indicator acquisition process.

[0110] The evaluation metric acquisition process may include machine learning processing (training) of the machine learning model and validation processing of the trained machine learning model using validation information 31b described later. Examples of the evaluation metric acquisition process include the first and second examples below.

[0111] (Example 1 of the evaluation metric acquisition process) In the machine learning process of the first example, the false alarm rate calculation unit 44 may train the category recognition model 31a to convert the product appearance image into a category according to the division result each time a division result is output from the division unit 42.

[0112] The training data may be generated based on product information 41a and the segmentation results (corresponding information). For example, the false alarm rate calculation unit 44 may use the appearance image included in the product information 41a as input data, and generate training data using the converted category obtained by converting the product name included in the product information 41a into a category based on the corresponding information as the correct label. Figure 9 shows an example in which the product information 41a and corresponding information included in the training data are provided from the server 4 to the fraud detection device 3 for convenience.

[0113] In the verification process of the first example, the false alarm rate calculation unit 44 may obtain from the fraud detection device 3 an evaluation index regarding the results of the fraud detection process when each camera image of multiple products included in the verification information 31b is input to the trained category recognition model 31a.

[0114] In the following explanation, in the first example of the evaluation index acquisition process, we assume that the fraud detection device 3 calculates the evaluation index in response to instructions from the false alarm rate calculation unit 44, and in the explanation of the example of the functional configuration of the fraud detection device 3, we will explain the details of the evaluation index calculation process. Note that the calculation of the evaluation index in the evaluation index acquisition process may be performed, for example, by the server 4 (false alarm rate calculation unit 44) using a category recognition model or image recognition model stored in the server 4.

[0115] (Second example of the evaluation metric acquisition process) In the machine learning process of the second example, the false alarm rate calculation unit 44 may train a product recognition model to output the product name of a product when an image of the appearance of each of the multiple products to be processed is input based on the product information 41a. The product recognition model may be a different image recognition model (machine learning model) from the category recognition model 31a and may be provided on the server 4. In the second example, training of the product recognition model for the multiple products to be processed may be performed only once at the first time the category generation process is performed, or beforehand.

[0116] In the verification process of the second example, the false alarm rate calculation unit 44 may acquire verification information 31b (see dashed arrow in Figure 9) and calculate an evaluation index for the results of the fraud detection process when each of the camera images of the multiple products included in the verification information 31b is input to the trained product recognition model.

[0117] For example, the false alarm rate calculation unit 44 inputs the camera images of each of the multiple products included in the verification information 31b into the product recognition model and converts the product names obtained into categories based on the division results.

[0118] The false alarm rate calculation unit 44 may then calculate the false alarm rate E by determining whether each of the categories obtained through conversion matches for each of the multiple products included in the verification information 31b.

[0119] The determination unit 45 determines whether the calculated undetected loss amount L and false alarm rate E satisfy predetermined conditions. If the predetermined conditions are met, it determines that the division result at that time is the optimal category information and stores the category information in the corresponding information 41b. The corresponding information 41b is information indicating the category of each of the multiple products when the predetermined conditions are met. The corresponding information 41b may be output from the server 4 to the fraud detection device 3 as training data together with the product information 41a.

[0120] The determination by the decision unit 45 as to whether the undetected loss amount L and false alarm rate E satisfy predetermined conditions may be based on the optimization process described above, for example, the decision logic in the multi-objective optimization process. As an example, the decision unit 45 may determine that the predetermined conditions are met when the undetected loss amount L and false alarm rate E of a certain division pattern are both below their respective thresholds, and when both the undetected loss amount L and false alarm rate E are smaller compared to other division patterns. Alternatively, the decision unit 45 may perform the determination by first calculating the undetected loss amount L and false alarm rate E for all division patterns and then comparing the undetected loss amount L and false alarm rate E of each division pattern.

[0121] Furthermore, if the predetermined conditions include only condition (a) of condition (a) and condition (b), the determination unit 45 may determine whether one or more statistical quantities calculated for each category (for example, the average value of these, which is the undetected loss amount L) satisfy the predetermined conditions. In this case, the function of the false alarm rate calculation unit 44 may be omitted in the server 4. Similarly, if the predetermined conditions include only condition (b) of condition (a) and condition (b), the function of the undetected loss amount calculation unit 43 may be omitted in the server 4.

[0122] If the calculated undetected loss amount L and false alarm rate E do not meet predetermined conditions, or if there are undetermined division patterns, the determination unit 45 instructs the division unit 42 to change the division pattern, thereby changing the division pattern of the division into product categories performed by the division unit 42. The undetected loss amount calculation unit 43 and the false alarm rate calculation unit 44 each obtain the division results corresponding to the changed division pattern from the division unit 42 and execute the process of calculating the undetected loss amount L and false alarm rate E.

[0123] (Explanation of an example of the functional configuration of the fraud detection device 3) Next, with reference to Figure 9, an example of the functional configuration of the fraud detection device 3 will be explained.

[0124] The training unit 32 uses multiple training data input from the server 4 to train (machine learning process) the category recognition model 31a. The category recognition model 31a is an example of an image recognition model (machine learning model) trained to recognize product categories.

[0125] Multiple training data sets may include, for example, information that uniquely identifies each of the multiple products to be processed (e.g., product name), an image of the product's appearance, and the product's category. The product name and appearance image are information included in product information 41a and are examples of input data to the category recognition model 31a. The category is a category obtained by converting from the product name based on the above-described segmentation result (corresponding information) or corresponding information 41b, and is an example of ground truth data (ground truth label). Multiple training data sets for multiple products to be processed may be called a training dataset.

[0126] The training unit 32, for example, inputs appearance images included in the training data into the category recognition model 31a and obtains the categories output from the category recognition model 31a in accordance with the input training data. The training unit 32 then updates (optimizes) the parameters of the category recognition model 31a so that the categories obtained from the category recognition model 31a match the categories included in the training data (ground truth data) (the error is minimized).

[0127] The training unit 32 trains the category recognition model 31a by performing the above-described process for each of the multiple training data in the machine learning process in the first example of the evaluation index acquisition process, or in the machine learning process executed after the completion of the category generation process of the server 4. Various known methods may be used to determine the completion of the machine learning process for the category recognition model 31a.

[0128] The inference unit 33 (category recognition unit 30), the determination unit 34, and the output unit 35 execute the verification process (calculation of evaluation indicators) in the first example of the evaluation indicator acquisition process, in response to instructions from the false alarm rate calculation unit 44 of the server 4.

[0129] Verification information 31b may include multiple verification data used in the verification process. The multiple verification data may include, for example, information that uniquely identifies each of the multiple products to be verified (e.g., product name) and an image of the product's appearance.

[0130] In the following explanation, the external images included in verification information 31b are assumed to be images of the product captured by an imaging device such as camera 2, or images that have been processed, and may be referred to as "camera images."

[0131] Furthermore, the inference unit 33 (category recognition unit 30), the determination unit 34, and the output unit 35 perform fraud detection processing during the operation phase of the fraud detection device 3. Fraud detection processing is a process that detects fraudulent activity using image information 31c and scan results input from the POS device 5.

[0132] Image information 31c may include images of the appearance of each of the multiple products, for example, multiple camera images. The multiple camera images included in image information 31c are an example of multiple captured images obtained by photographing multiple products that are subject to fraud detection. Note that each of the multiple camera images included in image information 31c may be an image that has been processed from the captured image.

[0133] The inference unit 33 (category recognition unit 30) performs inference processing using the trained category recognition model 31a trained by the training unit 32.

[0134] For example, in the verification process, the inference unit 33 may input the camera image included in the verification information 31b to the category recognition model 31a and obtain the category output from the category recognition model 31a. Alternatively, for example, in the fraud detection process, the inference unit 33 may input the camera image included in the image information 31c to the category recognition model 31a and obtain the category output from the category recognition model 31a. The inference unit 33 may output the category obtained from the category recognition model 31a to the determination unit 34.

[0135] The determination unit 34 converts the input product name to obtain a category. The determination unit 34 then compares the converted category with the category input from the inference unit 33 and outputs a result (comparison result) indicating whether the two match to the output unit 35. This result may include at least one type of information from the two categories, the camera image, and the product name input from the POS device 5.

[0136] In the process of converting categories, the determination unit 34 may perform the following processes, for example. In the verification process, for example, the determination unit 34 may convert the product names included in the verification information 31b into categories based on the correspondence between product names and categories input from the false alarm rate calculation unit 44. In addition, in the fraud detection process, for example, the determination unit 34 may convert the product names included in the scan results input from the POS device 5 into categories based on the correspondence between product names and categories (optimal categories) included in the correspondence information 41b.

[0137] Based on the judgment result input from the determination unit 34, the output unit 35 outputs evaluation indicators such as recognition rate or false positive rate to the false alarm rate calculation unit 44 during the verification process, and outputs an alert to the POS device 5 during the fraud detection process. The alert is an example of information indicating that fraudulent activity has been detected regarding the scan results. The output unit 35 may also output the alert to terminal devices used by employees of the retailer (store), such as PCs (personal computers), smartphones, tablet devices, etc.

[0138] [C] Example of operation of one embodiment Next, an example of the operation of System 1 according to one embodiment will be described with reference to Figures 13 to 17.

[0139] [C-1] Example of Category Generation Process Operation Figure 13 is a flowchart illustrating an example of the category generation process. As illustrated in Figure 13, the division unit 42 of the server 4 divides multiple products into one or more categories based on the product information 41a (step S1).

[0140] The undetected loss amount calculation unit 43 calculates the undetected loss amount L based on the product information 41a and the division results (step S2).

[0141] The false alarm rate calculation unit 44 executes the evaluation index acquisition process (step S3) and calculates the false alarm rate E based on the acquired evaluation index (step S4). Note that the process in step S2 and the processes in steps S3 and S4 may be executed in reverse order, or at least a portion of them may be executed in parallel.

[0142] The determination unit 45 determines whether the optimization processing result, including the undetected loss amount L and the false alarm rate E, satisfies predetermined conditions (step S5). If the optimization processing result does not satisfy the predetermined conditions (NO in step S5), the process moves to step S1, and the division unit 42 outputs the division result using a different division pattern.

[0143] If the optimization processing result satisfies the predetermined conditions (YES in step S5), the determination unit 45 outputs the division result from the division unit 42 at that time as correspondence information 41b between the product name and the optimal category (step S6), and the process ends.

[0144] [C-2] Example of operation of the evaluation index acquisition process Figure 14 is a flowchart illustrating the operation of the first example of the evaluation index acquisition process. As illustrated in Figure 14, the false alarm rate calculation unit 44 of the server 4 acquires training data including the product name, appearance image, and category in the first example of the evaluation index acquisition process shown in step S3 of Figure 13 (step S11). The category may be obtained by converting the product name contained in the product information 41a based on the corresponding information (splitting result). The false alarm rate calculation unit 44 may output the training data to the fraud detection device 3 and have it calculate the evaluation index.

[0145] The training unit 32 of the fraud detection device 3 takes the appearance images in the training data as input and uses the categories as correct labels to train the category recognition model 31a (step S12).

[0146] The inference unit 33 acquires verification data (verification information 31b) including the product name and camera image (step S13).

[0147] The inference unit 33 converts the product names in the validation data into categories based on the correspondence information (step S14). Then, the inference unit 33 inputs the camera images in the validation data into the trained category recognition model 31a and obtains the category as the recognition result (step S15).

[0148] The determination unit 34 compares the recognition result with the converted category, outputs an evaluation index including the comparison result to the false alarm rate calculation unit 44 (step S16), and the process ends.

[0149] Figure 15 is a flowchart illustrating the operation of the second example of the evaluation index acquisition process. As illustrated in Figure 15, the false alarm rate calculation unit 44 of the server 4 acquires training data including the product name, appearance image, and category in the second example of the evaluation index acquisition process shown in step S3 of Figure 13 (step S21).

[0150] The false alarm rate calculation unit 44 determines whether the product recognition model has been trained (step S22). If it has been trained (YES in step S22), the process proceeds to step S24. If it has not been trained (NO in step S22), the false alarm rate calculation unit 44 takes the appearance image from the training data as input and the product name from the training data as the correct label to train the product recognition model (step S23).

[0151] The false alarm rate calculation unit 44 acquires verification data (verification information 31b) including the product name and camera image (step S24).

[0152] The false alarm rate calculation unit 44 converts the product names in the verification data into categories based on the corresponding information (step S25). Then, the false alarm rate calculation unit 44 inputs the camera images in the verification data into the trained product recognition model and obtains the product names as recognition results (step S26).

[0153] The false alarm rate calculation unit 44 converts the recognition result into a category based on the corresponding information (step S27). Then, the false alarm rate calculation unit 44 compares the converted category of the recognition result with the converted category of the product name in the verification data, obtains an evaluation index including the comparison result (step S28), and the process ends.

[0154] [C-3] Example of machine learning processing operation Figure 16 is a flowchart illustrating an example of machine learning processing. As illustrated in Figure 16, the training unit 32 of the fraud detection device 3 acquires training data from the server 4, including product names, appearance images, and optimal categories (step S31). The optimal category may be obtained by converting the product name contained in the product information 41a based on the corresponding information 41b.

[0155] The training unit 32 takes appearance images from the training data as input and uses the optimal category as the correct label to train the category recognition model 31a (step S32), and the process ends.

[0156] [C-4] Example of fraud detection process operation Figure 17 is a flowchart illustrating an example of the operation of the fraud detection process. As illustrated in Figure 17, the fraud detection device 3 acquires a camera image from camera 2 (step S41) and stores it in image information 31c.

[0157] The inference unit 33 inputs the camera image contained in the image information 31c to the trained category recognition model 31a, obtains the category as a recognition result (step S42), and outputs the recognition result to the determination unit 34.

[0158] The determination unit 34 obtains the scan result, including the product name of the scanned product, from the POS device 5 (step S43), and converts the obtained product name into a category based on the correspondence between the product name and category (optimal category) contained in the correspondence information 41b (step S44).

[0159] The determination unit 34 compares the recognition result with the converted category and determines whether they match (step S45), and outputs the determination result to the output unit 35.

[0160] If the judgment result indicates a match between the two (YES in step S45), the process ends. In this case, the output unit 35 may also notify the POS device 5 that no fraudulent activity has been detected.

[0161] On the other hand, if the judgment result indicates a mismatch between the two (NO in step S45), the output unit 35 outputs an alert to the POS device 5 (step S46), and the process ends.

[0162] [D] Application Examples Next, an example of the application of System 1 according to one embodiment will be explained using fraud detection processing for label switching and banana trick as examples.

[0163] [D-1] First example of fraud detection processing Figure 18 is a diagram illustrating a first example of fraud detection processing. As illustrated in Figure 18, the correspondence information 41b includes {product name, category} such as {fine wine A (premium wine 110), W}, {snack B (sweets 120), X}. As shown by arrow C1, assume that the label of premium wine 110 is obscured by the label of sweets 120 when captured by camera 2, and that the label of sweets 120 is scanned by POS device 5.

[0164] In this case, as shown by arrow C2, when the category recognition unit 30 receives a camera image, it outputs category W corresponding to the premium wine 110 to the determination unit 34. Meanwhile, the determination unit 34 receives the product name of the confectionery 120 as a scan result from the POS device 5. The determination unit 34 converts the product name of the confectionery 120 to category X based on the correspondence information 41b. The determination unit 34 compares category W and category X and outputs a comparison result indicating a mismatch to the output unit 35. The output unit 35 outputs an alert to the POS device 5 based on the comparison result.

[0165] Figure 19 shows an example of an alert output in a first example of fraud detection processing. The POS device 5 may include an operation screen 50 such as a touch panel and a fixed scanner device 51. If the confectionery 120 is presented in a way that covers the label of the premium wine 110 within the scan area 52, which is the scan range of the scanner device 51, an alert 501 is displayed on the operation screen 50.

[0166] Alert 501 may include, for example, wording indicating a mismatch between the scanned product and the product recognized by image recognition, and wording prompting the customer to scan again. Alert 501 may also display a button 502 for the customer to choose whether or not to scan again. If "No" is selected with button 502, the POS device 5 may output various notifications, in other words, "issue an alert," such as calling an employee. On the other hand, if "Yes" is selected with button 502, the POS device 5 may cancel the registration of the confectionery 120 that was the subject of Alert 501 (or deduct the price of confectionery 120 from the registered total amount) and resume the process of registering the product.

[0167] Thus, according to the method of one embodiment, fraudulent activity using label switches can be detected. Furthermore, even when the normal registration of a product fails due to customer error or other reasons, such activity (fraudulent activity) can be detected.

[0168] [D-2] Second example of fraud detection processing Figure 20 is a diagram illustrating a second example of fraud detection processing. As illustrated in Figure 20, the correspondence information 41b includes {Shine Muscat 130, Y} and {Banana 140, Z} as {product name, category}. As shown by arrow D1, we assume that an unlabeled Shine Muscat 130 is captured by camera 2, and that a banana 140 is registered (selected: see screen 102 in Figure 2) in place of the Shine Muscat 130 in the POS device 5.

[0169] In this case, as shown by arrow D2, when the category recognition unit 30 receives a camera image, it outputs category Y corresponding to Shine Muscat 130 to the determination unit 34. Meanwhile, the determination unit 34 receives the product name of banana 140 as a scan result (selection result) from the POS device 5. The determination unit 34 converts the product name of banana 140 to category Z based on the correspondence information 41b. The determination unit 34 compares category Y and category Z and outputs a comparison result indicating a mismatch to the output unit 35. The output unit 35 outputs an alert to the POS device 5 based on the comparison result.

[0170] Figure 21 shows an example of an alert output in the second example of fraud detection processing. If bananas 140 are registered in the POS device 5 instead of Shine Muscat grapes 130, alert 501 is displayed on the operation screen 50.

[0171] If "Yes" is selected on button 502 of alert 501, the POS device 5 may cancel the registration of the banana 140 that was the subject of alert 501 (or deduct the price of banana 140 from the registered total amount). In this case, the POS device 5 may, for example, display a screen 503 for registering products on the operation screen 50. This allows the customer to correctly select Shine Muscat grapes 130 on screen 503.

[0172] Thus, according to the method of one embodiment, fraudulent activity using the banana trick can be detected. Furthermore, even when the normal registration of a product fails due to customer error or other reasons, such activity (fraudulent activity) can be detected.

[0173] As described above, according to the server 4 of one embodiment, it is possible to generate artificial categories that satisfy predetermined conditions based on objective criteria. For example, the server 4 can define the predetermined conditions as an objective function and generate categories that optimally satisfy the conditions by optimizing that objective function. As a result, for example, if the predetermined conditions include conditions (a) and (b), it is possible to generate product categories that reduce false alarms and minimize the amount of loss when fraudulent activity is overlooked.

[0174] Furthermore, the fraud detection device 3, which includes a category recognition model 31a trained based on the generated categories, can perform fraud detection processing that reduces false alarms and minimizes the amount of loss incurred when fraudulent activity is overlooked.

[0175] Furthermore, since categories are determined according to predetermined conditions, it is possible to generate pseudo (artificial) categories that are difficult for humans to understand. As a result, even if a malicious customer tries to register products in the same category (e.g., daily necessities) as the products they intend to fraudulently obtain through fraudulent activities such as label switching or the banana trick, the fraud detection process can appropriately detect the fraudulent activity.

[0176] [E] Other The technology according to the above embodiment can be implemented by modifying and changing it as follows.

[0177] For example, the division unit 42, undetected loss amount calculation unit 43, false alarm rate calculation unit 44, and determination unit 45 of the server 4 shown in Figure 9 may be merged in any combination or each may be separated. Similarly, the training unit 32, inference unit 33, judgment unit 34, and determination unit 45 of the fraud detection device 3 shown in Figure 9 may be merged in any combination or each may be separated. Furthermore, the fraud detection device 3 may be merged with the server 4 (see Figure 7) or with the POS device 5. If the fraud detection device 3 is merged with the POS device 5, the functional configuration of the fraud detection device 3 shown in Figure 9 may be provided in the POS device 5.

[0178] Furthermore, the output unit 35 is configured to send a message as an alert to the POS device 5, but is not limited to this. The alert may be an audio message, buzzer sound, or the like, prompting the customer to re-register the product in which fraudulent activity was detected, instead of or in addition to the message. The message is not limited to wording prompting the customer to re-register the product, but may also be wording indicating that registration failed (in other words, a screen prompting the customer to re-register the product), or wording suggesting that the customer call an employee, etc. In addition, the alert may include a command (control information) instructing the POS device 5 to temporarily suspend the functions of product registration and accounting processing.

[0179] Furthermore, for example, one or both of the server 4 and the fraud detection device 3 shown in Figure 9 may be configured so that multiple devices cooperate with each other via a network to realize their respective processing functions. For example, the division unit 42, the undetected loss amount calculation unit 43, the false alarm rate calculation unit 44, and the determination unit 45 of the server 4 may be implemented by an application server or a web server, and the storage area for storing product information 41a and corresponding information 41b may be implemented by a DB (Database) server. Also, the training unit 32, the inference unit 33, the judgment unit 34, and the output unit 35 of the fraud detection device 3 may be implemented by an application server or a web server, and the storage area for storing the category recognition model 31a, verification information 31b, and image information 31c may be implemented by a DB (Database) server. In these cases, the web server, application server, and DB server may cooperate with each other via a network to realize the processing functions of one or both of the server 4 and the fraud detection device 3.

[0180] [F] Note With respect to the above embodiment, the following additional information is disclosed.

[0181] (Note 1) Based on the multiple appearance images stored in a storage area (41a) that stores the appearance images and prices of each of the multiple products to be judged in association with each other, the multiple products are divided into one or more categories. Based on the multiple prices stored in the memory area, for each of the one or more divided categories, a statistical value of the prices of multiple products within the category is calculated. The division pattern is changed and the division process and calculation process are executed so that one or more of the calculated statistics satisfy the predetermined conditions. When the predetermined conditions are met, category information indicating the category of each of the multiple products is output. An information processing program that causes a computer to perform a task.

[0182] (Note 2) When a machine learning model that outputs the category of a product to be judged, taking an image of the product to be judged as input, is trained based on one or more divided categories, the false alarm rate of the machine learning model is calculated. The division pattern is changed and the division process and calculation process are executed so that the calculated statistical quantity of 1 or more and the calculated false alarm rate satisfy the predetermined conditions. An information processing program as described in Appendix 1, which causes the computer to perform the processing.

[0183] (Note 3) The process of changing the division pattern and performing the division process and the calculation process includes a multi-objective optimization process that optimizes the objective function for the one or more statistical quantities and the objective function for the false alarm rate. The information processing program described in Appendix 2.

[0184] (Note 4) The aforementioned division process is, The image feature quantities of each of the plurality of appearance images stored in the memory area are obtained, The process includes clustering the plurality of appearance images based on the similarity of the aforementioned image features. An information processing program described in any one of the items in Appendix 1 to Appendix 3.

[0185] (Note 5) The process for outputting the category information includes a process for outputting the appearance images of each of the multiple products stored in the memory area and the category of each of the multiple products indicated by the category information, as training data for a machine learning model that takes the captured images of the products to be judged as input and outputs the category of the products to be judged. Using the aforementioned training data, the machine learning process of the machine learning model is executed. The computer is made to perform the process. An information processing program described in any one of the items in Appendix 1 to Appendix 4.

[0186] (Note 6) The category of the product registered in the automated checkout device is obtained based on the category information, The system determines whether the category obtained by inputting the captured image of the product to be judged into the trained machine learning model matches the category of the registered product. If they do not match, an alert is sent to the automatic payment device. An information processing program as described in Appendix 5, which causes the computer to perform the processing.

[0187] (Note 7) Based on the multiple appearance images stored in a memory area that associates the appearance images and prices of each of the multiple products to be judged, the multiple products are divided into one or more categories. Based on the multiple prices stored in the memory area, for each of the one or more divided categories, a statistical value of the prices of multiple products within the category is calculated. The division pattern is changed and the division process and calculation process are executed so that one or more of the calculated statistics satisfy the predetermined conditions. When the predetermined conditions are met, category information indicating the category of each of the multiple products is output. An information processing device equipped with a control unit.

[0188] (Note 8) The control unit, When a machine learning model that outputs the category of a product to be judged, taking an image of the product to be judged as input, is trained based on one or more divided categories, the false alarm rate of the machine learning model is calculated. The division pattern is changed and the division process and calculation process are executed so that the calculated statistical quantity of 1 or more and the calculated false alarm rate satisfy the predetermined conditions. The information processing device described in Appendix 7.

[0189] (Note 9) The control unit performs a multi-objective optimization process in which it changes the division pattern and executes the division process and the calculation process, optimizing the objective function for the one or more statistical quantities and the objective function for the false alarm rate. The information processing device described in Appendix 8.

[0190] (Note 10) The control unit, in the division process, The image feature quantities of each of the plurality of appearance images stored in the memory area are obtained, Clustering of the multiple appearance images is performed based on the similarity of the aforementioned image features. An information processing device as described in any one of the items 7 to 9 of the appendix.

[0191] (Note 11) The control unit, In the process of outputting the category information, the following are output as training data for a machine learning model that takes the captured image of the product to be determined as input and outputs the appearance image of each of the multiple products stored in the memory area and the category of each of the multiple products indicated by the category information: Using the aforementioned training data, the machine learning process of the machine learning model is executed. An information processing device as described in any one of the items in Appendix 7 to Appendix 10.

[0192] (Note 12) The control unit, The category of the product registered in the automated checkout device is obtained based on the category information, The system determines whether the category obtained by inputting the captured image of the product to be judged into the trained machine learning model matches the category of the registered product. If they do not match, an alert is sent to the automatic payment device. The information processing device described in Appendix 11.

[0193] (Note 13) An information processing system comprising an automated payment device and a fraud detection device that performs fraud detection in the automated payment device using an image recognition model, Based on the multiple appearance images stored in a memory area that associates the appearance images and prices of each of the multiple products to be judged, the multiple products are divided into one or more categories. Based on the multiple prices stored in the memory area, for each of the one or more divided categories, a statistical value of the prices of multiple products within the category is calculated. The division pattern is changed and the division process and calculation process are executed so that one or more of the calculated statistics satisfy the predetermined conditions. When the predetermined conditions are met, category information indicating the category of each of the multiple products is output. Equipped with a control unit, The fraud detection device performs fraud detection using the image recognition model trained with the category information. Information processing system.

[0194] (Note 14) The control unit, When a machine learning model that outputs the category of a product to be judged, taking an image of the product to be judged as input, is trained based on one or more divided categories, the false alarm rate of the machine learning model is calculated. The division pattern is changed and the division process and calculation process are executed so that the calculated statistical quantity of 1 or more and the calculated false alarm rate satisfy the predetermined conditions. The information processing system described in Appendix 13.

[0195] (Note 15) The control unit performs a multi-objective optimization process in which it changes the division pattern and executes the division process and the calculation process, optimizing the objective function for the one or more statistical quantities and the objective function for the false alarm rate. The information processing system described in Appendix 14.

[0196] (Note 16) The control unit, in the division process, The image feature quantities of each of the plurality of appearance images stored in the memory area are obtained, Clustering of the multiple appearance images is performed based on the similarity of the aforementioned image features. An information processing system described in any one of the items in Appendix 13 to Appendix 15.

[0197] (Note 17) The control unit, In the process of outputting the category information, the following are output as training data for a machine learning model that takes an image of the product to be determined as input: the appearance images of each of the multiple products stored in the memory area and the categories of each of the multiple products indicated by the category information. An information processing system as described in any one of the items in Appendix 13 to Appendix 16.

[0198] (Note 18) The fraud detection device is The category of the product registered in the automatic payment device is obtained based on the category information, The system determines whether the category obtained by inputting the captured image of the product to be judged into the trained machine learning model matches the category of the registered product. If they do not match, an alert is sent to the automatic payment device. The information processing system described in Appendix 17. [Explanation of Symbols]

[0199] 1 System 10 Computers 2 cameras 3. Fraud detection device 30 Category Recognition Unit 31a Category Recognition Model 31b Verification Information 31c Image Information 32 Training Department 33 Reasoning part 34 Judgment section 35 Output section 36,46 Control Unit 4 servers 41a Product Information 41b Compatibility Information 42 Division 43. Undetected Loss Amount Calculation Unit 44 False alarm rate calculation section 45 Decision Section 5 POS device 50 Operation screen 51 Scanner device

Claims

1. Based on the multiple appearance images stored in a memory area that stores the appearance images and prices of each of the multiple products to be judged in association with each other, the multiple products are divided into one or more categories. Based on the multiple prices stored in the memory area, for each of the one or more divided categories, a statistical value of the prices of multiple products within the category is calculated. The division pattern is changed and the division process and calculation process are executed so that one or more of the calculated statistics satisfy the predetermined conditions. When the predetermined conditions are met, category information indicating the category of each of the multiple products is output. An information processing program that causes a computer to perform a task.

2. When a machine learning model that outputs the category of a product to be judged, taking an image of the product to be judged as input, is trained based on one or more divided categories, the false alarm rate of the machine learning model is calculated. The division pattern is changed and the division process and calculation process are executed so that the calculated statistical quantity of 1 or more and the calculated false alarm rate satisfy the predetermined conditions. The information processing program according to claim 1, which causes the computer to perform the processing.

3. The process of changing the division pattern and performing the division process and the calculation process includes a multi-objective optimization process that optimizes the objective function for one or more statistical quantities and the objective function for the false alarm rate. The information processing program according to claim 2.

4. The aforementioned division process is, The image feature quantities of each of the plurality of appearance images stored in the memory area are obtained, The process includes clustering the plurality of appearance images based on the similarity of the aforementioned image features. An information processing program according to any one of claims 1 to 3.

5. The process for outputting the category information includes a process for outputting the appearance images of each of the multiple products stored in the memory area and the category of each of the multiple products indicated by the category information, as training data for a machine learning model that takes the captured images of the products to be judged as input and outputs the category of the products to be judged. Using the aforementioned training data, the machine learning process of the machine learning model is executed. The computer is made to perform the process. An information processing program according to any one of claims 1 to 3.

6. The category of the product registered in the automated checkout device is obtained based on the category information, The system determines whether the category obtained by inputting the captured image of the product to be judged into the trained machine learning model matches the category of the registered product. If they do not match, an alert is sent to the automatic payment device. The information processing program according to claim 5, which causes the computer to perform the processing.

7. Based on the multiple appearance images stored in a memory area that stores the appearance images and prices of each of the multiple products to be judged in association with each other, the multiple products are divided into one or more categories. Based on the multiple prices stored in the memory area, for each of the one or more divided categories, a statistical value of the prices of multiple products within the category is calculated. The division pattern is changed and the division process and calculation process are executed so that one or more of the calculated statistics satisfy the predetermined conditions. When the predetermined conditions are met, category information indicating the category of each of the multiple products is output. An information processing device equipped with a control unit.

8. An information processing system comprising an automated payment device and a fraud detection device that performs fraud detection in the automated payment device using an image recognition model, Based on the multiple appearance images stored in a memory area that stores the appearance images and prices of each of the multiple products to be judged in association with each other, the multiple products are divided into one or more categories. Based on the multiple prices stored in the memory area, for each of the one or more divided categories, a statistical value of the prices of multiple products within the category is calculated. The division pattern is changed and the division process and calculation process are executed so that one or more of the calculated statistics satisfy the predetermined conditions. When the predetermined conditions are met, category information indicating the category of each of the multiple products is output. Equipped with a control unit, The fraud detection device performs fraud detection using the image recognition model trained with the category information. Information processing system.

Images