Information processing program, information processing method, and information processing device.

Abstract

To detect an error or a fraud of a user in usage of an accounting machine.SOLUTION: An information processor acquires picture data of a person who scans the code of an item on an accounting machine. The information processor identifies the whole region of an item and the position of the code of the item from the picture data by analyzing the acquired picture data. The information processor generates an alert showing an abnormality in an action of registering the item in the accounting machine on the basis of the identified position of the code of the item to the whole region of the item.SELECTED DRAWING: Figure 2

Description

Technical Field

[0004] , ,

[0006] , , , ,

[0005] , , , , ,

[0001] The present invention relates to an information processing program, an information processing method, and an information processing apparatus.

Background Art

[0002] Self-checkout has become widespread in stores such as supermarkets and convenience stores. Self-checkout is a POS (Point Of Sale) cash register system in which the user who purchases goods performs operations from reading the barcode of the goods to settlement. For example, by introducing self-checkout, it is possible to improve the labor shortage due to population decline and suppress labor costs.

Prior Art Documents

[0007] Furthermore, with self-checkout systems, scanning product codes and payment are left to the user, making it difficult to detect fraudulent activity. For example, even if one were to apply image recognition AI (Artificial Intelligence) to detect fraudulent activity, training the AI ​​would require a large amount of training data. However, stores such as supermarkets and convenience stores have a large variety of products, and the lifecycle of each product is short, resulting in frequent product changes. It is difficult to tune image recognition AI to match such product lifecycles, or to train new image recognition AI.

[0008] In one aspect, the objective is to provide an information processing program, information processing method, and information processing device that can detect user errors or fraud in accounting machines. [Means for solving the problem]

[0009] In the first proposal, the information processing program is characterized by causing a computer to perform the following processing: acquire video data of a person scanning a product code into a checkout machine; analyze the acquired video data to identify the entire area of ​​the product and the location of the code on the product from the video data; and generate an alert indicating an abnormality in the action of registering the product into the checkout machine based on the location of the code on the product relative to the identified entire area of ​​the product. [Effects of the Invention]

[0010] According to one embodiment, user errors or fraud can be detected in an accounting machine. [Brief explanation of the drawing]

[0011] [Figure 1] FIG. 1 is a diagram showing an overall configuration example of the self-checkout system according to Embodiment 1. [Figure 2] FIG. 2 is a diagram for explaining an example of fraud detection according to Embodiment 1. [Figure 3] FIG. 3 is a functional block diagram showing the functional configuration of the information processing apparatus according to Embodiment 1. [Figure 4] FIG. 4 is a diagram for explaining the machine learning of the machine learning model. [Figure 5] FIG. 5 is a diagram for explaining the identification process. [Figure 6] FIG. 6 is a diagram for explaining the identification result. [Figure 7] FIG. 7 is a diagram for explaining an example of fraud detection based on the positional relationship. [Figure 8] FIG. 8 is a diagram for explaining an example of the detected fraud. [Figure 9] FIG. 9 is a diagram for explaining an example of fraud in the set products. [Figure 10] FIG. 10 is a diagram for explaining an example of an illegal operation. [Figure 11] FIG. 11 is a diagram for explaining an example of alert notification. [Figure 12] FIG. 12 is a flowchart showing the process flow. [Figure 13] FIG. 13 is a functional block diagram showing the functional configuration of the information processing apparatus according to Embodiment 2. <00°0007%FIG. 14 is a diagram showing an example of information stored in the pattern DB. [Figure 15] FIG. 15 is a diagram for explaining the machine learning of the generation model. [Figure 16] FIG. 16 is a diagram for explaining fraud detection. [Figure 17] FIG. 17 is a diagram for explaining an example of the hardware configuration. [Figure 18] FIG. 18 is a diagram for explaining an example of the hardware configuration of the self-checkout.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the information processing program, information processing method, and information processing apparatus disclosed in the present application will be described in detail based on the drawings. Note that the present invention is not limited by these embodiments. Also, the respective embodiments can be appropriately combined within a range without contradiction.

Embodiment

[0013] <Explanation of Self-checkout System> FIG. 1 is a diagram showing an overall configuration example of a self-checkout system 5 according to Embodiment 1. As shown in FIG. 1, the self-checkout system 5 includes a camera 30, a self-checkout 50, an administrator terminal 60, and an information processing apparatus 100.

[0014] The information processing apparatus 100 is an example of a computer connected to the camera 30 and the self-checkout 50. The information processing apparatus 100 is connected to the administrator terminal 60 via a network 3 that can adopt various communication networks regardless of wired or wireless. The camera 30 and the self-checkout 50 may be connected to the information processing apparatus 100 via the network 3.

[0015] The camera 30 is an example of a camera that captures an image of an area including the self-checkout 50. The camera 30 transmits the image data of the video to the information processing apparatus 100. In the following description, the image data of the video may be referred to as "video data" or simply "video".

[0016] The video data includes a plurality of image frames in time series. Each image frame is assigned a frame number in ascending order of time series. One image frame is the image data of a still image captured by the camera 30 at a certain timing.

[0017] Self-checkout 50 is an example of a POS system or payment machine where user 2, who purchases goods, handles everything from scanning the product's barcode to payment. For example, when user 2 moves the items to be purchased to the scanning area of ​​self-checkout 50, self-checkout 50 scans the barcodes of the items and registers them as purchased items.

[0018] As mentioned above, Self-Checkout 50 is an example of a self-checkout system where customers register their purchases (checkout process) and make payments themselves. It is also known as Self checkout, automated checkout, self-checkout machine, or self-checkout register. A barcode is a type of identifier that represents numbers or letters using the thickness of striped lines. Self-Checkout 50 can scan (read) barcodes to identify the price and type of product (e.g., food). Barcodes are just one example of codes; other two-dimensional codes such as QR (Quick Response) codes, which have the same function, can also be used.

[0019] User 2 repeatedly performs the above product registration operation, and once the product scanning is complete, operates the touch panel or other controls of the self-checkout register 50 to request payment. Upon receiving the payment request, the self-checkout register 50 displays the number of items to be purchased, the purchase amount, etc., and performs the payment process. The self-checkout register 50 stores the information of the items scanned between the time User 2 starts scanning and the time the payment request is made in its memory unit, and transmits this information as self-checkout data (product information) to the information processing device 100. The user places the items to be purchased, selected in the store, into a shopping basket or cart and carries the items to the self-checkout register 50.

[0020] The administrator terminal 60 is an example of a terminal device used by a store manager. The administrator terminal 60 receives notifications from the information processing device 100, such as alerts indicating that fraudulent activity has occurred regarding the purchase of goods.

[0021] In this configuration, the information processing device 100 acquires video data of a person scanning a product's barcode at the self-checkout register 50. By analyzing the acquired video data, the information processing device 100 identifies the entire area of ​​the product and the location of the barcode on the product from the video data. Based on the identified location of the barcode on the product relative to the entire area of ​​the product, the information processing device 100 generates an alert indicating an abnormality in the action of registering the product at the self-checkout register 50.

[0022] Figure 2 illustrates an example of fraud detection according to Embodiment 1. As shown in Figure 2, the information processing device 100 identifies product A from the video data and identifies the entire area of ​​product A and the location A1 of product A's barcode. The information processing device 100 then determines that there is a high possibility of fraudulent activity when it detects a location A2 different from the barcode location A1 of product A at the scan location where product A is scanned and registered by the self-checkout register 50, and generates an alert to notify the user.

[0023] Generally, the relative positions of labels representing product names or patterns and the product's barcode can be pre-determined and mapped, for example, the barcode not being in the middle of the label. Therefore, the information processing device 100 identifies the positions representing product features such as labels and patterns from the video data, and determines whether the scan operation is normal or abnormal based on the positional relationship (relative position) between the position representing the features and the position of the product to be scanned.

[0024] As a result, the information processing device 100 can detect fraud at self-checkout counters by analyzing video data without requiring complex equipment.

[0025] <Functional Configuration> Figure 3 is a functional block diagram showing the functional configuration of the information processing device 100 according to Embodiment 1. As shown in Figure 3, the information processing device 100 has a communication unit 101, a storage unit 102, and a control unit 110.

[0026] The communication unit 101 is a processing unit that controls communication with other devices, and is implemented, for example, by a communication interface. For example, the communication unit 101 receives video data from the camera 30 and transmits the processing result from the control unit 110 to the administrator terminal 60.

[0027] The memory unit 102 is a processing unit that stores various data and programs executed by the control unit 110, and is implemented using memory or a hard disk. The memory unit 102 stores the training data DB 103, the machine learning model 104, and the video data DB 105.

[0028] The training data DB103 is a database that stores the data used to train the machine learning model 104. For example, the training data stored in the training data DB103 is supervised data with "product image data" as the explanatory variable and "product barcode location" as the dependent variable.

[0029] The machine learning model 104 is a machine learning model that estimates the location of barcodes on products in video data in response to video data input. For example, the machine learning model 104 outputs an estimation result that estimates the location of the barcode on a product for each image data (each frame) in the video data.

[0030] Furthermore, the machine learning model 104 can also be a machine learning model trained to output the entire area of ​​the product and the location of the product's barcode in the video data in response to the input video data.

[0031] The video data DB105 is a database that stores video data captured by cameras 30 installed on self-checkout registers 50. For example, the video data DB105 stores video data for each self-checkout register 50 or for each camera 30.

[0032] The control unit 110 is a processing unit that oversees the entire information processing device 100 and is implemented by, for example, a processor. This control unit 110 includes a machine learning unit 111, an image acquisition unit 112, an identification unit 113, a fraud detection unit 114, and a warning control unit 115. The machine learning unit 111, the image acquisition unit 112, the identification unit 113, the fraud detection unit 114, and the warning control unit 115 are implemented by electronic circuits and processes executed by the processor.

[0033] (Machine Learning) The machine learning unit 111 is a processing unit that performs machine learning on the machine learning model 104 using each training data stored in the training data DB 103. Figure 4 is a diagram illustrating the machine learning of the machine learning model 104. As shown in Figure 4, the machine learning unit 111 inputs training data in which "image data" is the explanatory variable and "barcode position" is the target variable into the machine learning model 104, and calculates error information between the output result "barcode position" of the machine learning model 104 and the target variable "barcode position". Then, the machine learning unit 111 performs machine learning to update the parameters of the machine learning model 104 by backpropagation in order to reduce the error.

[0034] (Video acquisition) The video acquisition unit 112 is a processing unit that acquires video data from the camera 30. For example, the video acquisition unit 112 continuously acquires video data from the camera 30 installed on the self-checkout machine 50 and stores it in the video data DB 105.

[0035] (identification) The identification unit 113 is a processing unit that analyzes the video data acquired by the video acquisition unit 11 using a machine learning model or the like to identify the overall area of ​​the product and the location of the code that the product possesses from the video data.

[0036] Figure 5 is a diagram illustrating the identification process. As shown in Figure 5, the identification unit 113 uses a machine learning model 104 and pattern matching used in image analysis to identify the entire area of ​​the product and the location of the product's barcode for each image data within the video data. The identification unit 113 then stores the identification result in the storage unit 102 and outputs it to the fraud detection unit 114.

[0037] Here, the identification result obtained by the identification unit 113 will be explained. Figure 6 is a diagram illustrating the identification result. Figure 6 shows an image of a packaged, flat-shaped chocolate being scanned. As shown in Figure 6, the identification unit 113 identifies the region B, which includes the entire flat-shaped chocolate, and the position B1 of the barcode on the flat-shaped chocolate.

[0038] Furthermore, the identification unit 113 sets multiple products, each having its own code, as a single product, and identifies the entire area of ​​a set product having a code as a single product, as well as the location of the code on the set product. For example, the identification unit 113 identifies the entire area of ​​a set of six canned beer products and the location of the barcode assigned to the set of canned beer.

[0039] (Fraud detection) The fraud detection unit 114 is a processing unit that detects abnormalities in the act of registering a product in the self-checkout 50 based on the position of the barcode of the product relative to the entire area of ​​the product identified by the identification unit 113. Specifically, the fraud detection unit 114 detects fraudulent behavior based on the positional relationship between the product name and the feature area (feature position) represented by the product name and features of the product, and the barcode of the product identified from the video data.

[0040] In other words, the fraud detection unit 114 detects fraudulent activity if a predetermined area of ​​a product located at the scanning position of the self-checkout register 50 does not contain the product's barcode. When the fraud detection unit 114 detects fraudulent activity, it notifies the warning control unit 115.

[0041] Figure 7 illustrates an example of fraud detection based on positional relationships. As shown in Figure 7, the fraud detection unit 114 detects fraudulent scanning operations based on the positional relationship between the product's characteristic area (characteristic position) and the product's barcode identified from the video data. For example, as shown in Figure 7, if the product is instant ramen, the top surface typically has the name of the ramen, distinctive shapes, and distinctive words printed on it, but the product's barcode is not printed on this top surface. Therefore, by analyzing the image data, the fraud detection unit 114 determines that if the side or bottom of the ramen is located at the scanning position, it is a normal scanning operation, and if the top surface of the ramen is located at the scanning position, it is a fraudulent scanning operation.

[0042] The fraud detected in Figure 7 is a type of fraud where a user attempts to purchase an expensive item by having the barcode of a less expensive item scanned instead (e.g., label switching). Figure 8 illustrates an example of the detected fraud. As shown in Figure 8, even though the top surface of the expensive item C is positioned at the scan location, if the item is scanned successfully (registered as a purchased item), it is likely that the user placed the less expensive item D on top of the expensive item C and had the barcode of item D scanned instead of item C. Therefore, the fraud detection unit 114 detects fraud based on the positional relationship shown in Figure 7, and thus can detect the fraudulent behavior shown in Figure 8.

[0043] Furthermore, after the identification unit 113 identifies the set product and the barcode position of the set product, the fraud detection unit 114 detects a fraudulent scan operation if the code of any product within the set product is located at the scan position for registering the product in the self-checkout register 50.

[0044] Figure 9 illustrates an example of fraud involving a set product. As shown in Figure 9, a possible fraudulent attempt is for a user to scan the barcode on an individual item included in the set product, rather than scanning the barcode on the set product E itself, thereby making it appear as if they purchased the set product E using the lower-priced individual items. The fraud detection unit 114 can identify the barcode location on the set product using the identification unit 113, and can detect fraud if the barcode is scanned at a location different from the identified barcode location. Therefore, it can detect the fraudulent behavior shown in Figure 9.

[0045] For example, a set product is packaged so that six cans of alcoholic beverage can be carried together, with the cans arranged in two rows of three using packaging material. In this case, both the packaging material that packages the set of alcoholic beverage cans and the cans of alcoholic beverage packaged within the packaging material have barcodes. The information processing device 100 notifies an alert when the barcode of the alcoholic beverage packaged in the packaging material, rather than the barcode of the packaging material, is scanned by the payment machine.

[0046] Furthermore, the fraud detection unit 114 detects fraudulent activity if, despite the entire area of ​​the product and the location of the product barcode being identified, the product is registered in the self-checkout register 50 without any scanning operation being detected.

[0047] Figure 10 illustrates an example of fraudulent activity. As shown in Figure 10, a possible fraudulent activity is when a user selects a low-priced item on the touch panel of the self-checkout 50 and registers it as a purchase, even though the item has a barcode. The fraud detection unit 114 can detect fraud when an item is registered even though the barcode of the identified item is not located at the scanning position, and thus can detect the fraudulent activity shown in Figure 10.

[0048] (Alert notification) The warning control unit 115 is a processing unit that generates an alert and executes alert notification control when fraudulent behavior (fraudulent operation) is detected by the fraud detection unit 114. For example, the warning control unit 115 generates an alert indicating that there are items that the person has not registered in the self-checkout register 50, or that the items the person has registered in the self-checkout register 50 are abnormal, and outputs it to the self-checkout register 50 or the administrator terminal 60.

[0049] Furthermore, if the warning control unit 116 generates an alert regarding an abnormality in the action of registering items at the self-checkout register 50, it outputs an audio or screen message from the self-checkout register 50 to prompt the person located at the self-checkout register 50 to register any items that may have been missed.

[0050] Figure 11 illustrates an example of an alert notification. As shown in Figure 11, the warning control unit 115 displays a message such as, "Are there any items you forgot to scan? Please scan the items again," on the touch panel or other display screen of the self-checkout register 50.

[0051] Furthermore, the warning control unit 115 may illuminate a warning light installed on the self-checkout register 50, display the identifier of the self-checkout register 50 and a message indicating the possibility of fraud on the administrator terminal 60, or send the identifier of the self-checkout register 50 and a message indicating the occurrence of fraud and the need for verification to the terminals of store employees in the store.

[0052] Furthermore, when the warning control unit 115 generates an alert regarding an abnormality in the act of registering items at the self-checkout register 50, it causes the camera 30 on the self-checkout register 50 to photograph the person, and stores the image data of the photographed person in the memory unit in association with the alert. In this way, information on fraudulent individuals who engage in fraudulent behavior can be collected, which can be used for various measures to prevent fraudulent behavior, such as detecting customers who have engaged in fraudulent behavior at the store entrance. In addition, the warning control unit 115 can generate a machine learning model using supervised learning with the image data of fraudulent individuals, enabling it to detect fraudulent individuals from the image data of people using the self-checkout register 50 and to detect fraudulent individuals at the store entrance. The warning control unit 115 can also acquire and store the credit card information of individuals who have engaged in fraudulent behavior from the self-checkout register 50.

[0053] <Processing flow> Figure 12 is a flowchart showing the processing flow. As shown in Figure 12, the information processing device 100 acquires video data as it progresses (S101).

[0054] Next, when the information processing device 100 is instructed to start the fraud detection process (S102: Yes), it acquires frames from the video data (S103). At this point, if no video data exists, the information processing device 100 terminates the process. On the other hand, if video data exists, the information processing device 100 identifies the entire product and the location of the barcode using the machine learning model 104 and pattern matching (S104).

[0055] Here, if the information processing device 100 does not detect any abnormal behavior based on the position of the barcode on the entire product (S105: No), it repeats steps S103 onwards. On the other hand, if the information processing device 100 detects abnormal behavior (S105: Yes), it detects fraud (S106), issues an alert (S107), and terminates the process.

[0056] <Effects> As described above, the information processing device 100 identifies the entire area of ​​the product and the location of the barcode on the product, and generates an alert indicating an abnormality in the action of registering the product at the self-checkout register 50 based on the location of the barcode on the product relative to the identified entire area of ​​the product. Therefore, the information processing device 100 can detect fraud at the self-checkout register 50.

[0057] Furthermore, the information processing device 100 sets multiple products, each with its own code, as a single product, and identifies the entire area of ​​the set product, as well as the location of the barcodes on the set product. The information processing device 100 then generates an alert if the code of any product within the set product is located at the location where the product is to be registered in the self-checkout register 50. Thus, the information processing device 100 can detect fraudulent purchases of the set product at the price of the individual items.

[0058] Furthermore, the information processing device 100 estimates the location of the barcode on a product in the video data by inputting the video data into a machine learning model that has learned the location of the code for each product. The information processing device 100 then generates an alert if the estimated location of the barcode on the product does not match the location of the code on the product located at the self-checkout counter 50. Therefore, the information processing device 100 can detect fraud such as picking up high-priced and low-priced items together and having the low-priced item scanned to pretend to have purchased the high-priced item.

[0059] Furthermore, the information processing device 100 generates an alert indicating that there are items that the person has not registered in the self-checkout register 50, or that the items the person has registered in the self-checkout register 50 are abnormal. Therefore, by using the information processing device 100, store employees can take measures such as questioning the person who has committed fraudulent acts before they leave the store.

[0060] Furthermore, if the information processing device 100 generates an alert regarding an abnormality in the action of registering items at the self-checkout register 50, it outputs an audio or screen message from the self-checkout register 50 to the person located at the self-checkout register 50 prompting them to register items that have been missed. Therefore, the information processing device 100 can directly alert the person scanning items, whether it is an unavoidable mistake or intentional fraud, thereby reducing mistakes and intentional fraud.

[0061] Furthermore, when an alert is generated regarding an abnormality in the act of registering products at the self-checkout register 50, the information processing device 100 uses the camera on the self-checkout register 50 to photograph the person, and stores the image data of the photographed person in a memory unit, associating it with the alert. Therefore, the information processing device 100 can collect and store information on fraudulent individuals who engage in fraudulent activities, and by detecting the presence of fraudulent individuals from the image data captured by the camera that photographs customers, it can be used to implement various measures to prevent fraudulent activities. In addition, the information processing device 100 can also obtain and store the credit card information of the person who engaged in fraudulent activity from the self-checkout register 50, so if fraudulent activity is confirmed, charges can be billed through the credit card company. [Examples]

[0062] Incidentally, the information processing device 100 can improve the accuracy of detecting abnormal behavior by analyzing video data to generate 2D images and 3D models of products, and by improving the accuracy of identifying the barcode position of products.

[0063] <Functional Configuration> Figure 13 is a functional block diagram showing the functional configuration of the information processing device 100 according to Embodiment 2. As shown in Figure 13, it has a communication unit 101, a storage unit 102, and a control unit 110. Here, we will describe the pattern DB 106 and the generation model 107, which are different functions from Embodiment 1, and the processing units related to them.

[0064] Pattern DB106 is a database that stores patterns indicating the location of barcodes on products. Figure 14 shows an example of the information stored in Pattern DB106. As shown in Figure 14, Pattern DB106 stores the "product face, pattern, and barcode location" in association.

[0065] The "pattern" stored here is an identifier that identifies the pattern. "Product side" is information indicating the side of the product where the label is printed. "Barcode position" is information indicating the position of the barcode on the product. In the example in Figure 14, Pattern 1 is a pattern where the barcode is in the "opposite position from the label" when the product label is located on the "top" and "side". Pattern 2 is a pattern where the barcode is on the side of the product when the product label is located on the "top". Note that each pattern can be set for each product, or for each product type.

[0066] The generative model 107 is a machine learning model that generates a 3D model of a product shown in video data in response to the input video data. For example, the generative model 107 generates and outputs a 3D model of a product for each image data (each frame) in the video data.

[0067] The machine learning unit 111 is a processing unit that performs machine learning on the generative model 107 using each training data. Figure 15 is a diagram illustrating the machine learning of the generative model 107. As shown in Figure 15, the machine learning unit 111 inputs training data in which "image data" is the explanatory variable and "3D model" is the target variable (ground truth information) into the generative model 107, and calculates error information between the output result "3D model" of the machine learning model 104 and the target variable "3D model". Then, the machine learning unit 111 performs machine learning to update the parameters of the generative model 107 by backpropagation to reduce the error.

[0068] The identification unit 113 analyzes the video data to recognize patterns on the surface of the product in the 2D image and to identify the position of the code relative to the pattern. For example, the identification unit 113 obtains a 2D image of the product from the image data in the video data, performs pattern matching on the 2D image, and identifies the position of the product's label and the product's barcode.

[0069] The fraud detection unit 114 detects fraudulent activity using the identification results from the identification unit 113. Specifically, the fraud detection unit 114 obtains the "location of the product label and the location of the product barcode" from the identification unit 113 and detects fraudulent activity based on whether or not it matches a pattern stored in the pattern DB 106.

[0070] To give a specific example, suppose the fraud detection unit 114 obtains "product label position = top surface" and "product barcode position = top surface" from the identification unit 113. In this case, the fraud detection unit 114 refers to the pattern DB 106 to identify "pattern 1" which corresponds to "product label position = top surface," and then identifies the "position opposite to the label" which corresponds to pattern 1. Then, the fraud detection unit 114 determines that the action is fraudulent because the barcode position "top surface" identified from the product is different from the "position opposite to the label" in pattern 1.

[0071] Furthermore, the fraud detection unit 114 can also input video data into a machine learning model to generate a 3D model of the product, and detect fraudulent activity when the position of the barcode in the 3D model differs from a pre-set position.

[0072] Figure 16 illustrates fraud detection. As shown in Figure 16, the fraud detection unit 114 inputs image data into the generation model 107 to generate a 3D model of the product in the image data. The fraud detection unit 114 then performs fraud detection by comparing it with the relative position and the pattern described above, as explained in Example 1.

[0073] <Effects> As described above, the information processing device 100 analyzes video data to recognize the surface pattern of the product in the 2D image and identifies the position of the code relative to the pattern. When the position of the code relative to the identified pattern differs from the position pre-set for each pattern, the information processing device 100 generates an alert indicating an abnormality in the action of registering the product in the self-checkout 50. Therefore, even if fraud that has not occurred in the past or new fraud that was not anticipated occurs, the information processing device 100 can detect fraud based on patterns that are based on positional relationships, etc.

[0074] Furthermore, the information processing device 100 generates a 3D model of the product by inputting video data into the generation model 107. When the position of the code in the generated 3D model differs from a pre-set position, the information processing device 100 generates an alert indicating an abnormality in the action of registering the product in the self-checkout 50. Therefore, the information processing device 100 can improve the accuracy of relative position and pattern comparison, and thus improve the accuracy of detecting fraudulent actions. In addition, since the barcode position can be visually identified using the 3D model, store employees can visually determine whether the barcode is located at the scanning position while viewing the video data. [Examples]

[0075] Now, although embodiments of the present invention have been described, the present invention may be implemented in various other forms besides those described above.

[0076] (Numerical values, etc.) The number of self-checkout machines and cameras used in the above examples, numerical examples, training data examples, number of training data points, machine learning models, class names, number of classes, and data formats are merely examples and can be changed as needed. Furthermore, the processing flow described in each flowchart can be modified as appropriate within a consistent range. Additionally, each model can be one generated using various algorithms, such as neural networks.

[0077] Furthermore, the information processing device 100 can also use known technologies such as other machine learning models for detecting location, object detection technologies, and location detection technologies to determine the scan location and the location of the shopping basket. For example, the information processing device 100 can detect the location of the shopping basket based on the difference between frames (image data) and the time series changes of the frames, so it may use this method for detection, or it may use this method to generate a different model. In addition, by pre-specifying the size of the shopping basket, the information processing device 100 can also identify the location of the shopping basket when an object of that size is detected from the image data. Since the scan location is a somewhat fixed location, the information processing device 100 can also identify a location specified by an administrator or the like as the scan location.

[0078] (system) Unless otherwise specified, the processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed at will.

[0079] Furthermore, the specific forms of distribution and integration of the components of each device are not limited to those shown in the figures. For example, the identification unit 113 and the fraud detection unit 114 may be integrated. In other words, all or part of the components may be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. Moreover, all or any part of the processing functions of each device may be implemented by a CPU and a program that is analyzed and executed by the CPU, or by hardware using wired logic.

[0080] Furthermore, each processing function performed by each device may be implemented, in whole or in part, by a CPU and a program executed for analysis by that CPU, or by hardware using wired logic.

[0081] (Hardware) Figure 17 illustrates an example of a hardware configuration. Here, as an example, the information processing device 100 will be described. As shown in Figure 17, the information processing device 100 includes a communication device 100a, an HDD (Hard Disk Drive) 100b, memory 100c, and a processor 100d. Furthermore, each of the components shown in Figure 17 is interconnected by a bus or the like.

[0082] The communication device 100a is a network interface card or the like, and communicates with other devices. The HDD 100b stores programs and databases that operate the functions shown in Figure 3.

[0083] The processor 100d operates a process that performs the functions described in Figure 3 by reading a program that performs the same processing as each processing unit shown in Figure 3 from the HDD 100b or the like and loading it into memory 100c. For example, this process performs the same functions as each processing unit of the information processing device 100. Specifically, the processor 100d reads a program that has the same functions as the machine learning unit 111, the video acquisition unit 112, the identification unit 113, the fraud detection unit 114, the warning control unit 115, etc. from the HDD 100b or the like. Then, the processor 100d executes a process that performs the same processing as the machine learning unit 111, the video acquisition unit 112, the identification unit 113, the fraud detection unit 114, the warning control unit 115, etc.

[0084] Thus, the information processing device 100 operates as an information processing device that executes an information processing method by reading and executing a program. Furthermore, the information processing device 100 can also achieve the same functionality as the above-described embodiment by reading the program from a recording medium using a media reader and executing the read program. It should be noted that the program referred to in this other embodiment is not limited to being executed by the information processing device 100. For example, the above embodiment may also be applied similarly when another computer or server executes the program, or when they collaborate to execute the program.

[0085] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), or DVD (Digital Versatile Disc), and executed by being read from the recording medium by a computer.

[0086] Figure 18 illustrates an example of the hardware configuration of the self-checkout machine 50. As shown in Figure 18, the self-checkout machine 50 has a communication interface 400a, an HDD 400b, memory 400c, a processor 400d, an input device 400e, and an output device 400f. Furthermore, each of the components shown in Figure 18 is interconnected by a bus or the like.

[0087] The communication interface 400a is a network interface card or similar device that communicates with other information processing devices. The HDD 400b stores the programs and data that operate the various functions of the self-checkout machine 50.

[0088] The processor 400d is a hardware circuit that operates the processes that execute each function of the self-checkout machine 50 by reading programs that perform the processing of each function of the self-checkout machine 50 from the HDD 400b or other source and loading them into memory 400c. In other words, this process performs the same functions as the processing units of the self-checkout machine 50.

[0089] Thus, the self-checkout register 50 operates as an information processing device that performs operation control processing by reading and executing programs that perform the processing of each function of the self-checkout register 50. Furthermore, the self-checkout register 50 can also realize each function of the self-checkout register 50 by reading a program from a recording medium using a media reader and executing the read program. It should be noted that the programs referred to in these other embodiments are not limited to those executed by the self-checkout register 50. For example, this embodiment may also be applied to cases where another computer or server executes a program, or where these devices collaborate to execute a program.

[0090] Furthermore, the programs that execute the processing of each function of the Self-Checkout 50 can be distributed via networks such as the Internet. These programs can also be recorded on computer-readable storage media such as hard disks, floppy disks, CD-ROMs, MOs, and DVDs, and executed by being read from these media by a computer.

[0091] The input device 400e detects various user input operations, such as input operations to a program executed by the processor 400d. These input operations include, for example, touch operations. In the case of touch operations, the self-checkout 50 further includes a display unit, and the input operation detected by the input device 400e may be a touch operation on the display unit. The input device 400e may be, for example, a button, a touch panel, or a proximity sensor. The input device 400e also reads barcodes. The input device 400e may be, for example, a barcode reader. The barcode reader has a light source and a light sensor and scans barcodes.

[0092] The output device 400f outputs data from a program executed by the processor 400d via an external device connected to the self-checkout register 50, such as an external display device. Note that if the self-checkout register 50 is equipped with a display unit, it does not need to be equipped with the output device 400f. [Explanation of symbols]

[0093] 30 Cameras 50 Self-checkout 60 Administrator terminals 100 Information Processing Devices 101 Communications Department 102 Storage section 103 Training Data Database 104 Machine Learning Models 105 Video Data Database 110 Control Unit 111 Machine Learning Department 112 Video Acquisition Unit 113 Identification Unit 114 Fraud Detection Unit 115 Warning Control Unit

Claims

1. On the computer, We obtained video data of a person scanning the product code into the checkout machine. By analyzing the acquired video data, the overall area of ​​the product and the location of the code on the product are identified from the video data. Based on the location of the code of the product within the entire area of ​​the identified product, an alert is generated indicating an anomaly in the act of registering the product in the accounting machine. An information processing program characterized by executing a process.

2. The aforementioned identification process is, By analyzing the acquired video data, the surface pattern of the product in the two-dimensional image is recognized. Identify the location of the code for the aforementioned pattern, The aforementioned generation process is, When the location of the code for the identified pattern differs from the location pre-set for each pattern, an alert is generated indicating an abnormality in the action of registering the product in the accounting machine. The information processing program according to feature 1.

3. The aforementioned identification process is, By inputting the acquired video data into a machine learning model, a 3D model of the product is generated. The process that generates the aforementioned alert is: If the position of the code in the generated 3D model differs from a pre-set position, an alert is generated indicating an anomaly in the action of registering the product in the accounting machine. The information processing program according to feature 1.

4. The aforementioned identification process is, Multiple products, each having its own code, are set as a single product, and the entire area of ​​the set product having the code of the single product, and the location of the code of the set product are identified, respectively. The aforementioned generation process is, The alert is generated when the code of any of the products in the set of products is located at the position for registering products in the accounting machine. The information processing program according to feature 1.

5. The aforementioned identification process is, By inputting the acquired video data into a machine learning model that has learned the location of the code for each product, the location of the code on the product in the video data is estimated. The aforementioned generation process is, The alert is generated when the location of the code on the estimated product does not match the location of the code on the product located at the location where the product is registered in the accounting machine. The information processing program according to feature 1.

6. The process that generates the aforementioned alert is: As an alert related to an abnormality in the act of registering products in the aforementioned accounting machine, an alert is generated indicating that there are products that the person has not registered in the aforementioned accounting machine, or that the products that the person has registered in the aforementioned accounting machine are abnormal. The information processing program according to feature 1.

7. The information processing program according to claim 1, characterized in that when an alert is generated regarding an abnormality in the act of registering goods in the accounting machine, the computer is instructed to perform a process to notify the terminal held by the store clerk, associating the identification information of the accounting machine with the generated alert.

8. The process that generates the aforementioned alert is If an alert is generated regarding an abnormality in the act of registering products with the accounting machine, the accounting machine will output an audio or screen prompting the person located at the accounting machine to register the product. The information processing program according to feature 1.

9. When an alert is generated regarding an abnormality in the act of registering goods in the aforementioned accounting machine, the camera of the accounting machine will take a picture of the person. The information processing program according to claim 1, characterized in that it causes the computer to perform a process of associating the image data of the person that has been photographed with the alert and storing it in a memory unit.

10. The aforementioned set product has a barcode attached to both the packaging material that wraps the collection of multiple cans and the cans wrapped in the packaging material. The aforementioned generation process is, The information processing program according to claim 4, characterized in that it causes the computer to perform a process to notify the alert when the barcode of the can, rather than the barcode of the packaging material, is scanned by the accounting machine.

11. The information processing program according to claim 1, characterized in that the accounting machine is a self-checkout terminal.

12. Computers We obtained video data of a person scanning the product code into the checkout machine. By analyzing the acquired video data, the overall area of ​​the product and the location of the code on the product are identified from the video data. Based on the location of the code of the product within the entire area of ​​the identified product, an alert is generated indicating an anomaly in the act of registering the product in the accounting machine. An information processing method characterized by performing a process.

13. We obtained video data of a person scanning the product code into the checkout machine. By analyzing the acquired video data, the overall area of ​​the product and the location of the code on the product are identified from the video data. Based on the location of the code of the product within the entire area of ​​the identified product, an alert is generated indicating an anomaly in the act of registering the product in the accounting machine. An information processing device characterized by having a control unit.

Images