Information processing program, information processing method, and information processing device.

Abstract

To detect an error or a fraud of a user in usage of an accounting machine.SOLUTION: An information processor acquires picture data of a person who scans the code of an item by an accounting machine. The information processor analyzes the acquired picture data and specifies the item that the person held in the range of an area set to scan the code of the item by the accounting machine. The information processor acquires item information registered in the accounting machine by scanning the code of the item by the scanning machine. The information processor compares the acquired item information and the item held by the specified person and generates an alert related to an abnormality in an action of registering the item in the accounting machine.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing apparatus. [Background Art]

[0002] Self-checkout systems have become widespread in stores such as supermarkets and convenience stores. A self-checkout system is a POS (Point Of Sale) register system in which the user purchasing a product personally performs processing from reading the product barcode to settlement. For example, the introduction of self-checkout systems can improve labor shortages caused by population decline and curb labor costs. [Prior Art Literature] [Patent Literature]

[0003] [Patent Literature 1] Japanese Patent Laid-Open No. 2020-53019 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, with the above-mentioned technology, it is difficult to detect fraud. For example, in an accounting machine such as a self-checkout system, there are unavoidable errors caused by the user, intentional fraud, and the like, resulting in unpaid amounts and the like.

[0005] Examples of unavoidable errors include scanning omissions in which a user forgets to scan a product and moves the product from a shopping basket to a shopping bag. Examples of intentional fraud include bar code hiding, where a user pretends to scan a product while hiding only the bar code with a finger, and reading errors, for example, a case of six canned beer as a set has barcodes on both the beer box and each can, and the user mistakenly causes the machine to read the barcode of a single can.

[0006] While it is conceivable to install weight sensors or similar devices at each self-checkout counter to automatically count items and detect fraud, the cost would be excessive and impractical, especially for large stores or nationwide chains.

[0007] Furthermore, with self-checkout systems, scanning product codes and payment are left to the user, making it difficult to detect fraudulent activity. For example, even if one were to apply image recognition AI (Artificial Intelligence) to detect fraudulent activity, training the AI ​​would require a large amount of training data. However, stores such as supermarkets and convenience stores have a large variety of products, and the lifecycle of each product is short, resulting in frequent product changes. It is difficult to tune image recognition AI to match such product lifecycles, or to train new image recognition AI.

[0008] One aspect of this invention is to provide an information processing program, information processing method, and information processing device that can detect user errors and fraud in accounting machines. [Means for solving the problem]

[0009] In the first proposal, the information processing program is characterized by causing a computer to perform the following processes: acquire video data of a person scanning a product code into a checkout machine; analyze the acquired video data to identify the product held by the person within the area set for scanning the product code into the checkout machine; acquire product information registered in the checkout machine when the checkout machine scans the product code; and compare the acquired product information with the identified product held by the person to generate an alert related to an abnormality in the act of registering a product into the checkout machine. [Effects of the Invention]

[0010] According to one embodiment, user errors and fraud can be detected in an accounting machine. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 shows an example of the overall configuration of the self-checkout system according to Example 1. [Figure 2] Figure 2 illustrates an example of abnormal behavior detection according to Example 1. [Figure 3] Figure 3 is a functional block diagram showing the functional configuration of the information processing device according to Embodiment 1. [Figure 4] Figure 4 is a diagram illustrating the training data. [Figure 5] Figure 5 illustrates the machine learning process of the first machine learning model. [Figure 6] Figure 6 illustrates the machine learning process of the second machine learning model. [Figure 7] Figure 7 illustrates the machine learning process of the third machine learning model. [Figure 8] Figure 8 illustrates an example of abnormal behavior detection according to Example 1. [Figure 9] Figure 9 illustrates an example of abnormal behavior detection according to Example 1, specifically example 2. [Figure 10] Figure 10 is a diagram illustrating product detection. [Figure 11] Figure 11 illustrates an example of abnormal behavior detection 3 according to Example 1. [Figure 12] Figure 12 illustrates an example of an alert notification. [Figure 13] Figure 13 shows an example of an alert display on a self-checkout machine. [Figure 14] Figure 14 shows an example of an alert display for store employees. [Figure 15] Figure 15 is a flowchart showing the processing flow. [Figure 16] Figure 16 illustrates an example of a hardware configuration. [Figure 17] Figure 17 illustrates an example of a self-checkout hardware configuration. MODE FOR CARRYING OUT THE INVENTION

[0012] Hereinafter, embodiments of an information processing program, an information processing method and an information processing apparatus disclosed in the present application will be described in detail with reference to the drawings. It should be noted that the present invention is not limited by these embodiments. In addition, each embodiment can be appropriately combined within a consistent scope. Example

[0013] <Description of Self-checkout System> Fig. 1 is a diagram showing an example of the overall configuration of a self-checkout system 5 according to the first embodiment. As shown in Fig. 1, the self-checkout system 5 includes a camera 30, a self-checkout 50, an administrator terminal 60, and an information processing apparatus 100.

[0014] The information processing apparatus 100 is an example of a computer connected to the camera 30 and the self-checkout 50. The information processing apparatus 100 is connected to the administrator terminal 60 via the network 3 that can adopt various communication networks regardless of wired or wireless. The camera 30 and the self-checkout 50 may be connected to the information processing apparatus 100 via the network 3.

[0015] The camera 30 is an example of a camera that captures a video of an area including the self-checkout 50. The camera 30 transmits video data to the information processing apparatus 100. In the following description, video data may be referred to as "video data" or simply "video".

[0016] The video data includes a plurality of time-series image frames. Each image frame is assigned a frame number in ascending chronological order. One image frame is image data of a still image captured by the camera 30 at a certain timing. In the following description, image data may be simply referred to as "image".

[0017] Self-checkout 50 is an example of a POS system or payment machine where user 2, who purchases goods, handles everything from scanning the product's barcode to payment. For example, when user 2 moves the items to be purchased to the scanning area of ​​self-checkout 50, self-checkout 50 scans the barcodes of the items and registers them as purchased items.

[0018] As mentioned above, Self-Checkout 50 is an example of a self-checkout system where customers register their purchases (checkout process) and make payments themselves. It is also known as Self checkout, automated checkout, self-checkout machine, or self-checkout register. A barcode is a type of identifier that represents numbers or letters using the thickness of striped lines. Self-Checkout 50 can scan (read) barcodes to identify the price and type of product (e.g., food). Barcodes are just one example of codes; other two-dimensional codes such as QR (Quick Response) codes, which have the same function, can also be used.

[0019] User 2 repeatedly performs the above product registration operation, and once the product scanning is complete, operates the touch panel or other controls of the self-checkout register 50 to request payment. Upon receiving the payment request, the self-checkout register 50 displays the number of items to be purchased, the purchase amount, etc., and performs the payment process. The self-checkout register 50 stores the information of the items scanned between the time User 2 starts scanning and the time the payment request is made in its storage unit and transmits it to the information processing device 100 as self-checkout data (product information). User 2 transports the selected items to be purchased in the store to the self-checkout register using a shopping basket, shopping cart, or other transport device.

[0020] The administrator terminal 60 is an example of a terminal device used by a store manager. The administrator terminal 60 receives notifications from the information processing device 100, such as alerts indicating that fraudulent activity has occurred regarding the purchase of goods.

[0021] In this configuration, the information processing device 100 acquires video data of a person scanning a product's barcode at the self-checkout register 50. The information processing device 100 inputs the acquired video data into a first machine learning model to identify the product the person has grasped within the area set for scanning the product's barcode at the self-checkout register 50. Alternatively, the product the person has grasped can be identified using image analysis or other methods instead of a machine learning model. The information processing device 100 acquires product information registered at the self-checkout register 50 when the self-checkout register 50 scans the product's barcode, and by comparing the acquired product information with the product the person has grasped, it generates an alert related to any abnormality in the person's action of registering the product at the self-checkout register 50.

[0022] Figure 2 illustrates an example of abnormal behavior detection according to Embodiment 1. As shown in Figure 2, the information processing device 100 uses AI (Artificial Intelligence) to identify the product to be scanned from video data capturing the scanning position where the barcode of the product is read at the self-checkout register 50. At the same time, the information processing device 100 retrieves the products actually registered at the self-checkout register 50. The information processing device 100 then compares the product identified from the video data with the product retrieved from the self-checkout register 50 to achieve immediate detection of fraudulent behavior.

[0023] In other words, the information processing device 100 determines whether a different product from the one picked up by user 2 at the self-checkout register 50 has been registered at the self-checkout register 50. As a result, the information processing device 100 can detect fraud at the self-checkout register 50 by detecting the behavior of scanning a low-priced product instead of a high-priced product and pretending that the scanning of the high-priced product has been completed.

[0024] <Functional Configuration> Figure 3 is a functional block diagram showing the functional configuration of the information processing device 100 according to Embodiment 1. As shown in Figure 3, the information processing device 100 has a communication unit 101, a storage unit 102, and a control unit 110.

[0025] The communication unit 101 is a processing unit that controls communication with other devices, and is implemented, for example, by a communication interface. For example, the communication unit 101 receives video data from the camera 30 and transmits the processing result from the control unit 110 to the administrator terminal 60.

[0026] The memory unit 102 is a processing unit that stores various data and programs executed by the control unit 110, and is implemented by memory or a hard disk. The memory unit 102 stores the training data DB 103, the first machine learning model 104, the second machine learning model 105, the third machine learning model 106, and the video data DB 107.

[0027] The training data DB103 is a database that stores the training data used to train each machine learning model. For example, Figure 4 illustrates an example where a Human Object Interaction Detection (HOID) model is used for the third machine learning model 106. Figure 4 is a diagram illustrating the training data. As shown in Figure 4, each training data set consists of image data that serves as input data and correct information (labels) set for that image data.

[0028] The correct answer information includes the class of the person and object being detected, the class indicating the interaction between the person and the object, and the Bbox (Bounding Box: object region information) indicating the region of each class. For example, the correct answer information might include region information for the Something class, which indicates an object other than a shopping bag, region information for the person class, which indicates a user purchasing the product, and a relationship (grasping class) indicating the interaction between the Something class and the person class. In other words, the correct answer information includes information about an object being grasped by a person. Note that the person class is an example of the first class, the Something class is an example of the second class, the region information for the person class is an example of the first region, the region information for the Something class is an example of the second region, and the relationship between the person and the object is an example of interaction.

[0029] Furthermore, the correct answer information includes domain information for the plastic bag class, which represents the plastic bag; domain information for the human class, which represents the user using the plastic bag; and a relationship (grasping class) that shows the interaction between the plastic bag class and the human class. In other words, the correct answer information is set to include information about the plastic bag being held by the person.

[0030] Generally, when you create a Something class using standard object recognition, it detects everything unrelated to the task, such as the background, clothing, and accessories. Furthermore, since they are all categorized as Something, the image data simply contains a large number of Bboxes, and nothing useful is revealed. In the case of HOID, however, it is understood that the object has a specific relationship to a person (it may also have other relationships, such as sitting or operating it), so it can be used as meaningful information for a task (for example, a self-checkout fraud detection task). After detecting the object as Something, a shopping bag, for example, is identified as a unique class called Bag. This shopping bag is valuable information for a self-checkout fraud detection task, but not important information for other tasks. Therefore, it is valuable and effective to use it based on the unique insight of the self-checkout fraud detection task, which is that products are taken out of the basket and placed in a bag.

[0031] Returning to Figure 3, the first machine learning model 104 is an example of a machine learning model trained to identify products held by a person in the image data. For example, in response to the input of video data, the first machine learning model 104 identifies and outputs products held by a person within a set area (e.g., the scanning position) for scanning the product code at the self-checkout 50.

[0032] The second machine learning model 105 is an example of a machine learning model trained to identify attributes related to the appearance of an item being held by a person in an image file. For example, in response to video data input, the second machine learning model 105 identifies and outputs attributes related to the appearance of the item being held by the person, such as the size of the item, the shape of the item, the color of the item, the texture of the item, the weight of the item, and the number of items.

[0033] The third machine learning model 106 is an example of a machine learning model trained to identify people and objects (e.g., people and storage, people and products) in image data. Specifically, the third machine learning model 106 is a machine learning model that identifies people, products, and the relationship between people and products from input image data and outputs the identification result. For example, the third machine learning model 106 can employ a model for HOID, or it can employ a machine learning model using various neural networks. In the case of HOID, the output will be "people's class and region information, product (object)'s class and region information, and people-product interactions."

[0034] The video data DB107 is a database that stores video data captured by cameras 30 installed on self-checkout registers 50. For example, the video data DB107 stores video data for each self-checkout register 50 or for each camera 30.

[0035] The control unit 110 is the processing unit that oversees the entire information processing device 100, and is implemented by, for example, a processor. This control unit 110 includes a machine learning unit 111, an image acquisition unit 112, a fraud processing unit 113, and a warning control unit 114. The machine learning unit 111, the image acquisition unit 112, the fraud processing unit 113, and the warning control unit 114 are implemented by electronic circuits and processes executed by the processor.

[0036] (Machine Learning) The machine learning unit 111 is a processing unit that performs machine learning on each machine learning model using the training data stored in the training data DB 103.

[0037] Figure 5 illustrates the machine learning process of the first machine learning model 104. As shown in Figure 5, the machine learning unit 111 inputs training data, in which "image data" is the explanatory variable and "product" is the target variable, into the first machine learning model 104 and obtains the output result "product" from the first machine learning model 104. The machine learning unit 111 then calculates error information between the output result "product" from the first machine learning model 104 and the target variable "product". Subsequently, the machine learning unit 111 performs machine learning by updating the parameters of the first machine learning model 104 through backpropagation to reduce the error. Here, "product" includes various information that identifies the product, such as the product itself, the product name, the type or category of the product, and the product's features.

[0038] Figure 6 illustrates the machine learning process of the second machine learning model 105. As shown in Figure 6, the machine learning unit 111 inputs training data, in which "image data" is the explanatory variable and "product attributes" is the target variable, into the second machine learning model 105, and obtains the output result "product attributes" from the second machine learning model 105. The machine learning unit 111 then calculates error information between the output result "product attributes" from the second machine learning model 105 and the target variable "product attributes". Subsequently, the machine learning unit 111 performs machine learning by updating the parameters of the second machine learning model 105 through backpropagation to reduce the error.

[0039] Figure 7 illustrates the machine learning process of the third machine learning model 106. Figure 7 shows an example of using HOID in the third machine learning model 106. As shown in Figure 7, the machine learning unit 111 inputs the training data into HOID and obtains the output result of HOID. This output result includes the human class, object class, and human-object interaction detected by HOID. The machine learning unit 111 then calculates the error information between the correct information of the training data and the output result of HOID, and performs machine learning to update the parameters of HOID by backpropagation to reduce the error.

[0040] (Fraud detection process) The fraud processing unit 113 includes a product identification unit 113a, a scan information acquisition unit 113b, and a fraud detection unit 113c, and is a processing unit that detects fraudulent user behavior occurring at the self-checkout 50.

[0041] (Fraud detection process: Identification of the product) The product identification unit 113a is a processing unit that identifies the product held by a person and the attributes of the product held by the person from the video data acquired by the video acquisition unit 112. The product identification unit 113a then outputs the identified information to the fraud detection unit 113c.

[0042] For example, the product identification unit 113a identifies products held by a person within the area set for scanning the product barcode at the self-checkout 50 by inputting video data into the first machine learning model 104. In other words, the product identification unit 113a identifies products that have been taken out of the shopping basket and held up to the scanning position, products that are held up at the scanning position, or products that are placed around the scanning position.

[0043] Furthermore, the product identification unit 113a uses a third machine learning model 106 to identify image data from each image data within the video data that captures the state in which the product is located at the scan location. The product identification unit 113a can then input the image data capturing the state in which the product is located at the scan location into the first machine learning model 104, thereby identifying the product that a person is holding at the scan location.

[0044] Furthermore, the product identification unit 113a identifies attributes related to the appearance of the product held by the person by inputting the video data into the second machine learning model 105. Specifically, the product identification unit 113a identifies the size, shape, color, texture, weight, and quantity of the product that was taken out of the shopping basket and held up to the scanning position. In other words, the product identification unit 113a identifies the external characteristics of the held product.

[0045] (Fraud detection process: scan information) The scan information acquisition unit 113b is a processing unit that acquires product information registered in the self-checkout register 50 when the self-checkout register 50 scans the barcode of a product. For example, the scan information acquisition unit 113b acquires product information such as product name, price, and product attributes (e.g., product size, product shape, product color, product texture, product weight, product quantity, etc.).

[0046] Furthermore, the scan information acquisition unit 113b can also estimate product attributes from product names, prices, etc., and can also estimate the attributes of other products from product names, prices, and the attributes of at least one product. For example, the scan information acquisition unit 113b can estimate product attributes using a table that associates product names, prices, and product attributes, or a machine learning model that estimates product attributes from product names, prices, etc.

[0047] Furthermore, the scan information acquisition unit 113b can acquire not only scan information registered in the self-checkout register 50 when a barcode is scanned, but also, for example, product information selected and registered on the touch panel of the self-checkout register 50 as scan information.

[0048] (Fraud detection process: Fraud detected) The fraud detection unit 113c is a processing unit that detects abnormalities (fraud) in the act of registering products in the self-checkout register 50 by comparing product information acquired by the scan information acquisition unit 113b with the product held by the person identified by the product identification unit 113a. For example, the fraud detection unit 113c detects that fraudulent activity has occurred when the products do not match and notifies the warning control unit 114 of information regarding the self-checkout register 50 where the fraud occurred and information about the fraud. Here, a specific example of fraud detection will be explained.

[0049] (Fraud detection process: Specific example 1) Specific example 1 is a fraud detection method based on image data analysis and scan information. Figure 8 is a diagram illustrating example 1 of abnormal behavior detection according to Example 1. As shown in Figure 8, the fraud processing unit 113 acquires image data captured by the camera 30 and inputs the acquired image data into the first machine learning model 104 to identify the product grasped by the person at the scan location. Meanwhile, the fraud processing unit 113 acquires information on the product that was actually scanned from the self-checkout 50. The fraud processing unit 113 then compares the two products and detects fraudulent behavior if the products do not match.

[0050] (Fraud detection process: Specific example 2) Specific Example 2 is a method for more robustly detecting fraudulent behavior by performing a detailed fraud detection when Specific Example 1 determines that the behavior is not fraudulent. Note that Specific Example 2 may be executed only, not just after executing Specific Example 1. Figure 9 is a diagram illustrating Specific Example 2 of abnormal behavior detection according to Specific Example 1. As shown in Figure 9, when the product matches using the method of Specific Example 1, the fraud processing unit 113 inputs the image data captured by the camera 30, which has been input to the first machine learning model 104, into the second machine learning model 105 to identify the attributes of the product grasped by the person at the scan location. Meanwhile, the fraud processing unit 113 acquires the attributes of the product actually scanned by the self-checkout 50 via the scan information acquisition unit 113b. The fraud processing unit 113 then compares the attributes of both products and detects fraudulent behavior if the product attributes do not match.

[0051] (Fraud detection process: Specific example 3) Specific example 3 is a method that aims to speed up fraud detection processing by inputting HOID (Host of Information) from each image data of the video data into the model.

[0052] Figure 10 illustrates the product detection process. Figure 10 shows the image data input to HOID and the output results of HOID. Furthermore, in Figure 10, the Bbox of a person is shown with a solid line frame, and the Bbox of an object is shown with a dashed line frame. As shown in Figure 10, the output results of HOID include the Bbox of a person, the Bbox of an object, the probability value of the interaction between the person and the object, and the class name. Referring to the Bbox of the object, the fraud processing unit 113 extracts the region of the product being held by the person by cutting out the Bbox of the object, i.e., the partial image corresponding to the dashed line frame in Figure 10, from the image data.

[0053] In other words, the fraud processing unit 113 can input image data into the HOID to detect the area of ​​a person, the area of ​​a product, the relationship between the person and the product, and the action of the person grasping the product at a known scan location. In this way, the fraud processing unit 113 can identify from among multiple image data in the video data acquired at any given time the image data in which the action of the person grasping the product at a known scan location was detected.

[0054] Figure 11 illustrates an example of abnormal behavior detection example 3 according to Example 1. Figure 11 shows image data 1 to 7, which are input data to HOID, and the HOID's detection content when image data 1 to 7 are input in order. Note that in Figure 11, the descriptions written above the image data are information captured in the image data, which is unknown information as input to HOID, and is the information that HOID is targeting for detection.

[0055] As shown in Figure 11, the fraud processing unit 113 acquires image data 1 showing a person taking an item out of a shopping basket, inputs it to the HOID, and obtains the output result. The fraud processing unit 113 then detects the area of ​​the person, the area of ​​the item, the relationship between the person and the item (grasping), and the actions of user 2 who took the item.

[0056] Next, the fraud detection unit 113 acquires image data 2 showing a person holding an item taken out of a shopping basket, inputs it to the HOID, and obtains the output result. The fraud detection unit 113 then detects the area of ​​the person, the area of ​​the item, the relationship between the person and the item (holding), and the actions of user 2 holding the item.

[0057] Next, the fraud detection unit 113 acquires image data 3 showing a person grasping the product at the scanning location, inputs it to the HOID, and obtains the output result. The fraud detection unit 113 then detects the area of ​​the person, the area of ​​the product, the relationship between the person and the product (grasping), and the actions of user 2 grasping the product at the scanning location. At this point, since "user 2 grasping the product at the scanning location" has been detected, the fraud detection unit 113 identifies image data 3 as the target for HOID determination.

[0058] The fraud processing unit 113 then inputs the identified image data 3 into the first machine learning model 104 to identify the product that the person grasped at the scanning location. Meanwhile, the fraud processing unit 113 obtains information on the product that was actually scanned from the self-checkout 50 at the same time as the image data 3 (for example, with a time difference of within a predetermined time). The fraud processing unit 113 then compares the two products and detects fraudulent activity if the products do not match. It should be noted that specific examples 3 and 2 can also be combined.

[0059] (Alert notification) The warning control unit 117 is a processing unit that generates an alert and executes alert notification control when fraudulent behavior (fraudulent operation) is detected by the fraud detection unit 116. For example, the warning control unit 117 generates an alert indicating that an item registered by a person in the self-checkout 50 is abnormal and outputs it to the self-checkout 50 and the administrator terminal 60.

[0060] For example, the warning control unit 117 displays a message on the self-checkout register 50 as an alert. Figure 12 illustrates an example of alert notification. As shown in Figure 12, the warning control unit 117 displays a message such as, "Are there any items you forgot to scan? Please scan the items again." on the touch panel or other display screen of the self-checkout register 50.

[0061] Furthermore, if the warning control unit 117 detects fraud (the so-called banana trick) in which a customer manually enters low-priced items on the self-checkout 50's touch panel without scanning high-priced items, thereby making it appear as though they have purchased high-priced items at a low price, the warning control unit 117 can also notify the self-checkout 50 of an alert with specific information.

[0062] Figure 13 shows an example of an alert display on the self-checkout register 50. Figure 13 shows the alert displayed on the self-checkout register 50 when a banana trick is detected. As shown in Figure 13, an alert window 230 is displayed on the touch panel 51 of the self-checkout register 50. This alert window 230 displays the product item "banana" registered at the register via manual input and the product item "wine" identified by image analysis of each machine learning model in a comparative manner. In addition, the alert window 230 can include a notification prompting the user to redo the corrected input. By displaying such an alert window 230, the user can be warned of the detection of a banana trick, in which "banana" is manually entered at the register instead of "wine" which should be manually entered at the register. As a result, it is possible to encourage the user to stop the settlement due to the banana trick, thereby suppressing the damage to the store caused by the banana trick. The warning control unit 117 can also output the content of the alert shown in Figure 13 as audio.

[0063] Furthermore, the warning control unit 117 may illuminate a warning light installed on the self-checkout register 50, display the identifier of the self-checkout register 50 and a message indicating the possibility of fraud on the administrator terminal 60, or send the identifier of the self-checkout register 50 and a message indicating the occurrence of fraud and the need for verification to the terminals of store employees in the store.

[0064] Figure 14 shows an example of an alert displayed to a store employee. Figure 14 shows the alert displayed on the administrator terminal 60 when a banana trick is detected. As shown in Figure 14, an alert window 250 is displayed on the administrator terminal 60. This alert window 250 displays the product item "banana" and price "350 yen" that were registered in the register via manual input, and the product item "wine" and price "4500 yen" that were identified through image analysis, in a way that allows for comparison. Furthermore, the alert window 250 displays the type of fraud "banana trick," the register number "2" where the banana trick occurred, and the estimated amount of damage caused by the banana trick settlement "4150 yen (= 4500 yen - 350 yen)." In addition, the alert window 250 displays GUI components 241-243 that accept requests such as displaying a facial photograph of user 2 using the self-checkout register 50 at register number "2," in-store announcements, or reporting to the police. The display of this alert window 240 allows for notification of the occurrence of the banana trick, assessment of the extent of the damage, and presentation of various countermeasures. As a result, it is possible to encourage user 2 to take action against the banana trick, thereby reducing the damage to stores caused by the banana trick.

[0065] Furthermore, when the warning control unit 117 generates an alert regarding an abnormality in the act of registering items at the self-checkout register 50, it causes the camera 30 on the self-checkout register 50 to photograph the person, and stores the image data of the photographed person in the memory unit in association with the alert. In this way, information on fraudulent individuals who engage in fraudulent behavior can be collected, which can be used for various measures to prevent fraudulent behavior, such as detecting customers who have engaged in fraudulent behavior at the store entrance. In addition, the warning control unit 117 can generate a machine learning model using supervised learning with the image data of fraudulent individuals, enabling it to detect fraudulent individuals from the image data of people using the self-checkout register 50, as well as detect fraudulent individuals at the store entrance. The warning control unit 117 can also acquire and store the credit card information of individuals who have engaged in fraudulent behavior from the self-checkout register 50.

[0066] <Processing flow> Figure 15 is a flowchart showing the processing flow. As shown in Figure 15, the information processing device 100 acquires video data as it progresses (S101).

[0067] Next, when the information processing device 100 is instructed to start processing (S102: Yes), it acquires frames from the video data (S103). At this point, if no video data exists, the information processing device 100 terminates processing. On the other hand, if video data exists, the information processing device 100 uses a third machine learning model 106 to identify areas of people, product areas, etc. (S104).

[0068] Here, if the information processing device 100 does not detect an area of ​​a product at the scan location (S105: No), it repeats steps S103 onwards. On the other hand, if the information processing device 100 detects an area of ​​a product at the scan location (S105: Yes), it inputs the frame (image data) in which the area of ​​a product was detected at the scan location into the first machine learning model 104 to identify the product at the scan location (S106).

[0069] The information processing device 100 then acquires scan information from the self-checkout register 50 and identifies the scanned product (S107), and compares the product identified from the scan information with the product identified by the first machine learning model 104 (S108).

[0070] Subsequently, the information processing device 100 issues an alert (S109) if the compared products do not match (S108: No), and terminates the process if the compared products match (S108: Yes).

[0071] <Effects> As described above, the information processing device 100 acquires video data of a person scanning a product code at the self-checkout register 50. The information processing device 100 inputs the acquired video data into a first machine learning model to identify the product the person has grasped within the area set for scanning the product code at the self-checkout register 50. The information processing device 100 acquires product information registered at the self-checkout register 50 when the self-checkout register 50 scans the product code, and detects fraudulent activity by comparing the acquired product information with the product the person has grasped. As a result, the information processing device 100 can detect fraud at the self-checkout register 50. Furthermore, since the information processing device 100 can detect fraud without performing complex processing, the fraud detection process can be accelerated.

[0072] Furthermore, the information processing device 100 detects fraudulent activity by comparing the attributes of the products. As a result, the information processing device 100 can detect fraud in two stages, thus reducing the likelihood of missing fraudulent activity even if the activity cannot be determined as fraudulent based on the product alone, or if unknown fraudulent activities that have not yet occurred occur.

[0073] Furthermore, when an alert is generated regarding an abnormality in the act of registering products at the self-checkout register 50, the information processing device 100 uses the camera on the self-checkout register 50 to photograph the person, and stores the image data of the photographed person in association with the alert in the memory unit. Therefore, the information processing device 100 can collect and store information on fraudulent individuals who engage in fraudulent activities, and by detecting the presence of fraudulent individuals from the image data captured by the camera that photographs customers, it can be used to implement various measures to prevent fraudulent activities.

[0074] Furthermore, the information processing device 100 can also obtain and store the credit card information of the person who committed the fraudulent act from the self-checkout 50, so that if fraudulent activity is confirmed, the charges can be billed through the credit card company.

[0075] Furthermore, since the information processing device 100 uses HOID to identify when a product is located at the scanning position, it can narrow down the image data used to identify the product, thereby enabling the detection of fraud at the self-checkout 50 in terms of both speeding up processing and improving accuracy.

[0076] Furthermore, the information processing device 100 generates an alert indicating that there are items that the person has not registered in the self-checkout register 50, or that the items the person has registered in the self-checkout register 50 are abnormal. Therefore, by using the information processing device 100, store employees can take measures such as questioning the person who has committed fraudulent acts before they leave the store.

[0077] Furthermore, if the information processing device 100 generates an alert regarding an abnormality in the action of registering items at the self-checkout register 50, it outputs an audio or screen message from the self-checkout register 50 to the person located at the self-checkout register 50 prompting them to register items that have been missed. Therefore, the information processing device 100 can directly alert the person scanning items, whether it is an unavoidable mistake or intentional fraud, thereby reducing mistakes and intentional fraud. [Examples]

[0078] Now, although embodiments of the present invention have been described, the present invention may be implemented in various other forms besides those described above.

[0079] (Numerical values, etc.) The number of self-checkout machines and cameras used in the above examples, numerical examples, training data examples, number of training data points, machine learning models, class names, number of classes, and data formats are merely examples and can be changed as needed. Furthermore, the processing flow described in each flowchart can be modified as appropriate within a consistent range. Additionally, each model can be one generated using various algorithms, such as neural networks.

[0080] Furthermore, the information processing device 100 can also use known technologies such as other machine learning models for detecting location, object detection technologies, and location detection technologies to determine the scan location and the location of the shopping basket. For example, the information processing device 100 can detect the location of the shopping basket based on the difference between frames (image data) and the time series changes of the frames, so it may use this method for detection, or it may use this method to generate a different model. In addition, by pre-specifying the size of the shopping basket, the information processing device 100 can also identify the location of the shopping basket when an object of that size is detected from the image data. Since the scan location is a somewhat fixed location, the information processing device 100 can also identify a location specified by an administrator or the like as the scan location.

[0081] (system) Unless otherwise specified, the processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed at will.

[0082] Furthermore, the specific forms of distribution and integration of the components of each device are not limited to those shown in the diagram. For example, the fraud detection unit 113 and the warning control unit 114 may be integrated. In other words, all or part of the components may be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. Moreover, all or any part of the processing functions of each device may be implemented by a CPU and a program that is analyzed and executed by the CPU, or as hardware using wired logic.

[0083] Furthermore, each processing function performed by each device may be implemented, in whole or in part, by a CPU and a program executed for analysis by that CPU, or by hardware using wired logic.

[0084] (Hardware) Figure 16 illustrates an example of a hardware configuration. Here, as an example, the information processing device 100 will be described. As shown in Figure 16, the information processing device 100 includes a communication device 100a, an HDD (Hard Disk Drive) 100b, memory 100c, and a processor 100d. Furthermore, each of the parts shown in Figure 16 is interconnected by a bus or the like.

[0085] The communication device 100a is a network interface card or the like, and communicates with other devices. The HDD 100b stores programs and databases that operate the functions shown in Figure 3.

[0086] The processor 100d operates a process that performs the functions described in Figure 3 by reading a program that performs the same processing as each processing unit shown in Figure 3 from the HDD 100b or the like and loading it into memory 100c. For example, this process performs the same functions as each processing unit of the information processing device 100. Specifically, the processor 100d reads a program that has the same functions as the machine learning unit 111, the video acquisition unit 112, the fraud processing unit 113, the warning control unit 114, etc., from the HDD 100b or the like. Then, the processor 100d executes a process that performs the same processing as the machine learning unit 111, the video acquisition unit 112, the fraud processing unit 113, the warning control unit 114, etc.

[0087] Thus, the information processing device 100 operates as an information processing device that executes an information processing method by reading and executing a program. Furthermore, the information processing device 100 can also achieve the same functionality as the above-described embodiment by reading the program from a recording medium using a media reader and executing the read program. It should be noted that the program referred to in this other embodiment is not limited to being executed by the information processing device 100. For example, the above embodiment may also be applied similarly when another computer or server executes the program, or when they collaborate to execute the program.

[0088] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), or DVD (Digital Versatile Disc), and executed by being read from the recording medium by a computer.

[0089] Figure 17 illustrates an example of the hardware configuration of the self-checkout machine 50. As shown in Figure 17, the self-checkout machine 50 has a communication interface 400a, an HDD 400b, memory 400c, a processor 400d, an input device 400e, and an output device 400f. Furthermore, each of the components shown in Figure 17 is interconnected by a bus or the like.

[0090] The communication interface 400a is a network interface card or similar device that communicates with other information processing devices. The HDD 400b stores the programs and data that operate the various functions of the self-checkout machine 50.

[0091] The processor 400d is a hardware circuit that operates the processes that execute each function of the self-checkout machine 50 by reading programs that perform the processing of each function of the self-checkout machine 50 from the HDD 400b or other source and loading them into memory 400c. In other words, this process performs the same functions as the processing units of the self-checkout machine 50.

[0092] Thus, the self-checkout register 50 operates as an information processing device that performs operation control processing by reading and executing programs that perform the processing of each function of the self-checkout register 50. Furthermore, the self-checkout register 50 can also realize each function of the self-checkout register 50 by reading a program from a recording medium using a media reader and executing the read program. It should be noted that the programs referred to in these other embodiments are not limited to those executed by the self-checkout register 50. For example, this embodiment may also be applied to cases where another computer or server executes a program, or where these devices collaborate to execute a program.

[0093] Furthermore, the programs that execute the processing of each function of the Self-Checkout 50 can be distributed via networks such as the Internet. These programs can also be recorded on computer-readable storage media such as hard disks, floppy disks, CD-ROMs, MOs, and DVDs, and executed by being read from these media by a computer.

[0094] The input device 400e detects various user input operations, such as input operations to a program executed by the processor 400d. These input operations include, for example, touch operations. In the case of touch operations, the self-checkout 50 further includes a display unit, and the input operation detected by the input device 400e may be a touch operation on the display unit. The input device 400e may be, for example, a button, a touch panel, or a proximity sensor. The input device 400e also reads barcodes. The input device 400e may be, for example, a barcode reader. The barcode reader has a light source and a light sensor and scans barcodes.

[0095] The output device 400f outputs data from a program executed by the processor 400d via an external device connected to the self-checkout register 50, such as an external display device. Note that if the self-checkout register 50 is equipped with a display unit, it does not need to be equipped with the output device 400f. [Explanation of Symbols]

[0096] 30 Cameras 50 Self-checkout 60 Administrator terminals 100 Information Processing Devices 101 Communications Department 102 Storage section 103 Training Data Database 104 The First Machine Learning Model 105 The Second Machine Learning Model 106 The Third Machine Learning Model 107 Video Data Database 110 Control Unit 111 Machine Learning Department 112 Video Acquisition Unit 113 Illegal Processing Unit 113a Product Specification Department 113b Scan information acquisition unit 113c Fraud detection unit 114 Warning Control Unit

Claims

1. On the computer, We obtained video data of a person scanning the product code into the checkout machine. Based on the first region containing the person's hand and the second region containing the product, it is determined that the person was grasping the product. By analyzing the video data in which it is identified that the person grasped the product, the product grasped by the person is identified within the area set for scanning the product code at the checkout machine. The accounting machine obtains product information registered in the accounting machine by scanning the code of the product. By comparing the acquired product information with the product held by the identified person, an alert is generated related to an anomaly in the act of registering the product in the accounting machine. An information processing program characterized by executing a process.

2. The process of identifying the goods held by the aforementioned person is as follows: By inputting the acquired video data into a first machine learning model, the system identifies the product held by the person within the area set for scanning the product code at the checkout machine. The information processing program according to feature 1.

3. The process of identifying the goods held by the aforementioned person is as follows: By inputting the acquired video data into a second machine learning model, the attributes related to the appearance of the product held by the person are identified. The process that generates the aforementioned alert is: Based on the product information obtained from the accounting machine, the attributes of the product are estimated. The system generates the alert when the estimated attributes of the product differ from the attributes of the product held by the person identified using the second machine learning model. The information processing program according to feature 2.

4. The process of identifying that the person has grasped the product is: By inputting the acquired video data into a third machine learning model, the relationship between the first region, the second region, and the relationship between the first region and the second region is identified. Based on the identified first area, the second area, and the relationship, it is determined that the person was holding the goods. The process to be identified is, By inputting the video data in which the person has been identified as grasping the product using the third machine learning model into the first machine learning model, the product grasped by the person within the area is identified. The information processing program according to feature 2.

5. The process that generates the aforementioned alert is: As an alert related to an abnormality in the act of registering products in the aforementioned accounting machine, an alert is generated indicating that there are products that the person has not registered in the aforementioned accounting machine, or that the products that the person has registered in the aforementioned accounting machine are abnormal. The information processing program according to feature 1.

6. The information processing program according to claim 1, characterized in that when an alert is generated regarding an abnormality in the act of registering goods in the accounting machine, the computer is instructed to perform a process to notify a terminal held by a store employee, associating the identification information of the accounting machine with the generated alert.

7. The process that generates the aforementioned alert is If an alert is generated regarding an abnormality in the act of registering products with the accounting machine, the accounting machine will output an audio or screen prompting the person located at the accounting machine to register the products. The information processing program according to feature 1.

8. When an alert is generated regarding an abnormality in the act of registering goods in the aforementioned accounting machine, the camera of the accounting machine will take a picture of the person. The information processing program according to claim 1, characterized in that it causes the computer to perform a process of associating the image data of the person that has been photographed with the alert and storing it in a memory unit.

9. By inputting the acquired video data into the third machine learning model, the computer is made to perform a process to identify the first region, the second region, and the relationship between them. The information processing program according to claim 4, characterized in that the third machine learning model is a Human Object Interaction Detection (HOID) model in which machine learning is performed to identify a first class indicating a person purchasing a product and first region information indicating a region in which the person appears, a second class indicating an object containing a product and second region information indicating a region in which the object appears, and the interaction between the first class and the second class.

10. The process that generates the aforementioned alert is: When the acquired product information does not match the product held by the identified person, an alert is generated related to the abnormality of the act of registering the product in the accounting machine. The information processing program according to feature 1.

11. The information processing program according to claim 1, characterized in that the accounting machine is a self-checkout terminal.

12. Computers We obtained video data of a person scanning the product code into the checkout machine. Based on the first region containing the person's hand and the second region containing the product, it is determined that the person was grasping the product. By analyzing the video data in which it is identified that the person grasped the product, the product grasped by the person is identified within the area set for scanning the product code at the checkout machine. The accounting machine obtains product information registered in the accounting machine by scanning the code of the product. By comparing the acquired product information with the product held by the identified person, an alert is generated related to an anomaly in the act of registering the product in the accounting machine. An information processing method characterized by performing a process.

13. We obtained video data of a person scanning the product code into the checkout machine. Based on the first region containing the person's hand and the second region containing the product, it is determined that the person was grasping the product. By analyzing the video data in which it is identified that the person grasped the product, the product grasped by the person is identified within the area set for scanning the product code at the checkout machine. The accounting machine obtains product information registered in the accounting machine by scanning the code of the product. By comparing the acquired product information with the product held by the identified person, an alert is generated related to an anomaly in the act of registering the product in the accounting machine. An information processing device characterized by having a control unit.

Images