Information processing program, information processing method, and information processing device

Abstract

To detect a fraud in a self register.SOLUTION: An information processor acquires image data obtained by photographing a predetermined area in front of an accounting machine for registering and accounting a commodity by a user himself / herself. The information processor inputs the image data to a machine learning model trained so as to identify the commodity and a storage for storing the commodity, and acquires an output result. The information processor identifies an action for the commodity of the user using the commodity and the storage included in the output result.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] Self-checkout registers are becoming common in supermarkets, convenience stores, and other stores. A self-checkout register is a Point of Sale (POS) cash register system in which the user himself performs all the steps from scanning the product's barcode to paying. For example, introducing self-checkout registers can help alleviate labor shortages caused by population decline and prevent labor costs from being reduced. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-53019 Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is difficult to detect fraud using the above technology. For example, at self-checkout registers, users may commit unavoidable errors or intentional fraud, resulting in unpaid bills.

[0005] Examples of unavoidable mistakes include forgetting to scan an item and moving it from the basket to a shopping bag, misreading the barcode on the can, for example, when a beer box containing six cans has a barcode on both the box and each can, and intentional fraud such as hiding the barcode, where a user pretends to scan an item while covering only the barcode with their finger.

[0006] It is possible to automatically count the number of items and detect fraud by installing weight sensors at each self-checkout register, but this would be too costly and unrealistic, especially for large stores or stores with nationwide operations.

[0007] In one aspect, an object of the present invention is to provide an information processing program, an information processing method, and an information processing device that can detect fraud at self-checkout registers. [Means for solving the problem]

[0008] In the first proposal, the information processing program causes a computer to acquire image data of a specified area in front of a cash register where a user registers products and performs checkout, input the image data into a machine learning model trained to identify products and storage units in which the products are stored, obtain output results, and use the products and storage units included in the output results to identify the user's behavior toward the products. [Effects of the Invention]

[0009] According to one embodiment, fraud at self-checkouts can be detected. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a self-checkout system according to a first embodiment. [Figure 2] FIG. 2 is a functional block diagram of the information processing apparatus according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating the training data. [Figure 4] FIG. 4 is a diagram illustrating machine learning of a machine learning model. [Figure 5] FIG. 5 is a diagram illustrating behavior identification using HOID. [Figure 6] FIG. 6 is a diagram illustrating an example of counting up purchased products. [Figure 7]FIG. 7 is a diagram illustrating fraud detection. [Figure 8] FIG. 8 is a flowchart illustrating the flow of the self-checkout process according to the first embodiment. [Figure 9] FIG. 9 is a flowchart showing the flow of the product number counting process. [Figure 10] FIG. 10 is a diagram illustrating behavior identification using a machine learning model. [Figure 11] FIG. 11 is a diagram illustrating behavior classification using a combination of HOID and a machine learning model. [Figure 12] FIG. 12 is a diagram illustrating an example of a hardware configuration. [Figure 13] FIG. 13 is a diagram illustrating an example of the hardware configuration of a self-checkout register. DETAILED DESCRIPTION OF THE INVENTION

[0011] The following describes in detail embodiments of the information processing program, information processing method, and information processing device disclosed herein with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. Furthermore, the embodiments can be combined as appropriate within a consistent range. [Example]

[0012] [Overall configuration] 1 is a diagram illustrating an example of the overall configuration of a self-checkout system 5 according to Example 1. As illustrated in FIG. 1, the self-checkout system 5 includes a camera 30, a self-checkout 50, an administrator terminal 60, and an information processing device 100.

[0013] The information processing device 100 is an example of a computer connected to the camera 30 and the self-checkout 50. The information processing device 100 is connected to the administrator terminal 60 via a network 3 which can employ various communication networks, whether wired or wireless. The camera 30 and the self-checkout 50 may be connected to the information processing device 100 via the network 3.

[0014] Camera 30 is an example of a camera that captures video of an area including self-checkout 50. Camera 30 transmits video data to information processing device 100. In the following description, video data may be referred to as "video data."

[0015] The video data includes multiple image frames in chronological order. Each image frame is assigned a frame number in ascending chronological order. One image frame is image data of a still image captured by the camera 30 at a certain timing.

[0016] The self-checkout register 50 is an example of a POS register system or accounting machine that allows a user 2 purchasing a product to perform operations from reading the product's barcode to paying. For example, when the user 2 moves the product to be purchased into the scanning area of ​​the self-checkout register 50, the self-checkout register 50 scans the product's barcode and registers it as the product to be purchased.

[0017] User 2 repeatedly performs the above-described product registration operation, and when the scanning of the products is complete, he or she operates the touch panel or the like of the self-register 50 to request payment. When the self-register 50 accepts the payment request, it presents the number of products to be purchased, the purchase amount, etc., and executes the payment process. The self-register 50 stores information about the products scanned by User 2 from the time that User 2 starts scanning until the time that User 2 requests payment in a memory unit, and transmits this information to the information processing device 100 as self-register data (product information).

[0018] The manager terminal 60 is an example of a terminal device used by a store manager. The manager terminal 60 receives, from the information processing device 100, an alert notification or the like indicating that fraud has occurred in relation to the purchase of a product.

[0019] In this configuration, the information processing device 100 acquires image data of a predetermined area in front of the self-checkout register 50 where the user 2 registers products and performs checkout. The information processing device 100 then inputs the image data into a machine learning model trained to distinguish between products and storage (such as a plastic bag) for storing the products, and acquires an output result. The information processing device 100 then uses the products and storage included in the output result to identify the user's behavior toward the products.

[0020] That is, the information processing device 100 detects products and the user 2's interaction with the products (for example, the act of holding the products), and counts the number of products removed from the shopping cart, the number of products that pass through the scanning position of the self-checkout, or the number of products that are placed in a shopping bag.The information processing device 100 then compares the counted number of products with the number of products scanned by the self-checkout 50, and detects fraudulent purchases of products.

[0021] As a result, the information processing device 100 does not require the introduction of a weight sensor or the like, so that the introduction cost can be reduced and fraud at self-checkout registers can be detected.

[0022] [Function Configuration] 2 is a functional block diagram illustrating a functional configuration of the information processing device 100 according to the first embodiment. As illustrated in FIG. 2, the information processing device 100 includes a communication unit 101, a storage unit 102, and a control unit 110.

[0023] The communication unit 101 is a processing unit that controls communication with other devices, and is realized by, for example, a communication interface, etc. For example, the communication unit 101 receives video data from the camera 30 and transmits the processing result by the control unit 110 to the administrator terminal 60.

[0024] The storage unit 102 is a processing unit that stores various data and programs executed by the control unit 110, and is realized by a memory, a hard disk, etc. The storage unit 102 stores a training data DB 103, a machine learning model 104, a video data DB 105, and a self-checkout data DB 106.

[0025] The training data DB 103 is a database that stores data used for training the machine learning model 104. For example, an example in which HOID (Human Object Interaction Detection) is adopted in the machine learning model 104 will be described with reference to FIG. 3. FIG. 3 is a diagram illustrating training data. As shown in FIG. 3, each training data has image data that serves as input data and correct answer information set for the image data.

[0026] The correct answer information includes the classes of the person and object to be detected, a class indicating the interaction between the person and the object, and a Bbox (Bounding Box: object area information) indicating the area of ​​each class. For example, the correct answer information includes area information of a Something class indicating an object such as a product other than a plastic bag, area information of a human class indicating a user who purchases the product, and a relationship (grasping class) indicating the interaction between the Something class and the human class. That is, the correct answer information includes information about an object being held by a person. The human class is an example of a first class, the Something class is an example of a second class, the area information of the human class is an example of first area information, the area information of the Something class is an example of second area information, and the interaction between a person and an object is an example of an interaction.

[0027] Furthermore, as the correct answer information, area information of the plastic bag class indicating the plastic bag, area information of the human class indicating the user who uses the plastic bag, and a relationship (holding class) indicating the interaction between the plastic bag class and the human class are set. That is, information about the plastic bag being held by a person is set as the correct answer information.

[0028] Generally, creating a Something class using standard object identification (object recognition) will result in detecting all backgrounds, clothing, accessories, and other items unrelated to the task. Furthermore, because these are all Somethings, all that is identified is a large number of B boxes in the image data, and nothing else is known. With HOID, it is clear that these are objects held by a person, which is a special relationship (there may also be other relationships, such as sitting or operating the object), and this information can be used as meaningful information for a task (for example, a fraud detection task at a self-checkout). After detecting an object as Something, a plastic bag is identified as a unique class called Bag. While this plastic bag is valuable information for the fraud detection task at a self-checkout, it is not important information for other tasks. Therefore, it is valuable to use it based on the unique knowledge of the fraud detection task at a self-checkout, in which items are removed from a shopping cart and placed in a bag, and useful results can be obtained.

[0029] Returning to FIG. 2, machine learning model 104 is an example of a machine learning model trained to distinguish between products shown in training data and the containers (such as plastic bags) that store the products. Specifically, machine learning model 104 is a machine learning model that distinguishes between people, products, and relationships between people and products from input image data and outputs the distinction results. For example, machine learning model 104 can employ HOID, or can also employ machine learning models that use various neural networks. In the case of HOID, "human class and area information, product (object) class and area information, and interaction between people and products" are output.

[0030] The video data DB 105 is a database that stores video data captured by the cameras 30 installed at the self-checkout registers 50. For example, the video data DB 105 stores video data for each self-checkout register 50 or for each camera 30.

[0031] The self-checkout data DB 106 is a database that stores various data acquired from the self-checkout 50. For example, the self-checkout data DB 106 stores, for each self-checkout 50, the number of products registered as items to be purchased, the billing amount, which is the total price of all items to be purchased, and the like.

[0032] The control unit 110 is a processing unit that controls the entire information processing device 100, and is realized by, for example, a processor. The control unit 110 has a machine learning unit 111, an image acquisition unit 112, a fraud detection unit 113, and a warning unit 114. The machine learning unit 111, the image acquisition unit 112, the fraud detection unit 113, and the warning unit 114 are realized by electronic circuits included in the processor, processes executed by the processor, etc.

[0033] The machine learning unit 111 is a processing unit that executes machine learning for the machine learning model 104 using each piece of training data stored in the training data DB 103. FIG. 4 is a diagram illustrating machine learning for the machine learning model 104. FIG. 4 illustrates an example in which HOID is used for the machine learning model 104. As shown in FIG. 4, the machine learning unit 111 inputs training data input data into HOID and obtains the output result of HOID. This output result includes the human class, object class, and interaction between the human and object detected by HOID. The machine learning unit 111 then calculates error information between the correct answer information for the training data and the output result of HOID, and executes machine learning for HOID by error backpropagation so as to reduce the error.

[0034] The video acquisition unit 112 is a processing unit that acquires video data from the camera 30. For example, the video acquisition unit 112 acquires video data from the camera 30 installed in the self-checkout 50 at any time, and stores the data in the video data DB 105.

[0035] The fraud detection unit 113 is a processing unit that detects fraud, such as forgetting to scan a product, based on video data captured around the self-checkout 50. Specifically, the fraud detection unit 113 acquires image data captured of a predetermined area in front of the self-checkout 50 where the user 2 registers the product and performs the checkout. The fraud detection unit 113 then inputs the image data into the machine learning model 104 to obtain an output result, and identifies the behavior of the user 2 regarding the product using the product and the shopping bag included in the output result.

[0036] For example, the fraud detection unit 113 obtains "person class and area information, product (object) class and area information, and human-product interaction" from the HOID output results. The fraud detection unit 113 then counts the number of products for which a specific behavior, such as holding (interacting with) by user 2 (human), has taken place. The fraud detection unit 113 then compares the counted number of products with the number of scans (number of registered products) scanned and registered in the self-checkout 50, and if there is a difference between the two, it detects fraud and outputs the result to the warning unit 114.

[0037] FIG. 5 is a diagram illustrating behavior identification using HOID. As shown in FIG. 5, the fraud detection unit 113 inputs each image data included in the video data into HOID and obtains the output result of HOID. As described above, the output result of HOID includes the Bbox of the person, the Bbox of the object, the probability value and class name of the interaction between the person and the object, etc. Then, based on the output result of HOID, the fraud detection unit 113 counts the number of products that were the target of the behavior when it identifies any of the following behaviors (a), (b), and (c):

[0038] For example, as shown in (a) of Figure 5, the fraud detection unit 113 counts the number of items removed from the shopping cart by identifying (specifying) a person, an item, and the person holding an item from the output result of the HOID. In other words, when a product in the shopping cart is picked up by user 2 from multiple image data and then the picked up item is moved to the top of the shopping cart, the fraud detection unit 113 counts the product as the number of items removed. Note that the location of the shopping cart is an example of a second area, and may be specified by an administrator or the like, or may be automatically specified using another machine learning model or the like.

[0039] Furthermore, as shown in FIG. 5(b), the fraud detection unit 113 counts the number of products that have passed through the scan position of each code, such as a barcode or two-dimensional code, at the self-checkout register 50 by identifying the person, the product, and whether the person is holding the product from the output result of the HOID. In other words, the fraud detection unit 113 counts the number of products that have passed the scan position after user 2 has held the product from multiple image data. Note that the scan position is an example of a first area, and may be specified by an administrator or the like, or may be automatically specified using another machine learning model or the like.

[0040] Furthermore, as shown in FIG. 5(c), the fraud detection unit 113 counts the number of items placed in a plastic bag by user 2 by distinguishing between a person, a product, a person holding a product, a person, a plastic bag, and a person holding a plastic bag from the HOID output results. In other words, when, based on multiple image data, user 2 holds a product, user 2 holds a plastic bag, and the held product is placed in the held plastic bag, the fraud detection unit 113 counts the product as the number of items placed in the plastic bag. Note that the position of the plastic bag may be specified by an administrator or the like, or may be specified automatically using another machine learning model or the like. Furthermore, if the position of the plastic bag is fixed, the fraud detection unit 113 is not limited to the plastic bag being held, and can also count the number of items placed in the fixed plastic bag.

[0041] Here, we will explain an example in which the fraud detection unit 113 counts the number of products purchased by user 2 using video data having multiple image data (frames). Figure 6 is a diagram explaining an example of counting up purchased products. Figure 6 illustrates image data 1 to 7, which are input data to the HOID, the detection content of the HOID when image data 1 to 7 are input in order, and the number of products counted. Note that in Figure 6, the explanation written above the image data is information that appears in the image data, is unknown information as input to the HOID, and is information that is the target of detection by the HOID.

[0042] 6, the fraud detection unit 113 acquires image data 1, which does not show any people or objects, inputs it into the HOID, and obtains an output result, but determines that no people or objects have been detected. Next, the fraud detection unit 113 acquires image data 2, which shows a person holding a shopping basket, inputs it into the HOID, and detects user 2 and the shopping basket held by user 2 according to the output result.

[0043] Next, the fraud detection unit 113 acquires image data 3 of a person removing an item from the shopping cart, inputs it into the HOID, and detects the behavior of user 2 who moved the item they were holding onto the shopping cart according to the output result. Here, since the detection result corresponds to (a) in Figure 5 above, the fraud detection unit 113 counts up the number of items removed.

[0044] Next, the fraud detection unit 113 acquires image data 4 showing the person scanning the product, inputs it into the HOID, and detects the behavior of the user 2 who moved the product they were holding to the scanning position according to the output result. Here, the fraud detection unit 113 counts up the number of scan targets because the detection result corresponds to (b) in Figure 5 above.

[0045] Next, the fraud detection unit 113 acquires image data 5 of a person putting items into a plastic bag, inputs it into the HOID, and detects the behavior of user 2 putting the items into the plastic bag according to the output result. Here, since the detection result corresponds to (c) in Figure 5 above, the fraud detection unit 113 counts up the number of plastic bags put into the bag.

[0046] Next, the fraud detection unit 113 acquires image data 6 of a person removing an item from the shopping cart, inputs it into the HOID, and detects the behavior of user 2 who moved the item they were holding onto the shopping cart according to the output result. Here, since the detection result corresponds to (a) in Figure 5 above, the fraud detection unit 113 counts up the number of items removed.

[0047] Next, the fraud detection unit 113 acquires image data 7 of the person scanning the product, inputs it into the HOID, and detects the behavior of the user 2 who moved the product they were holding to the scanning position according to the output result. Here, the fraud detection unit 113 counts up the number of scan targets because the detection result corresponds to (b) in Figure 5 above.

[0048] As described above, the fraud detection unit 113 inputs each frame of video data captured from when the user 2 brings the shopping cart to the self-checkout 50 until when the user pays for the items into the HOID, obtains an output result (detection result), and performs behavior identification (behavior recognition) of the items to be counted up according to the output result. As a result, the fraud detection unit 113 can count the number of items that the user 2 intends to purchase. Note that the timing to end counting the items can be, for example, when the self-checkout 50 notifies the user of the number of registered items, or when the self-checkout 50 notifies the user of the completion of product registration.

[0049] Thereafter, the fraud detection unit 113 acquires the number of registered products registered in the self-checkout register 50 from the self-checkout register 50, and compares the acquired number of registered products with the counted number of products to detect fraud by the user 2.

[0050] Fig. 7 is a diagram illustrating fraud detection. As shown in Fig. 7, fraud detection unit 113 counts the number of products taken out (T), the number of items to be scanned (S), and the number of plastic bags put in (Q) from the video data, and determines the largest number as the number of products (N). Fraud detection unit 113 then compares the number of scans registered in self-checkout 50 (number of registered products: P) with the counted number of products (N), and detects fraud if the number of products (N) is greater than the number of scans (P).

[0051] Returning to FIG. 2, the warning unit 114 is a processing unit that outputs a predetermined warning when fraud is detected by the fraud detection unit 113. For example, the warning unit 114 displays a warning on the self-checkout register 50, notifies a store clerk near the self-checkout register 50 of the detection of fraud by the user on their smartphone, or notifies the administrator terminal 60 of information about the self-checkout register 50 and the detection of fraud. In the example of FIG. 7, the warning unit 114 outputs a warning when it detects that the number of items (N) is greater than the number of scans (P).

[0052] [Processing flow] Fig. 8 is a flowchart showing the flow of the self-checkout process according to the first embodiment. As shown in Fig. 8, each time a product is picked up, the information processing device 100 counts the number of products (S101), the self-checkout 50 counts the number of scans (S102), and if the number of products is greater than the number of scans (S103: Yes), the information processing device 100 requests a correction operation (S104) and repeats S101 and subsequent steps. The correction operation may be, for example, a rescan of the product by the user.

[0053] On the other hand, if the number of items is not greater than the number of items scanned (S103: No) and the self-checkout register 50 performs the scan process without selecting the payment process through the operation of the user 2 (S105: No), S101 and subsequent steps are executed.

[0054] On the other hand, when the checkout process is selected by the operation of the user 2, the self-checkout 50 executes the checkout process (S106). Then, the information processing device 100 compares the number of items counted up to the checkout process with the number of scanned items registered in the self-checkout 50 up to the checkout process (S107).

[0055] If the number of items is greater than the number of scans (S107: Yes), the information processing device 100 detects fraud and takes warning action (S108) and requests correction (S109). On the other hand, if the number of items is not greater than the number of scans (S107: No), the information processing device 100 ends the process.

[0056] [Counting the number of items] Next, the process of counting the number of items will be described, taking as an example an example where the information processing device 100 counts the number of items removed from a shopping cart and the number of items placed in a shopping bag.

[0057] Fig. 9 is a flowchart showing the flow of the product number counting process. As shown in Fig. 9, the information processing device 100 executes detection of people and objects from image data using HOID (S201).

[0058] Next, when a person and an object are detected (S202: Yes), the information processing device 100 determines whether the action is to take an object out of the shopping basket (S203). Here, if the action is to take an object out of the shopping basket (S203: Yes), the information processing device 100 increments the PickUpFromBasket count (S204).

[0059] Thereafter, the information processing device 100 determines whether or not the action is to put something into a plastic bag (S205), and if the action is to put something into a plastic bag (S205: Yes), it counts up the PutInBag count (S206).

[0060] If the information processing device 100 continues scanning (S207: No), it repeats S201 and subsequent steps. On the other hand, if all scanning has been completed (S207: Yes), the information processing device 100 determines the number of products to be the larger of the PickUpFromBasket count and the PutInBag count (S208).

[0061] In addition, if no person or object is detected in S202 (S202: No), the information processing device 100 executes S207 without executing S203 to S206. If in S203 the action is not to take an object out of the shopping cart (S203: No), the information processing device 100 executes S205 without executing S204. In addition, if in S205 the action is not to put an object into a plastic bag (S205: No), the information processing device 100 executes S207 without executing S206.

[0062] [effect] As described above, by using HOID, the information processing device 100 can detect items such as products and shopping bags that interact with a user (human). At this time, the information processing device 100 detects shopping bags and shopping baskets that indicate whether products are being brought in or taken out, in order to count the number of products that the user brings to the self-checkout 50 for purchase, and can accurately count the number of products purchased by the user. As a result, the information processing device 100 can detect fraud at the self-checkout. Note that fraud does not only include intentionally not scanning products, but also forgetting to scan products.

[0063] Furthermore, general object identification cannot identify objects without a large amount of learning data for each product, and because it identifies objects that do not interact with humans, it also identifies objects in the background. On the other hand, unlike general object identification, the information processing device 100 uses HOID, which identifies only objects that interact with humans, even if they are identifiable as objects. This makes it possible to identify any object as "Something" (without being influenced by the appearance of the object itself) and estimate its object region (BBox). Furthermore, because plastic bags and shopping baskets frequently appear in image data at self-checkouts and their appearance does not change as frequently as products, it is also possible to reduce the cost of collecting training data for training HOID. [Example]

[0064] In the above embodiment, behavioral identification using HOID was described, but this is not limited to this. Instead of HOID, machine learning models using neural networks or machine learning models generated by deep learning can also be used.

[0065] Fig. 10 is a diagram illustrating behavior recognition using a machine learning model. The machine learning model shown in Fig. 10 is a model trained to detect area information of objects such as people, products, shopping baskets, and plastic bags that appear in the image data in response to input image data.

[0066] In this state, the information processing device 100 inputs image data A1 (frame) into the machine learning model and obtains an output result S1 in which the position information of the person is detected. Subsequently, the information processing device 100 inputs the next acquired image data A2 into the machine learning model and obtains an output result S2 in which the position information of the person and the position information of the product are detected.

[0067] Then, the information processing device 100 calculates the difference between the output result S1 and the output result S2 and performs behavior classification. For example, when a "person" is detected in both output results and a "product" is detected as the difference, the information processing device 100 counts the product as a purchase target.

[0068] In this way, the information processing device 100 can perform behavioral identification using inter-frame differences and count the number of products. Note that the method of fraud detection thereafter is the same as in the first embodiment, and therefore detailed description thereof will be omitted. As a result, the information processing device 100 can provide a simple system using a machine learning model.

[0069] Furthermore, the information processing device 100 can perform behavior classification by combining HOID and a machine learning model. FIG. 11 is a diagram illustrating behavior classification by combining HOID and a machine learning model. The machine learning model shown in FIG. 11 is a model trained to detect area information of a shopping basket or a plastic bag that appears in image data in response to input image data. That is, this machine learning model detects area information (location information) of objects that are not identified by HOID, or detects area information of objects that are not detected by HOID.

[0070] 11, the information processing device 100 detects a Bbox of a person, a Bbox of an object, a probability value of an interaction between a person and an object, and a class name from image data using HOID, as in Example 1. Meanwhile, the information processing device 100 detects area information of a shopping basket or a plastic bag from the same image data using a machine learning model, which is an example of a detection model.

[0071] By combining these, the information processing device 100 can identify a shopping cart or a plastic bag in the output result of the HOID and then execute the same processing as in Example 1. As a result, the information processing device 100 can accurately detect the position of the shopping cart or plastic bag, thereby improving the accuracy of detecting products and interactive operations and also improving the accuracy of detecting fraud. [Example]

[0072] Although the embodiments of the present invention have been described above, the present invention may be embodied in various different forms other than the above-described embodiments.

[0073] [Numbers, etc.] The number of self-checkouts and cameras, numerical examples, training data examples, number of training data, machine learning models, class names, number of classes, data formats, etc. used in the above examples are merely examples and can be changed as desired. The process flow described in each flowchart can also be changed as appropriate within a consistent range. Each model can be generated using various algorithms, such as a neural network.

[0074] Furthermore, the information processing device 100 can use known technologies, such as other machine learning models for detecting positions, object detection technologies, and position detection technologies, to determine the scan position and the shopping cart position. For example, the information processing device 100 can detect the position of a shopping cart based on the difference between frames (image data) and changes in the frames over time. This may be used for detection, or a different model may be generated using this. Furthermore, by specifying the size of the shopping cart in advance, the information processing device 100 can identify the location of an object of that size as the location of the shopping cart when it is detected from the image data. Note that the scan position is a fixed position to a certain extent, so the information processing device 100 can also identify a position specified by an administrator or the like as the scan position.

[0075] [system] The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings may be changed arbitrarily unless otherwise specified.

[0076] Furthermore, the specific form of distribution or integration of the components of each device is not limited to that shown in the figure. For example, the video acquisition unit 112 and the fraud detection unit 113 may be integrated. In other words, all or some of the components may be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions of each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0077] [Hardware] Fig. 12 is a diagram illustrating an example of a hardware configuration. Here, an information processing device 100 will be described as an example. As shown in Fig. 12, the information processing device 100 includes a communication device 100a, an HDD (Hard Disk Drive) 100b, a memory 100c, and a processor 100d. The components illustrated in Fig. 12 are connected to each other via a bus or the like.

[0078] The communication device 100a is a network interface card or the like, and communicates with other devices. The HDD 100b stores programs and DBs that operate the functions shown in FIG.

[0079] The processor 100d reads out from the HDD 100b or the like a program that executes the same processes as the respective processing units shown in FIG. 2 and loads the program into the memory 100c, thereby operating a process that executes each function described in FIG. 2 or the like. For example, this process executes the same functions as the respective processing units of the information processing device 100. Specifically, the processor 100d reads out from the HDD 100b or the like a program that has the same functions as the machine learning unit 111, the video acquisition unit 112, the fraud detection unit 113, the warning unit 114, and the like. Then, the processor 100d executes a process that executes the same processes as the machine learning unit 111, the video acquisition unit 112, the fraud detection unit 113, the warning unit 114, and the like.

[0080] In this way, the information processing device 100 operates as an information processing device that executes an information processing method by reading and executing a program. The information processing device 100 can also realize functions similar to those of the above-described embodiment by reading the program from a recording medium using a medium reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the information processing device 100. For example, the above-described embodiment may also be applied in the same way to cases where another computer or server executes the program, or where these execute the program in cooperation with each other.

[0081] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and may be read out from the recording medium and executed by a computer.

[0082] 13 is a diagram illustrating an example of the hardware configuration of a self-checkout register 50. As shown in Fig. 13, the self-checkout register 50 has a communication interface 400a, an HDD 400b, a memory 400c, a processor 400d, an input device 400e, and an output device 400f. The components shown in Fig. 13 are connected to each other via a bus or the like.

[0083] The communication interface 400a is a network interface card or the like, and communicates with other information processing devices. The HDD 400b stores programs and data that operate each function of the self-checkout 50.

[0084] Processor 400d is a hardware circuit that reads out from HDD 400b or the like a program that executes the processing of each function of self-checkout 50 and loads it into memory 400c, thereby operating a process that executes each function of self-checkout 50. In other words, this process executes the same functions as each processing unit that self-checkout 50 has.

[0085] In this way, the self-checkout register 50 operates as an information processing device that executes operation control processing by reading and executing a program that executes the processing of each function of the self-checkout register 50. The self-checkout register 50 can also realize each function of the self-checkout register 50 by reading a program from a recording medium using a media reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the self-checkout register 50. For example, this embodiment may also be applied in a similar manner to cases where another computer or server executes the program, or where these execute the program in cooperation with each other.

[0086] The program that executes the processing of each function of self-checkout 50 can be distributed via a network such as the Internet. The program can also be recorded on a computer-readable recording medium such as a hard disk, FD, CD-ROM, MO, or DVD, and can be executed by being read from the recording medium by a computer.

[0087] The input device 400e detects various input operations by the user, such as input operations for a program executed by the processor 400d. The input operations include, for example, touch operations. In the case of touch operations, the self-checkout 50 further includes a display unit, and the input operation detected by the input device 400e may be a touch operation on the display unit. The input device 400e may be, for example, a button, a touch panel, or a proximity sensor. The input device 400e also reads barcodes. For example, the input device 400e is a barcode reader. The barcode reader has a light source and an optical sensor and scans barcodes.

[0088] The output device 400f outputs data output from the program executed by the processor 400d via an external device, such as an external display device, connected to the self-checkout 50. Note that if the self-checkout 50 has a display unit, the self-checkout 50 does not need to have the output device 400f. [Explanation of symbols]

[0089] 30 Camera 50 Self-checkout 60 Administrator terminal 100 Information processing device 101 Communications Department 102 Storage section 103 Training Data DB 104 Machine Learning Models 105 Video Data DB 106 Self-checkout data DB 110 control section 111 Machine Learning Department 112 Video acquisition unit 113 Fraud Detection Unit 114 Warning section

Claims

1. On the computer, Acquire image data of a predetermined area in front of the cashier where the user registers the product and makes the payment; inputting the image data into a machine learning model trained to identify products and storage areas for storing the products, and obtaining an output result; Identifying the user's behavior regarding the product using the product and the storage included in the output result; Execute a process to detect fraud by the user using the behavior identification result; The machine learning model is the machine learning model for HOID (Human Object Interaction Detection) on which machine learning is performed to identify a first class indicating the user who purchases the product and first area information indicating the area in which the user appears, a second class indicating an object including the product and second area information indicating the area in which the object appears, and an interaction between the first class and the second class; and a detection model on which machine learning is performed to detect area information of each object included in the image data, including objects that are not targets of identification by the machine learning model for HOID; The acquiring process includes: Input each image data into the machine learning model for the HOID to obtain an identification result for each class and each interaction, and input each image data into the detection model to obtain a detection result including region information for each object; The identifying process includes: Identifying the location of a shopping basket for placing pre-purchase items and a shopping bag for placing pre-purchase items based on the region information of each object included in the detection result; When the first class and the second class having the interaction are detected at the position of the shopping cart, or when the first class and the second class having the interaction are detected at the position of the plastic bag, counting the products belonging to the second class as the number of products to be purchased; The detecting process includes: an information processing program that detects fraud by the user when the counted number of products is greater than the number of products registered in the accounting machine;

2. If fraud by the user is detected, a warning is displayed on the cash register or a store clerk is notified that fraud by the user has been detected.

2. The information processing program according to claim 1, wherein the information processing program causes the computer to execute processing.

3. The computer Acquire image data of a predetermined area in front of the cashier where the user registers the product and makes the payment; inputting the image data into a machine learning model trained to identify products and storage areas for storing the products, and obtaining an output result; Identifying the user's behavior regarding the product using the product and the storage included in the output result; Execute a process to detect fraud by the user using the behavior identification result; The machine learning model is the machine learning model for HOID (Human Object Interaction Detection) on which machine learning is performed to identify a first class indicating the user who purchases the product and first area information indicating the area in which the user appears, a second class indicating an object including the product and second area information indicating the area in which the object appears, and an interaction between the first class and the second class; and a detection model on which machine learning is performed to detect area information of each object included in the image data, including objects that are not targets of identification by the machine learning model for HOID; The acquiring process includes: Input each image data into the machine learning model for the HOID to obtain an identification result for each class and each interaction, and input each image data into the detection model to obtain a detection result including region information for each object; The identifying process includes: Identifying the location of a shopping basket for placing pre-purchase items and a shopping bag for placing pre-purchase items based on the region information of each object included in the detection result; When the first class and the second class having the interaction are detected at the position of the shopping cart, or when the first class and the second class having the interaction are detected at the position of the plastic bag, counting the products belonging to the second class as the number of products to be purchased; The detecting process includes: an information processing method characterized in that, if the counted number of products is greater than the number of products registered in the cash register, fraud by the user is detected;

4. Acquire image data of a predetermined area in front of the cashier where the user registers the product and makes the payment; inputting the image data into a machine learning model trained to identify products and storage areas for storing the products, and obtaining an output result; Identifying the user's behavior regarding the product using the product and the storage included in the output result; a control unit that detects fraud by the user using the behavior identification result; The machine learning model is the machine learning model for HOID (Human Object Interaction Detection) on which machine learning is performed to identify a first class indicating the user who purchases the product and first area information indicating the area in which the user appears, a second class indicating an object including the product and second area information indicating the area in which the object appears, and an interaction between the first class and the second class; and a detection model on which machine learning is performed to detect area information of each object included in the image data, including objects that are not targets of identification by the machine learning model for HOID; The control unit Input each image data into the machine learning model for the HOID to obtain an identification result for each class and each interaction, and input each image data into the detection model to obtain a detection result including region information for each object; Identifying the location of a shopping basket for placing pre-purchase items and a shopping bag for placing pre-purchase items based on the region information of each object included in the detection result; When the first class and the second class having the interaction are detected at the position of the shopping cart, or when the first class and the second class having the interaction are detected at the position of the plastic bag, counting the products belonging to the second class as the number of products to be purchased; An information processing device characterized in that if the counted number of products is greater than the number of products registered in the accounting machine, fraud by the user is detected.

Images