Information Processing Program, Information Processing Method, and Information Processing Apparatus

Abstract

To solve the problem in which: when detecting scanning omissions of products at a self-checkout from a photographed image by a monitoring camera, it is required to identify individual products; and it is unrealistic to learn products sold at a shop one by one as a method of identifying the products due to a lot of types of products, fast replacements with new products, etc.SOLUTION: An information processing program causes a computer to execute processing of: tracking an action taken by a user with respect to an object from a photographed image; identifying, based on the action, between a first action in which the user holds the object and a second action in which the user releases the object held by the user; storing, in a first storage unit, first identification information and first location information with respect to the first action, and storing, in a second storage unit, second identification information and second location information with respect to the second action; and in a case where the first action is detected at a location indicated in the second location information, storing the second identification information and the second location information on the first storage unit.SELECTED DRAWING: Figure 15

Description

Technical Field

[0001] The present invention relates to an information processing program, an information processing method, and an information processing apparatus.

Background Art

[0002] In stores such as convenience stores, self-checkout systems are being introduced in order to reduce labor costs by streamlining cashier operations and avoid cashier congestion, where customers scan and register products themselves and perform checkout. In such a system where customers scan products themselves, it is important to detect fraudulent behavior such as failure to scan products, so-called product scanning omissions.

[0003] As a system for detecting customers' fraudulent behavior in stores, for example, a system has been developed that uses surveillance cameras in the store to detect suspicious behavior of customers and fraudulent behavior such as shoplifting.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when detecting product scanning omissions from captured images by surveillance cameras, it is necessary for the system side to identify individual products, such as which products have been scanned. As a method for identifying products, for example, a machine learning model for identifying products from captured images can be considered, but it is unrealistic to learn each and every product sold in the store because of the large number of product types and the rapid turnover to new products.

[0006] On one side, an object is to provide an information processing program, an information processing method, and an information processing apparatus that can more accurately detect the behavior of customers using self-checkout.

Means for Solving the Problem

[0007] In one aspect, the information processing program tracks the user's actions on an object from a captured image, and based on the actions, discriminates a first action in which the user grasps the object and a second action in which the user releases the grasped object, stores first identification information and first position information for the first action in a first storage unit, stores second identification information and second position information for the second action in a second storage unit, and when the first action is detected at the position indicated by the second position information, causes a computer to execute a process of storing the second identification information and the second position information in the first storage unit.

Effect of the Invention

[0008] On one side, the behavior of customers using self-checkout can be detected more accurately.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

MODE FOR CARRYING OUT THE INVENTION

[0010] Hereinafter, examples of the information processing program, information processing method, and information processing apparatus according to the present embodiment will be described in detail with reference to the drawings. Note that the present embodiment is not limited by this example. Also, each example can be appropriately combined within a non - contradictory range.

Example

[0011] First, an unauthorized detection system for implementing the present embodiment will be described. FIG. 1 is a diagram showing a configuration example of the unauthorized detection system according to Example 1. As shown in FIG. 1, the unauthorized detection system 1 is a system in which a management device 10 and a self - checkout terminal 100 are connected to be mutually communicable via a network 50.

[0012] In addition, the management device 10 is also connected to a camera device 200 and a store clerk terminal 300 to be mutually communicable via the network 50.

[0013] For the network 50, various communication networks such as an intranet used in a store such as a convenience store can be adopted regardless of whether it is wired or wireless. Also, the network 50 is not a single network, and for example, it may be configured by an intranet and the Internet via a network device such as a gateway or other devices (not shown).

[0014] The management device 10 is an information processing device such as a desktop PC (Personal Computer), a notebook PC, or a server computer installed in a store such as a convenience store and used by store staff, administrators, etc.

[0015] The management device 10 receives a plurality of images of a customer (hereinafter sometimes simply referred to as a "person" or "user") who is checked out by the self - checkout terminal 100 and captured by the camera device 200 from the camera device 200. Note that the plurality of images are strictly a series of frames of a video, that is, a video captured by the camera device 200.

[0016] In addition, the management device 10 identifies people and products from the captured images by using object detection technology or the like. Then, the management device 10 detects actions such as a person grasping a product and scanning the product barcode to register it in the self-checkout terminal 100.

[0017] In FIG. 1, the management device 10 is shown as a single computer, but it may be a distributed computing system composed of multiple computers. Further, the management device 10 may be a cloud computer device managed by a service provider that provides cloud computing services.

[0018] The self-checkout terminal 100 is an information processing terminal installed in a checkout area within a store such as a convenience store, where customers themselves scan and register products and perform checkout. The self-checkout terminal 100 is equipped with a code reader for scanning product barcodes and a touch panel display unit for displaying a settlement screen, or is communicably connected to an external code reader and display.

[0019] FIG. 2 is a diagram showing an example of the self-checkout terminal 100 according to the first embodiment. As shown in FIG. 2, the self-checkout terminal 100 is provided together with a temporary placement stand for placing products at the time of settlement, indicated by the temporary placement stand area 150. At the time of settlement, the customer places the product on the temporary placement stand, takes the product from the temporary placement stand, and scans the product barcode with a code reader indicated by the pre-checkout area 160, and registers the purchased product in the self-checkout terminal 100. Then, the customer operates the settlement screen displayed on the display of the self-checkout terminal 100 and pays for the products registered in the self-checkout terminal 100 using cash, electronic money, a credit card, or the like.

[0020] Returning to the description of FIG. 1, the camera device 200 is installed, for example, above the self-checkout terminal 100 and is a monitoring camera that captures customers who perform checkout at the self-checkout terminal 100. The video captured by the camera device 200 is transmitted to the management device 10.

[0021] The store employee terminal 300 may be a mobile terminal such as a smartphone or a tablet PC held by a store employee in a convenience store or the like, or may be an information processing device such as a desktop PC or a notebook PC installed at a predetermined position within the store. When the management device 10 detects improper behavior of a customer, such as a missed scan of a product, the store employee terminal 300 receives an alert from the management device 10. Note that, for example, there may be a plurality of store employee terminals 300 for each store employee in the store, but the terminal to which the alert is notified may be limited to, for example, a terminal held by a specific store employee.

[0022] Next, with reference to FIG. 3, a method in which a customer scans a product by himself / herself and registers it in the self-checkout terminal 100 (hereinafter sometimes referred to as "self-scanning") to purchase the product will be described step by step. FIG. 3 is a diagram showing an example of product purchase by self-checkout according to the first embodiment.

[0023] As shown in FIG. 3, first, the customer holds the product to be purchased and enters the checkout area where the self-checkout terminal 100 is installed (step S1). Next, the customer places the product on the temporary placement stand indicated by the temporary placement stand area 150 (step S2). In the example of FIG. 3, two products are placed on the temporary placement stand.

[0024] Next, in order to perform product scanning, the customer holds the product placed on the temporary placement stand (step S3). Then, the customer scans the product barcode of the held product with the barcode reader indicated by the area in front of the checkout 160 and registers the purchased product in the self-checkout terminal 100 (step S4).

[0025] Next, the customer places the product for which product scanning has been completed on the temporary placement stand again (step S5). Then, in order to complete product scanning for all the products to be purchased, steps 3 and 4 are repeated.

[0026] Then, when the product scanning for all products and the payment for the products registered in the self-checkout terminal 100 are completed, the customer holds the purchased products (step S6) and exits the checkout area (step S7).

[0027] As described above, the product purchase by self-scanning has been described with reference to FIG. 3. However, in self-scanning, for example, a customer can avoid paying for some products by scanning only a part of the purchased products and accounting for only the scanned products at the self-checkout terminal 100. In particular, when the number of products is small, fraudulent behavior is easily detected by a store clerk or the like. However, for example, when the number of products is large and some products are not scanned, it becomes difficult for a store clerk or the like to detect. Therefore, the management device 10 according to the present embodiment recognizes the customer's behavior from the video captured by the camera device 200 and detects fraudulent behavior.

[0028] [Functional Configuration of Management Device 10] Next, the functional configuration of the management device 10, which is the execution entity of the present embodiment, will be described. FIG. 4 is a diagram showing a configuration example of the management device 10 according to Example 1. As shown in FIG. 4, the management device 10 includes a communication unit 20, a storage unit 30, and a control unit 40.

[0029] The communication unit 20 is a processing unit that controls communication with other devices such as the self-checkout terminal 1000 and the camera device 200, and is, for example, a communication interface such as a network interface card.

[0030] The storage unit 30 has a function of storing various data and programs executed by the control unit 40, and is realized by a storage device such as a memory or a hard disk, for example. The storage unit 30 stores an image DB 31, a detection model DB 32, a temporary table ROI list 33, a pre-checkout ROI list 34, a tracked object list 35, a tracking paused object list 36, and the like. Here, ROI is an abbreviation of "Region Of Interest" meaning a target area, an area of interest, or the like.

[0031] The image DB 31 stores a plurality of captured images, which are a series of frames captured by the camera device 200.

[0032] The detection model DB 32 stores a machine learning model for detecting a person or an object from the captured image by the camera device 200. In the present embodiment, since there are many types of products sold in the store and the switch to new products is fast, it is considered unrealistic to learn each product. Therefore, in the present embodiment, for example, HOID (Human Object Interaction Detection), which is an existing technology capable of detecting an object with a small amount of training data, is used to detect a person or an object.

[0033] Here, HOID will be described. FIG. 5 is a diagram for explaining HOID. HOID detects the interaction between a person and an object. For example, as shown in FIG. 5, it is a technology for detecting a person and an object that are determined to be in an interaction relationship, such as a person holding an object, using the captured image 250 as input data. The detection of a person or an object is shown, for example, as a bounding box (BB), which is a rectangular area surrounding the detected person or object, as shown in FIG. 5. In the case of FIG. 5, the BBs of the person and the object are the person BB 170 and the object BB 180, respectively. Also, in HOID, a probability value of the interaction between a person and an object and a Class name (for example, "hold") are also output. In the present embodiment, HOID is used to detect an object held by a person as a product.

[0034] Returning to the description of FIG. 4, the temporary stage ROI list 33 stores data related to products entering and leaving the temporary stage area 150 described with reference to FIGS. 2 and 3.

[0035] The checkout ROI list 34 stores data related to products entering and leaving the area in front of the cash register 160 described with reference to FIGS. 2 and 3.

[0036] The object list 35 being tracked stores data regarding the products being detected by the management device 10 and tracked due to a person grasping the products, as well as their position information. Here, the position information of the product is, for example, the x and y coordinates of the upper left and lower right corners of the bounding box, which is a rectangular area surrounding the product in the captured image.

[0037] The object list 36 with tracking paused stores data regarding the products whose tracking has been paused and their position information, which are detected by the management device 10 when a person releases the product they were holding.

[0038] Note that the above information stored in the storage unit 30 is merely an example, and the storage unit 30 can store various other information in addition to the above information.

[0039] The control unit 40 is a processing unit that controls the entire management device 10 and is, for example, a processor or the like. The control unit 40 includes an acquisition unit 41, a detection unit 42, a tracking unit 43, a measurement unit 44, an output unit 45, and the like. Note that each processing unit is an example of an electronic circuit of the processor or an example of a process executed by the processor.

[0040] The acquisition unit 41 acquires a plurality of captured images, which are a series of frames captured by the camera device 200, from the camera device 200. The acquired captured images are stored in the image DB 31.

[0041] The detection unit 42 uses existing technologies such as HOID to detect interacting persons and objects from the captured images by the camera device 200. Note that the detected persons and objects may be indicated by bounding boxes.

[0042] Further, the detection unit 42 determines whether or not the similarity between the bounding box indicated by the position information of the product stored in the tracking pause object list 36 and the bounding box of the product in the HOID detection result is equal to or greater than a predetermined threshold. The details of the similarity of the bounding boxes will be described later. For example, it is the distance between the two bounding boxes, and the closer the distance, the more similar the two bounding boxes are determined to be. When the similarity between the two bounding boxes is equal to or greater than a predetermined threshold, the detection unit 42 detects the action that the person has re-grasped the product paused in tracking stored in the tracking pause object list 36.

[0043] Also, the detection unit 42 determines whether or not the similarity between the bounding box indicated by the position information of the product stored in the tracking object list 35 and the bounding box of the product in the HOID detection result is equal to or greater than a predetermined threshold. When the similarity between the two bounding boxes is equal to or greater than a predetermined threshold, the detection unit 42 determines that the product being tracked stored in the tracking object list 35 has not moved.

[0044] In addition, in a plurality of captured images continuously captured by the camera device 200, when it is determined that the product being tracked stored in the tracking object list 35 has not moved for a predetermined number of times or more, the action that the person has released the product being held is detected.

[0045] The tracking unit 43 tracks the actions of a person with respect to an object using the HOID from the captured image by the camera device 200. Further, the tracking unit 43 identifies, based on the tracked action, an action of a person grasping an object and an action of a person releasing the grasped object. Then, the tracking unit 43 stores the identification information and the position information of the object with respect to the action of a person grasping an object in the object list 35 being tracked. Further, the tracking unit 43 stores the identification information and the position information of the object with respect to the action of a person releasing the grasped object in the suspended object list 36. Furthermore, when the tracking unit 43 detects an action of a person grasping an object at the position indicated by the position information of the object stored in the suspended object list 36, the tracking unit 43 stores the identification information of the object stored in the suspended object list 36 and the second position information in the object list 35 being tracked.

[0046] The measurement unit 44 detects that the object being tracked has exited the temporary placement area 150 in the captured image and counts the number of times. Further, the measurement unit 44 detects that the object has entered the area in front of the cash register 160 in the captured image and counts the number of times.

[0047] The output unit 45 outputs, for example, the number of times the object has exited the temporary placement area 150 and the number of times the object has entered the area in front of the cash register 160 counted by the measurement unit 44. Note that the output of each count by the output unit 45 may be a notification by a message or the like to the store clerk terminal 300 or the like. Further, the content notified to the store clerk terminal 300 or the like does not have to be each count itself. This may be data generated based on each count, for example, when the number of times exiting the temporary placement area 150 and the number of times entering the area in front of the cash register 160 do not match and there may be a possibility of a scanning omission of a product.

[0048] As described above, the management device 10 tracks the actions of a person with respect to an object using the HOID from the captured image by the camera device 200. However, since the HOID has the following problems, in the present embodiment, these problems are also solved.

[0049] The problems of HOID will be described. Since HOID detects an article from a single captured image, in order to recognize that an article has entered or exited a predetermined area across a plurality of continuously captured images, it is necessary to assign an identifier (ID) to the article and track it. Also, since HOID detects when a person holds an object and does not detect when the person releases it, tracking starts when the person holds the object and ends when the person releases the hand. Therefore, there are problems such as a new ID being assigned when the person releases the object from the hand and then holds it again, or the ID of another object being transferred.

[0050] Therefore, when tracking a product using HOID, if the product is placed on a temporary stand, the hand is released, and then the product is held again, the ID assigned to the product will change. As a result, the count for one product will be erroneously performed multiple times. More specifically, for example, when the product is brought into the cashier area and passes in front of the cash register, and when the product is taken from the temporary stand and scanned, the number of times the product enters the area 160 in front of the cash register is counted twice, although it should be counted only once correctly.

[0051] FIG. 6 is a diagram showing an example of problems and solutions for product tracking by HOID. The upper route without ID management in FIG. 6 shows the problems of product tracking by HOID. In the upper route, when a person holds the product indicated by the object BB181, an ID of "00" is assigned. Therefore, the ID before the person releases the hand from the product is "00". Then, when the person releases the hand from the product, the object BB181 is no longer detected. When the same product is held again, the object BB181 is detected. However, when it is held again, a new ID is assigned, so the ID becomes "02". As a result, the management device 10 will recognize it as a different product from the product with the ID "00" before the hand was released.

[0052] Therefore, in the present embodiment, as shown in the lower route of FIG. 6, the ID is managed so that even when a person releases the hand from the product, the same ID is inherited and the same product can be tracked as the same product. Therefore, when the tracking of the product ends because the person has released the hand from the product, the management device 10 determines whether the product has been placed somewhere or lost. Next, when it is determined that the product has been placed, the management device 10 stores in the tracking suspension object list 36 the state at the end of the tracking, for example, the assigned ID and the coordinate position of the bounding box of the product in the captured image. Then, when a HOID detection result is obtained at the same position as the coordinate position stored in the tracking suspension object list 36, the product detected by the HOID is regarded as the same product as the product indicated by the ID stored in the tracking suspension object list 36, and the tracking is resumed by inheriting the stored ID.

[0053] Note that the case where a HOID detection result is obtained at the same position as the coordinate position stored in the tracking suspension object list 36 does not have to be exactly the same. For example, it may include a case where the similarity between the bounding box indicated by the coordinate position and the bounding box of the product in the HOID detection result is equal to or greater than a predetermined threshold.

[0054] FIG. 7 is a diagram showing an example of a product tracking method using HOID according to Example 1. In FIG. 7, the bounding box of the product detected by HOID is shown as object BB182, and the bounding box of the product being tracked indicated by the coordinate position stored in the tracking object list 35 is shown as object BB183.

[0055] As shown on the left side of FIG. 7, for example, when the distance between the center coordinates of object BB182 and object BB183 is within the range of the determination area 190, which is an area within 2 bounding boxes from the center coordinates of object BB183, the similarity between the two BBs is determined to be equal to or greater than a predetermined threshold. In this case, the management device 10 determines that the product of the HOID detection result indicated by object BB182 and the product being tracked indicated by object BB183 are the same product, and the tracking of the product is successful.

[0056] On the other hand, as shown on the right side of FIG. 7, for example, when the distance between the center coordinates of object BB182 and object BB183 is a distance outside the range of determination area 190, the similarity between the two BBs is less than a predetermined threshold value, and it is determined that the products indicated by the two BBs are different products. In this case, the product detected by HOID and indicated by object BB182 will be tracked as a product different from the product being tracked.

[0057] Although the determination method when a product is grasped and tracked using FIG. 7 has been described, next, the determination method when a person releases the hand from the product grasped will be described using FIG. 8. FIG. 8 is a diagram showing an example of the determination method for product tracking by HOID according to Embodiment 1. In FIG. 8, the bounding box of the product detected by HOID is shown as object BB182, and the bounding box of the product being tracked indicated by the coordinate position stored in the in-track object list 35 is shown as object BB183.

[0058] As shown on the left side of FIG. 8, for example, when the distance between the center coordinates of object BB182 and object BB183 is a distance within the range of determination area 190, it is determined that the similarity between the two BBs is equal to or greater than a predetermined threshold value. In this case, the product of the HOID detection result indicated by object BB182 is determined to be "not moving", that is, "placed". Note that it is not necessary to immediately determine "placed" when it is determined to be "not moving". For example, the fact that it is determined to be "not moving" may be counted in consecutive frames, and when it reaches 5 counts or more, it may be determined to be "placed". Also, the size of determination area 190 in FIG. 8 may be different from the size of determination area 190 in FIG. 7, and for example, it may be an area within 0.5 bounding boxes. And when it is determined to be "not moving", the tracking of the product being tracked indicated by object BB183 is paused.

[0059] On the other hand, as shown on the right side of FIG. 8, for example, when the distance between the center coordinates of the object BB182 and the object BB183 is a distance outside the range of the determination region 190, the similarity of both BBs is determined to be less than a predetermined threshold. In this case, the product of the HOID detection result indicated by the object BB182 is determined to be "moving". Note that when it is determined that the product is "moving", the counter counted when it is determined that the product is "not moving" may be reset.

[0060] In this way, the management device 10 tracks the product by using the bounding box of the product detected by HOID and the bounding box of the product indicated by the coordinate positions stored in the tracking object list 35 and the tracking suspended object list 36. The tracking object list 35 and the tracking suspended object list 36 will be described.

[0061] FIG. 13 is a diagram showing an example of data stored in the tracking object list 35 according to the first embodiment. The data of the tracking object list 35 is newly created, for example, when a product is detected by HOID and a product ID is assigned.

[0062] The tracking object list 35 stores, for example, "ID" indicating the product ID and "Bbox" indicating the position of the bounding box of the product being tracked in association with each other. Further, the tracking object list 35 stores, for example, "lost_count" indicating the number of times the product being tracked has been lost and "stay_count" indicating the number of times the product being tracked has been determined to be "not moving" in association with each other.

[0063] The "Bbox" in the list of objects being tracked 35 indicates the position of the bounding box of the product being tracked, and is updated, for example, according to the position of the bounding box of the product detected for each frame. Also, when the product being tracked is lost for each frame, the "lost_count" is incremented, and when a predetermined number of times, such as 4 times, is exceeded, the tracking is paused. Also, when it is determined that the product "is not moving" for each frame, the "stay_count" is incremented, and when a predetermined number of times, such as 5 times, is exceeded, the product is determined to be "placed".

[0064] FIG. 14 is a diagram showing an example of data stored in the list of objects with tracking paused 36 according to the first embodiment. The data in the list of objects with tracking paused 36 is newly created, for example, when it is determined that the product is "placed".

[0065] The list of objects with tracking paused 36 stores, for example, "ID" indicating the product ID, "Bbox" indicating the position of the bounding box of the product when the tracking is paused, and "restart" indicating whether the tracking has been restarted, in association with each other. The initial value of "restart" is set to "false", and is updated to "true" when the tracking of the product is restarted. Note that after the tracking of the product is restarted and the tracking is paused again, "Bbox" is the position of the bounding box of the product when the tracking is paused, and "restart" is updated to "false".

[0066] In this way, the data in the list of objects being tracked 35 and the list of objects with tracking paused 36 is updated according to the tracking status of the product. Next, with reference to FIGS. 15 and 16, the data transition of the list of objects being tracked 35 and the list of objects with tracking paused 36 will be described.

[0067] FIG. 15 is a diagram showing an example of data transition (1) of each tracking list according to Example 1. As shown in FIG. 15, first, when a customer holds a product to be purchased and enters the checkout area where the self-checkout terminal 100 is installed, the product is detected by the HOID, data is created in the tracking object list 35, and tracking of the product is started (step S11). In the example of FIG. 15, since the person is holding two products, two data items for product 1 and product 2 are created in the tracking object list 35. At this point, no data for product 1 and product 2 is created in the tracking paused object list 36, and it is empty.

[0068] Next, when the customer places each product on the temporary table and it is determined after several frames that each product has been "placed", the data in the tracking object list 35 is deleted, two data items for product 1 and product 2 are created in the tracking paused object list 36, and tracking of the product is paused (step S12). And since detection by the HOID is not performed when the customer is not holding product 1 or product 2, there is no change in the data in the tracking object list 35 and the tracking paused object list 36 (step S13).

[0069] Next, when the customer holds product 1 placed on the temporary table to perform a product scan, product 1 is detected by the HOID, data for product 1 is created in the tracking object list 35, and tracking of product 1 is resumed (step S14). Then, while the customer is holding product 1, the customer takes it out from the temporary table (step S15) and performs a product scan (step S16), and tracking of product 1 continues. During this period, the data for product 1 in the tracking object list 35 is updated based on the coordinates of the bounding box of product 1 detected for each frame.

[0070] FIG. 16 is a diagram showing an example of data transition (2) of each tracking list according to Example 1. As shown in FIG. 15, when the product scan of product 1 is performed (step S16), next, the customer returns product 1 to the temporary table and places it there (step S17). During this period, the data of product 1 in the tracking object list 35 is updated based on the coordinates of the bounding box of product 1 detected for each frame and the like.

[0071] Next, when it is determined that product 1 has been "placed" after several frames, the data of product 1 in the tracking object list 35 is deleted, the data of product 1 is created in the tracking paused object list 36, and the tracking of product 1 is paused (step S18).

[0072] Next, when the customer holds product 2 placed on the temporary table in order to perform a product scan, product 2 is detected by HOID, the data of product 2 is created in the tracking object list 35, and the tracking of product 2 is resumed (step S19). Then, similar to product 1, while the customer holds product 2, it is taken out from the temporary table and the tracking of product 2 continues until it is determined that the product scan has been performed and it has been "placed" back on the temporary table (step S20). When it is also determined that product 2 has been "placed", the data of product 2 in the tracking object list 35 is deleted, the data of product 2 is created in the tracking paused object list 36, and the tracking of product 2 is paused.

[0073] Next, when the customer holds product 1 and product 2 placed on the temporary table in order to leave the checkout area, both products are detected by HOID, the data of both products is created in the tracking object list 35, and the tracking of both products is resumed (step S21). Then, while the customer holds products 1 and 2, when they are taken out from the temporary table, the tracking of both products continues, and the data of both products in the tracking object list 35 is updated based on the coordinates of the bounding boxes of both products detected for each frame and the like (step S22).

[0074] Next, a method for counting products that have left the temporary placement area 150 or entered the area in front of the cash register 160, which is executed to detect scanning omissions of products and the like, will be described. FIG. 9 is a diagram showing an example of a method for counting products that have left the temporary placement ROI according to the first embodiment. In FIG. 9, the temporary placement ROI 151 indicates the temporary placement area 150 in the captured images 251 and 252. The management device 10 counts the products that have left the temporary placement ROI 151 based on the positional relationship of the products detected for each frame.

[0075] First, in the captured image 251, since a person is holding the product, the product is detected by the HOID and the object BB184 is shown. At this time, since the object BB184 is within the temporary placement ROI 151, it is determined that the product is within the temporary placement area 150.

[0076] Next, in the captured image 252, which is a frame after the captured image 251, the person moves the held product, and since the object BB184 is outside the temporary placement ROI 151, it is determined that the product is outside the temporary placement area 150. In this way, when the object BB184 that was within the temporary placement ROI 151 in the previous and subsequent frames moves outside, the management device 10 counts a counter indicating the number of times the product has left the temporary placement area 150.

[0077] However, counting the number of times a product has left the temporary placement area 150 is, for example, to compare with the number of times the product has been scanned and to detect scanning omissions of the product. Therefore, for example, it is desirable not to include in the count the number of times a person takes out a product from the temporary placement area 150 when leaving the cashier area. Therefore, in the present embodiment, it is managed whether each product has been counted, and if it has been counted, control is performed so that it is not counted repeatedly.

[0078] FIG. 11 is a diagram showing an example of the data stored in the temporary placement ROI list 33 according to the first embodiment. The data in the temporary placement ROI list 33 is newly created, for example, when a product is detected by the HOID and a product ID is assigned.

[0079] The temporary stand ROI list 33 stores, for example, "ID" indicating the product ID and "previous frame position" indicating the position of the product on the temporary stand ROI 151 in the previous frame in an associated manner. Further, the temporary stand ROI list 33 stores, for example, "counted" indicating whether the number of times the product has exited the temporary stand area 150 has been counted, in an associated manner.

[0080] For the "previous frame position" in the temporary stand ROI list 33, for example, when determining whether the product is inside or outside the temporary stand ROI 151, "OUT" is set if the product was outside the temporary stand ROI 151 in the previous frame of the frame, and "IN" is set if it was inside. Also, for "counted" in the temporary stand ROI list 33, "true" is set if the number of times the product has exited the temporary stand area 150 has been counted, and "false" is set if it has not been counted.

[0081] When a product is detected by the HOID and a product ID is assigned, for example, for the "previous frame position" in the temporary stand ROI list 33, "OUT" or "IN" is set according to the current position of the product relative to the temporary stand ROI 151, and "false" is set for "counted".

[0082] When "counted" in the temporary stand ROI list 33 is "false", "previous frame position" is "IN", and the position of the product in the current frame relative to the temporary stand ROI 151 is "OUT", the number of times the product has exited the temporary stand area 150 is counted. Also, at this time, since the number of times the product has exited the temporary stand area 150 has been counted, "counted" in the temporary stand ROI list 33 is updated to "true".

[0083] Next, a method for counting the products that have entered the area in front of the cashier will be described. FIG. 10 is a diagram showing an example of a method for counting the products that have entered the ROI in front of the cashier according to the first embodiment. In FIG. 10, the ROI 161 in front of the cashier indicates the area 160 in front of the cashier in the captured images 253 and 254. Also, the self-checkout BB 101 is, for example, the bounding box of the self-checkout terminal 100 detected by HOID when a person brings their hand close to the self-checkout terminal 100. A predetermined area within the front part of the self-checkout BB 101 may be set as the ROI 161 in front of the cashier. The management device 10 counts the products that enter the ROI 161 in front of the cashier based on the positional relationship of the products detected for each frame.

[0084] First, in the captured image 253, since the object BB 184 indicating the product held by the person is outside the ROI 161 in front of the cashier, it is determined that the product is outside the area 160 in front of the cashier.

[0085] Next, in the captured image 254, which is a frame after the captured image 253, the product held by the person is moved, and since the object BB 184 is inside the ROI 161 in front of the cashier, it is determined that the product is inside the ROI 161 in front of the cashier. In this way, when the object BB 184 that was outside the ROI 161 in front of the cashier in the previous and subsequent frames enters it, the management device 10 counts a counter indicating the number of times the product has entered the area 160 in front of the cashier.

[0086] However, counting the number of times a product has entered the area 160 in front of the cashier is, for example, to compare with the number of times the product has been scanned and to detect any missed scanning of the product. Therefore, for example, it is desirable not to include in the count the number of times a product has entered the area 160 in front of the cashier when a person brings the product into the checkout area and passes in front of the checkout counter. Thus, in this embodiment, it is managed whether each product has been counted, and if it has been counted, control is performed so that it is not counted repeatedly.

[0087] FIG. 12 is a diagram showing an example of data stored in the pre - register ROI list 34 according to Example 1. The data in the pre - register ROI list 34 is newly created, for example, when a product is detected by HOID and a product ID is assigned.

[0088] The pre - register ROI list 34 stores, for example, "ID" indicating the product ID and "previous - frame position" indicating the position of the product in the previous frame with respect to the pre - register ROI 161 in association. Further, the pre - register ROI list 34 stores, for example, "counted" indicating whether the number of times the product has entered the pre - register area 160 has been counted in association.

[0089] In the "previous - frame position" of the pre - register ROI list 34, for example, when it is determined whether the product is outside or inside the pre - register ROI 161, "OUT" is set if the product was outside the pre - register ROI 161 in the previous frame of the frame when making the determination, and "IN" is set if it was inside. Also, in the "counted" of the pre - register ROI list 34, "true" is set if the number of times the product has entered the pre - register area 160 has been counted, and "false" is set if it has not been counted.

[0090] When a product is detected by HOID and a product ID is assigned, for example, in the "previous - frame position" of the pre - register ROI list 34, "OUT" or "IN" is set according to the current position of the product with respect to the pre - register ROI 161, and "false" is set in "counted".

[0091] And when "counted" in the pre - register ROI list 34 is "false", "previous - frame position" is "OUT", and the position of the product in the current frame with respect to the pre - register ROI 161 is "IN", the number of times the product has entered the pre - register area 160 is counted. Also, at this time, since the number of times the product has entered the pre - register area 160 has been counted, "counted" in the pre - register ROI list 34 is updated to "true".

[0092] [Processing flow] Next, with reference to FIGS. 17 and 18, the flow of the product tracking process executed by the management device 10 will be described. FIG. 17 is a flowchart showing the flow of the product tracking process (1) according to the first embodiment. The product tracking process shown in FIG. 17 may be started, for example, when a customer enters a checkout area where the self-checkout terminal 100 is installed.

[0093] First, as shown in FIG. 17, the management device 10 initializes the tracking object list 35 and the tracking pause object list 36 (step S101).

[0094] Next, the management device 10 obtains the detection result of the HOID for the captured image obtained by the camera device 200 by imaging a predetermined imaging range such as in front of the self-checkout terminal 100 (step S102). Hereinafter, steps S102 to S109 are repeatedly executed for each frame.

[0095] Next, the management device 10 collates the HOID detection result obtained in step S102 with the bounding box indicated by the "Bbox" of the tracking pause object list 36, and determines the similarity of the bounding box of the product (step S103). If the similarity is equal to or greater than a predetermined threshold such as 1 in step S103 and it is determined that the product has been "placed", the "restart" of the corresponding entry in the tracking pause object list 36 is set to "true", and the entry is deleted from the HOID detection result.

[0096] Next, the management device 10 collates the HOID detection result obtained in step S102 with the bounding box indicated by the "Bbox" of the object list 35 being tracked, and determines the similarity of the bounding box of the product (step S104). If the similarity is equal to or greater than a predetermined threshold such as 2 in step S104, it is determined that the product detected by HOID is the product being tracked, and the object list 35 being tracked is updated based on the HOID detection result. When updating the object list 35 being tracked, if the similarity is equal to or greater than a predetermined threshold such as 3 and it is determined that the product being tracked is "not moving", the "stay_count" of the object list 35 being tracked is incremented, and if the similarity is less than the predetermined threshold, it is reset. Steps S105 and subsequent steps will be described with reference to FIG. 18.

[0097] FIG. 18 is a flowchart showing the flow of the product tracking process (2) according to the first embodiment. Next, for the product with the HOID detection result that could not be collated in step S104, the management device 10 assigns an ID to the product as a new detection and adds data to the object list 35 being tracked (step S105). At this time, the "stay_count" of the object list 35 being tracked is set to the initial value of 0.

[0098] Next, the management device 10 increments the "lost_count" of the product in the object list 35 being tracked that could not be collated in step S104 (step S106). When the "lost_count" exceeds a predetermined threshold such as 4 and the "stay_count" is equal to or greater than a predetermined threshold such as 5, the management device 10 determines that the product in the object list 35 being tracked that could not be collated in step S104 has been "placed". The product determined to have been "placed" is moved from the object list 35 being tracked to the object list 36 of suspended tracking. At this time, "false" is set for the "restart" of the object list 36 of suspended tracking. On the other hand, when the "lost_count" exceeds a predetermined threshold such as 4 and the "stay_count" is less than the predetermined threshold, the management device 10 deletes the data of the product in the object list 35 being tracked that could not be collated in step S104. The product is, for example, a product that has already left the sales area.

[0099] Next, the management device 10 moves the products in the tracking suspension object list 36 with "restart" being "true" to the tracking object list 35 and resumes tracking (step S107). At this time, "stay_count" in the tracking object list 35 is set to the initial value of 0.

[0100] Next, the management device 10 recognizes that the product being tracked has come out of the temporary placement stand ROI151 and counts up the number of times the product has come out of the temporary placement stand area 150 (step S108).

[0101] Next, the management device 10 recognizes that the product being tracked has entered from outside the self-checkout ROI161 and approached the self-checkout terminal 100, and counts up the number of times the product has been scanned, that is, the number of times it has entered the area in front of the checkout 160 (step S109).

[0102] Next, the management device 10 outputs the number of times the product has come out of the temporary placement stand area 150 and the number of times the product has been scanned, which were counted in steps S108 and S109 (step S110). After the execution of step S110, the product tracking process shown in FIGS. 17 and 18 ends. However, for example, when the difference between the number of times the product has come out of the temporary placement stand area 150 and the number of times the product has been scanned is equal to or greater than a predetermined number, it is determined that there may be a scanning omission of the product. In this case, the management device 10 can notify an alert to the store clerk terminal 300 or the like.

[0103] [Effect] As described above, the management device 10 tracks the user's actions on the object from the captured image, and based on the actions, identifies the first action of the user grasping the object and the second action of the user releasing the grasped object, stores the first identification information and the first position information for the first action in the first storage unit, stores the second identification information and the second position information for the second action in the second storage unit, and when the first action is detected at the position indicated by the second position information, stores the second identification information and the second position information in the first storage unit.

[0104] In this way, the management device 10 identifies the actions of gripping and releasing the product, stores the position and identifier where the product was released, and when a gripping action is detected at the position where the product was released, it inherits the stored identifier, enabling more accurate detection of customer behavior using self-checkout.

[0105] Further, the management device 10 detects that an object has exited from a first area in the captured image, counts a first counter, detects that the object has entered a second area in the captured image, counts a second counter, and outputs the first counter and the second counter.

[0106] Thereby, the management device 10 can provide information for more accurately detecting customer behavior using self-checkout.

[0107] Also, the process of tracking actions executed by the management device 10 uses the HOID to track the user's actions with respect to the object.

[0108] Thereby, the management device 10 can more accurately detect customer behavior using self-checkout.

[0109] Further, the management device 10 obtains the detection result of the HOID, and when the similarity between the second bounding box indicated by the second position information stored in the second storage unit and the third bounding box of the object in the detection result of the HOID is equal to or greater than a predetermined threshold, it detects a first action with respect to the object in the second bounding box.

[0110] Thereby, the management device 10 can more accurately detect customer behavior using self-checkout.

[0111] Also, the management device 10 obtains the detection result of the HOID, and when the similarity between the first bounding box indicated by the first position information stored in the first storage unit and the third bounding box of the object in the detection result of the HOID is equal to or greater than a predetermined threshold, it determines that the object in the first bounding box has not moved.

[0112] As a result, the management device 10 can more accurately detect the customer behavior of using self-checkout.

[0113] In addition, in the management device 10, when it is determined that the object is not moving, in a plurality of continuously captured imaging images, if it is determined that the object in the first bounding box is not moving for a predetermined number of times or more, a second action is detected for the object in the first bounding box.

[0114] As a result, the management device 10 can more accurately detect the customer behavior of using self-checkout.

[0115] [System] The processing procedures, control procedures, specific names, information including various data and parameters shown in the above documents and drawings may be arbitrarily changed unless otherwise specified. Also, the specific examples, distributions, numerical values, etc. described in the embodiments are merely examples and may be arbitrarily changed.

[0116] In addition, the specific forms of the dispersion and integration of the components of each device are not limited to those shown in the drawings. That is, all or part of the components may be functionally or physically dispersed and integrated in any unit according to various loads, usage situations, etc. Furthermore, each processing function of each device may be realized by all or any part of a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or may be realized as hardware by wired logic.

[0117] [Hardware] FIG. 19 is a diagram for explaining a hardware configuration example of the management device 10. As shown in FIG. 19, the management device 10 includes a communication interface 10a, an HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. Also, each part shown in FIG. 19 is interconnected by a bus or the like.

[0118] The communication interface 10a is, for example, a network interface card, and communicates with other information processing devices. The HDD 10b stores programs and data for operating the functions shown in FIG. 4.

[0119] The processor 10d is a hardware circuit that operates a process for executing each function described in FIG. 4, etc., by reading a program for executing the same processing as each processing unit shown in FIG. 4 from the HDD 10b or the like and expanding it in the memory 10c. That is, this process executes the same functions as each processing unit of the management device 10. Specifically, the processor 10d reads a program having the same functions as the acquisition unit 41, the detection unit 42, etc. from the HDD 10b or the like. Then, the processor 10d executes a process for executing the same processing as the acquisition unit 41, the detection unit 42, etc.

[0120] In this way, the management device 10 operates as an information processing device that executes operation control processing by reading and executing a program for executing the same processing as each processing unit shown in FIG. 4. Also, the management device 10 can read a program from a recording medium by a medium reader and execute the read program to realize the same functions as the above-described embodiments. Note that the program in this other embodiment is not limited to being executed by the management device 10. For example, the present embodiment may be similarly applied when another computer or server executes the program, or when these cooperate to execute the program.

[0121] Also, a program for executing the same processing as each processing unit shown in FIG. 4 can be distributed via a network such as the Internet. Also, this program is recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a MO (Magneto-Optical disk), a DVD (Digital Versatile Disc), etc., and can be executed by being read from the recording medium by a computer.

[0122] FIG. 20 is a diagram for explaining an example of the hardware configuration of the self-checkout terminal 100. As shown in FIG. 20, the self-checkout terminal 100 includes a communication interface 100a, an HDD 100b, a memory 100c, a processor 100d, an input unit 100e, and an output unit 100f. Further, each unit shown in FIG. 20 is interconnected by a bus or the like.

[0123] The communication interface 100a is a network interface card or the like and communicates with other information processing devices. The HDD 100b stores programs and data for operating each function of the self-checkout terminal 100.

[0124] The processor 100d is a hardware circuit that operates a process for executing each function of the self-checkout terminal 100 by reading a program for executing the processing of each function of the self-checkout terminal 100 from the HDD 100b or the like and expanding it in the memory 100c. That is, this process executes the same functions as each processing unit included in the self-checkout terminal 100.

[0125] In this way, the self-checkout terminal 100 operates as an information processing device that executes operation control processing by reading and executing a program for executing the processing of each function of the self-checkout terminal 100. Further, the self-checkout terminal 100 can also realize each function of the self-checkout terminal 100 by reading a program from a recording medium by a medium reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the self-checkout terminal 100. For example, the present embodiment may be similarly applied when another computer or server executes the program, or when these cooperate to execute the program.

[0126] In addition, the program for executing the processing of each function of the self-checkout terminal 100 can be distributed via a network such as the Internet. Also, this program can be recorded on a computer-readable recording medium such as a hard disk, FD, CD-ROM, MO, DVD, etc., and can be executed by being read from the recording medium by a computer.

[0127] The input unit 100e detects various input operations by the user, such as input operations for the program executed by the processor 100d. The input operations include, for example, touch operations. In the case of a touch operation, the self-checkout terminal 100 further includes a display unit, and the input operation detected by the input unit 100e may be a touch operation on the display unit. The input unit 100e may be, for example, a button, a touch panel, a proximity sensor, etc.

[0128] The output unit 100f outputs the data output from the program executed by the processor 100d via an external device connected to the self-checkout terminal 100, such as an external display device. Note that when the self-checkout terminal 100 includes a display unit, the self-checkout terminal 100 may not include the output unit 100f.

[0129] FIG. 21 is a diagram for explaining an example of the hardware configuration of the store clerk terminal 300. As shown in FIG. 21, the store clerk terminal 300 has a communication interface 300a, an HDD 300b, a memory 300c, a processor 300d, an input unit 300e, and a display unit 300f. Also, each part shown in FIG. 21 is interconnected by a bus or the like.

[0130] The communication interface 300a is a network interface card or the like and communicates with other information processing devices. The HDD 300b stores programs and data for operating each function of the store clerk terminal 300.

[0131] The processor 300d is a hardware circuit that operates the process of executing each function of the store clerk terminal 300 by reading a program for executing the processing of each function of the store clerk terminal 300 from the HDD 300b or the like and expanding it in the memory 300c. That is, this process executes the same functions as each processing unit of the store clerk terminal 300.

[0132] In this way, the store clerk terminal 300 operates as an information processing device that executes operation control processing by reading and executing a program for executing the processing of each function of the store clerk terminal 300. Further, the store clerk terminal 300 can also realize each function of the store clerk terminal 300 by reading a program from a recording medium by a medium reading device and executing the read program. Note that the program referred to in this other embodiment is not limited to being executed by the store clerk terminal 300. For example, the present embodiment may be similarly applied when another computer or server executes the program, or when these cooperate to execute the program.

[0133] Also, the program for executing the processing of each function of the store clerk terminal 300 can be distributed via a network such as the Internet. Further, this program is recorded on a computer-readable recording medium such as a hard disk, FD, CD-ROM, MO, DVD, and can be executed by being read from the recording medium by a computer.

[0134] The input unit 300e detects various input operations by the user, such as an input operation for a program executed by the processor 300d. The input operation includes, for example, a touch operation and the insertion of an earphone terminal into the store clerk terminal 300. Here, the touch operation refers to various contact operations on the display unit 300f, such as a tap, double tap, swipe, pinch, etc. Further, the touch operation includes an operation of bringing an object such as a finger close to the display unit 300f. The input unit 300e may be, for example, a button, a touch panel, a proximity sensor, or the like.

[0135] The display unit 300f displays various visual information based on the control by the processor 300d. The display unit 300f may be a liquid crystal display (LCD), an OLED (Organic Light Emitting Diode), or a so-called organic EL (Electro Luminescence) display, etc.

[0136] Regarding the embodiments including the above embodiments, the following additional notes are further disclosed.

[0137] (Appendix 1) Tracking the user's actions on an object from a captured image, Based on the actions, identifying a first action in which the user grasps the object and a second action in which the user releases the grasped object, Storing first identification information and first position information for the first action in a first storage unit, Storing second identification information and second position information for the second action in a second storage unit, When the first action is detected at the position indicated by the second position information, storing the second identification information and the second position information in the first storage unit An information processing program characterized by causing a computer to execute the processing.

[0138] (Appendix 2) Detecting that the object has exited a first region in the captured image and counting a first counter, Detecting that the object has entered a second region in the captured image and counting a second counter, Outputting the first counter and the second counter The information processing program according to Appendix 1, characterized by causing the computer to execute the processing.

[0139] (Appendix 3) The information processing program according to Appendix 1 or 2, characterized in that the processing for tracking the actions includes processing for tracking the user's actions on the object using HOID.

[0140] (Appendix 4) Obtain the detection result of the HOID, When the similarity between the second bounding box indicated by the second position information stored in the second storage unit and the third bounding box of the object in the detection result is equal to or greater than a predetermined threshold, detect the first action with respect to the object of the second bounding box The information processing program according to Appendix 3, characterized in that the computer is caused to execute the process.

[0141] (Appendix 5) Obtain the detection result of the HOID, When the similarity between the first bounding box indicated by the first position information stored in the first storage unit and the third bounding box of the object in the detection result is equal to or greater than a predetermined threshold, determine that the object of the first bounding box is not moving The information processing program according to Appendix 3, characterized in that the computer is caused to execute the process.

[0142] (Appendix 6) The process of determining that the object is not moving is In a plurality of the captured images captured continuously, when it is determined that the object of the first bounding box is not moving for a predetermined number of times or more, detect the second action with respect to the object of the first bounding box The information processing program according to Appendix 5, characterized by including the process.

[0143] (Appendix 7) Track the user's action with respect to the object from the captured image, Based on the action, identify the first action of the user grasping the object and the second action of the user releasing the grasped object, Store the first identification information and the first position information for the first action in the first storage unit, Store the second identification information and the second position information for the second action in the second storage unit, When the first action is detected at the position indicated by the second position information, the second identification information and the second position information are stored in the first storage unit. An information processing method, characterized in that a computer executes the processing.

[0144] (Appendix 8) Detect that the object has exited from the first area in the captured image, and count the first counter. Detect that the object has entered the second area in the captured image, and count the second counter. Output the first counter and the second counter. The information processing method according to Appendix 7, characterized in that a computer executes the processing.

[0145] (Appendix 9) The processing for tracking the action includes processing for tracking the user's action with respect to the object using HOID, and is characterized by the information processing method according to Appendix 7 or 8.

[0146] (Appendix 10) Obtain the detection result of the HOID. When the similarity between the second bounding box indicated by the second position information stored in the second storage unit and the third bounding box of the object in the detection result is equal to or greater than a predetermined threshold, detect the first action with respect to the object in the second bounding box. The information processing method according to Appendix 9, characterized in that a computer executes the processing.

[0147] (Appendix 11) Obtain the detection result of the HOID. When the similarity between the first bounding box indicated by the first position information stored in the first storage unit and the third bounding box of the object in the detection result is equal to or greater than a predetermined threshold, determine that the object in the first bounding box has not moved. The information processing method according to Appendix 9, characterized in that a computer executes the processing.

[0148] (Appendix 12) The process of determining that the object is not moving is in a plurality of the captured images captured continuously, when it is determined that the object in the first bounding box has not moved for a predetermined number of times or more, detecting the second action with respect to the object in the first bounding box The information processing method according to Appendix 11, characterized by including the process.

[0149] (Appendix 13) Tracking the user's action with respect to the object from the captured image, identifying, based on the action, a first action in which the user grasps the object and a second action in which the user releases the grasped object, storing first identification information and first position information for the first action in a first storage unit, storing second identification information and second position information for the second action in a second storage unit, when the first action is detected at the position indicated by the second position information, storing the second identification information and the second position information in the first storage unit An information processing apparatus having a control unit that executes the process.

[0150] (Appendix 14) Detecting that the object has exited a first region in the captured image and counting a first counter, detecting that the object has entered a second region in the captured image and counting a second counter, outputting the first counter and the second counter The information processing apparatus according to Appendix 13, characterized in that the control unit executes the process.

[0151] (Appendix 15) The process of tracking the action includes a process of tracking the user's action with respect to the object using HOID. The information processing apparatus according to Appendix 13 or 14, characterized by this.

[0152] (Appendix 16) Obtaining the detection result of the HOID, When the similarity between the second bounding box indicated by the second position information stored in the second storage unit and the third bounding box of the object in the detection result is equal to or greater than a predetermined threshold, the control unit detects the first action with respect to the object in the second bounding box. The information processing apparatus according to appended claim 15, wherein the control unit executes the processing.

[0153] (Appended claim 17) Obtaining the detection result of the HOID, When the similarity between the first bounding box indicated by the first position information stored in the first storage unit and the third bounding box of the object in the detection result is equal to or greater than a predetermined threshold, it is determined that the object in the first bounding box is not moving. The information processing apparatus according to appended claim 15, wherein the control unit executes the processing.

[0154] (Appended claim 18) The process of determining that the object is not moving is In a plurality of the captured images captured continuously, when it is determined that the object in the first bounding box is not moving for a predetermined number of times or more, the control unit detects the second action with respect to the object in the first bounding box. The information processing apparatus according to appended claim 17, comprising the processing.

[0155] (Appended claim 19) An information processing apparatus including a memory operably connected to the processor, and wherein the processor tracks an action of the user with respect to an object in a captured image, identifies, based on the action, a first action in which the user grasps the object and a second action in which the user releases the grasped object, stores first identification information and first position information for the first action in a first storage unit, stores second identification information and second position information for the second action in a second storage unit, When the first action is detected at the position indicated by the second position information, the second identification information and the second position information are stored in the first storage unit. An information processing apparatus characterized by executing the processing.

Explanation of Signs

[0156] 1 Anti-fraud detection system 10 Management device 10a Communication interface 10b HDD 10c Memory 10d Processor 20 Communication unit 30 Storage unit 31 Image DB 32 Detection model DB 33 Dummy stand ROI list 34 ROI list in front of the cash register 35 List of objects being tracked 36 List of objects with tracking paused 40 Control unit 41 Acquisition unit 42 Detection unit 43 Tracking unit 44 Measurement unit 45 Output unit 50 Network 100 Self-checkout terminal 100a Communication interface 100b HDD 100c Memory 100d Processor 100e Input unit 100f Output unit 101 Self-checkout BB 150 Dummy stand area 151 Dummy stand ROI 160 Area in front of the cash register 161 ROI in front of the cash register 170 Person BB 180~184 Object BB 190 Judgment area 200 Camera device 250~252 Captured images 300 Store Employee Terminal 300a Communication Interface 300b HDD 300c Memory 300d Processor 300e Input Unit 300f Display Unit

Claims

When a user who has selected an object to be purchased in a store enters an area including a cash register terminal where the user performs accounting by himself / herself, from a captured image captured by an imaging device that images the area, track the actions of the user with respect to the object performed in the area, Based on the actions, identify a first action in which the user holds the object and a second action in which the user releases the object held by the user, When the first action is identified, store in a first storage unit first identification information for the first action and first position information of a first position where the action identified by the first identification information is performed in the area, When the second action is identified, store in a second storage unit second identification information for the second action and second position information of a second position where the action identified by the first identification information is performed in the area, After the second identification information and the second position information are stored in the second storage unit, if the first action is detected at the position indicated by the second position information, store the second identification information and the second position information in the first storage unit An information processing program characterized by causing a computer to execute the processing.

2. Detect that the object has exited a first area in the captured image, and count a first counter, Detect that the object has entered a second area in the captured image, and count a second counter, Output the first counter and the second counter An information processing program according to claim 1, characterized by causing the computer to execute the processing.

3. The processing for tracking the actions includes processing for tracking the actions of the user with respect to the object using HOID (Human Object Interaction Detection), according to claim 1 or 2. An information processing program described. When a user who has selected an object to be purchased in a store enters an area including a cash register terminal where the user performs accounting by himself / herself, from a captured image captured by an imaging device that images the area, track the actions of the user with respect to the object performed in the area, Based on the actions, identify a first action in which the user holds the object and a second action in which the user releases the object held by the user, When the first action is identified, store in a first storage unit first identification information for the first action and first position information indicating a position in the area where the action identified by the first identification information was performed. When the second action is identified, store in a second storage unit second identification information for the second action and second position information indicating a position in the area where the action identified by the first identification information was performed. After the second identification information and the second position information are stored in the second storage unit, if the first action is detected at the position indicated by the second position information, store the second identification information and the second position information in the first storage unit. An information processing method, characterized in that a computer executes the processing.

5. When a user who has selected an object to be purchased in a store enters an area including a cash register terminal where the user performs checkout by himself / herself, track the action of the user on the object performed in the area from a captured image captured by an imaging device that captures the area. Based on the action, identify a first action in which the user grips the object and a second action in which the user releases the object that the user has gripped. When the first action is identified, store in a first storage unit first identification information for the first action and first position information indicating a position in the area where the action identified by the first identification information was performed. When the second action is identified, store in a second storage unit second identification information for the second action and second position information indicating a position in the area where the action identified by the first identification information was performed. After the second identification information and the second position information are stored in the second storage unit, if the first action is detected at the position indicated by the second position information, store the second identification information and the second position information in the first storage unit. An information processing apparatus, characterized by having a control unit that executes the processing.

6. Track the action of the user on the object using HOID (Human Object Interaction Detection) from the captured image. Based on the action, identify a first action in which the user grips the object and a second action in which the user releases the object that the user has gripped. Store the first identification information and the first position information for the first action in a first storage unit. Store the second identification information and the second position information for the second action in the second storage unit. When the first action is detected at the position indicated by the second position information, store the second identification information and the second position information in the first storage unit. Obtain the detection result of the HOID. When the similarity between the second bounding box indicated by the second position information stored in the second storage unit and the third bounding box of the object in the detection result is equal to or greater than a predetermined threshold, detect the first action for the object of the second bounding box. An information processing program characterized by causing a computer to execute the processing.

7. Track the action of the user on the object using HOID (Human Object Interaction Detection) from the captured image. Based on the action, identify the first action of the user grasping the object and the second action of the user releasing the grasped object. Store the first identification information and the first position information for the first action in the first storage unit. Store the second identification information and the second position information for the second action in the second storage unit. When the first action is detected at the position indicated by the second position information, store the second identification information and the second position information in the first storage unit. Obtain the detection result of the HOID. When the similarity between the first bounding box indicated by the first position information stored in the first storage unit and the third bounding box of the object in the detection result is equal to or greater than a predetermined threshold, determine that the object of the first bounding box is not moving. An information processing program characterized by causing a computer to execute the processing.

8. The process of determining that the object is not moving is as follows: In a plurality of the captured images captured continuously, when it is determined that the object of the first bounding box is not moving for a predetermined number of times or more, detect the second action for the object of the first bounding box. The information processing program according to claim 7, characterized by including the processing.

Images