Information processing program, information processing method, and information processing device

Abstract

To identify a product on a product shelve.SOLUTION: An information processing device identifies an action in which a particular user among a plurality of persons takes out a product from a shelf, from a video obtained by photographing an area including the shelf housing the products in a shop. The information processing device identifies each of the particular user and the accounting machine, from the video obtained by photographing the area including an accounting machine in the shop. The information processing device associates the particular user with the accounting machine, respectively, and stores them in a storage unit. The information processing device receives a purchase history transmitted from the accounting machine associated with the particular user, and identifies one or more products included in the purchase history of the accounting machine. The information processing device identifies the product associated with the shelf, so that a difference between a predicted value of the number of purchased products based on taking out behavior and an observed value is minimized, on the basis of the number of the identified products, and the number of the taking out behavior of the user associated with the accounting machine.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing program and the like. [Background technology]

[0002] 2. Description of the Related Art In recent years, customer purchasing behavior has been analyzed using images captured by cameras installed in stores.

[0003] By analyzing customer purchasing behavior based on camera footage, it is possible to detect shoplifting and provide customer service support. For example, if it is determined that a customer picked up two "cosmetics" from a shelf, but at the time of checkout there was only one "cosmetics" in the basket, shoplifting will be detected. Also, if it is determined that a customer looked at "curtains" for a while, then moved to the "beds" section, where an employee will be speaking to and serving the customer, it is possible to recommend that the store employee make a suggestion regarding "curtains" and provide customer service support.

[0004] When analyzing customer purchasing behavior, it is important to identify product information on each shelf based on camera footage. For example, in conventional technology, an image database is prepared in which product images are associated with product identification information, and products stored on shelves are identified by matching the product images included in the camera footage with the product images in the image database. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Hayato Akatsuka, Kazunari Nakamura, and Masaaki Taka, "Product Shelf Analysis Solution Using Image Recognition," NTT Docomo, Inc. [Non-patent document 2] NEC Corporation, "Sales Floor Information Image Analysis Solutions and Services," [online], [searched January 28, 2022], Internet<URL:https: / / jpn.nec.com / process / marketing / iasl4sa.html> Summary of the Invention [Problem to be solved by the invention]

[0006] However, the above-mentioned conventional technology has a problem in that it is not possible to identify products based on camera images.

[0007] For example, images of products captured by surveillance cameras installed in typical stores often have low resolution, making it difficult to match them with product images in an image database.

[0008] In one aspect, the present invention aims to provide an information processing program, an information processing method, and an information processing device that can identify products on a product shelf. [Means for solving the problem]

[0009] In the first proposal, a computer is caused to perform the following process. The computer identifies the behavior of a specific user from among multiple people taking products from a shelf from video captured of an area in a store including shelves that store products. The computer identifies each specific user and each payment machine from video captured of an area in the store including payment machines. The computer associates each specific user with each payment machine and stores the association in a memory unit. The computer receives a purchase history sent from the payment machine associated with the specific user, and identifies one or more products included in the purchase history of the payment machine. Based on the number of identified products and the number of picking actions of the user associated with the payment machine, the computer identifies products to associate with shelves so as to minimize the difference between the predicted and observed number of product purchases based on the picking actions. [Effects of the Invention]

[0010] Products on the shelves can be identified. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram for explaining the processing of the information processing device according to the present embodiment. [Figure 2] FIG. 2 is a diagram showing an example of a low-resolution product image. [Figure 3] FIG. 3 is a diagram for explaining the process (1) of identifying products stored on shelves. [Figure 4] FIG. 4 is a diagram for explaining the process (3) of identifying products stored on shelves. [Figure 5] FIG. 5 is a diagram illustrating a system according to this embodiment. [Figure 6] FIG. 6 is a functional block diagram showing the configuration of an information processing device according to this embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the data structure of shelf information. [Figure 8] FIG. 8 is a diagram illustrating an example of the data structure of the register information. [Figure 9] FIG. 9 is a diagram illustrating an example of the data structure of the behavior DB. [Figure 10] FIG. 10 is a diagram for explaining the second detection method. [Figure 11] FIG. 11 is a flowchart illustrating the detection process of the information processing device according to the present embodiment. [Figure 12] FIG. 12 is a flowchart illustrating the identification process of the information processing device according to the present embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an information processing program, an information processing method, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0013] FIG. 1 is a diagram for explaining the processing of an information processing device according to this embodiment. For example, as shown in FIG. 1, a sales floor 5 is provided with a camera 10 and shelves 20a, 20b, and 20c for storing products. A cash register area 6 is provided with a camera 11 and cash registers 30a and 30b. In the following description, the shelves 20a to 20c will be collectively referred to as "shelf 20" as appropriate. The cash registers 30a and 30b will be collectively referred to as "cash register 30."

[0014] The information processing device analyzes the video from the camera 10 to identify the behavior of a specific user taking products from the shelf 20, and generates product acquisition information D1. The product acquisition information D1 includes information that identifies the shelf 20 (such as the shelf number) and information that associates the number of times the product was taken. For example, if user C1 takes three products from shelf 20c, the product acquisition information D1 is set to include information such as "shelf 20c: 3 items." The information processing device also extracts personal characteristic information of user C1.

[0015] The information processing device analyzes the video from camera 11 to identify the same person as user C1 among multiple users in the cash register area 6 included in the video from camera 11. The information processing device compares the personal characteristic information of user C1 with the personal characteristic information of user C1' and the personal characteristic information of user C2 to identify that user C1 and user C1' are the same person, and that user C1 and user C2 are different people.

[0016] When user C1' pays at the register 30, the information processing device acquires purchase history information D2 from the register 30. The purchase history information D2 includes information associating product items with the purchase quantities of the corresponding items. For example, user C1' pays at the register 30b and breaks down the purchase quantity of vegetables to "3," meat to "2," beverage to "1," and alcohol to "3." In this case, the information "vegetables: 3, meat: 2, alcohol: 3" is set in the purchase history information D2.

[0017] The information processing device associates product acquisition information D1 of user C1 with purchase history information D2 of user C1', who is the same person as user C1, and registers the associated information in a record DB (Data Base) 145. For example, the product acquisition information D1 and purchase history information D2 related to user C1 described in FIG. 1 correspond to the first row of the record DB 145. The information processing device repeatedly executes the above process for multiple users, and registers the relationship between the product acquisition information D1 and the purchase history information D2 in the record DB 145.

[0018] The information processing device statistically associates the shelves 20 with the product items by using the information stored in the record DB 145. That is, the information processing device identifies the product items associated with the shelves 20 as the products stored on the shelves 20.

[0019] For example, in conventional technology, camera images are matched with images of specific products, so if the product images are low resolution, the product cannot be identified. Figure 2 is a diagram showing an example of a low-resolution product image. Image Im1 shown in Figure 2 is an image of a can of beer, but because it is low resolution, it is difficult to determine whether it matches a pre-prepared image of the specific product. For example, the degree of match with various cans other than canned beer contained in the image database is similar, so it is not possible to identify image Im1 as canned beer.

[0020] In contrast, the information processing device according to this embodiment analyzes the video from the camera 10 to identify the behavior of a specific user taking out a product from the shelf 20 and generate product acquisition information D1. Even with a low-resolution image, it is possible to identify the behavior of taking out some kind of product. Furthermore, the information processing device statistically associates the shelf 20 with the product type using the relationship between the product acquisition information D1 and the purchase history information D2 acquired from the cash register 30. This makes it possible to identify the products stored on the shelf 20. The information processing device 100 identifies the products to be associated with the shelf so as to minimize the difference between the predicted value of the number of products purchased based on the user's taking out behavior and the observed value.

[0021] Next, a specific description will be given of processes (1) to (3) in which the information processing device according to this embodiment statistically identifies products stored on the shelf 20 based on the information stored in the record DB 145. The information processing device may execute any of the processes (1) to (3) described below.

[0022] First, we will explain the process (1) for identifying products stored on the shelves 20. This process (1) is based on the premise that the cameras 10 for detecting the behavior of users taking out products cover all sales areas and there are no missed or false detections.

[0023] FIG. 3 is a diagram for explaining the process (1) of identifying products stored on a shelf. Here, shelves 20a to 20c are shown, but other shelves may also be included. Also, item-A (vegetables), item-B (meat), item-C (drinks), and item-D (alcohol) are shown as product types, but other items may also be included. The information processing device sets a variable k according to the combination of shelf and product type. A variable (or unknown) corresponding to "a certain shelf x" and "a certain product type Y" is defined as "variable k xY For example, the variable corresponding to the shelf 20a and the item-A (vegetables) is represented as variable k. aA Let's say.

[0024] For the number of shelves M, the number of items N, and the user l, the following formula (1) holds.i (l) indicates the number of items that user l has acquired from shelf i. j (l) indicates the number of items of product j purchased by user l.

[0025]

number

[0026] In equation (1), there are M×N unknowns, and N equations are set for each user. Therefore, a solution (variable k) can be calculated using data (product acquisition information D1, purchase history information D2) of M users. The information processing device identifies products stored on the shelf 20 based on the variable k whose value is equal to or greater than a threshold value among multiple variables k. For example, the information processing device identifies the product stored on the shelf 20 based on the variable k corresponding to the shelf 20a and item-A (vegetables). aA is equal to or greater than the threshold value, the item of the product stored on shelf 20a is identified as "vegetables."

[0027] In the process (1), the value of the variable k is either 0 or 1, and the information processing device therefore identifies the type of merchandise stored on the shelf 20 based on the variable k whose value is 1.

[0028] As described above, if the camera 10 covers all sales areas and there is no missed or false detection of product removal based on the image from the camera 10, the information processing device can identify the items of products stored on the shelf 20 by executing process (1).

[0029] Next, we will explain the process (2) for identifying products stored on the shelves 20. In this process (2), the cameras 10 for detecting the user's action of taking out a product cover all sales areas, but it is assumed that there are detection misses and false detections. If there are detection misses and false detections, t i (l) However, this may not be correct.

[0030] The above formula (1) is a linear formula, so it can be expressed as formula (2). In formula (2), T (l) is t i (l) is an (N, N×M) matrix with elements k=(k aA ,k bA ,k cA ,···), and the number of dimensions is N×M. p (l) is p (l) =(p A (l) ,p B (l) ,p C (l) ,···), and the number of dimensions is N.

[0031]

number

[0032] The information processing device obtains the relationship of formula (2) by combining data (product acquisition information D1, purchase history information D2) of L users (L>M). Here, t i (l) If there is an error in the equation (2), there is no solution to the equation (2). Therefore, the information processing device calculates an approximate solution for the variable k based on the equation (3). However, as a constraint, aA ≧0,k bA ≧0,k cA ≧0,···".

[0033]

number

[0034] The information processing device calculates (searches) a variable k that minimizes the difference between T×k and p, as shown in equation (3). The information processing device identifies the products stored on the shelf 20 based on the calculated variable k whose value is equal to or greater than a threshold. For example, the information processing device identifies the product stored on the shelf 20 based on the variable k corresponding to the shelf 20a and item-A (vegetables). aAis equal to or greater than the threshold value, the item of the product stored on shelf 20a is identified as "vegetables."

[0035] As described above, even if there is a possibility that the information processing device misses or misdetects the removal of a product based on the image of camera 10, as long as camera 10 covers all sales areas, it can execute process (2) and identify the items of the product stored on shelf 20.

[0036] Next, we will explain the process (3) for identifying products stored on the shelves 20. In this process (3), it is assumed that the cameras 10 for detecting the user's action of taking out a product do not cover all sales areas, and that there are cases of missed detections and false positives. If missed detections and false positives occur, t i (l) However, this may not be correct.

[0037] 4 is a diagram for explaining the process (3) of identifying products stored on a shelf. Based on each user's product acquisition information D1 and purchase history information D2, the information processing device uses data mining to detect items commonly purchased by multiple users who took products from the same shelf 20, and identifies the types of products stored on the shelf 20.

[0038] 4, it is assumed that user C1 acquires products from multiple shelves 20 including shelf 20a, user C2 acquires products from multiple shelves 20 including shelf 20a, and user C3 acquires products from multiple shelves 20 including shelf 20a. In this case, users C1 to C3 are users who have in common the fact that they acquired products from shelf 20a.

[0039] For example, purchase history information D2-1 of user C1 includes a purchase history of alcohol, fish, and meat. Purchase history information D2-2 of user C2 includes a purchase history of frozen foods and alcohol. Purchase history information D2-3 of user C2 includes a purchase history of meat, alcohol, and prepared foods. The information processing device compares the purchase history information D2-1, D2-2, and D2-3, and identifies the common product item "alcohol" as the product item stored on shelf 20a. Note that the above process can also be performed using data of L users (product acquisition information D1, purchase history information D2) to find item X that satisfies equation (4), which will be described later.

[0040] For example, process (3) can be formulated by equation (4). Equation (4) is an equation for shelf i, and exists for the number of shelves (=M). In equation (4), t i (l) is the number of items acquired from shelf i by user l. x (l) is the number of items X purchased by user l.

[0041]

number

[0042] s(x) included in equation (4) is expressed by equation (5). s(x) is 1 when x is 1 or greater.

[0043]

number

[0044] s(t i (l) ), if user l has acquired the product from shelf i at least once, then s(t i (l) )=1, and if not acquired, s(t i (l) )=0.

[0045] s(p x (l)), if user l has purchased at least one item of item X, then s(p x (l) )=1, and if no purchase is made, s(p x (l) )=0.

[0046] "s(t i (l) )-s(p x (l) The value of "X" is 0 if the combination is correct and 1 if it is inappropriate. The information processing device identifies the X that minimizes the total value of all users as the product item that is most likely to be correct.

[0047] As described above, the information processing device may miss or misdetect product removal based on the image from camera 10, and even if camera 10 does not cover all sales areas, it can perform process (3) to identify the items of products stored on shelf 20.

[0048] Next, an example of a system according to this embodiment will be described. Fig. 5 is a diagram showing a system according to this embodiment. As shown in Fig. 5, this system has a camera 10, a camera 11, a cash register 30, and an information processing device 100. The camera 10, the camera 11, and the cash register 30 are connected to each other via a network 15.

[0049] The camera 10 is a camera that captures images of the shelves 20 installed in the sales floor 5 described in Fig. 1. The camera 10 transmits information about the captured images to the information processing device 100.

[0050] Camera 11 is a camera that captures an image of cash register area 6 described in Fig. 1. The image capturing range of camera 11 includes cash register 30. Camera 11 transmits information about the captured image to information processing device 100.

[0051] When a user pays at the register 30, the register 30 generates purchase history information D2 and transmits the purchase history information D2 to the information processing device 100. The purchase history information D2 may also include information that identifies the register 30. When the register 30 receives a request from a store clerk or a user to start a transaction, it transmits transaction start information to the information processing device 100. A register number that identifies the register 30 is set in the transaction start information.

[0052] 1 to 4, the information processing device 100 identifies the types of merchandise stored on the shelf 20. An example of the configuration of the information processing device 100 will be described below.

[0053] 6 is a functional block diagram showing the configuration of an information processing device according to this embodiment. As shown in FIG. 6, the information processing device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0054] The communication unit 110 transmits and receives information between the cameras 10 and 11, the cash register 30, etc. via the network 15. For example, the communication unit 110 is realized by a NIC (Network Interface Card) or the like.

[0055] The input unit 120 is realized using input devices such as a keyboard and a mouse, and inputs various information to the control unit 150 in response to input operations by an operator.

[0056] The display unit 130 is realized by a display device such as a liquid crystal display. For example, the display unit 130 may display the items of merchandise stored on the shelf 20, which have been identified by the control unit 150.

[0057] The storage unit 140 has a first video buffer 141, a second video buffer 142, shelf information 143a, register information 143b, a behavior DB 144, and a record DB 145. The storage unit 140 is realized by, for example, a semiconductor memory element such as a flash memory, or a storage device such as a hard disk or an optical disk.

[0058] The first video buffer 141 is a buffer that stores video information received from the camera 10 installed in the sales floor 5. In the following description, the video information received from the camera 10 will be referred to as "first video information." The first video information includes time-series images (still images).

[0059] Second video buffer 142 is a buffer that stores video information received from camera 11 installed in cash register area 6. In the following description, the video information received from camera 11 will be referred to as "second video information." The second video information includes time-series images (still images).

[0060] The shelf information 143a is information that indicates the area of ​​the shelf 20 on the image (first video information) captured by the camera 10. FIG. 7 is a diagram showing an example of the data structure of the shelf information. As shown in FIG. 7, the shelf information 143a associates a shelf number with shelf area information. The shelf number is a number that uniquely identifies the shelf 20. For example, shelf numbers T20a, T20b, and T20c correspond to the shelf 20a, shelf 20b, and shelf 20c, respectively. The shelf area information is information that indicates the area of ​​the shelf 20 on the image (first video information). For example, the shelf area information has the two-dimensional coordinates of the upper left corner and the two-dimensional coordinates of the lower right corner of the shelf 20.

[0061] Cash register information 143b is information that indicates the area of ​​cash register 30 on the image (second video information) captured by camera 11. FIG. 8 is a diagram showing an example of the data structure of cash register information. As shown in FIG. 8, cash register information 143b associates cash register numbers with cash register area information. Cash register numbers are numbers that uniquely identify cash register 30. For example, cash register numbers T30a and T30b correspond to cash register 30a and cash register 30b, respectively. Cash register area information is information that indicates the area of ​​cash register 30 on the image (second video information). For example, cash register area information has two-dimensional coordinates of the upper left corner and two-dimensional coordinates of the lower right corner of cash register 30.

[0062] The behavior DB 144 stores various information when a user's behavior of removing a product from the shelf 20 is identified based on the first video information. Fig. 9 is a diagram showing an example of the data structure of the behavior DB. As shown in Fig. 9, the behavior DB 144 associates an item number, a time, a shelf number, and person characteristic information.

[0063] The item number is a number that identifies each record in the behavior DB 144. The time is the time when the user specified the behavior of taking a product from the shelf 20. The shelf number is a number that uniquely identifies the shelf 20. The person characteristic information is vector information that indicates the characteristics of the user who took the product. For example, the person characteristic information is 512-dimensional vector information obtained by person re-identification or the like.

[0064] 1, the record DB 145 stores information that associates product acquisition information D1 with purchase history information D2. For example, the record DB 145 associates, for each user, the number of times the user has taken out a product from the shelf 20 with the number of items purchased for each product item.

[0065] Returning to the explanation of Fig. 6, the control unit 150 has a receiving unit 151, a detecting unit 152, and an identifying unit 153. The control unit 150 is realized by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The control unit 150 may also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0066] Receiving unit 151 receives the first video information from camera 10. Receiving unit 151 stores the first video information in first video buffer 141.

[0067] Receiving unit 151 receives the second video information from camera 11. Receiving unit 151 stores the second video information in second video buffer 142. Note that receiving unit 151 may start receiving the second video information when identification unit 153, which will be described later, receives transaction start information from cash register 30.

[0068] The detection unit 152 detects the behavior of the user taking out a product based on the first video information stored in the first video buffer 141. When the detection unit 152 detects the behavior of taking out a product, the detection unit 152 associates the time, the shelf number of the shelf 20 from which the product was taken out, and the personal characteristic information of the user, and registers them in the behavior DB 144. The detection unit 152 acquires time information from a timer or the like.

[0069] Here, an example of a process in which the detection unit 152 detects the user's behavior of taking out a product will be described. A first detection method and a second detection method will be described below. The detection unit 152 detects the user's behavior of taking out a product using the first detection method or the second detection method.

[0070] The first detection method will be described. The detection unit 152 detects the user's action of taking out a product using HOID (Human Object Interaction Detection). HOID is a technology that uses a moving image (first video information) as input and recognizes the interaction behavior between a user and an object. For example, the detection unit 152 detects the user's action of taking out a product using a learning model (HOID) trained using training data that uses video information of the shelf 20 as input and outputs whether or not the product has been taken out.

[0071] The second detection method will now be described. FIG. 10 is a diagram illustrating the second detection method. The detection unit 152 analyzes the first video information to acquire posture information of the target user C1. The posture information only needs to include the area of ​​the user C1's hand. The detection unit 152 detects the timing when the hand of the user C1 enters the area of ​​the shelf 20a on the image or approaches the area of ​​the shelf 20a. The shelf number and shelf area information related to the area of ​​the shelf 20a are registered in the shelf information 143a, and the detection unit 152 uses this shelf area information.

[0072] For example, as shown in FIG. 10, the area near shelf 20a is defined as "area 25." An image of area 25 when user C1 reaches his / her hand out to shelf 20a is defined as image 25a. An image of area 25 when user C1 removes his / her hand from shelf 20a is defined as image 25b. The detection unit 152 uses a discrimination model 26 to determine whether or not a product is present in user C1's hand. The discrimination model 26 is a model that identifies whether or not a product is present in the user's hand, and may be a learning model based on the above-mentioned HOID.

[0073] The detection unit 152 detects that the user C1 has taken the action of taking out a product when the image 25a is input into the identification model 26 and is identified as "no product" and the image 25b is input into the identification model 26 and is identified as "product present."

[0074] The detection unit 152 detects that the user C1 has performed the action of returning the product when the image 25a is input into the discrimination model 26 and is recognized as "product present" and the image 25b is input into the discrimination model 26 and is recognized as "product absent." When the detection unit 152 detects that the user C1 has performed the action of returning the product within a predetermined time after detecting that the user C1 has performed the action of taking out the product, the detection unit 152 invalidates the previous detection of the action of the user C1 taking out the product.

[0075] When detecting the action of the user C1 taking out a product using the first or second detection method, the detection unit 152 extracts personal characteristic information of the user C1 included in the first video information.

[0076] For example, when extracting person characteristic information, the detection unit 152 uses a learning model based on Person Re-Identification. When the first video information is input to the learning model based on Person Re-Identification, the characteristic information of the user C1 included in the first video information is output as a high-dimensional vector. That is, the detection unit 152 obtains the person characteristic information of the user C1 by inputting the first video information to the learning model based on Person Re-Identification.

[0077] In addition to the above processing, the detection unit 152 may extract personal characteristic information from the first video information using the following method. For example, the detection unit 152 may detect the user's belongings (bag, hat, glasses, etc.) using object detection technology and use the detected personal characteristic information. The detection unit 152 may estimate the user's attributes (gender, age) from the first video information and use the detected personal characteristic information. The detection unit 152 may add information about companions (e.g., a group of three) and the characteristics of the companions to the user's personal characteristic information. The detection unit 152 determines the companions based on the similarity of movement trajectories, interactions with the same shopping cart, etc.

[0078] The detection unit 152 repeatedly executes the above process each time it detects the user's behavior of taking out a product, and registers the time when the behavior of taking out the product was detected, the shelf number of the shelf 20 from which the product was taken out, and the user's personal characteristic information in the behavior DB 144 in association with each other.

[0079] Returning to the explanation of Fig. 6, the identification unit 153 performs the process described below to identify the item of merchandise stored on the shelf 20. The identification unit 153 may output the relationship between the shelf 20 and the item of merchandise to the display unit 130 for display, or may notify a designated external device.

[0080] When the identification unit 153 acquires transaction start information from the cash register 30, it identifies the cash register area information of the cash register 30 where the transaction has started based on the cash register number set in the transaction start information and the cash register information 143b.

[0081] The identification unit 153 obtains from the second video buffer 142 second video information from the timing when transaction start information is obtained from the cash register 30, detects users located near the cash register area of ​​the cash register 30, and extracts the detected personal characteristic information. This associates the cash register 30 where the transaction (checkout) has started with the personal characteristic information of the user who will be making the payment. The identification unit 153 may associate the cash register number of the cash register 30 where the transaction has started with the personal characteristic information of the user who will be making the payment at that cash register, and register this in the storage unit 140. Note that the identification unit 153 excludes a predetermined area where a store clerk is located from the user detection range.

[0082] The identification unit 153 inputs the second video information into a learning model based on Person Re-Identification to obtain personal feature information of a user located near the cash register 30. The explanation of the learning model based on Person Re-Identification is the same as that above. In the following explanation, the personal feature information of a user located near the cash register 30 identified based on the second video information will be referred to as "query feature information."

[0083] The identification unit 153 calculates the similarity (cosine similarity) between the query feature information and each piece of person feature information stored in the behavior DB 144, and identifies records whose similarity is equal to or greater than a predetermined similarity. The identification unit 153 classifies each identified record by shelf number, and based on the classification result, identifies the number of times a product has been acquired for each shelf, and generates product acquisition information D1. For example, if there are three records for shelf number "20c", the identification unit 153 sets the number of times a product corresponding to shelf number "20c" to "3".

[0084] The identification unit 153 receives purchase history information D2 from the cash register 30, associates the purchase history information D2 with the product acquisition information D1, and registers the associated information in the record DB 145. The identification unit 153 may associate the purchase history information D2 with the product acquisition information D1 based on the cash register number set in the transaction start information and the cash register number set in the purchase history information D2.

[0085] The identification unit 153 repeatedly executes the above process to register a plurality of pairs of the product acquisition information D1 and the purchase history information D2 in the record DB 145.

[0086] Next, the identification unit 153 executes one of the processes (1) to (3) for identifying the product stored on the shelf 20 described above.

[0087] When the specifying unit 153 executes the process (1), the specifying unit 153 uses the formula (1). The specifying unit 153 determines t in the formula (1) from the relationship between the product acquisition information D1 and the purchase history information D2 set in the record DB 145. i (l) , p j (l) and calculates the value of the variable k. The identification unit 153 identifies the product stored on the shelf 20 based on the variable k whose value is equal to or greater than the threshold value among the multiple variables k.

[0088] When the specifying unit 153 executes the process (2), the specifying unit 153 uses the formula (3). The specifying unit 153 determines T in the formula (3) from the relationship between the product acquisition information D1 and the purchase history information D2 set in the record DB 145. (l) , p (l) and calculates an approximate solution for the variable k. The identification unit 153 identifies the product stored on the shelf 20 based on the variable k whose value is equal to or greater than a threshold value among the multiple variables k.

[0089] When the identification unit 153 executes the process (3), it uses data mining to detect items commonly purchased by multiple users who have taken products from the same shelf 20, and identifies the items of the products stored on the shelf 20. For example, based on equations (4) and (5), the identification unit 153 searches for item X that minimizes the value of equation (4), and identifies it as the item of the product that is most likely to be correct.

[0090] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 11 is a flowchart showing the detection processing of the information processing device according to this embodiment. As shown in Fig. 11, the receiving unit 151 of the information processing device 100 receives the first video information from the camera 10 in the sales floor 5 and registers it in the first video buffer 141 (step S101).

[0091] The detection unit 152 of the information processing device 100 detects the removal of a product based on the first video information (step S102). If the detection unit 152 detects the act of returning the product (step S103, Yes), it invalidates the previous removal of the product (step S104) and returns to step S102.

[0092] On the other hand, if the detection unit 152 does not detect the behavior of returning the product (No at Step S103), the process proceeds to Step S105. The detection unit 152 identifies the shelf number (Step S105). The detection unit 152 extracts person characteristic information based on the first video information (Step S106).

[0093] The detection unit 152 registers the time, shelf number, and person characteristic information in the behavior DB 144 (step S107). If the detection unit 152 continues the process (step S108, Yes), the detection unit 152 proceeds to step S102 again. If the detection unit 152 does not continue the process (step S108, No), the detection unit 152 ends the detection process.

[0094] 12 is a flowchart showing the identification process of the information processing device according to this embodiment. As shown in FIG. 12, the identification unit 153 of the information processing device 100 acquires transaction start information from the cash register 30 (step S201). The receiving unit 151 of the information processing device 100 receives second video information from the camera 11 in the cash register area 6 and registers it in the second video buffer 142 (step S202).

[0095] The identification unit 153 detects the user from the second video information and extracts person characteristic information (step S203). The identification unit 153 generates product acquisition information D1 based on the query characteristic information and the behavior DB 144 (step S204).

[0096] The identification unit 153 acquires the purchase history information D2 from the cash register 30 (step S205). The identification unit 153 registers a set of the product acquisition information D1 and the purchase history information D2 in the record DB 145 (step S206).

[0097] If the number of records in the record DB 145 is not equal to or greater than the predetermined number (step S207, No), the identifying unit 153 proceeds to step S201. If the number of records in the record DB 145 is equal to or greater than the predetermined number (step S207, Yes), the identifying unit 153 proceeds to step S208.

[0098] The identification unit 153 executes any one of processes (1) to (3) to identify the item of the product stored on the shelf (step S208). The identification unit 153 outputs the relationship between the shelf 20 and the item of the product stored on the shelf 20 (step S209).

[0099] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 identifies the behavior of a specific user taking products from the shelf 20 by analyzing the video captured by the camera 10, and generates product acquisition information D1. The information processing device 100 uses the relationship between the product acquisition information D1 and purchase history information D2 acquired from the cash register 30 to identify the types of products stored on the shelf 20 so as to minimize the difference between the predicted and observed number of products purchased based on the taking-out behavior. This makes it possible to identify the products stored on the shelf 20 even if the video captured by the camera 10 has low resolution.

[0100] The information processing device 100 identifies the types of products stored on the shelf 20 based on process (1). For example, the information processing device 100 sets a variable k related to the combination of a shelf and a predetermined product, and calculates the value of the variable k based on a simultaneous equation in which the value obtained by multiplying a predicted value (the product of the variable and the number of times the product was acquired) is equal to the observed value (the number of items purchased). The information processing device 100 identifies, as a product to be associated with the shelf, a predetermined product that is paired with a shelf related to the variable k for which the calculated value is equal to or greater than a threshold. This makes it possible to identify the products stored on the shelf 20 even if the image from the camera 10 has low resolution.

[0101] The information processing device 100 identifies the types of merchandise stored on the shelves 20 based on process (2). For example, the information processing device 100 calculates a variable vector that minimizes the value obtained by subtracting the observed value vector from the predicted value vector (the product of the variable vector and a matrix having the number of times a product is taken as an element) according to equation (3), and identifies the merchandise to be associated with the shelves based on the values ​​of each element of the calculated variable vector. As a result, even if there is a possibility that the camera 10 may miss or falsely detect the removal of a product based on the video from the camera 10, the information processing device 10 can execute process (2) to identify the types of merchandise stored on the shelves 20, as long as the camera 10 covers all sales areas.

[0102] The information processing device 100 identifies the types of merchandise stored on the shelf 20 based on process (3). For example, the information processing device 100 uses data mining to detect items commonly purchased by multiple users who removed products from the same shelf 20, based on each user's product acquisition information D1 and purchase history information D2, and identifies the types of merchandise stored on the shelf 20. For example, the information processing device 100 identifies the merchandise items on the relevant shelf according to equation (4) so ​​as to minimize the difference between the predicted value (whether or not a product was acquired, i.e., whether or not the product was likely to be purchased) and the observed value (whether or not a product was purchased), for each shelf. As a result, even if there is a possibility of missed or false detection of product removal based on the video from the camera 10 and the camera 10 does not cover all sales areas, it is possible to perform process (3) and identify the types of merchandise stored on the shelf 20.

[0103] Furthermore, the information processing device 100 uses the relationship between the product acquisition information D1 and the purchase history information D2 to identify the types of products stored on the shelf 20 so as to minimize the difference between the predicted and observed number of products purchased based on the picking behavior. This allows the product types to be identified with simpler processing than when identifying the products stored on the shelf 20 by analyzing high-resolution video of the shelf 20.

[0104] The above-described processing by the information processing device 100 is merely an example, and other processing may be executed by the information processing device 100. The following describes other processing executed by the information processing device 100.

[0105] The detection unit 152 of the information processing device 100 identifies the area of ​​the shelf 20 included in the first video information using the shelf information 143a prepared in advance, but the area of ​​the shelf 20 may also be identified using a learning model based on Semantic Segmentation or the like.

[0106] When receiving transaction start information and generating product acquisition information D1, the identification unit 153 of the information processing device 100 compares the query feature information with all records registered in the behavior DB 144, but this is not limited to this. The information processing device 100 may also compare records from within the most recent T hours from the time the transaction start information was received. The identification unit 153 may also perform person matching using a camera installed at the entrance or exit of the store, and generate product acquisition information D1 by limiting the records in the behavior DB 144 from the time the relevant user entered the store to the time they left the store.

[0107] The identification unit 153 of the information processing device 100 may use the location information of multiple cameras set up in the store to track the movement of users between the cameras, and based on the tracking results, may associate the user in the sales floor 5 with the user in the cash register area 6. The identification unit 153 of the information processing device 100 may use multiple beacons and receiving terminals placed in the store to track user location information, and may associate the user in the sales floor 5 with the user in the cash register area 6.

[0108] Next, an example of the hardware configuration of a computer that realizes the same functions as the information processing device 100 described in the above embodiment will be described. Fig. 13 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0109] 13, computer 300 includes CPU 301 for executing various types of arithmetic processing, input device 302 for receiving data input from a user, and display 303. Computer 300 also includes communication device 304 for transmitting and receiving data to and from external devices via a wired or wireless network, and interface device 305. Computer 300 also includes RAM 306 for temporarily storing various types of information, and hard disk drive 307. Devices 301 to 307 are connected to bus 308.

[0110] The hard disk drive 307 stores a receiving program 307a, a detecting program 307b, and a specifying program 307c. The CPU 301 reads out each of the programs 307a to 307c and loads them into the RAM 306.

[0111] The receiving program 307a functions as the receiving process 306a, the detecting program 307b functions as the detecting process 306b, and the identifying program 307c functions as the identifying process 306c.

[0112] The processing of the reception process 306a corresponds to the processing of the reception unit 151. The processing of the detection process 306b corresponds to the processing of the detection unit 152. The processing of the identification process 306c corresponds to the processing of the identification unit 153.

[0113] It should be noted that each of the programs 307a to 307c does not necessarily have to be stored in the hard disk drive 307 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 300. Then, the computer 300 may read and execute each of the programs 307a to 307c.

[0114] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0115] (Appendix 1) From a video of an area in a store that includes shelves that store products, the action of a specific user among multiple people taking a product from the shelf is identified, Identifying the specific user and the cash register from a video of an area in the store that includes the cash register; storing the specific user and the payment machine in association with each other in a storage unit; receiving a purchase history transmitted from the payment machine associated with the specific user; identifying one or more items included in the purchase history of the cash register; Based on the number of identified products and the number of take-out actions of the user associated with the checkout machine, products to be associated with the shelf are identified so as to minimize the difference between the predicted and observed number of purchases of products based on the take-out actions. An information processing program that causes a computer to execute a process.

[0116] (Appendix 2) The information processing program described in Appendix 1 is characterized in that the process of identifying a product to be associated with the shelf sets a variable related to the combination of the shelf and a specified product, calculates the value of the variable based on a simultaneous equation in which the predicted value obtained by multiplying the variable by the number of actions to be taken is equal to the observed value, and identifies the specified product that is paired with the shelf related to the variable for which the calculated value is equal to or greater than a threshold as the product to be associated with the shelf.

[0117] (Appendix 3) The information processing program described in Appendix 1 is characterized in that the process of identifying the product to be associated with the shelf includes setting a predicted value vector obtained by multiplying a variable vector having elements that are variables related to the combination of the shelf and a specified product by a matrix having elements that are the number of actions to be taken, and an observed value vector having elements that are the observed values, calculating a variable vector that minimizes the value obtained by subtracting the observed value vector from the predicted value vector, and identifying the product to be associated with the shelf based on the values ​​of each element of the calculated variable vector.

[0118] (Appendix 4) The information processing program described in Appendix 1 is characterized in that the process of identifying products to be associated with the shelf identifies products purchased in common by multiple users who took the products from the same shelf based on the purchase history as products obtained from the same shelf.

[0119] (Appendix 5) From a video of an area in a store that includes shelves that store products, the action of a specific user among multiple people taking a product from the shelf is identified, Identifying the specific user and the cash register from a video of an area in the store that includes the cash register; storing the specific user and the payment machine in association with each other in a storage unit; receiving a purchase history transmitted from the payment machine associated with the specific user; identifying one or more items included in the purchase history of the cash register; Based on the number of identified products and the number of take-out actions of the user associated with the checkout machine, products to be associated with the shelf are identified so as to minimize the difference between the predicted and observed number of purchases of products based on the take-out actions. An information processing method characterized in that the processing is executed by a computer.

[0120] (Appendix 6) The information processing method described in Appendix 5 is characterized in that the process of identifying a product to be associated with the shelf involves setting a variable related to the combination of the shelf and a specified product, calculating the value of the variable based on a simultaneous equation in which the predicted value obtained by multiplying the variable by the number of actions to be taken is equal to the observed value, and identifying the specified product that is paired with the shelf related to the variable for which the calculated value is equal to or greater than a threshold as the product to be associated with the shelf.

[0121] (Appendix 7) The information processing method described in Appendix 5 is characterized in that the process of identifying the product to be associated with the shelf involves setting a predicted value vector obtained by multiplying a variable vector having elements that are variables related to the combination of the shelf and a specified product by a matrix having elements that are the number of actions to be taken, and an observed value vector having elements that are the observed values, calculating a variable vector that minimizes the value obtained by subtracting the observed value vector from the predicted value vector, and identifying the product to be associated with the shelf based on the values ​​of each element of the calculated variable vector.

[0122] (Appendix 8) The information processing method described in Appendix 5 is characterized in that the process of identifying products to be associated with the shelf identifies products purchased in common by multiple users who took the products from the same shelf based on the purchase history as products obtained from the same shelf.

[0123] (Appendix 9) From a video of an area in a store that includes shelves that store products, the action of a specific user among multiple people taking a product from the shelf is identified, Identifying the specific user and the cash register from a video of an area in the store that includes the cash register; storing the specific user and the payment machine in association with each other in a storage unit; receiving a purchase history transmitted from the payment machine associated with the specific user; identifying one or more items included in the purchase history of the cash register; Based on the number of identified products and the number of take-out actions of the user associated with the checkout machine, products to be associated with the shelf are identified so as to minimize the difference between the predicted and observed number of purchases of products based on the take-out actions. An information processing device having a control unit that executes processing.

[0124] (Appendix 10) The information processing device described in Appendix 9 is characterized in that the process of identifying a product to be associated with the shelf sets a variable related to the combination of the shelf and a predetermined product, calculates the value of the variable based on a simultaneous equation in which the predicted value obtained by multiplying the variable by the number of actions to be taken is equal to the observed value, and identifies the predetermined product that is paired with the shelf related to the variable for which the calculated value is equal to or greater than a threshold as the product to be associated with the shelf.

[0125] (Appendix 11) The information processing device described in Appendix 9 is characterized in that the process of identifying the product to be associated with the shelf includes setting a predicted value vector obtained by multiplying a variable vector having elements that are variables related to the combination of the shelf and a specified product by a matrix having elements that are the number of actions to be taken, and an observed value vector having elements that are the observed values, calculating a variable vector that minimizes the value obtained by subtracting the observed value vector from the predicted value vector, and identifying the product to be associated with the shelf based on the values ​​of each element of the calculated variable vector.

[0126] (Appendix 12) The information processing device described in Appendix 9 is characterized in that the process of identifying products to be associated with the shelf identifies products purchased in common by multiple users who took the products from the same shelf based on the purchase history as products obtained from the same shelf. [Explanation of symbols]

[0127] 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 First video buffer 142 Second Video Buffer 143a Shelf Information 143b Cashier Information 144 Behavior DB 145 Record DB 150 control section 151 Receiving unit 152 Detection unit 153 Specific part

Claims

1. Identifying, from a video image of an area in a store including shelves that store products, the behavior of a specific user among a plurality of people taking a product from the shelves; Identifying the specific user and the cash register from a video of an area in the store that includes the cash register; storing the specific user and the payment machine in association with each other in a storage unit; receiving a purchase history transmitted from the payment machine associated with the specific user; Identifying one or more items included in the purchase history of the cash register; A predicted value vector is set by multiplying a variable vector having elements that are variables related to the combination of the shelf and a predetermined product by a matrix having elements that are the number of actions to be taken, and an observed value vector has elements that are observed values ​​that are the number of products identified from the purchase history, and a variable vector is calculated that minimizes the value obtained by subtracting the observed value vector from the predicted value vector, and the product to be associated with the shelf is identified based on the values ​​of each element of the calculated variable vector. An information processing program that causes a computer to execute a process.

2. The information processing program described in Claim 1 is characterized in that the computer is further made to execute the process of setting a variable related to the combination of the shelf and a specified product, calculating the value of the variable based on a simultaneous equation in which the predicted value of the number of purchases of the product, calculated by multiplying the variable by the number of actions to be taken, is equal to the observed value, and identifying the specified product that is paired with the shelf related to the variable for which the calculated value is equal to or greater than a threshold, as the product to be associated with the shelf.

3. The information processing program according to claim 1, characterized in that the process of identifying products to be associated with the shelf identifies products purchased in common by multiple users who removed the products from the same shelf based on the purchase history as products obtained from the same shelf.

4. Identifying, from a video image of an area in a store including shelves that store products, the behavior of a specific user among a plurality of people taking a product from the shelves; Identifying the specific user and the cash register from a video of an area in the store that includes the cash register; storing the specific user and the payment machine in association with each other in a storage unit; receiving a purchase history transmitted from the payment machine associated with the specific user; Identifying one or more items included in the purchase history of the cash register; A predicted value vector is set by multiplying a variable vector having elements that are variables related to the combination of the shelf and a predetermined product by a matrix having elements that are the number of actions to be taken, and an observed value vector has elements that are observed values ​​that are the number of products identified from the purchase history, and a variable vector is calculated that minimizes the value obtained by subtracting the observed value vector from the predicted value vector, and the product to be associated with the shelf is identified based on the values ​​of each element of the calculated variable vector. An information processing method characterized in that the processing is executed by a computer.

5. Identifying, from a video image of an area in a store including shelves that store products, the behavior of a specific user among a plurality of people taking a product from the shelves; Identifying the specific user and the cash register from a video of an area in the store that includes the cash register; storing the specific user and the payment machine in association with each other in a storage unit; receiving a purchase history transmitted from the payment machine associated with the specific user; Identifying one or more items included in the purchase history of the cash register; A predicted value vector is set by multiplying a variable vector having elements that are variables related to the combination of the shelf and a predetermined product by a matrix having elements that are the number of actions to be taken, and an observed value vector has elements that are observed values ​​that are the number of products identified from the purchase history, and a variable vector is calculated that minimizes the value obtained by subtracting the observed value vector from the predicted value vector, and the product to be associated with the shelf is identified based on the values ​​of each element of the calculated variable vector. An information processing device having a control unit that executes processing.

Images