Article estimation device, article estimation method, and program

Abstract

To improve discrimination accuracy of an article taken out from a shelf.SOLUTION: An article estimation device includes a determination unit and an output unit. The determination unit determines a height of a hand of a person when stretching the hand to a shelf on which a plurality of articles can be placed, by using a depth sensor which can detect movement of the hand of the person located in a space (shelf front space) in front of the shelf. The output unit executes alert processing when a positional relation between a position of the shelf specified based on a detection value of a weight sensor provided on the shelf and the height of the hand does not satisfy a criterion.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an article estimation device, an article estimation method, and a program. [Background technology]

[0002] In recent years, technological developments have been underway to reduce labor in stores, factories, etc. For example, Patent Document 1 describes a method for measuring the total weight of the items 5 stored on a stock shelf during the task of packing multiple types of items taken out of a stock shelf into a box as a set, and using the measurement result to determine whether or not to issue a warning.

[0003] Patent Document 2 also describes that, in order to manage the handling of goods, inventory data is generated using the results of processing images of barcodes or QR codes (registered trademark) of goods. The inventory data is information that indicates the handling status of goods, and is information that associates, for example, the identification information of the photographed goods, the date and time when the goods were verified, the location where the goods were installed, and the identification information of the user who handled the goods. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-218289 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-210310 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to promote labor savings, it is preferable to be able to automatically identify an item taken from a shelf. One of the objects of the present invention is to improve the accuracy of identifying an item taken from a shelf. [Means for solving the problem]

[0006] The product estimation device in the present disclosure includes: a determining means for determining the height of a person's hand when the person extends the hand toward the shelf, using a depth sensor capable of detecting the movement of the hand of the person positioned in a front-shelf space, which is a space in front of the shelf on which a plurality of items can be placed; an output means for executing an alert process when the positional relationship between the height of the hand and the position of the shelf specified based on the detected value of a weight sensor provided on the shelf does not satisfy a standard; Equipped with.

[0007] The article estimation method in the present disclosure includes: The computer a depth sensor capable of detecting hand movement of a person positioned in a shelf space in front of a shelf on which a plurality of items can be placed is used to determine the height of the hand of the person when the person reaches the shelf; executing an alert process when the positional relationship between the height of the hand and the position of the shelf specified based on the detection value of a weight sensor provided on the shelf does not satisfy a criterion; This includes:

[0008] The first program in the present disclosure is Computer, a determining means for determining the height of a person's hand when the person extends the hand toward the shelf, using a depth sensor capable of detecting the movement of the hand of the person positioned in a front-shelf space, which is a space in front of the shelf on which a plurality of items can be placed; an output means for executing an alert process when the positional relationship between the height of the hand and the position of the shelf specified based on the detected value of a weight sensor provided on the shelf does not satisfy a standard; Function as. [Effects of the Invention]

[0009] According to the present invention, the accuracy of identifying an article taken out from a shelf is improved. [Brief explanation of the drawings]

[0010] The above-mentioned objects, as well as other objects, features and advantages, will become more apparent from the preferred embodiments described below and the accompanying drawings.

[0011] [Figure 1] 1 is a diagram showing a functional configuration of an article information estimation device according to an embodiment of the present invention, together with an environment in which the article information estimation device is used. [Figure 2] FIG. 10 is a diagram illustrating an example of data stored in a shelf allocation information storage unit. [Figure 3] 2 is a block diagram illustrating a hardware configuration of the product information estimation device shown in FIG. 1. FIG. [Figure 4] 10 is a flowchart illustrating an example of the operation of the product information estimation device. [Figure 5] 10 is a flowchart illustrating details of the process performed in step S104. [Figure 6] FIG. 10 is a diagram showing an example of a layout of shelves and weight sensors according to a modified example. [Figure 7] FIG. 10 is a diagram showing the functional configuration of an article information estimation device according to a second embodiment, together with the environment in which the article information estimation device is used. [Figure 8] FIG. 11 is a plan view for explaining the layout of a weight sensor according to a third embodiment. [Figure 9] 10 is a flowchart for explaining details of the item identification process (step S104 in FIG. 4) in the third embodiment. [Figure 10] FIG. 10 is a diagram showing the functional configuration of an article information estimation device according to a fourth embodiment, together with the environment in which the article information estimation device is used. [Figure 11] 13 is a flowchart illustrating details of step S104 in the fourth embodiment. [Figure 12] FIG. 10 is a diagram showing the functional configuration of an article information estimation device according to a fifth embodiment, together with the environment in which the article information estimation device is used. [Figure 13] 13 is a flowchart illustrating an example of the operation of the product information estimation device according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0013] [First embodiment] 1 is a diagram showing the functional configuration of an article information estimation device 10 according to this embodiment, together with the environment in which the article information estimation device 10 is used. The article information estimation device 10 according to this embodiment is a device that estimates an article 200 that a person has taken out from a shelf 20, and is used together with a weight sensor 30 and a depth sensor 40. In this figure, the shelf 20 is shown as viewed from the side.

[0014] The shelf 20 can hold a plurality of items 200. For example, if the shelf 20 is located in a store or a logistics center, the shelf 20 is a merchandise shelf, the items 200 are merchandise, and the person who takes out the items 200 is a customer or a store clerk (employee). If the shelf 20 is located in a pharmacy, the shelf 20 is a medicine shelf, the items 200 are medicines, and the person who takes out the items 200 is a pharmacist.

[0015] In this embodiment, the articles 200 are arranged on each of multiple shelves 20. Multiple types of articles 200 are placed on the multiple shelves 20. The shelf 20 on which each article 200 is placed is determined in advance. Therefore, if the shelf 20 from which the article 200 was taken out is known, the type of the article 200 can be estimated. However, the shelf 20 may be a single shelf.

[0016] The depth sensor 40 has a detection range that includes the space in front of the shelf 20 (hereinafter referred to as the shelf-front space) and generates data indicating the hand movement of a person located in the shelf-front space. For example, the depth sensor 40 is disposed above the shelf-front space, but it may be disposed in both the shelf-front spaces or below the shelf-front space. The depth sensor 40 generates data indicating the position of the hand in the xy plane (i.e., horizontal plane) and the position of the hand in the z direction (i.e., height direction), and outputs this data to the item information estimation device 10. Therefore, when a person places their hand on a shelf 20, the item information estimation device 10 can identify the shelf 20 by using the data generated by the depth sensor 40. The depth sensor 40 may be, for example, a stereo camera or a LiDAR (Light Detection and Ranging) sensor. The item information estimation device 10 may also generate data indicating the hand position by processing the output data from the depth sensor 40.

[0017] Furthermore, even if it is detected that the total weight of the items placed on the shelf 20 has decreased by more than a reference value, i.e., the weight of the shelf 20 has decreased by more than a reference value, it is possible to estimate the item 200 that has been removed from the shelf 20. Specifically, the weight sensor 30 detects the total weight of the shelf 20. The detected value of the weight sensor 30 is output to the item information estimation device 10 together with weight sensor identification information assigned to that weight sensor 30. Then, by using this weight sensor identification information, the item information estimation device 10 can estimate the type of the item 200 that has been removed.

[0018] <Example of functional configuration> The product information estimation device 10 includes an acquisition unit 110 and an output unit 120 .

[0019] The acquiring unit 110 acquires data (hereinafter referred to as weight change data) based on changes in the detected value of the weight sensor 30. For example, the acquiring unit 110 generates weight change data by chronologically organizing the data acquired from the weight sensor 30. However, a data processing device that generates weight change data using the data generated by the weight sensor 30 may be provided outside the product information estimation device 10. In this case, the acquiring unit 110 acquires the weight change data from this data processing device.

[0020] The acquisition unit 110 also acquires data indicating hand movements of a person positioned in the shelf space (hereinafter referred to as movement data). The acquisition unit 110 generates the movement data, for example, by chronologically organizing the data output from the depth sensor 40 to the item information estimation device 10.

[0021] The output unit 120 uses the weight change data and the action data to output item identification information of an item that is estimated to have been picked up by the hand of a person located in the space in front of the shelf. In this embodiment, the item information estimation device 10 has a shelf allocation information storage unit 130. The shelf allocation information storage unit 130 stores, for each shelf 20, item identification information that identifies the item placed on that shelf 20. The output unit 120, for example, identifies the shelf 20 on which the picked product was placed, reads out the item identification information corresponding to the identified shelf 20 from the shelf allocation information storage unit 130, and outputs the read item identification information. The item identification information is, for example, an ID (which may be code information) assigned to the item, or the item name of the item (for example, the product name).

[0022] FIG. 2 is a diagram showing an example of data stored in the shelf allocation information storage unit 130. In this embodiment, the shelf allocation information storage unit 130 stores, for each piece of information indicating the position of a shelf 20 (hereinafter referred to as shelf position information), weight sensor identification information of the weight sensor 30 attached to that shelf 20, item identification information of the item 200 placed at that position, and threshold information. The shelf position information includes information specifying the height of the shelf 20 (e.g., height from the floor or number of shelves from the bottom). The threshold information is a value that is expected to represent the amount of decrease in the detection value of the weight sensor 30 when one item 200 is removed from that shelf, and is set to, for example, a value between 90% and 110% of the weight of the item 200. The threshold indicated by this threshold information is used by the output unit 120, as will be described later.

[0023] <Hardware configuration example> Fig. 3 is a block diagram illustrating an example of the hardware configuration of the product information estimation device 10 shown in Fig. 1. The product information estimation device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0024] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0025] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0026] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0027] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules that realize each function (e.g., the acquisition unit 110 and the output unit 120) of the product information estimation device 10. The processor 1020 loads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to the program module.

[0028] The input / output interface 1050 is an interface for connecting the product information estimation device 10 to various input / output devices.

[0029] The network interface 1060 is an interface for connecting the product information estimation device 10 to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The network interface 1060 may be connected to the network via a wireless connection or a wired connection.

[0030] The article information estimation device 10 is connected to necessary devices (for example, a group of sensors such as the weight sensor 30 and the depth sensor 40) via the input / output interface 1050 or the network interface 1060.

[0031] <Example of operation> 4 is a flowchart illustrating an example of the operation of the product information estimation device 10. In the example shown in this figure, the weight sensor 30 continuously transmits data and weight sensor identification information to the product information estimation device 10. The depth sensor 40 also continuously transmits data to the product information estimation device 10. The acquisition unit 110 continuously acquires these data, i.e., weight change data and operation data. The acquisition unit 110 also continuously stores the acquired data in storage as needed.

[0032] The output unit 120 analyzes the detection values ​​of the weight sensors 30 acquired by the acquisition unit 110, and identifies the weight sensor identification information of the weight sensors 30 whose detection values ​​(i.e., weights) have decreased by a reference value or more (step S102). This reference value is stored in the shelf allocation information storage unit 130, for example, as shown in Fig. 2. The output unit 120 then performs an item identification process using the weight sensor identification information identified in step S102 and the detection values ​​of the depth sensor 40 (step S104).

[0033] 5 is a flowchart for explaining the details of the processing performed in step S104. First, the output unit 120 reads out the shelf position information corresponding to the weight sensor identification information identified in step S102 from the shelf allocation information storage unit 130 (step S202). Next, the output unit 120 analyzes the data output by the depth sensor 40 and detects the height of the hand inserted into the shelf 20 (step S204).

[0034] Next, the output unit 120 determines whether the relationship between the height of the hand detected in step S204 and the shelf position information read out in step S202 satisfies a criterion (step S206). For example, the output unit 120 determines that the criterion is satisfied if the height of the hand detected in step S204 is between the height indicated by the shelf position information and the height of the shelf 20 above it. Note that this criterion may be stored in the shelf allocation information storage unit 130 for each shelf 20. In this case, the output unit 120 reads out and uses the criterion linked to the shelf position information identified in step S102. Furthermore, the shelf allocation information storage unit 130 may store a range of possible heights for a hand inserted into that shelf 20 instead of the shelf position information. In this case, the output unit 120 determines whether the height just obtained by the depth sensor 40 is within this range.

[0035] If the relationship between the hand height and the shelf position information satisfies the criteria (step S206: Yes), the output unit 120 estimates the item 200 taken out by the person by reading out the item identification information corresponding to the weight sensor identification information identified in step S102 from the shelf allocation information storage unit 130 (step S208). Then, the output unit 120 outputs the read-out item identification information.

[0036] On the other hand, if the relationship between the hand height and the shelf position information does not satisfy the criteria (step S206: No), the output unit 120 performs an alert process. This alert process may, for example, display a predetermined screen on the terminal of the manager of the item 200 (for example, a store clerk if the shelf 20 is in a store) (step S210). Note that, together with or instead of this alert process, data generated by the depth sensor 40 or images captured by the first imaging unit 70 and the second imaging unit 80 described in the embodiments below may be transmitted to the manager's terminal. In this case, the manager may estimate the item 200 that has been taken out by checking the images, etc., and transmit the result to the output unit 120 via the terminal.

[0037] In this embodiment, the output unit 120 may first detect that the hand position has reached a height corresponding to one of the shelves 20, and then read out the item identification information corresponding to that shelf 20 when the weight change of that shelf 20 meets a criterion.

[0038] <Variation 1> FIG. 6 is a diagram showing an example layout of shelves 20 and weight sensors 30 according to a modified example. In this figure, the shelf 20 is shown as seen from the front. In the example shown in this figure, the shelf 20 has a plurality of partial areas 22 on at least one level. In at least one of the plurality of partial areas 22, an item 200 different from that in the other partial areas 22 is placed. A weight sensor 30 is provided for each partial area 22. The shelf allocation information storage unit 130 stores the information shown in FIG. 2, i.e., shelf position information, weight sensor identification information, item specification information, and threshold value information, for each of the plurality of partial areas 22.

[0039] Furthermore, the shelf-front space is set for each partial area 22, and a depth sensor 40 is also provided for each partial area 22. Each of the depth sensors 40 stores depth sensor identification information that distinguishes the depth sensor 40 from the other depth sensors 40. The depth sensors 40 then transmit this depth sensor identification information along with data to the item information estimation device 10. The shelf allocation information storage unit 130 stores, for each piece of shelf position information, the depth sensor identification information of the depth sensor 40 that corresponds to that shelf position. The item information estimation device 10 identifies the combination of the data transmitted from the depth sensor 40 and the data transmitted from the weight sensor 30 by using the combination of the weight sensor identification information and the depth sensor identification information stored in the shelf allocation information storage unit 130.

[0040] 4 and 5 for each combination of data, i.e., for each partial region 22. According to this modification, even if multiple types of articles 200 are placed on shelves 20 at the same height, it is possible to estimate the article 200 that the person has picked up.

[0041] As described above, according to this embodiment, when the position of the shelf 20 where the weight change was detected and the height of the hand identified by the depth sensor 40 meet the criteria, it is determined that the item 200 on that shelf 20 has been taken by that hand. Therefore, the accuracy of identifying the item 200 taken from the shelf 20 is improved.

[0042] [Second embodiment] <Example of functional configuration> Fig. 7 is a diagram showing the functional configuration of the product information estimation device 10 according to this embodiment together with the environment in which the product information estimation device 10 is used, and corresponds to Fig. 1 of the first embodiment. The product information estimation device 10 according to this embodiment has the same configuration as the product information estimation device 10 according to the first embodiment, except for the following points.

[0043] First, the product information estimation device 10 acquires, from the person tracking device 50, the person identification information of a person present in the shelf space in front of the shelf 20.

[0044] The person tracking device 50 generates, for each person, movement line information indicating the movement line of the person, for example, by analyzing images sent from multiple imaging units each having an imaging range of a different location. The person tracking device 50 then stores the movement line information in association with person identification information. The person identification information is, for example, a feature obtained from an image of the person. Furthermore, if the shelf 20 is installed in a store, the person identification information may be a customer ID such as a membership number. Then, when a person stays in the space in front of the shelf 20 for a certain period of time, the person tracking device 50 outputs the person identification information of the person to the item information estimation device 10.

[0045] The product information estimation device 10 includes a storage processing unit 140. The storage processing unit 140 associates the product identification information acquired by the output unit 120 in step S208 of Fig. 5 with the person identification information acquired from the person tracking device 50 and stores the information in the registered product storage unit 60.

[0046] For example, if the shelf 20 is installed in a store, the product information estimation device 10 and the registered product storage unit 60 can be used as a product registration device for a Point of Sale (POS) system or a store server. The POS settlement device then performs payment processing using the information stored in the registered product storage unit 60.

[0047] For example, the person tracking device 50 stores facial features of a customer who enters a store. In this case, the person tracking device 50 acquires an image from an imaging device whose imaging range includes the store entrance, and processes this image to acquire and store the facial features of the customer.

[0048] As described above, the person tracking device 50 generates customer movement line information using these feature amounts. The movement line information is linked to the feature amounts or the customer ID linked to these feature amounts. Furthermore, the storage processing unit 140 of the product information estimation device 10 stores product identification information of the products picked up by the customer in the registered product storage unit 60, linking the product identification information to the customer's feature amounts (or the customer ID linked to these feature amounts). This process is repeated until the customer completes the payment process. Therefore, if the customer picks up multiple products, the registered product storage unit 60 stores the product identification information of these multiple products, linking the product identification information to the customer's feature amounts (or the customer ID linked to these feature amounts).

[0049] Furthermore, a customer can read out information stored in the registered item storage unit 60 using the customer's terminal. For example, the customer's terminal transmits the customer's characteristic quantities (or customer ID) to the storage processing unit 140. The storage processing unit 140 reads out the item identification information linked to the transmitted characteristic quantities (or customer ID) from the registered item storage unit 60, and transmits this item identification information to the customer's terminal as a list of items. At this time, the item identification information may be converted into product names using a database. Furthermore, the price of the item may be transmitted together with the product identification information (or product name). In the latter case, the total price of the registered items may also be transmitted to the customer's terminal.

[0050] The customer's terminal then displays the list of products that has been sent to it. This screen includes, for example, an input button for making a payment.

[0051] Then, the customer, for example, operates the customer's terminal to send information indicating that they will pay for the product along with the customer's feature amount (or customer ID) to the settlement device. The settlement device reads out the product identification information corresponding to the received feature amount (or customer ID) from the registered product storage unit 60 and performs the settlement process using the read out information. Thereafter, the settlement device generates an electronic receipt and sends it to the customer's terminal. Note that this settlement device may be incorporated into the product information estimation device 10.

[0052] The payment information may be input from a terminal installed in the store. In this case, the terminal may capture an image of the customer's face to generate features and transmit the features to the payment device.

[0053] Furthermore, if the shelf 20 is installed in a distribution center or a pharmacy, the information stored in the registered article storage unit 60 can be used to identify the person who has taken out the article 200.

[0054] 7, the registered item storage unit 60 is located outside the item information estimation device 10, but the registered item storage unit 60 may be part of the item information estimation device 10. Furthermore, the person identification information may be input by a person using an input device (e.g., a card reader) attached to the shelf 20, for example.

[0055] As with the first embodiment, this embodiment also improves the accuracy of identifying an item 200 taken out from a shelf 20. Furthermore, the registered item storage unit 60 stores the item identification information of an item 200 taken out by a person in association with the person's personal identification information. Therefore, it is possible to confirm who took out which item 200.

[0056] [Third embodiment] <Example of functional configuration> 8 is a plan view illustrating the layout of the weight sensors 30 according to this embodiment. In this embodiment, a plurality of weight sensors 30 are provided separately from one another on one shelf 20 or partial area 22 (hereinafter referred to as shelf 20). In the example shown in this figure, the shelf 20 is rectangular, and the weight sensors 30 are provided at each of the four corners of the shelf 20.

[0057] The weight change data used by the output unit 120 is based on changes in the detection values ​​of the multiple weight sensors 30. As an example, the weight change data indicates changes over time in the detection values ​​of the multiple weight sensors 30. The output unit 120 of the item information estimation device 10 determines that an item 200 on a shelf 20 has been removed when the change in the detection values ​​of the multiple weight sensors 30 meets a criterion. For example, the output unit 120 determines that an item 200 has been removed when the total amount of decrease in the detection values ​​of the multiple weight sensors 30 meets a criterion. In this case, the output unit 120 uses the amount of decrease in the detection values ​​of the multiple weight sensors 30 to determine from which position on the shelf 20 the item 200 has been removed.

[0058] The weight sensor identification information of multiple weight sensors 30 provided on the same shelf 20 is linked to each other in the shelf allocation information storage unit 130 and managed as a set of weight sensors 30. For example, the weight sensor identification information of multiple weight sensors 30 provided on the same shelf 20 is linked to information that identifies that shelf 20 from other shelves 20, such as shelf position information. Therefore, the output unit 120 can perform the above-mentioned processing by using the information stored in the shelf allocation information storage unit 130.

[0059] <Example of operation> The product information estimation device 10 first identifies a set of weight sensors 30 whose change in detection value satisfies a criterion (step S102 in FIG. 4), and then executes product identification processing using the detection results of the identified set of weight sensors 30 (step S104 in FIG. 4).

[0060] 9 is a flowchart for explaining details of the item identification process (step S104 in FIG. 4) in this embodiment. First, the output unit 120 reads out the shelf position information corresponding to the weight sensor identification information of the weight sensor 30 identified in step S102 (step S222).

[0061] Next, the output unit 120 uses the changes in the detected values ​​of the multiple weight sensors 30 to estimate the position on the shelf 20 where the weight change occurred, i.e., the position where the removed item 200 was placed. For example, the output unit 120 regards the amount of change in each weight sensor 30 as weight, and estimates the position that is the center of gravity of these weights as the above-mentioned position (step S224).

[0062] Furthermore, the output unit 120 determines the height of the hand and the direction in which the hand is stretched, using the data transmitted from the depth sensor 40. For example, if the depth sensor 40 outputs a depth map that shows height information in two dimensions, the output unit 120 determines the height and direction of the hand by using this depth map (step S226).

[0063] The output unit 120 then determines whether the relationship between the hand height and the shelf position information satisfies the criteria, and whether the relationship between the hand direction and the position of the item 200 identified in step S224 satisfies the criteria. The determination of whether the relationship between the hand height and the shelf position information satisfies the criteria is the same as the determination described in step S206 of Fig. 5. Regarding the relationship between the hand direction and the position of the item 200, it is determined that the criteria are met if, for example, the hand direction overlaps with the position of the item 200 or the shortest distance therebetween is equal to or less than a criteria value (step S228).

[0064] If the result of step S228 is Yes, the output unit 120 estimates the item 200 taken out by the person (step S230). The process performed in step S230 is the same as the process performed in step S208 of FIG. 5. On the other hand, if the result of step S224 is No, the output unit 120 performs alert processing (step S232). The process performed in step S232 is the same as the process performed in step S210 of FIG. 5.

[0065] As with the first embodiment, this embodiment also improves the accuracy of identifying an item 200 taken out from a shelf 20. Furthermore, the item information estimation device 10 uses not only the relationship between the hand height and shelf position information (i.e., the relationship in the vertical direction) but also the relationship between the hand direction and the position of the item 200 (i.e., the relationship in the horizontal plane) when estimating the item 200. This further improves the accuracy of identifying an item 200 taken out from a shelf 20.

[0066] [Fourth embodiment] <Example of functional configuration> Fig. 10 is a diagram showing the functional configuration of the product information estimation device 10 according to this embodiment, together with the environment in which the product information estimation device 10 is used. The product information estimation device 10 according to this embodiment has the same configuration as the product information estimation device 10 according to any of the first to third embodiments, except that the product information estimation device 10 repeatedly acquires images (hereinafter referred to as first images) from a first imaging unit 70 and identifies the product 200 using these first images. Fig. 10 shows the same case as the first embodiment.

[0067] The first imaging unit 70 includes in its imaging area at least a portion of the in-shelf space, which is the space in front of the shelf 20. Therefore, the first image generated by the first imaging unit 70 includes at least a portion of the in-shelf space and also includes the item 200 taken out from the shelf 20.

[0068] The output unit 120 then uses the image of the item 200 included in the first image to estimate the item 200 that the person has taken out from the shelf 20. Specifically, the shelf allocation information storage unit 130 stores feature amounts on the image of the item 200 along with the item identification information. The output unit 120 then estimates the item 200 using the result of matching the feature amounts with the first image.

[0069] <Example of operation> The process performed by the product information estimation device 10 shown in Fig. 10 is the same as that described in the first embodiment with reference to Fig. 4. However, the details of the process shown in step S104 are different from those in the first embodiment.

[0070] Fig. 11 is a flowchart for explaining the details of step S104 in this embodiment. The processes performed in steps S202, S204, S206, S208, and S210 are the same as those described with reference to Fig. 5. However, in step S208, the output unit 120 also reads out the feature amount of the item 200 along with the item identification information.

[0071] The output unit 120 then processes the first image within a reference time (for example, within 10 seconds) after the change in the detection value of the weight sensor 30, and extracts feature amounts of the item 200 included in the first image. If the extracted feature amounts match the feature amounts read out in step S208, for example, if the score is equal to or greater than a reference value (step S209: Yes), the output unit 120 outputs the item identification information read out in step S208 as is. On the other hand, if these feature amounts do not match each other (step S209: No), an alert process is performed (step S210).

[0072] When the above-described process is applied to the product information estimation device 10 shown in the third embodiment, the process shown in step S209 is performed after step S230 in FIG.

[0073] According to this embodiment, similar to the first embodiment, the estimation accuracy of the item 200 taken by a person from the shelf 20 is improved. Furthermore, the first image includes the item 200 taken by the person. The output unit 120 of the item information estimation device 10 further verifies the item 200 estimated based on the detection values ​​of the depth sensor 40 and the weight sensor 30 using the first image. Therefore, the estimation accuracy of the item 200 is further improved.

[0074] [Fifth embodiment] <Example of functional configuration> Fig. 12 is a diagram showing the functional configuration of the product information estimation device 10 according to this embodiment, together with the environment in which the product information estimation device 10 is used. The product information estimation device 10 according to this embodiment has the same configuration as the product information estimation device 10 according to any one of the first to fourth embodiments, except that it repeatedly acquires images (hereinafter referred to as second images) from the second imaging unit 80 and identifies the product 200 using these multiple second images. Fig. 12 shows the same case as the fourth embodiment.

[0075] The second imaging unit 80 captures an image of the shelf 20 from the front (for example, diagonally from above). Therefore, the second image includes the item 200 placed on the shelf 20. Furthermore, when the second imaging unit 80 captures an image of the shelf 20 from diagonally from above, it can also capture an image of the item 200 located at the back of the shelf 20. The output unit 120 of the item information estimation device 10 then further uses changes in the second image to estimate the item 200 that the person has taken out of the shelf 20. Specifically, the output unit 120 estimates the item 200 using the difference between the second image taken before the depth sensor 40 detects the person's hand (i.e., before the person comes into the space in front of the shelf) and the second image taken after the depth sensor 40 no longer detects the person's hand (i.e., after the person has left the space in front of the shelf).

[0076] <Example of operation> The process performed by the product information estimation device 10 shown in Fig. 12 is the same as that described in the first embodiment with reference to Fig. 4. However, the details of the process shown in step S104 are different from those in the first embodiment.

[0077] 13 is a flowchart illustrating an example of the operation of the product information estimation device 10 according to this embodiment. The processes performed in steps S202, S204, S206, S208, S209, and S210 are the same as those described with reference to FIG.

[0078] If the feature amount of the item 200 included in the first image matches the feature amount of the item read from the shelf allocation information storage unit 130 (step S209: Yes), the output unit 120 of the item information estimation device 10 processes the second image and determines whether correction based on the second image is required for the item identification information read in step S208 (step S212). If correction is required (step S212: Yes), the output unit 120 executes the correction (step S214).

[0079] For example, the output unit 120 extracts the difference between the second image taken before the depth sensor 40 detects a human hand (i.e., before the person enters the space in front of the shelf) and the second image taken after the depth sensor 40 no longer detects a human hand (i.e., after the person leaves the space in front of the shelf), and performs a matching process on this difference to determine whether the item 200 corresponding to the item identification information read out in step S208 has been moved to a shelf 20 different from the shelf 20 where it should have been. In this process, the position of the item 200 after the movement is identified by, for example, a matching process using the feature amounts of the item 200. If this movement is detected (step S212: Yes), the output unit 120 does not output the item identification information. For example, if this function is added to the second embodiment, the item identification information of this item 200 is not stored in the registered item storage unit 60 (step S214).

[0080] Additionally, the output unit 120 pre-stores a combination of the detection result of the weight sensor 30, the detection result of the depth sensor 40, and the processing result of the second image for each movement pattern of the article 200 by a person. When the output unit 120 detects a result corresponding to this combination, it estimates that a movement pattern corresponding to that combination has occurred.

[0081] On the other hand, if correction is not required (step S212: No), the output unit 120 outputs the item identification information read out in step S208.

[0082] This product identification information is then used in the payment process for the product at the store, as described in the second embodiment, for example.

[0083] According to this embodiment, similarly to the first embodiment, the estimation accuracy of the item 200 that a person has taken out from the shelf 20 is improved. Furthermore, the output unit 120 of the item information estimation device 10 identifies the item 200 that has been moved within the shelf 20. Therefore, when there is an item 200 that a person has moved within the shelf 20, it is possible to prevent the item 200 from being mistakenly recognized as having been taken out by the person.

[0084] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.

[0085] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed to the extent that the content is not affected. Furthermore, the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0086] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. An acquisition means for acquiring weight change data, which is data based on changes in the detected values ​​of a weight sensor provided on a shelf capable of placing multiple items, and motion data, which indicates the hand movements of a person positioned in a shelf front space, which is a space in front of the shelf; an output means for outputting item identification information of the item presumed to have been picked up by the hand, using the weight change data and the movement data; An article estimation device comprising: 2. In the product estimation device described in 1 above, the acquiring means generates the operation data using a detection value of a depth sensor that includes the shelf front space in its detection range, The output means is an item estimation device that outputs the item identification information linked to the shelf when the relationship between the height of the shelf where the detection value of the weight sensor has changed and the detection value of the depth sensor that can acquire the operation data meets a criterion. 3. In the product estimation device described in 2 above, There are a plurality of said shelves of different heights, At least one of the item identification information is linked to each of the plurality of shelves, The weight change data indicates fluctuations in weight for each of the plurality of shelves. 4. In the product estimation device according to any one of 1 to 3 above, The output means is an item estimation device that outputs the item identification information linked to a partial area where a change in weight in the partial area of ​​the shelf satisfies a criterion. 5. In the product estimation device according to any one of 1 to 4 above, The article estimation device further includes a storage processing means for storing the article identification information output by the output means in a storage means in association with person identification information that identifies the person. 6. In the product estimation device described in 5 above, The storage processing means is an article estimation device that acquires the person identification information from a person tracking device that tracks the movement of the person. 7. In the product estimation device according to any one of 1 to 6 above, The acquisition means repeatedly acquires a first image that is an image including at least a portion of the shelf front space, The output means further includes an item estimation device that estimates the item picked up by the hand using an image of the item included in the first image. 8. In the product estimation device according to any one of 2 to 6 above, the acquisition means repeatedly acquires second images which are images of the shelf taken from the front, The output means further uses the change in the second image to identify the item identification information of the item picked up by the hand. 9. In the product estimation device according to any one of 1 to 8 above, a plurality of weight sensors are provided on the shelf at intervals; The acquisition means is an article estimation device that generates the weight change data using detection values ​​of the plurality of weight sensors. 10. In the product estimation device according to any one of 1 to 9 above, The shelf is installed in a store, the person is a customer; a payment processing means for performing payment processing using the item identification information output by the output means; an electronic receipt output unit that outputs an electronic receipt based on the settlement process; An article estimation device comprising: 11. The computer weight change data, which is data based on changes in detected values ​​of a weight sensor provided on a shelf on which a plurality of items can be placed, and motion data, which indicates hand movements of a person positioned in a shelf front space, which is a space in front of the shelf, are acquired; an item estimation method that uses the weight change data and the motion data to output item identification information of the item that is estimated to have been picked up by the hand; 12. In the article estimation method described in 11 above, The computer generating the operation data using a detection value of a depth sensor that includes the shelf front space in its detection range; An item estimation method that outputs the item identification information linked to the shelf when the relationship between the height of the shelf where the detection value of the weight sensor has changed and the detection value of the depth sensor that can acquire the operation data meets a criterion. 13. In the article estimation method described in 12 above, There are a plurality of said shelves of different heights, At least one of the item identification information is linked to each of the plurality of shelves, An article estimation method, wherein the weight change data indicates fluctuations in weight for each of the plurality of shelves. 14. In the article estimation method according to any one of 11 to 13 above, An item estimation method in which the computer outputs the item identification information linked to a partial area where a change in weight in multiple partial areas of the shelf satisfies a criterion. 15. In the article estimation method according to any one of 11 to 14 above, The computer stores the output item identification information in a storage unit in association with person identification information that identifies the person. 16. In the article estimation method described in 15 above, The computer acquires the person identification information from a person tracking device that tracks the movement of the person. 17. In the article estimation method according to any one of 11 to 16 above, The computer repeatedly acquiring a first image that is an image including at least a portion of the shelf front space; The object estimation method further estimates the object picked up by the hand using an image of the object included in the first image. 18. In the article estimation method according to any one of 12 to 16 above, The computer repeatedly acquires a second image, which is an image of the shelf taken from the front, and further uses changes in the second image to identify the item identification information of the item picked up by the hand. 19. In the article estimation method according to any one of 11 to 18 above, a plurality of weight sensors are provided on the shelf at intervals; The computer generates the weight change data using the detected values ​​of the plurality of weight sensors. 20. In the article estimation method according to any one of 11 to 19 above, The shelf is installed in a store, the person is a customer; The computer performs a settlement process using the output item identification information and outputs an electronic receipt based on the settlement process. 21.To the computer, A function of acquiring weight change data, which is data based on changes in the detected values ​​of a weight sensor provided on a shelf on which a plurality of items can be placed, and motion data, which indicates the hand movements of a person positioned in a space in front of the shelf, which is a space in front of the shelf; a function of outputting item identification information of the item presumed to have been picked up by the hand using the weight change data and the motion data; A program that allows you to have 22. In the program described in 21 above, The computer, a function of generating the operation data using a detection value of a depth sensor that includes the shelf front space in its detection range; a function of outputting the item identification information associated with the shelf when the relationship between the height of the shelf where the change in the detected value of the weight sensor occurred and the detected value of the depth sensor capable of acquiring the operation data satisfies a criterion; A program that allows you to have 23. In the program according to 22 above, There are a plurality of said shelves of different heights, At least one of the item identification information is linked to each of the plurality of shelves, The weight change data indicates fluctuations in weight for each of the plurality of shelves. 24. In the program according to any one of 21 to 23 above, A program that gives the computer the function of outputting the item identification information linked to a partial area where the change in weight in multiple partial areas of the shelf meets a criterion. 25. In the program according to any one of 21 to 24 above, A program that causes the computer to have a function of storing the output item identification information in a storage means in association with person identification information that identifies the person. 26. In the program according to 25 above, A program that allows the computer to have a function of acquiring the person identification information from a person tracking device that tracks the movement of the person. 27. In the program according to any one of 21 to 26 above, The computer, a function of repeatedly acquiring a first image that is an image including at least a portion of the shelf front space; Further, a function of estimating the item picked up by the hand using an image of the item included in the first image; A program that allows you to have 28. In the program according to any one of 22 to 26 above, A program that gives the computer the function of repeatedly acquiring a second image, which is an image of the shelf taken from the front, and further using changes in the second image to identify the item identification information of the item picked up by the hand. 29. In the program according to any one of 21 to 28 above, a plurality of weight sensors are provided on the shelf at intervals; A program that causes the computer to have the function of generating the weight change data using the detected values ​​of the plurality of weight sensors. 30. In the program according to any one of 21 to 29 above, The shelf is installed in a store, the person is a customer; A program that causes the computer to perform a settlement process using the output item identification information and output an electronic receipt based on the settlement process.

[0087] This application claims priority based on Japanese Patent Application No. 2019-037829, filed March 1, 2019, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. a determining means for determining the height of a person's hand when the person extends the hand toward the shelf, using a depth sensor capable of detecting the movement of the hand of the person positioned in a front-shelf space, which is a space in front of the shelf on which a plurality of items can be placed; an output means for executing an alert process when the positional relationship between the height of the hand and the position of the shelf specified based on the detected value of a weight sensor provided on the shelf does not satisfy a standard; An article estimation device comprising:

2. the output means outputs information about the item corresponding to the shelf position when the positional relationship between the shelf position and the hand height satisfies the criterion. The article estimation device according to claim 1 .

3. The information about the item includes the name of the item and the price of the item. The article estimation device according to claim 2 .

4. the output means outputs a list including information on the items and a total price of the items included in the list. The product estimation device according to claim 2 or 3.

5. the output means outputs information about the item to a terminal device of the person. The article estimation device according to claim 2 .

6. The information on the article is stored in a storage unit in association with personal identification information that identifies the person. The article estimation device according to claim 2 .

7. the storage processing means acquires the person identification information from a person tracking device that tracks the movement of the person; The article estimation device according to claim 6.

8. The alert processing includes a process of outputting alert information including at least one of a screen for the alert processing, data generated by the depth sensor, and a photographed image of the shelf front space. The article estimation device according to claim 1 .

9. the output means outputs the alert information to a terminal of a manager who manages the plurality of items. The article estimation device according to claim 8 .

10. The computer a depth sensor capable of detecting hand movement of a person positioned in a shelf space in front of a shelf on which a plurality of items can be placed is used to determine the height of the hand of the person when the person reaches the shelf; executing an alert process when the positional relationship between the height of the hand and the position of the shelf specified based on the detection value of a weight sensor provided on the shelf does not satisfy a criterion; A method for estimating an item, comprising:

11. Computer, a determining means for determining the height of a person's hand when the person extends the hand toward the shelf, using a depth sensor capable of detecting the movement of the hand of the person positioned in a front-shelf space, which is a space in front of the shelf on which a plurality of items can be placed; an output means for executing an alert process when the positional relationship between the height of the hand and the position of the shelf specified based on the detected value of a weight sensor provided on the shelf does not satisfy a standard; A program to function as a

Images