Article estimation device, article estimation method, and program

Abstract

To enhance a determination precision for articles picked up from a rack.SOLUTION: An article estimation device includes a determining unit and an output unit. The determining unit determines that a person extends an arm to a position of a rack where a change in detection value by a weight sensor provided at the rack is detected using a height of the arm or a direction in which the arm is extended which is detected by a depth sensor capable of detecting a motion of the arm of the person located at a front space (a rack front space) of the rack where multiple articles can be placed. The output unit outputs information on the articles corresponding to the position of the rack where the person extends the arm among the multiple articles placed on the rack.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an article estimation device, an article estimation method, and a program.

Background Art

[0002] In recent years, technological developments for labor saving in stores, factories, etc. have been promoted. For example, in Patent Document 1, in the operation of packing a plurality of types of articles taken out from a stock shelf into a box as a set, the total weight of the articles 5 stored in the stock shelf is measured, and it is described whether to issue a warning using this measurement result is determined.

[0003] Also, in Patent Document 2, in order to manage the handling of articles, it is described that inventory data is generated using the result of processing an image obtained by photographing a barcode or QR code (registered trademark) of an article. The inventory data is information indicating the handling status of articles. For example, it is information in which the identification information of the photographed article, the date and time when the collation of the article was performed, the location where the article was installed, and the identification information of the user who handled the article are associated with each other.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In order to promote labor saving, it is preferable to be able to automatically discriminate articles taken out from the shelf. One of the purposes of the present invention is to improve the discrimination accuracy of articles taken out from the shelf.

Means for Solving the Problems

[0006] The first article estimation device in the present disclosure is using the height of the hand detected by a depth sensor capable of detecting the movement of the hand of a person located in the front-of-shelf space, which is the space in front of a shelf on which a plurality of articles can be placed, to determine that the person has extended the hand to the position of the shelf at which a change in the detection value of a weight sensor provided on the shelf has been detected; a determination means and output means for outputting information on an article corresponding to the position of the shelf at which the person has extended the hand among the plurality of articles. It is provided with. Also, the second article estimation device in the present disclosure is using the direction in which the hand detected by a depth sensor capable of detecting the movement of the hand of a person located in the front-of-shelf space, which is the space in front of a shelf on which a plurality of articles can be placed, is extended to determine that the person has extended the hand to the position of the shelf at which a change in the detection value of a weight sensor provided on the shelf has been detected; a determination means and output means for outputting information on an article corresponding to the position of the shelf at which the person has extended the hand among the plurality of articles. It is provided with.

[0007] The first article estimation method in the present disclosure is a computer using the height of the hand detected by a depth sensor capable of detecting the movement of the hand of a person located in the front-of-shelf space, which is the space in front of a shelf on which a plurality of articles can be placed, to determine that the person has extended the hand to the position of the shelf at which a change in the detection value of a weight sensor provided on the shelf has been detected, and outputting information on an article corresponding to the position of the shelf at which the person has extended the hand among the plurality of articles. This includes. Also, the second article estimation method in the present disclosure is a computer Using the direction in which the hand detected by a depth sensor capable of detecting the movement of a hand of a person located in the front-shelf space, which is the space in front of a shelf on which a plurality of articles can be placed, is extended, it is determined that the person has extended the hand to the position of the shelf at which a change in the detection value of a weight sensor provided on the shelf has been detected. Outputting information on an article corresponding to the position of the shelf at which the person has extended the hand among the plurality of articles. Including this.

[0008] The first program in the present disclosure Causes a computer To function as determination means for determining that the person has extended the hand to the position of the shelf at which a change in the detection value of a weight sensor provided on the shelf has been detected, using the height of the hand detected by a depth sensor capable of detecting the movement of a hand of a person located in the front-shelf space, which is the space in front of a shelf on which a plurality of articles can be placed. And output means for outputting information on an article corresponding to the position of the shelf at which the person has extended the hand among the plurality of articles. To function as such. Also, the second program in the present disclosure Causes a computer To determine that the person has extended the hand to the position of the shelf at which a change in the detection value of a weight sensor provided on the shelf has been detected, using the direction in which the hand detected by a depth sensor capable of detecting the movement of a hand of a person located in the front-shelf space, which is the space in front of a shelf on which a plurality of articles can be placed, is extended. And output means for outputting information on an article corresponding to the position of the shelf at which the person has extended the hand among the plurality of articles. To function as such.

Advantages of the Invention

[0009] According to the present invention, the discrimination accuracy of an article taken out from a shelf is improved.

Brief Description of the Drawings

[0010] The above-described object, as well as other objects, features, and advantages, will become even more apparent from the preferred embodiments described below and the accompanying drawings.

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0013] [First Embodiment] FIG. 1 is a diagram showing the functional configuration of the article information estimation device 10 according to the present embodiment together with the usage environment of the article information estimation device 10. The article information estimation device 10 according to the embodiment is a device that estimates an article 200 taken out by a person from a shelf 20, and is used together with a weight sensor 30 and a depth sensor 40. In this figure, the shelf 20 is shown in a state as seen from the side.

[0014] The shelf 20 can accommodate a plurality of articles 200. For example, when the shelf 20 is arranged in a store or a logistics center, the shelf 20 is a merchandise shelf, the article 200 is a commodity, and the person taking out the article 200 is a customer or a store employee (staff). Also, when the shelf 20 is arranged in a pharmacy, the shelf 20 is a medicine shelf, the article 200 is a medicine, and the person taking out the article 200 is a pharmacist.

[0015] In the present embodiment, the articles 200 are arranged on each of the plurality of shelves 20. A plurality of types of articles 200 are placed on the plurality of shelves 20. And for each article 200, the shelf 20 on which the article 200 is placed is predetermined. Therefore, if the shelf 20 from which the article 200 has been taken out is known, the type of the article 200 can be estimated. However, the shelf 20 may be single-tier.

[0016] The depth sensor 40 includes the space in front of the shelf 20 (hereinafter referred to as the front-shelf space) within its detection range, and generates data indicating the movement of a person's hand located in the front-shelf space. For example, the depth sensor 40 is arranged above the front-shelf space, but it may also be arranged on both sides of the front-shelf space, or below the front-shelf space. Then, the depth sensor 40 generates data indicating the position of the hand in the xy plane (i.e., the horizontal plane) and the position of the hand in the z direction (i.e., the height direction), and outputs this data to the article information estimation device 10. Therefore, when a person puts their hand into the shelf 20, the article information estimation device 10 can identify the shelf 20 by using the data generated by the depth sensor 40. The depth sensor 40 may be, for example, one using a stereo camera, or one using LiDAR (Light Detection and Ranging). Also, the article information estimation device 10 may generate data indicating the position of the hand by processing the output data from the depth sensor 40.

[0017] Also, even if it is detected that the total weight of the articles placed on the shelf 20 has decreased by a reference value or more, that is, the weight of the shelf 20 has decreased by a reference value or more, the article 200 taken out from the shelf 20 can be estimated. Specifically, the weight sensor 30 detects the total weight of the shelf 20. The detection value of the weight sensor 30 is output to the article information estimation device 10 together with the weight sensor identification information assigned to the weight sensor 30. By using this weight sensor identification information, the article information estimation device 10 can estimate the type of the taken-out article 200.

[0018] <Functional configuration example> The article information estimation device 10 includes an acquisition unit 110 and an output unit 120.

[0019] The acquisition unit 110 acquires data (hereinafter referred to as weight change data) based on the change in the detection value of the weight sensor 30. For example, the acquisition unit 110 generates weight change data by arranging the data acquired from the weight sensor 30 in time series. However, an external data processing device that generates weight change data using the data generated by the weight sensor 30 may be provided outside the article information estimation device 10. In this case, the acquisition unit 110 acquires the weight change data from this data processing device.

[0020] In addition, the acquisition unit 110 acquires data (hereinafter referred to as motion data) indicating the movement of the hand of a person located in front of the shelf. The acquisition unit 110 generates motion data, for example, by arranging the data output from the depth sensor 40 to the article information estimation device 10 in time series.

[0021] The output unit 120 outputs article identification information of an article estimated to have been taken out by the hand of a person located in front of the shelf using the weight change data and the motion data. In the present embodiment, the article information estimation device 10 has a shelf division information storage unit 130. The shelf division information storage unit 130 stores article identification information for identifying the articles arranged on each shelf 20. The output unit 120, for example, identifies the shelf 20 on which the taken-out product was placed, reads out the article identification information corresponding to the identified shelf 20 from the shelf division information storage unit 130, and outputs the read-out article identification information. The article identification information is, for example, the ID assigned to the article (which may be code information) or the name of the article (for example, the product name).

[0022] FIG. 2 is a diagram showing an example of data stored in the shelf division information storage unit 130. In the present embodiment, the shelf division information storage unit 130 stores weight sensor identification information of the weight sensor 30 attached to the shelf 20, article identification information of the article 200 placed at that position, and threshold information, separately for information indicating the position of the shelf 20 (hereinafter referred to as shelf position information). The shelf position information includes information for specifying the height of the shelf 20 (for example, the height from the floor or the number of steps from the bottom). The threshold information is a value assumed as the amount of decrease in the detection value of the weight sensor 30 when one article 200 is taken out from the shelf, and is set to a value of 90% or more and 110% or less of the weight of the article 200, for example. The threshold indicated by this threshold information is used in the output unit 120, as will be described later.

[0023] <Hardware Configuration Example> FIG. 3 is a block diagram illustrating the hardware configuration of the article information estimation device 10 shown in FIG. 1. The article information estimation device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0024] The bus 1010 is a data transmission path for the processor 1020, the memory 1030, the storage device 1040, the input / output interface 1050, and the network interface 1060 to transmit and receive data to and from each other. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0025] The processor 1020 is a processor implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0026] The memory 1030 is a main storage device implemented by a RAM (Random Access Memory) or the like.

[0027] The storage device 1040 is an auxiliary storage device realized by, for example, a HDD (Hard Disk Drive), SSD (Solid State Drive), memory card, or ROM (Read Only Memory). The storage device 1040 stores program modules that implement each function of the article information estimation device 10 (for example, the acquisition unit 110 and the output unit 120). When the processor 1020 reads and executes these program modules onto the memory 1030, each function corresponding to the program module is realized.

[0028] The input / output interface 1050 is an interface for connecting the article information estimation device 10 and various input / output devices.

[0029] The network interface 1060 is an interface for connecting the article information estimation device 10 to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method by which the network interface 1060 connects to the network may be a wireless connection or a wired connection.

[0030] Then, the article information estimation device 10 is connected to necessary devices (for example, a sensor group such as the weight sensor 30 and the depth sensor 40) via the input / output interface 1050 or the network interface 1060.

[0031] <Operation example> FIG. 4 is a flowchart for explaining an operation example of the article information estimation device 10. In the example shown in this figure, the weight sensor 30 continuously transmits data and weight sensor identification information to the article information estimation device 10. Also, the depth sensor 40 continuously transmits data to the article information estimation device 10. The acquisition unit 110 continuously acquires these data, that is, the weight change data and the motion data. Also, the acquisition unit 110 continuously stores the acquired data in the storage as necessary.

[0032] The output unit 120 analyzes the detection value of the weight sensor 30 acquired by the acquisition unit 110, and identifies the weight sensor identification information of the weight sensor 30 whose detection value (i.e., weight) has decreased by a reference value or more (step S102). This reference value is stored in the shelf division information storage unit 130 as shown in FIG. 2, for example. Then, the output unit 120 performs item identification processing using the weight sensor identification information identified in step S102 and the detection value of the depth sensor 40 (step S104).

[0033] FIG. 5 is a flowchart for explaining the details of the processing performed in step S104. First, the output unit 120 reads out the shelf position information corresponding to the weight sensor identification information identified in step S102 from the shelf division information storage unit 130 (step S202). Next, the output unit 120 analyzes the data output by the depth sensor 40 and detects the height of the hand inserted into the shelf 20 (step S204).

[0034] Next, the output unit 120 determines whether the relationship between the hand height detected in step S204 and the shelf position information read in step S202 satisfies a criterion (step S206). For example, the output unit 120 determines that the criterion is satisfied when the hand height detected in step S204 is between the height indicated by the shelf position information and the height of the shelf 20 above it. Note that this criterion may be stored in the shelf division information storage unit 130 for each shelf 20. In this case, the output unit 120 reads out and uses the criterion associated with the shelf position information identified in step S102. Also, the shelf division information storage unit 130 may store the range within which the hand height that can be inserted into the shelf 20 can be taken instead of the shelf position information. In this case, the output unit 120 determines whether the height of the hand acquired by the depth sensor 40 is within this range.

[0035] When the relationship between the hand height and the shelf position information meets the standard (step S206: Yes), the output unit 120 estimates the article 200 taken out by the person by reading out the article identification information corresponding to the weight sensor identification information specified in step S102 from the shelf division information storage unit 130 (step S208). Then, the output unit 120 outputs the read article identification information.

[0036] On the other hand, when the relationship between the hand height and the shelf position information does not meet the standard (step S206: No), the output unit 120 performs an alert process. This alert process is, for example, to cause a predetermined screen to be marked on the terminal of the administrator of the article 200 (for example, a store clerk when the shelf 20 is a store) (step S210). In addition, together with or instead of this alert process, data generated by the depth sensor 40, an image by the first imaging unit 70 described in the embodiment below, or an image by the second imaging unit 80 may be transmitted to the administrator's terminal. In this case, the administrator may estimate the article 200 taken out by checking the image or the like and transmit the result to the output unit 120 via the terminal.

[0037] In addition, in the present embodiment, the output unit 120 may first detect that the hand position has reached a height corresponding to any of the shelves 20, and then read out the article identification information corresponding to the shelf 20 when the weight change of the shelf 20 meets the standard.

[0038] <Modification Example 1> FIG. 6 is a diagram showing a layout example of the shelf 20 and the weight sensor 30 according to the modification example. In this figure, the shelf 20 is shown in a state seen from the front. In the example shown in this figure, in at least one stage, the shelf 20 has a plurality of partial regions 22. In at least one of the plurality of partial regions 22, an article 200 different from the other partial regions 22 is placed. The weight sensor 30 is provided for each partial region 22. And the shelf division information storage unit 130 stores the information shown in FIG. 2, that is, the shelf position information, the weight sensor identification information, the article identification information, and the threshold information for each of the plurality of partial regions 22.

[0039] In addition, the space in front of the shelf is set separately for each sub-region 22, and the depth sensors 40 are also provided separately for each sub-region 22. Each of the plurality of depth sensors 40 stores depth sensor identification information for identifying the depth sensor 40 from other depth sensors 40. Then, the depth sensor 40 transmits this depth sensor identification information to the article information estimation device 10 together with data. In the shelf division information storage unit 130, the depth sensor identification information of the depth sensor 40 corresponding to each shelf position is stored for each shelf position information. The article information estimation device 10 specifies a combination of the data transmitted from the depth sensor 40 and the data transmitted from the weight sensor 30 by using the combination of the weight sensor identification information and the depth sensor identification information stored in the shelf division information storage unit 130.

[0040] Then, the article information estimation device 10 performs the processes shown in FIGS. 4 and 5 for each combination of this data, that is, in units of the sub-region 22. According to this modification, even when a plurality of types of articles 200 are arranged on the shelf 20 at the same height, the article 200 taken out by a person can be estimated.

[0041] As described above, according to the present embodiment, when the position of the shelf 20 where a weight change is detected and the hand height specified by the depth sensor 40 satisfy the criteria, it is determined that the article 200 on the shelf 20 has been taken out by that hand. Therefore, the discrimination accuracy of the article 200 taken out from the shelf 20 is improved.

[0042] [Second Embodiment] <Functional Configuration Example> FIG. 7 is a diagram showing the functional configuration of the article information estimation device 10 according to the present embodiment together with the usage environment of the article information estimation device 10, and corresponds to FIG. 1 of the first embodiment. The article information estimation device 10 according to the present embodiment has the same configuration as the article information estimation device 10 according to the first embodiment, except for the following points.

[0043] First, the article information estimation device 10 acquires the person identification information of a person existing in the space in front of the shelf 20 from the person tracking device 50.

[0044] The person tracking device 50 generates movement line information indicating the movement line of each person by analyzing images sent from a plurality of imaging units having different imaging ranges, for example. Then, the person tracking device 50 stores the movement line information in association with person identification information. The person identification information is, for example, a feature amount obtained from an image of a person. When the shelf 20 is installed in a store, the person identification information may be a customer ID such as a membership number. When a person stays in the shelf front space in front of the shelf 20 for a certain period of time, the person tracking device 50 outputs the person identification information of that person to the article information estimation device 10.

[0045] The article information estimation device 10 has a storage processing unit 140. The storage processing unit 140 stores the article identification information acquired by the output unit 120 in step S208 of FIG. 5 in the registered article storage unit 60 in association with the person identification information acquired from the person tracking device 50.

[0046] For example, when the shelf 20 is installed in a store, the article information estimation device 10 and the registered article storage unit 60 can be used as a product registration device of a POS (Point of Sale system) or a store server. Then, the settlement device of the POS performs a settlement process using the information stored in the registered article storage unit 60.

[0047] For example, the person tracking device 50 stores the feature amount of the face of a customer who has entered the store. In this case, the person tracking device 50 acquires an image from an imaging device including, for example, the entrance of the store in the imaging range, and acquires and stores the feature amount of the customer's face by processing this image.

[0048] And as described above, the person tracking device 50 generates customer movement information using this feature amount. The movement information is associated with the feature amount or the customer ID associated with this feature amount. Further, the storage processing unit 140 of the article information estimation device 10 causes the registered article storage unit 60 to store the article identification information of the product taken out by the customer, in association with the customer's feature amount (or the customer ID associated with this feature amount). Since this process is repeated until the customer performs the settlement process, when the customer takes out a plurality of products, the registered article storage unit 60 stores the article identification information of these plurality of products in association with the customer's feature amount (or the customer ID associated with this feature amount).

[0049] Also, the customer can read out the information stored in the registered article storage unit 60 using the customer's terminal. For example, the customer's terminal transmits the customer's feature amount (or customer ID) to the storage processing unit 140. The storage processing unit 140 reads out the article identification information associated with the transmitted feature amount (or customer ID) from the registered article storage unit 60, and transmits this article identification information to the customer's terminal as a list of products. At this time, the article identification information may be converted into a product name using a database. Also, together with the article identification information (or product name), the price of the product may be sent. In the latter case, the total amount of the registered products may be further transmitted to the customer's terminal.

[0050] And the customer's terminal displays the transmitted list of products. This screen includes, for example, an input button for performing settlement.

[0051] Then, the customer transmits information indicating that the customer performs settlement of the product, together with the customer's feature amount (or customer ID), to the payment device, for example, by operating the customer's terminal. The payment device reads out the article identification information corresponding to the received feature amount (or customer ID) from the registered article storage unit 60, and performs a settlement process using the read information. Thereafter, the payment device generates an electronic receipt and transmits it to the customer's terminal. Note that this payment device may be incorporated in the article information estimation device 10.

[0052] Note that information indicating that a product is to be settled may be input from a terminal installed in the store. In this case, this terminal may capture an image of the customer's face to generate feature amounts and transmit these feature amounts to the settlement device.

[0053] Also, when the shelf 20 is installed in a logistics center or a pharmacy, the person who took out the article 200 can be identified by using the information stored in the registered article storage unit 60.

[0054] Note that in the example shown in FIG. 7, the registered article storage unit 60 is outside the article information estimation device 10, but the registered article storage unit 60 may be a part of the article information estimation device 10. Further, the person identification information may be input by a person using an input device (for example, a card reader) attached to the shelf 20.

[0055] Also according to this embodiment, as in the first embodiment, the discrimination accuracy of the article 200 taken out from the shelf 20 is improved. Further, the registered article storage unit 60 stores the article identification information of the article 200 taken out by a certain person in association with the person identification information of that person. Therefore, it is possible to confirm who took out which article 200.

[0056] [Third Embodiment] [Functional Configuration Example] FIG. 8 is a plan view for explaining the layout of the weight sensors 30 according to this embodiment. In this embodiment, a plurality of weight sensors 30 are provided at intervals from each other on one shelf 20 or partial area 22 (hereinafter referred to as the shelf 20). In the example shown in this figure, the shelf 20 is rectangular, and the weight sensors 30 are provided at each of the four corners of the shelf 20.

[0057] The weight change data used by the output unit 120 is based on the changes in the detection values of these multiple weight sensors 30. As an example, the weight change data shows the temporal change of the detection values of the multiple weight sensors 30. When the change in the detection values of the multiple weight sensors 30 meets the standard, the output unit 120 of the article information estimation device 10 determines that the article 200 on the shelf 20 has been taken out. For example, when the total value of the decrease in the detection values of the multiple weight sensors 30 meets the standard, the output unit 120 determines that the article 200 has been taken out. At this time, the output unit 120 uses the decrease in the detection values of the multiple weight sensors 30 to determine which position of the article 200 on the shelf 20 has been taken out.

[0058] Note that the weight sensor identification information of the multiple weight sensors 30 provided on the same shelf 20 is associated with each other in the shelf division information storage unit 130 and managed as a set of weight sensors 30. For example, the weight sensor identification information of the multiple weight sensors 30 provided on the same shelf 20 is associated with information for identifying that shelf 20 from other shelves 20, such as shelf position information. Therefore, the output unit 120 can perform the above-described processing by using the information stored in the shelf division information storage unit 130.

[0059] <Operation Example> The article information estimation device 10 first identifies a set of weight sensors 30 whose change in detection value meets the standard (step S102 in FIG. 4). Next, using the detection results of the identified set of weight sensors 30, the article identification process is executed (step S104 in FIG. 4).

[0060] FIG. 9 is a flowchart for explaining the details of the article identification process (step S104 in FIG. 4) in the present embodiment. First, the output unit 120 reads out the shelf position information corresponding to the weight sensor identification information of the weight sensors 30 identified in step S102 (step S222).

[0061] Next, the output unit 120 estimates the position where the weight has changed among the shelves 20, that is, the position where the taken-out article 200 was placed, using the changes in the detection values of the plurality of weight sensors 30. For example, the output unit 120 regards the change amount of each weight sensor 30 as a weight, and estimates the position that is the center of gravity of these weights as the above-described position (step S224).

[0062] Also, the output unit 120 specifies the hand height and the direction in which the hand is extended using the data transmitted from the depth sensor 40. For example, when the depth sensor 40 outputs a depth map showing height information two-dimensionally, the output unit 120 specifies the hand height and direction by using this depth map (step S226).

[0063] Then, the output unit 120 determines whether the relationship between the hand height and the shelf position information satisfies the standard, and whether the relationship between the hand direction and the position of the article 200 specified in step S224 satisfies the standard. The determination as to whether the relationship between the hand height and the shelf position information satisfies the standard is the same as the determination described in step S206 of FIG. 5. Regarding the relationship between the hand direction and the position of the article 200, for example, when the hand direction overlaps the position of the article 200 or the shortest distance therebetween is equal to or less than a reference value, it is determined that the standard is satisfied (step S228).

[0064] If Yes in step S228, the output unit 120 estimates the article 200 taken out by the person (step S230). The process performed in step S230 is the same as the process performed in step S208 of FIG. 5. On the other hand, if No in step S224, the output unit 120 performs alert processing (step S232). The process performed in step S232 is the same as the process performed in step S210 of FIG. 5.

[0065] Also according to this embodiment, similar to the first embodiment, the discrimination accuracy of the article 200 taken out from the shelf 20 is improved. Further, the article information estimation device 10 uses not only the relationship between the hand height and the shelf position information (i.e., the relationship in the height direction), but also the relationship between the hand direction and the position of the article 200 (i.e., the relationship in the horizontal plane) when estimating the article 200. Therefore, the discrimination accuracy of the article 200 taken out from the shelf 20 is further improved.

[0066] [Fourth Embodiment] <Functional Configuration Example> FIG. 10 is a diagram showing the functional configuration of the article information estimation device 10 according to this embodiment together with the usage environment of the article information estimation device 10. The article information estimation device 10 according to this embodiment has the same configuration as the article information estimation device 10 according to any one of the first to third embodiments, except that it repeatedly acquires images (hereinafter referred to as first images) from the first imaging unit 70 and identifies the article 200 using these first images. FIG. 10 shows the same case as the first embodiment.

[0067] The first imaging unit 70 includes at least a part of the space in front of the shelf 20, i.e., the front-shelf space, in the imaging area. Therefore, the first image generated by the first imaging unit 70 includes at least a part of the front-shelf space and the article 200 taken out from the shelf 20.

[0068] Then, the output unit 120 estimates the article 200 taken out by a person from the shelf 20 using the image of the article 200 included in the first image. Specifically, the shelf division information storage unit 130 stores the feature amounts on the image of the article 200 together with the article identification information. Then, the output unit 120 estimates the article 200 using the result of collating this feature amount with the first image.

[0069] <Operation Example> The processing performed by the article information estimation device 10 shown in FIG. 10 is as described with reference to FIG. 4 in the first embodiment. However, the details of the processing shown in step S104 are different from those in the first embodiment.

[0070] FIG. 11 is a flowchart for explaining the details of step S104 in the present embodiment. The processes performed in steps S202, S204, S206, S208, and S210 are as described with reference to FIG. 5. However, the output unit 120 reads out the feature amount of the article 200 together with the article identification information in step S208.

[0071] Then, the output unit 120 processes the first image within a reference time (for example, within 10 seconds) after the detection value of the weight sensor 30 changes, and extracts the feature amount of the article 200 included in the first image. If the extracted feature amount matches the feature amount read in step S208, for example, if the score is equal to or higher than the reference value (step S209: Yes), the article identification information read in step S208 is output as it is. On the other hand, if these feature amounts do not match each other (step S209: No), alert processing is performed (step S210).

[0072] When the above-described processing is applied to the article information estimation device 10 shown in the third embodiment, the processing shown in step S209 is performed after step S230 in FIG. 9.

[0073] According to the present embodiment, similar to the first embodiment, the estimation accuracy of the article 200 taken out by a person from the shelf 20 is improved. Further, the first image includes the article 200 taken out by the person. Then, the output unit 120 of the article information estimation device 10 further verifies the article 200 estimated from the detection values of the depth sensor 40 and the weight sensor 30 using the first image. Therefore, the estimation accuracy of the article 200 is further improved.

[0074] [Fifth Embodiment] [Functional Configuration Example] FIG. 12 is a diagram showing the functional configuration of the article information estimation device 10 according to the present embodiment, together with the usage environment of the article information estimation device 10. The article information estimation device 10 according to the present embodiment has the same configuration as the article information estimation device 10 according to any one of the first to fourth embodiments, except that it repeatedly acquires images (hereinafter referred to as second images) from the second imaging unit 80 and identifies the article 200 using these multiple second images. FIG. 12 shows the same case as the fourth embodiment.

[0075] The second imaging unit 80 images the shelf 20 from the front (for example, obliquely upward in the front). For this reason, the second image includes the article 200 placed on the shelf 20. Also, when the second imaging unit 80 images from obliquely upward in the front of the shelf 20, the article 200 located at the back of the shelf 20 can also be imaged. Then, the output unit 120 of the article information estimation device 10 further uses the change in the second image to estimate the article 200 taken out by a person from the shelf 20. Specifically, the output unit 120 estimates the article 200 using the difference between the second image before the depth sensor 40 detects a person's hand (that is, before a person comes to the front shelf space) and the second image after the depth sensor 40 stops detecting a person's hand (that is, after a person leaves the front shelf space).

[0076] <Operation example> The processing performed by the article information estimation device 10 shown in FIG. 12 is as described with reference to FIG. 4 in the first embodiment. However, the details of the processing shown in step S104 are different from those in the first embodiment.

[0077] FIG. 13 is a flowchart for explaining an operation example of the article information estimation device 10 according to the present embodiment. The processing performed in steps S202, S204, S206, S208, S209, and S210 is as described with reference to FIG. 11.

[0078] And when the output unit 120 of the article information estimation device 10 determines that the feature amount of the article 200 included in the first image matches the feature amount of the article read from the shelf division information storage unit 130 (step S209: Yes), it processes the second image and determines whether correction based on the second image is necessary for the article identification information read in step S208 (step S212). When correction is necessary (step S212: Yes), the output unit 120 executes the correction (step S214).

[0079] For example, the output unit 120 extracts the difference between the second image before the depth sensor 40 detects a human hand (i.e., before a person comes to the front of the shelf), and the second image after the depth sensor 40 stops detecting a human hand (i.e., after the person leaves the front of the shelf), and performs a matching process on this difference to determine whether the article 200 corresponding to the article identification information read in step S208 has moved to a shelf 20 different from the shelf 20 where it should originally be. In this process, the position of the article 200 after the movement is specified by, for example, a matching process using the feature amount of the article 200. When this movement is detected (step S212: Yes), the output unit 120 does not output the article identification information. For example, when this function is added to the second embodiment, the article identification information of this article 200 is not stored in the registered article storage unit 60 (step S214).

[0080] In addition, for each movement pattern of the article 200 by a person, the output unit 120 stores in advance a combination of the detection result of the weight sensor 30, the detection result of the depth sensor 40, and the processing result of the second image. And when a result corresponding to this combination is detected, the output unit 120 estimates that a movement pattern corresponding to this combination has occurred.

[0081] On the other hand, when correction is not necessary (step S212: No), the output unit 120 outputs the article identification information read in step S208.

[0082] And this article identification information is used, for example, for the settlement process of products in the store as described in the second embodiment.

[0083] According to the present embodiment, similar to the first embodiment, the estimation accuracy of the article 200 taken out by a person from the shelf 20 is improved. Further, the output unit 120 of the article information estimation device 10 identifies the article 200 moved in the shelf 20. Therefore, when there is an article 200 moved by a person in the shelf 20, it is possible to suppress misrecognition that the person has taken out the article 200.

[0084] As described above, the embodiments of the present invention have been described with reference to the drawings. These are examples of the present invention, and various configurations other than the above can also be adopted.

[0085] In addition, in the plurality of flowcharts used in the above description, a plurality of steps (processes) are described in order. However, the execution order of the steps executed in each embodiment is not limited to the described order. In each embodiment, the order of the illustrated steps can be changed within a range that does not interfere with the content. Also, the above-described embodiments can be combined within a range where the contents do not conflict.

[0086] Some or all of the above embodiments can be described as follows in the appended claims, but are not limited thereto. 1. Acquisition means for acquiring weight change data, which is data based on a change in a detection value of a weight sensor provided on a shelf on which a plurality of articles can be placed, and motion data indicating the movement of a person's hand located in a pre-shelf space, which is a space in front of the shelf; Output means for outputting article identification information of the article estimated to have been taken out by the hand, using the weight change data and the motion data; An article estimation device comprising: 2. The article estimation device according to 1 above, wherein the acquisition means generates the motion data using a detection value of a depth sensor that includes the pre-shelf space in a detection range, and the output means outputs the article identification information associated with the shelf when a relationship between a height of the shelf in which a change in the detection value of the weight sensor has occurred and a detection value of the depth sensor capable of acquiring the motion data satisfies a criterion. 3. In the article estimating device according to 2 above, There are a plurality of the shelves with different heights, At least one of the article identification information is associated with each of the plurality of the shelves, The weight change data indicates fluctuations in weights for each of the plurality of the shelves. An article estimating device. 4. In the article estimating device according to any one of 1 to 3 above, The output means outputs the article identification information associated with the partial area where the change in weight in a plurality of partial areas of the shelf satisfies a criterion. An article estimating device. 5. In the article estimating device according to any one of 1 to 4 above, The article estimating device further includes a storage processing means for associating the article identification information output by the output means with person identification information for identifying the person and storing the result in a storage means. 6. In the article estimating device according to 5 above, The storage processing means acquires the person identification information from a person tracking device that tracks the movement of the person. An article estimating device. 7. In the article estimating device according to any one of 1 to 6 above, The acquisition means repeatedly acquires a first image that is an image including at least a part of the space in front of the shelf, The output means further estimates the article taken out by the hand using an image of the article included in the first image. An article estimating device. 8. In the article estimating device according to any one of 2 to 6 above, The acquisition means repeatedly acquires a second image that is an image of the shelf taken from the front, The output means further identifies the article identification information of the article taken out by the hand using a change in the second image. An article estimating device. 9. In the article estimating device according to any one of 1 to 8 above, A plurality of weight sensors are provided on the shelf at intervals from each other, The acquisition means generates the weight change data using detection values of the plurality of weight sensors. An article estimating device. 10. In the article estimating device according to any one of 1 to 9 above, the shelf is installed in the store, the person is a customer, a settlement processing means for performing settlement processing using the article identification information output by the output means; an electronic receipt output means for outputting an electronic receipt based on the settlement processing; An article estimating device comprising: 11. A computer, obtains weight change data which is data based on a change in a detection value of a weight sensor provided on a shelf on which a plurality of articles can be placed, and motion data indicating a movement of a hand of a person located in a front-shelf space which is a space in front of the shelf, and using the weight change data and the motion data, outputs article identification information of the article estimated to have been taken out by the hand. An article estimating method. 12. In the article estimating method according to 11 above, the computer generates the motion data using a detection value of a depth sensor whose detection range includes the front-shelf space, and outputs the article identification information associated with the shelf when a relationship between a height of the shelf where a change in the detection value of the weight sensor has occurred and a detection value of the depth sensor from which the motion data can be obtained satisfies a criterion. An article estimating method. 13. In the article estimating method according to 12 above, there are a plurality of the shelves having different heights, at least one of the article identification information is associated with each of the plurality of shelves, and the weight change data indicates fluctuations in weights for each of the plurality of shelves. An article estimating method. 14. In the article estimating method according to any one of 11 to 13 above, the computer outputs the article identification information associated with the partial area when a change in weight in a plurality of partial areas of the shelf satisfies a criterion. An article estimating method. 15. In the article estimating method according to any one of 11 to 14 above, The computer stores the output item identification information in a storage means, linked to the person identification information for identifying the person, in an item estimation method. 16. In the item estimation method according to 15 above, The computer acquires the person identification information from a person tracking device that tracks the movement of the person, in an item estimation method. 17. In the item estimation method according to any one of 11 to 16 above, The computer Repeatedly acquires a first image that is an image including at least a part of the space in front of the shelf, And further estimates the item taken out by the hand using the image of the item included in the first image, in an item estimation method. 18. In the item estimation method according to any one of 12 to 16 above, The computer repeatedly acquires a second image that is an image of the shelf taken from the front, and further uses the change in the second image to identify the item identification information of the item taken out by the hand, in an item estimation method. 19. In the item estimation method according to any one of 11 to 18 above, A plurality of weight sensors are provided on the shelf at intervals from each other, The computer generates the weight change data using the detection values of the plurality of weight sensors, in an item estimation method. 20. In the item estimation method according to any one of 11 to 19 above, The shelf is installed in a store, The person is a customer, The computer performs a settlement process using the output item identification information and outputs an electronic receipt based on the settlement process, in an item estimation method. 21. A computer has A function of acquiring weight change data, which is data based on a change in detection values of weight sensors provided on a shelf on which a plurality of items can be placed, and motion data indicating the movement of a person's hand located in the space in front of the shelf, which is the shelf front space, A function of outputting article identification information of the article estimated to be taken out by the hand, using the weight change data and the operation data, A program that provides this. 22. In the program according to 21 above, For the computer, A function of generating the operation data using the detection value of a depth sensor whose detection range includes the space in front of the shelf, A function of outputting the article identification information associated with the shelf when the relationship between the height of the shelf where the detection value of the weight sensor has changed and the detection value of the depth sensor from which the operation data can be acquired satisfies a standard, A program that provides this. 23. In the program according to 22 above, There are a plurality of the shelves with different heights, At least one of the article identification information is associated with each of the plurality of shelves, A program in which the weight change data indicates the weight fluctuations for each of the plurality of shelves. 24. In the program according to any one of 21 to 23 above, A program that provides the computer with a function of outputting the article identification information associated with the partial area where the change in weight in a plurality of partial areas of the shelf satisfies a standard. 25. In the program according to any one of 21 to 24 above, A program that provides the computer with a function of associating the output article identification information with person identification information for identifying the person and storing it in a storage means. 26. In the program according to 25 above, A program that provides the computer with a function of acquiring the person identification information from a person tracking device that tracks the movement of the person. 27. In the program according to any one of 21 to 26 above, For the computer, A function of repeatedly acquiring a first image that is an image including at least a part of the space in front of the shelf, Furthermore, using the image of the article included in the first image, a function of estimating the article taken out by the hand, and a program having the same. 28. In the program according to any one of 22 to 26 above, the computer repeatedly acquires a second image which is an image of the shelf taken from the front, and further has a function of specifying the article identification information of the article taken out by the hand by using the change of the second image. 29. In the program according to any one of 21 to 28 above, a plurality of weight sensors are provided on the shelf at intervals from each other, and a program having the computer generate the weight change data by using the detection values of the plurality of weight sensors. 30. In the program according to any one of 21 to 29 above, the shelf is installed in a store, the person is a customer, and a program having the computer perform a settlement process by using the output article identification information and output an electronic receipt based on the settlement process.

[0087] This application claims priority based on Japanese Patent Application No. 2019-037829 filed on March 1, 2019, and incorporates the entire disclosure thereof herein.

Claims

1. a determining means for determining, using a height of a hand detected by a depth sensor capable of detecting hand movement of a person positioned in a front-shelf space, which is a space in front of a shelf on which a plurality of items can be placed, that the person has reached out his / her hand to the position of the shelf where a change in the detection value of a weight sensor provided on the shelf has been detected; an output means for outputting information on an item among the plurality of items that corresponds to the position of the shelf to which the person extends his / her hand; An article estimation device comprising:

2. The information on the item includes the name of the item and the price of the item. The article estimation device according to claim 1 .

3. The output means outputs a list including information on the items and a total price of the items included in the list. The article estimation device according to claim 1 .

4. The output means outputs information about the item to a terminal device of the person. The article estimation device according to claim 1 .

5. The determination means determines that the person has reached out his / her hand to the position of the shelf where the change in the detection value of the weight sensor has occurred, when a relationship between the height of the shelf where the change in the detection value of the weight sensor has been detected and the height of the hand detected by the depth sensor satisfies a criterion. The article estimation device according to claim 1 .

6. The information on the article is stored in a storage unit in association with personal identification information for identifying the person. The article estimation device according to claim 1 .

7. The storage processing means acquires the person identification information from a person tracking device that tracks the movement of the person. The article estimation device according to claim 6.

8. the determination means determines, using the detected height of the hand and the direction in which the hand is being extended detected by the depth sensor, that the person has extended his / her hand to the position of the shelf where the change in the detection value of the weight sensor has been detected. The article estimation device according to claim 1 .

9. a determination means for determining that a person has extended his / her hand to the position of the shelf where a change in the detection value of a weight sensor provided on the shelf has been detected, using a direction in which the person's hand is extended, the direction being detected by a depth sensor capable of detecting the movement of the hand of the person positioned in a front-shelf space, which is a space in front of the shelf on which a plurality of items can be placed; an output means for outputting information on an item among the plurality of items that corresponds to the position of the shelf to which the person extends his / her hand; An article estimation device comprising:

10. The computer using the height of a hand detected by a depth sensor capable of detecting hand movement of a person positioned in a front-shelf space, which is a space in front of a shelf on which a plurality of items can be placed, to determine that the person has reached out their hand to the position of the shelf where a change in the detection value of a weight sensor provided on the shelf has been detected; outputting information on an item among the plurality of items that corresponds to the position of the shelf to which the person extends his / her hand; The method for estimating an item includes:

11. The computer using a direction in which a hand of a person positioned in a front-shelf space, which is a space in front of a shelf on which a plurality of items can be placed, detected by a depth sensor capable of detecting the movement of the hand, to determine that the person has reached out their hand to the position of the shelf where a change in the detection value of a weight sensor provided on the shelf has been detected; outputting information on an item among the plurality of items that corresponds to the position of the shelf to which the person extends his / her hand; The method for estimating an item includes:

12. Computer, a determination means for determining, using the height of a hand detected by a depth sensor capable of detecting the movement of a hand of a person positioned in a front-shelf space, which is a space in front of a shelf on which a plurality of items can be placed, that the person has reached out his / her hand to the position of the shelf where a change in the detection value of a weight sensor provided on the shelf has been detected; an output means for outputting information on an item among the plurality of items that corresponds to the position of the shelf to which the person extends his / her hand; A program to function as a

13. Computer, a determination means for determining that the person has extended his / her hand to the position of the shelf where a change in the detection value of a weight sensor provided on the shelf has been detected, using a direction in which the hand of the person located in a front-shelf space, which is a space in front of the shelf on which a plurality of items can be placed, is extended, the direction being detected by a depth sensor capable of detecting the movement of the hand of the person; an output means for outputting information on an item among the plurality of items that corresponds to the position of the shelf to which the person extends his / her hand; A program to function as a

Images