Item identification system, item identification method, item identification program, and item acquisition determination system

Abstract

To provide an article specification system, an article specification method, an article specification program, and an article acquisition determination system that can precisely specify articles picked up from different position in the same shelf at the same time.SOLUTION: An article specification system includes: a weight sensor installed in a shelf; a distance measuring sensor acquiring distance data on a person; and an article specifying device including an information processing device acquiring weight measurement data from the weight sensor and distance data on the person from the distance measuring sensor. The information processing device calculates, based upon the weight measurement data, a first article picking-up position of a plurality of picked-up estimated articles and accuracy representing how accurate estimation of the picked-up estimation articles is. The information processing device acquires a second article picking-up position based upon the distance data on the person. The information processing device corrects the accuracy of the picked-up estimated articles based upon the first article picking-up position and the second article picking-up position to calculates correction accuracy, and specifies picked-up articles based upon the correction accuracy.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an item identification system, an item identification method, an item identification program, and an item acquisition determination system. [Background technology]

[0002] Patent Document 1 discloses an item estimation device (hereinafter referred to as the "conventional device") that estimates an item that has been picked up from a shelf on which multiple items are placed. The conventional device acquires weight change data based on changes in the detection values ​​of weight sensors installed on the shelf on which multiple items are placed, and motion data that indicates the hand movements of a person positioned in the shelf space, which is the space in front of the shelf. The conventional device uses the weight change data and the motion data to estimate an item that has been picked up by a person's hand from shelves at different heights. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 179480 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional devices have been unable to estimate items that have been picked up simultaneously from different positions on the same shelf. The present invention has been made to solve the above problem. That is, one of the objects of the present invention is to provide an item identification system, an item identification method, an item identification program, and an item acquisition determination system that can accurately identify items that have been picked up simultaneously from different positions on the same shelf. [Means for solving the problem]

[0005] In order to solve the above problem, the item identification system of the present invention is an item identification system including a weight sensor that is installed on a shelf on which the item is placed and that measures the weight of the item, a distance measuring sensor that includes the shelf and a predetermined area in front of the shelf in its measurement range and that measures the distance to a measurement object that is present in the measurement range, and an item identification device that includes an information processing device that acquires weight measurement data that is the weight measured by the weight sensor and acquires distance data that is the distance that is measured by the distance measuring sensor and that represents a person that is present in the measurement range, and the information processing device estimates an item that has been picked up from the shelf based on the weight measurement data when the item is picked up from the shelf, and For each of a plurality of estimated taken items, which are items that are predicted to have been taken, a first item taking position which is the estimated taking position of the estimated taken item and an accuracy which indicates the degree of accuracy of the estimation of the estimated taken item are calculated, a second item taking position which is based on the position of a predetermined part of the person when the person takes the item from the shelf based on the distance data of the person is obtained, for each of the plurality of estimated taken items, an accuracy is calculated by correcting the accuracy based on the first item taking position and the second item taking position, and based on the accuracy, the picked item which is the one or more of the items that were picked up from the shelf is identified from the plurality of estimated taken items.

[0006] The item identification method of the present invention is a method of identifying an item using an item identification device including a weight sensor that is installed on a shelf on which the item is placed and that measures the weight of the item, a distance measuring sensor that includes the shelf and a predetermined area in front of the shelf in its measurement range and that measures the distance to a measurement object that is present in the measurement range, and an information processing device that acquires weight measurement data that is the weight measured by the weight sensor and acquires distance data that is the distance that is measured by the distance measuring sensor and that represents a person that is present in the measurement range, and the information processing device estimates an item that has been picked up from the shelf based on the weight measurement data when the item is picked up from the shelf, and For each of the multiple estimated taken items, a first item pick-up position, which is the estimated pick-up position of the estimated taken item, and an accuracy indicating the degree of accuracy of the estimation of the estimated taken item are calculated, and based on the distance data of the person, a second item pick-up position based on the position of a specific part of the person when the person picks up the item from the shelf is obtained, and for each of the multiple estimated taken items, a corrected accuracy is calculated by correcting the accuracy based on the first item pick-up position and the second item pick-up position, and based on the corrected accuracy, the picked item is identified as one or more of the items picked up from the shelf from the multiple estimated taken items.

[0007] The item identification program of the present invention is an item identification program that causes a computer to execute processing using weight measurement data, which is the weight measured by a weight sensor that is installed on a shelf on which the item is placed and is acquired from the weight sensor, and distance data, which is the distance measured by the distance sensor that includes the shelf and a predetermined area in front of the shelf in its measurement range and measures the distance to a measurement object that is present in the measurement range and is acquired from the distance sensor, and which is the distance that represents a person present within the measurement range measured by the distance sensor, and causes the computer to estimate an item that has been picked up from the shelf based on the weight measurement data when the item is picked up from the shelf, and to identify a plurality of the estimated items. For each of the multiple estimated taken items, a first item pick-up position, which is the estimated pick-up position of the estimated taken item, and an accuracy indicating the degree of accuracy of the estimation of the estimated taken item are calculated, and based on the distance data of the person, a second item pick-up position based on the position of a specific part of the person when the person picks up the item from the shelf is obtained, and for each of the multiple estimated taken items, a correction accuracy is calculated by correcting the accuracy based on the first item pick-up position and the second item pick-up position, and a process is executed to identify the taken item, which is one or more of the items picked up from the shelf, from the multiple estimated taken items based on the correction accuracy.

[0008] The item acquisition determination system of the present invention includes a plurality of weight sensors spaced apart from one another on a shelf capable of placing a plurality of items thereon, the weight sensors detecting the weight of the items; a first detection unit that uses the detection results of the weight sensors to detect information including the weight distribution on the shelf on which the items are placed; a first determination unit that uses the information detected by the first detection unit to determine an item candidate acquired from the shelf and the validity of the item candidate, the first determination unit determining the validity of a plurality of the item candidates; a distance measurement sensor that detects the distance to an item and a person near the shelf; a second detection unit that uses the detection results of the distance measurement sensor to detect items, people, and people's actions near the shelf; and a second determination unit that uses the detection results of the second detection unit to determine which person has made an action toward which item; and a third judgment unit that uses the judgment results of the judgment unit and the second judgment unit to determine which person has acquired which item, wherein in a first situation in which a first item is placed in a first area of ​​the shelf in one direction, a second area and a third area sandwiching the first area on both sides in the one direction, a second item is placed in the second area, and a third item is placed in the third area, the first judgment unit determines that the multiple candidate items include one second item and one third item, and that the most appropriate candidate item is one second item and the next most appropriate candidate item is one third item, and the second judgment unit determines that the first person has performed an action toward the third item, the third judgment unit determines that the first person has acquired the third item.

[0009] The item acquisition determination system of the present invention includes a plurality of weight sensors spaced apart on a shelf capable of placing a plurality of items thereon, the weight sensors detecting the weight of the items; a first detection unit that uses the detection results of the weight sensors to detect information including a weight distribution on the shelf on which the items are placed; a first determination unit that uses the information detected by the first detection unit to determine an item candidate acquired from the shelf and the validity of the item candidate, the first determination unit determining the validity of a plurality of the item candidates; a distance measurement sensor that detects distances to items and people near the shelf; a second detection unit that uses the detection results of the distance measurement sensor to detect items, people, and people's actions near the shelf; a second determination unit that uses the detection results of the second detection unit to determine which person has made an action toward which item; and a third determination unit that uses the determination results of the first determination unit and the second determination unit to determine which person has acquired which item, In a second situation in which a first item is placed in a first area in the approximate center of the shelf in the one direction, and second and third areas sandwiching the first area on both sides in the one direction, a second item is placed in the second area, and a third item is placed in the third area, and the weights of the second and third items are approximately equal and the weight of the first item is approximately twice the weight of the second item, the first determination unit determines that the multiple candidate items are one of the first item, one of the second item, and one of the third item. and the most appropriate candidate item is one of the first items, and the next most appropriate candidate items are one of the second items and one of the third items, and if the second judgment unit determines that the first person has performed an action toward the second item and the second person has performed an action toward the third item, the third judgment unit determines that the first person has acquired the second item and the second person has acquired the third item.

[0010] The item acquisition determination system of the present invention includes a plurality of weight sensors spaced apart on a shelf capable of placing a plurality of items thereon, the weight sensors detecting the weight of the items; a first detection unit that uses the detection results of the weight sensors to detect information including the weight distribution on the shelf on which the items are placed; a first determination unit that uses the information detected by the first detection unit to determine an item candidate acquired from the shelf and the validity of the item candidate, the first determination unit determining the validity of a plurality of the item candidates; a distance measurement sensor that detects the distance to an item and a person near the shelf; a second detection unit that uses the detection results of the distance measurement sensor to detect items, people, and people's actions near the shelf; a second determination unit that uses the detection results of the second detection unit to determine which person has made an action toward which item; and a second determination unit that uses the determination results of the first determination unit and the second determination unit to determine which person has acquired which item. and a third judgment unit that judges whether the first person has acquired the first item, and a third judgment unit that judges whether the first person has acquired the first item, when in a second situation in which a first item is placed in a first area approximately in the center of the shelf in one direction of the shelf, and second and third areas sandwiching the first area on both sides in the one direction, a second item is placed in the second area, and a third item is placed in the third area, and the weights of the second item and the third item are approximately equal and the weight of the first item is approximately twice the weight of the second item, the first judgment unit judges that the multiple candidate items are one of the first item, one of the second item, and one of the third item, and that the most appropriate candidate item is one of the second item and one of the third item, and that the next most appropriate candidate item is one of the first item, and the second judgment unit judges that the first person has performed an action toward the first item,

[0011] The item acquisition determination system of the present invention includes a plurality of weight sensors spaced apart on a shelf capable of placing a plurality of items thereon, the weight sensors detecting the weight of the items; a first detection unit that uses the detection results of the weight sensors to detect information including the weight distribution on the shelf on which the items are placed; a first determination unit that uses the information detected by the first detection unit to determine an item candidate acquired from the shelf and the validity of the item candidate, the first determination unit determining the validity of a plurality of the item candidates; a distance measurement sensor that detects distances to items and people near the shelf; a second detection unit that uses the detection results of the distance measurement sensor to detect items, people, and people's actions near the shelf; a second determination unit that uses the detection results of the second detection unit to determine which person has made an action toward which item; and a third determination unit that uses the determination results of the first determination unit and the second determination unit to determine which person has acquired which item. and in a second situation in which a first item is placed in the first area, a second item is placed in the second area, and a third item is placed in the third area, the weights of the second item and the third item are approximately equal, and the weight of the first item is approximately twice the weight of the second item, the first determination unit determines that the multiple candidate items are one of the first item, one of the second item, and one of the third item, and that the most appropriate candidate item is one of the first item and the next most appropriate candidate item is one of the second item and one of the third item, and when the second determination unit determines that a first person has performed an action toward the second item and the third item, the third determination unit determines that the first person has acquired one of the second items and one of the third items.

[0012] The item acquisition determination system of the present invention includes a plurality of weight sensors installed spaced apart on a shelf capable of placing a plurality of items, the weight sensors detecting the weight of the items; a first detection unit that uses the detection results of the weight sensors to detect information including the weight distribution on the shelf on which the items are placed; a first determination unit that uses the information detected by the first detection unit to determine item candidates acquired from the shelf and the validity of the item candidates, the first determination unit determining the validity of a plurality of item candidates; a distance measurement sensor that detects distances to items and people near the shelf; and a distance measurement sensor that uses the detection results of the distance measurement sensor to determine the items, people, and people near the shelf. a second detection unit that detects the movement of a person, a second judgment unit that uses the detection result of the second detection unit to judge which person made the movement toward which item, and a third judgment unit that uses the judgment result of the first judgment unit and the judgment result of the second judgment unit to judge which person acquired which item, wherein when the item judged to have the highest validity by the first judgment unit does not match the item judged by the second judgment unit and the item judged to have the second highest validity by the first judgment unit matches the item judged by the second judgment unit, the third judgment unit judges that the item judged by the second judgment unit has been acquired. [Effects of the Invention]

[0013] According to the present invention, it is possible to accurately identify articles that are picked up simultaneously from different positions on the same shelf. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram for explaining an example of an unmanned merchandise store to which an item identification system according to a first embodiment of the present invention is applied. [Figure 2] FIG. 2 is a flowchart showing the actions a user takes from the time they enter an unmanned merchandise store until they leave. [Figure 3] FIG. 3 is a schematic diagram showing an example of the configuration of an item identification system according to the first embodiment of the present invention. [Figure 4] FIG. 4 is a diagram for explaining product management information. [Figure 5] FIG. 5 is a diagram for explaining information about shelves stored in the product management information. [Figure 6] FIG. 6 is a schematic diagram illustrating an example of the hardware configuration of the item identification device. [Figure 7] FIG. 7 is a diagram for explaining the problem of the item identification system of the reference example. [Figure 8A] FIG. 8A is a diagram for explaining a specific example 1 of the operation of the article identification device. [Figure 8B] FIG. 8B is a diagram for explaining specific example 1 of the operation of the article identification device. [Figure 9A] FIG. 9A is a diagram for explaining a second specific example of the operation of the article identification device. [Figure 9B] FIG. 9B is a diagram for explaining a second specific example of the operation of the article identification device. [Figure 10A] FIG. 10A is a diagram for explaining a specific example 3 of the operation of the article identification device. [Figure 10B] FIG. 10B is a diagram for explaining a specific example 3 of the operation of the article identification device. [Figure 11A] FIG. 11A is a diagram for explaining a fourth specific example of the operation of the article identifying device. [Figure 11B] FIG. 11B is a diagram for explaining a fourth specific example of the operation of the article identifying device. [Figure 12A] FIG. 12A is a diagram for explaining a fifth specific example of the operation of the article identifying device. [Figure 12B] FIG. 12B is a diagram for explaining a fifth specific example of the operation of the article identifying device. [Figure 13] FIG. 13 is a flowchart showing a processing flow executed by the item identification system. [Figure 14] FIG. 14 is a flowchart showing a processing flow executed by the product position determination unit (for integration) of the product identification device. [Figure 15] FIG. 15 is a diagram for explaining the operation of the item identification device of the item identification system according to the second embodiment. [Figure 16] FIG. 16 is a diagram for explaining the operation of the item identification device of the item identification system according to the third embodiment. [Figure 17] FIG. 17 is a diagram for explaining another modified example of an item acquisition determination system. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. In all the drawings of the embodiments, the same or corresponding parts may be denoted by the same reference numerals.

[0016] In the following description, various types of information may be described in table format or using records or other expressions, but the various types of information may also be expressed using data structures other than these. Furthermore, when describing identification information, expressions such as "name," "identification ID," and "number" are used, but these are interchangeable. Furthermore, in the following description, processing may be described using a program or functional block as the subject, but the subject of the processing may be a CPU or information processing device instead of a program or functional block.

[0017] <<First Embodiment>> Fig. 1 is a diagram illustrating an example of an unmanned product sales store SP1 to which an item identification system according to a first embodiment of the present invention is applied, and Fig. 2 is a flowchart showing the operations of a user from entering the unmanned product sales store SP1 to leaving the store.

[0018] As shown in FIG. 1, the unmanned retail store SP1 is equipped with a check-in terminal 100, distance measurement sensors 200a and 200b mounted on the ceiling (not shown), a product shelf SH1 including shelves SB1 to SB3 on which products are placed, and a product shelf SH2 including shelves SB11 to SB13 on which products are placed. Product shelf SH1 and product shelf SH2 may also be referred to as "product shelf SH" when there is no need to distinguish between them. Shelves SB1 to SB3 and shelves SB11 to SB13 may also be referred to as "shelves SB" when there is no need to distinguish between them. "Shelves SB" are also referred to as shelf boards. Products may also be referred to as "items."

[0019] Check-in terminal 100 is a terminal for user authentication. Distance measurement sensor 200a and distance measurement sensor 200b measure the distance (distance data) to a measurement target (object) present within the measurement range.

[0020] Distance measurement sensor 200a is a distance measurement sensor for acquiring distance data for detecting the position of a hand extended toward a product (item) placed on shelf SB of product shelf SH. Distance measurement sensor 200b is a distance measurement sensor for tracking a person's movement line, acquiring distance data for generating a person's movement line (a line showing the person's movement trajectory) in the store. In this example, distance measurement sensors 200a and 200b are TOF (Time Of Flight) sensors.

[0021] As shown in Figures 1 and 2, user Us1 enters unmanned merchandise store SP1 (S101) and moves along arrow a1. User Us1 performs personal authentication using check-in terminal 100 (S102). User Us1 selects a product placed on shelf SB of merchandise shelves SH and picks it up by hand (S103). At this time, the product picked up by user Us1 (referred to as the "picked product") is determined (identified) by item identification device 400 (see Figure 3). Automatic payment for the picked product is made by payment processing device 500 (see Figure 3). User Us1 leaves unmanned merchandise store SP1 (S104) and receives payment information at user terminal 110 (S105).

[0022] FIG. 3 is a schematic diagram showing an example of the configuration of an item identification system according to a first embodiment of the present invention. As shown in FIG. 3, this item identification system includes a distance measurement sensor 200a, a distance measurement sensor 200b, a weight sensor 300, an item identification device 400, and a payment processing device 500. These are communicatively connected via a network (not shown). Therefore, the item identification device 400 is capable of receiving information (data) from the distance measurement sensor 200a, the distance measurement sensor 200b, and the weight sensor 300, and the payment processing device 500 is capable of receiving information from the item identification device 400. Note that the item identification system may also be configured without the payment processing device 500. The item identification system may also be referred to as an "item acquisition determination system."

[0023] The distance measurement sensor 200a is installed on the ceiling surface (not shown) of the unmanned merchandise store SP1. The measurement range of the distance measurement sensor 200a includes a predetermined area in front of the product shelf SH1 and the product shelf SH2.

[0024] The weight sensor 300 is installed on each shelf SB of the product shelf SH1 and measures the weight of the product to obtain the weight change (change in total weight and change in weight balance) when the product is picked up from the product shelf SH1. For convenience, the weight (measured value) measured by the weight sensor 300 may also be referred to as "weight measurement data."

[0025] In this example, the product shelf SH1 includes shelves SB1 to SB3 each having a rectangular planar shape, and a weight sensor 300 is installed at each of the four corners of each of the shelves SB1 to SB3. The product shelf SH2 includes shelves SB11 to SB13 each having a rectangular planar shape, and a weight sensor 300 is installed at each of the four corners of each of the shelves SB11 to SB13.

[0026] The item identification device 400 includes an item position determination unit (for shelf) 410, a reach position determination unit 420, a final item position determination unit (for integration) 430, and item management information 440.

[0027] The product position determination unit (for shelf) 410 determines (determines, acquires) the position of the estimated product (hereinafter referred to as "estimated picked-up product") that is estimated to have been picked up from the shelf SB based on the weight change (change in total weight and change in weight balance) detected by the weight sensor 300. This position is also referred to as the "product acquisition occurrence event position." The "X-direction position Xw of the product acquisition occurrence event position" may also be simply referred to as the "X-direction position Xw." For convenience, the X-direction position Xw may also be referred to as the "first item pick-up position." The item identification device 400 defines the width direction of the product shelf SH as the X direction, and the direction perpendicular to the X direction in a horizontal plane parallel to the ground as the Y direction.

[0028] The hand reaching position determination unit 420 determines (determines, acquires) the position of the hand of the user Us1 when the user Us1 picks up an item from the product shelf SH. The hand reaching position determination unit 420 acquires, as this hand position, the position where the point cloud representing the hand of the user Us1 in the distance data (point cloud) of the user Us1 detected by the distance measurement sensor 200a and the distance measurement sensor 200b passes through a predetermined position in front of the product shelf SH.

[0029] This position is also referred to as the "item pickup position Xt." For convenience, the item pickup position Xt may also be referred to as the "second item pickup position." The X-direction position Xw and the item pickup position Xt described above are expressed by coordinate values ​​in a common coordinate system.

[0030] Fig. 4 is a diagram illustrating product management information 440. As shown in Fig. 4, the product management information 440 includes columns for storing information (values): #441, product name 442, barcode (ID) 443, weight 444, price 445, shelf number 446, shelf board number 447, shelf allocation number 448, shelf allocation start position 449, and shelf allocation end position 450. In the product management information 440, information corresponding to each column for managing products placed on a product shelf SH is associated with each other and stored as row-by-row information (records).

[0031] Specifically, #441 stores the row number. Product name 442 stores the name of the product. Barcode (id) 443 stores the product identification ID. Weight 444 stores the weight of the product. Price 445 stores the price of the product. Shelf number 446 stores the shelf numbers corresponding to product shelves SH1 and SH2 shown in Figure 5. Shelf number 447 stores the shelf numbers corresponding to each shelf SB shown in Figure 5. Shelf allocation number 448 stores the shelf allocation number of each shelf allocation column shown in Figure 5. Shelf allocation start position 449 stores the start position of the range of shelf allocation columns shown in Figure 5. Shelf allocation end position 450 stores the end position of the range of shelf allocation columns shown in Figure 5.

[0032] The payment processing device 500 includes a payment processing unit 510. The payment processing unit 510 performs processing to pay for the product identified by the product identification device 400.

[0033] Fig. 6 is a schematic diagram showing an example of the hardware configuration of the item identification device 400. As shown in Fig. 6, the item identification device 400 includes a CPU 2001, a ROM 2002, a RAM 2003, a storage device 2004, a network interface 2005, an input / output interface 2006, etc. These are communicably connected to each other via a bus 2007. For convenience, the device including the CPU 2001, the ROM 2002, the RAM 2003, the storage device 2004, the network interface 2005, the input / output interface 2006, and the bus 2007 is also referred to as an "information processing device." The information processing device may be a plurality of information processing devices, or may be a virtual information processing device on the cloud.

[0034] The CPU 2001 loads various programs (not shown) stored in the ROM 2002 and / or the storage device 2004 into the RAM 2003 and executes the programs loaded into the RAM 2003 to realize various functions. As described above, various programs executed by the CPU 2001 are loaded into the RAM 2003, and data used when the CPU 2001 executes the various programs is temporarily stored. The ROM 2002 is a non-volatile storage medium in which various programs are stored. The storage device 2004 is a non-volatile storage device that can read and write data. The network interface 515 is an interface for connecting the item identification device 400 to a network. The input / output interface 2006 is an interface for connecting to external devices (for example, operating devices such as a keyboard and a mouse, and a display (display device)).

[0035] The product position determination unit (for shelves) 410, the hand reaching position determination unit 420, and the product position determination unit (for integration) 430 are configured by various programs stored in the ROM 2002 and / or the storage device 2004 and executed by the CPU 2001. The product management information 440 is configured by a database stored in the storage device 2004. An example of the hardware configuration of the payment processing device 500 is the same as that shown in FIG. 6. The payment processing unit 510 is configured by various programs stored in the ROM 2002 and / or the storage device 2004 and executed by the CPU 2001.

[0036] <Problems with the reference example's item identification device> To facilitate understanding of the present invention, the problems of the reference example of an item identification system that uses only a weight sensor will be described. Fig. 7 is a diagram for explaining the problems of the reference example of an item identification system. In the example described below, an example will be described in which an item is picked up from the top shelf SB1 of the product shelf SH1 with shelf board number 1 (the same applies to specific examples 1 to 5 described below).

[0037] As shown in Fig. 7, shelf SB1 is divided into a shelf allocation row a having an X-direction position (widthwise position) of 0 or more and less than 0.3, a shelf allocation row b having an X-direction position of 0.3 or more and less than 0.6, and a shelf allocation row c having an X-direction position of 0.6 or more and less than 1.0. Mark MK1 in Fig. 7 indicates the position where a product acquisition event occurs. The position where a product acquisition event occurs is a position calculated (estimated) using the measurement value of the weight sensor 300. The method for calculating the position where a product acquisition event occurs will be described in detail later.

[0038] A plurality of items C are placed in the shelf layout row a, a plurality of items B are placed in the shelf layout row b, and a plurality of items A are placed in the shelf layout row c. The item identification device can obtain information indicating the type, weight, price, etc. of the items placed in each shelf layout row by referring to the item management information 440.

[0039] If user Us1 picks up product A located near the right side of X-axis position 0.6, a product acquisition event may occur at X-axis position Xw0.55 indicated by mark MK1. In this case, if the picked product is determined based only on X-axis position Xw, the picked product will be product B. Therefore, even though product A is actually picked up from shelf SB1, product B may be determined (identified) as the picked product. As such, the item identification device of the reference example has the problem of low accuracy in identifying the picked product (item).

[0040] <Overview of the operation of the present invention> To solve such problems, the item identification device 400 of the item identification system of the present invention uses the measurement value of the weight sensor 300 to estimate the estimated picked-up product, and calculates the accuracy (in this example, likelihood Pw) for each estimated picked-up product, which is a parameter indicating the position of the estimated picked-up product and the degree of accuracy of the estimation of the estimated picked-up product.

[0041] Furthermore, the item identification device 400 corrects the accuracy (in this example, likelihood Pw) of each estimated picked-up item based on the X-direction position Xw and the hand position of user Us1 when picking up the item, detected by distance measurement sensor 200a and distance measurement sensor 200b. The item identification device 400 identifies (determines) a picked-up item from among the estimated picked-up items based on the corrected accuracy (in this example, corrected likelihood Pnew). This allows the item identification device 400 to improve the accuracy of identifying the picked-up item.

[0042] (Calculation of product acquisition event occurrence location and likelihood Pw) The following describes a method for calculating the location of a product acquisition event occurrence and the likelihood Pw executed by the product position determination unit (for shelves) 410 of the product identification device 400. Weight sensors 300 are installed at the four corners of each shelf SB of the product shelves SH. When a product is picked up from the shelf SB, the weight balance of the product and shelf SB changes, causing the measurement values ​​of the weight sensors 300 at the four corners to change. The product identification device 400 estimates the product that is likely to be picked up based on the changes in the total weight and weight balance of the product and shelf SB measured by each weight sensor 300, and calculates the location of the estimated picked up product (i.e., the location of a product acquisition event occurrence) and the likelihood Pw of the estimated picked up product for each estimated picked up product.

[0043] As an example, when a product is picked up from shelf SB, the estimated product to be picked up, the position of each estimated product to be picked up (the position where the product acquisition event occurred), and the likelihood Pw can be calculated using a computational model that takes the measurement values ​​of each weight sensor 300 within a specified measurement time as input and outputs the estimated product to be picked up, the position where the product acquisition event occurred, and the likelihood Pw of each estimated product to be picked up.

[0044] The likelihood Pw indicates the probability that the predicted item to be picked up (one predicted item to be picked up or multiple predicted items to be picked up simultaneously) will actually be picked up, depending on the measurement values ​​of each weight sensor 300 within a predetermined time period (changes in the total weight and weight balance of the shelf SB (shelf SB with items placed on it) within a predetermined time period). The likelihood Pw is expressed as a numerical value ranging from 0 to 1. The likelihood Pw is an example of accuracy, which is a parameter indicating the degree of accuracy of the prediction of the predicted item to be picked up, and the accuracy may be a parameter other than the likelihood Pw.

[0045] The weight measurement by each weight sensor 300 does not have to be performed just once, and the change in weight may be measured continuously, periodically, or randomly for a predetermined time or a predetermined number of times. The product position determination unit (for shelf) 410 may calculate the weight distribution on the shelf SB, map and evaluate this, as disclosed in the specification of U.S. Patent Application Publication No. US2021 / 0148751, etc., to calculate the estimated picked-up products, the product acquisition event occurrence location and likelihood (event likeliness score) of each estimated picked-up product.

[0046] In reality, the measurements of the four weight sensors 300 contain errors, and the calculated total weight and weight balance of the products and shelf SB also contain errors. For example, when user Us1 presses on the shelf SB when picking up a product or when user Us1 touches the shelf SB, the shelf SB vibrates or swings, causing the weight measurements of each weight sensor 300 to fluctuate, making it impossible to accurately measure changes in total weight and weight balance due solely to the product being picked up. As an example of a countermeasure, the following may be taken: The product position determination unit (for shelves) 410 measures weight multiple times using the weight sensors 300 continuously, periodically, or randomly over a predetermined period of time or a predetermined number of times. If the measurement value of each weight sensor 300 fluctuates beyond a predetermined threshold, the product position determination unit (for shelves) 410 remeasures the weight multiple times or for a predetermined period of time. If the measurement value of each weight sensor 300 falls within the predetermined threshold as a result of the remeasurement, the product position determination unit (for shelf) 410 adopts the remeasurement result (measurement value of each weight sensor 300).

[0047] For predicted products with a likelihood Pw lower than a predetermined value, user Us1 may be asked to confirm the product at checkout (before payment) or user Us1 may be allowed to modify the product. For example, if the difference in likelihood Pw (or price) between the initially presented candidate product (predicted product) and the product modified by user Us1 is within a predetermined range, the customer's modification may be accepted. Furthermore, if the difference is outside the predetermined range, user Us1 may be prompted to reconfirm or a store clerk may be called to confirm.

[0048] (How to determine the hand position) The hand reaching position determination unit 420 of the item identification device 400 detects the X-direction position when the person's hand indicated by the distance measurement data (point cloud) passes through a virtual surface (also called a "virtual screen") parallel to the height direction of the product shelf SH installed (set) at a predetermined position in front of the product SH (shelf SB) using the distance measurement sensors 200a and 200b, as the "hand position when user Us1 picks up the product from the shelf SB (hereinafter simply referred to as the "hand reaching position")," and acquires the detected X-direction position as the item pick-up position Xt.

[0049] (Likelihood correction) The product position determination unit (for integration) 430 of the product identification device 400 corrects the likelihood Pw for the estimated picked-up product by applying the likelihood Pw, the X-direction position Xw of the product acquisition event occurrence location, and the product pickup position Xt to the following calculation formula (1), and calculates the corrected likelihood Pnew.

[0050] Pnew=Pw-α×|Xw-Xt|...Calculation formula (1) (In the calculation formula (1), Pnew is the corrected likelihood Pnew. Xw is the X-direction position Xw of the product acquisition event occurrence position. Xt is the item pickup position Xt. α is a weighting coefficient.) The value of α in the calculation formula (1) used in the specific examples 1 to 5 described below is set to "2".

[0051] <Example 1> Specific example 1 of the operation of item identification device 400 will be described using Figures 8A and 8B. Figures 8A and 8B are diagrams for explaining specific example 1 of the operation of item identification device 400. As shown in Figure 8A, assume a situation in which distance measurement sensors 200a and 200b detect an outstretched hand position at position P11, weight sensor 300 detects a weight change (a change in total weight and a change in weight balance), and a product acquisition event occurs at positions P21, P22, and P23. In Figure 8A, the position of the virtual screen is indicated by dashed line L1.

[0052] The product position determination unit (for shelves) 410 of the product identification device 400 calculates the weight sensor information shown in Fig. 8B based on the measurement value of the weight sensor 300 and the product management information 440. The weight sensor information includes columns for storing information (values): #801, X direction (0 to 1) Xw802, Y direction (0 to 1) 803, estimated product 804, and likelihood (0 to 1) Pw805.

[0053] In the weight sensor information, information corresponding to each column based on the measurement value of the weight sensor 300 is associated with each other and stored as information (records) in row units.

[0054] Specifically, #801 stores the row number. X direction (0-1) Xw 802 stores the X direction position Xw of the location where the product acquisition event occurred. Y direction (0-1) 803 stores the Y direction position of the location where the product acquisition event occurred. Estimated product 804 stores information (product name) indicating the estimated product to be picked up at X direction position Xw. Likelihood (0-1) Pw 805 stores the likelihood Pw of the estimated picked up product.

[0055] The hand reaching position determination unit 420 of the item identification device 400 calculates the TOF sensor information shown in FIG. 8B based on the position of a predetermined part of the user Us1 (the outstretched hand position in specific example 1) detected by the distance measurement sensors 200a and 200b and the item management information 440. The TOF sensor information includes #811 and X-direction (0 to 1) Xt 812 as columns for storing information (values). In the TOF sensor information, information corresponding to each column based on the information detected by the distance measurement sensors 200a and 200b is associated with each other and stored as row-by-row information (records). #811 stores the row number. The X-direction (0 to 1) Xt 812 stores the outstretched hand position as the item pick-up position Xt. The estimated item 813 stores information (item name) indicating the estimated item picked up at the item pick-up position Xt.

[0056] In specific example 1, the weight sensor information includes information on row number 1, information on row number 2, and information on row number 3. The X-direction position Xw of the information on row number 1 is 0.55, the estimated picked-up product is B, and the likelihood Pw is 0.8. The X-direction position Xw of the information on row number 2 is 0.80, the estimated picked-up product is A, and the likelihood Pw is 0.6. The X-direction position Xw of the information on row number 3 is 0.20, the estimated picked-up product is C, and the likelihood Pw is 0.3.

[0057] The TOF sensor information includes information of row number 1. The item pick-up position Xt of the information of row number 1 is 0.65, and the item estimated to be picked up is A.

[0058] The commodity position determination unit (for integration) 430 of the item identification device 400 calculates corrected likelihood information by applying the weight sensor information and the TOF sensor information to the calculation formula (1). That is, the commodity position determination unit (for integration) 430 calculates the corrected likelihood Pnew for each estimated picked-up commodity using the calculation formula (1).

[0059] The corrected likelihood information includes columns for storing information (values): # 821, estimated product 822, corrected likelihood (Pnew) 823, and Pnew 824. In the corrected likelihood information, information corresponding to each column for calculating the corrected likelihood Pnew is associated with each other and stored as row-by-row information (records).

[0060] Specifically, #821 stores the line number. Estimated product 822 stores information indicating the estimated product to be picked (product name). Corrected likelihood (Pnew) 823 stores formula (1) in which numerical values ​​are substituted for each variable. Pnew 824 stores the value of corrected likelihood Pnew calculated by formula (1).

[0061] Specifically, the method for calculating the corrected likelihood information is as follows: the product position determination unit (for integration) 430 calculates the corrected likelihood of the row with row number 1 of the corrected likelihood information using formula (1) from the information of row number 1 of the weight sensor information and the information of row number 1 of the TOF sensor information. The product position determination unit (for integration) 430 calculates the corrected likelihood of the row with row number 2 of the corrected likelihood information using formula (1) from the information of row number 2 of the weight sensor information and the information of row number 1 of the TOF sensor information. The product position determination unit (for integration) 430 calculates the corrected likelihood of the row with row number 3 of the corrected likelihood information using formula (1) from the information of row number 3 of the weight sensor information and the information of row number 1 of the TOF sensor information.

[0062] In specific example 1, the calculated corrected likelihood information includes information in row number 1, information in row number 2, and information in row number 3. The estimated product to be picked up in the information in row number 1 is B, and the corrected likelihood Pnew is 0.6. The information in row number 1 corresponds to the information when user Us1 picked up product B. The estimated product to be picked up in the information in row number 2 is A, and the corrected likelihood Pnew is 0.3. The information in row number 2 corresponds to the information when user Us1 picked up product A. The estimated product to be picked up in the information in row number 3 is C, and the corrected likelihood Pnew is -0.6. The information in row number 3 corresponds to the information when user Us1 picked up product C.

[0063] The product position determination unit (for integration) 430 determines (specifies) the estimated picked-up product with the highest corrected likelihood Pnew from among the estimated picked-up products as the picked-up product from shelf SB1. In specific example 1, the product position determination unit (for integration) 430 determines product B as the picked-up product because the estimated picked-up product B has the highest corrected likelihood Pnew.

[0064] In specific example 1, the estimated taken product with the highest likelihood among the estimated taken products in the weight sensor information is estimated product B, and the estimated taken product with the highest likelihood among the estimated taken products in the corrected likelihood information is estimated product B. The estimated taken products with the highest likelihood in each piece of information match.

[0065] <Example 2> 9A and 9B will be used to explain specific example 2 of the operation of item identification device 400. Figures 9A and 9B are diagrams for explaining specific example 2 of the operation of item identification device 400. As shown in Figure 9A, assume a situation in which distance measurement sensors 200a and 200b detect an outstretched hand position at position P31, weight sensor 300 detects a weight change (a change in total weight and a change in weight balance), and a product acquisition event occurs at positions P41, P42, and P43.

[0066] The product position determination unit (for shelves) 410 and the hand-stretching position determination unit 420 of the item identification device 400 calculate weight sensor information and TOF sensor information, similar to the first specific example.

[0067] In specific example 2, the weight sensor information includes information on row number 1, information on row number 2, and information on row number 3. The X-direction position Xw of the information on row number 1 is 0.55, the estimated picked-up product is B, and the likelihood Pw is 0.8. The X-direction position Xw of the information on row number 2 is 0.80, the estimated picked-up product is A, and the likelihood Pw is 0.6. The X-direction position Xw of the information on row number 3 is 0.20, the estimated picked-up product is C, and the likelihood Pw is 0.3.

[0068] The TOF sensor information includes information of row number 1. The item pick-up position Xt of the information of row number 1 is 0.95, and the item estimated to be picked up is A.

[0069] The commodity position determination unit (for integration) 430 of the item identification device 400 calculates corrected likelihood information by applying the weight sensor information and the TOF sensor information to calculation formula (1). That is, the commodity position determination unit (for integration) 430 calculates the corrected likelihood Pnew for each estimated pickup commodity using calculation formula (1). The calculation method for this corrected likelihood Pnew is the same as in specific example 1.

[0070] In specific example 2, the calculated corrected likelihood information includes information in row number 1, information in row number 2, and information in row number 3. The estimated product to be picked up in the information in row number 1 is B, and the corrected likelihood Pnew is 0. The information in row number 1 corresponds to the information when user Us1 picked up product B. The estimated product to be picked up in the information in row number 2 is A, and the corrected likelihood Pnew is 0.3. The information in row number 2 corresponds to the information when user Us1 picked up product A. The estimated product to be picked up in the information in row number 3 is C, and the corrected likelihood Pnew is -1.2. The information in row number 3 corresponds to the information when user Us1 picked up product C.

[0071] The product position determination unit (for integration) 430 determines (specifies) the estimated picked-up product with the highest corrected likelihood Pnew from among the estimated picked-up products as the picked-up product from shelf SB1. In specific example 2, the product position determination unit (for integration) 430 determines product A as the picked-up product because the estimated picked-up product A has the highest corrected likelihood Pnew.

[0072] In Specific Example 2, the estimated taken product with the highest likelihood among the estimated taken products in the weight sensor information is estimated product B, and the estimated taken product with the highest corrected likelihood Pnew among the estimated taken products in the corrected likelihood information is estimated product A. In other words, if the taken product is identified based only on the weight sensor information as in the reference example, the taken product will be identified as product B, and the wrong product will be identified as the taken product. In contrast, in Specific Example 2, the likelihood Pw of the weight sensor information is corrected, and the taken product is identified based on the corrected likelihood Pnew, thereby identifying the correct product A as the taken product.

[0073] <Example 3> Specific example 3 of the operation of item identification device 400 will be described using Figures 10A and 10B. Figures 10A and 10B are diagrams for explaining specific example 3 of the operation of item identification device 400. As shown in Figure 10A, assume a situation in which distance measurement sensors 200a and 200b detect the outstretched hand position at position P51, weight sensor 300 detects weight changes (changes in total weight and weight balance), and product acquisition events occur at positions P61a, P61b, and P62. Based on the weight changes (changes in total weight and weight balance), it is detected (estimated) that two products, one at position P61a and one at position P61b, have been picked up simultaneously.

[0074] The product position determination unit (for shelves) 410 and the hand reaching position determination unit 420 of the product identification device 400 acquire weight sensor information and TOF sensor information, similar to Specific Example 1. Note that information about multiple products picked up at the same time (in this example, product C and product A) is summarized into information about one row number including two rows.

[0075] In specific example 3, the weight sensor information includes information on row number 1 and information on row number 2, which include two rows. The X-axis position Xw of the information on one row of row number 1 is 0.20, the predicted prize to be taken is C, and the likelihood Pw is 0.8. The X-axis position Xw of the information on another row of row number 1 is 0.70, the predicted prize to be taken is A, and the likelihood Pw is 0.8. The X-axis position Xw of the information on row number 2 is 0.5, the predicted prize to be taken is B, and the likelihood Pw is 0.7.

[0076] The TOF sensor information includes information of row number 1. The item pick-up position Xt of the information of row number 1 is 0.5, and the estimated item is B.

[0077] The product position determination unit (for integration) 430 of the product identification device 400 calculates corrected likelihood information by applying the weight sensor information and TOF sensor information to calculation formula (1). That is, the product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for each estimated picked-up product using calculation formula (1). Furthermore, the product position determination unit (for integration) 430 calculates the average Pnew' (average corrected likelihood Pnew') of the corrected likelihoods Pnew of estimated picked-up product C and estimated product A, which are estimated to have been picked up at the same time. The corrected likelihood information further includes Pnew' 825 as a column for storing information. The average Pnew' of the corrected likelihoods Pnew is stored in Pnew' 825.

[0078] Specifically, the product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for each estimated picked product using formula (1). The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for row number 1 of the corrected likelihood information using formula (1) from the information in one row of row number 1 of the weight sensor information and the information in row number 1 of the TOF sensor information. The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for the other rows of row number 1 of the corrected likelihood information using formula (1) from the information in the other rows of row number 1 of the weight sensor information and the information in row number 1 of the TOF sensor information. The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for row number 2 of the corrected likelihood information using formula (1) from the information in row number 2 of the weight sensor information and the information in row number 1 of the TOF sensor information. The product position determination unit (for integration) 430 calculates the average Pnew' of the corrected likelihoods Pnew of estimated product C and estimated product A by finding the average Pnew' of the corrected likelihoods Pnew of the first row of row number 1 and the other rows, and stores it in the rows corresponding to the first row and the other rows of row number 1 of Pnew' 824a.

[0079] In the third specific example, the calculated corrected likelihood information includes information on row number 1, row number 2, and row number 3, which include two rows.

[0080] The estimated product to be picked up in the information in the first row of row number 1 is C, the corrected likelihood Pnew is 0.2, and the average corrected likelihood Pnew' is 0.3. The information in the first row of row number 1 corresponds to information when user Us1 picked up product C at the same time as A. The estimated product to be picked up in the information in the other rows of row number 1 is A, the corrected likelihood Pnew is 0.4, and the average corrected likelihood Pnew' is 0.3. The information in the other rows of row number 1 corresponds to information when user Us1 picked up product A at the same time as C. The estimated product to be picked up in the information in row number 2 is B, and the corrected likelihood Pnew is 0.7. The information in row number 2 corresponds to information when user Us1 picked up product B.

[0081] The product position determination unit (for integration) 430 determines (specifies) the estimated product to be picked up that has the highest corrected likelihood Pnew or average corrected likelihood Pnew' from among the estimated products to be picked up as the product to be picked up from shelf SB1. In specific example 3, the product position determination unit (for integration) 430 determines product B as the product to be picked up because the estimated product B has the highest corrected likelihood Pnew.

[0082] <Example 4> Specific example 4 of the operation of item identification device 400 will be described using Figures 11A and 11B. Figures 11A and 11B are diagrams for explaining specific example 4 of the operation of item identification device 400. As shown in Figure 11A, assume a situation in which distance measurement sensors 200a and 200b detect outstretched hand positions at positions P71 and P72, weight sensor 300 detects weight changes (changes in total weight and weight balance), and product acquisition events occur at positions P81, P82a, and P82b. Based on the weight changes (changes in total weight and weight balance), it is detected that the product at position P82a and the product at position P82b have been picked up simultaneously.

[0083] The product position determination unit (for shelves) 410 and the hand reaching position determination unit 420 of the product identification device 400 acquire the weight sensor information and the TOF sensor information shown in Fig. 11B, as in Example 1. Note that information about multiple products picked up at the same time (in this example, product C and product A) is summarized into information about one row number including two rows.

[0084] In specific example 4, the weight sensor information includes information in row number 1 and information in row number 2, which includes two rows. The X-direction position Xw of the information in row number 1 is 0.50, the predicted product to be picked up is B, and the likelihood Pw is 0.8. The X-direction position Xw of the information in one row of row number 2 is 0.20, the predicted product to be picked up is C, and the likelihood Pw is 0.7. The X-direction position Xw of the information in the other row of row number 2 is 0.7, the predicted product to be picked up is A, and the likelihood Pw is 0.7.

[0085] The TOF sensor information includes information for row number 1, which includes two rows. The item pick-up position Xt for the information in one row of row number 1 is 0.25, and the estimated item to be picked up is C. The item pick-up position Xt for the information in the other row of row number 1 is 0.65, and the estimated item to be picked up is A.

[0086] The product position determination unit (for integration) 430 of the product identification device 400 calculates corrected likelihood information by applying the weight sensor information and the TOF sensor information to calculation formula (1). The corrected likelihood information further includes Pnew selection 824a and Pnew_sum 826 as columns for storing information. Pnew selection 824a stores information indicating whether the corrected likelihood Pnew of that row has been selected. Pnew_sum 826 stores the sum Pnew_sum of the corrected likelihoods Pnew.

[0087] Specifically, the product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for each estimated picked product using formula (1). The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for the first row of row number 1 of the corrected likelihood information using formula (1) from the information in row number 1 of the weight sensor information and the information in the first row of row number 1 of the TOF sensor information. The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for the other rows of row number 1 of the corrected likelihood information using formula (1) from the information in the first row of row number 1 of the weight sensor information and the information in the other rows of row number 1 of the TOF sensor information. Then, the product position determination unit (for integration) 430 selects the larger of the corrected likelihood Pnew in one row of row number 1 of the corrected likelihood information and the corrected likelihood Pnew in another row of row number 1 of the corrected likelihood information, and stores information indicating that the larger corrected likelihood has been selected, "In this example, O", in the row corresponding to the selected corrected likelihood Pnew in Pnew selection 824a.

[0088] The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for the first row of row number 2 of the corrected likelihood information using formula (1) from the information in the first row of row number 2 of the weight sensor information and the information in the first row of row number 1 of the TOF sensor information. The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for the second row of row number 2 of the corrected likelihood information using formula (1) from the information in the first row of row number 2 of the weight sensor information and the information in the other row of row number 1 of the TOF sensor information. Then, the product position determination unit (for integration) 430 selects the larger of the corrected likelihood Pnew for the first row of row number 2 of the corrected likelihood information and the corrected likelihood Pnew for the second row of row number 2 of the corrected likelihood information, and stores information indicating that the larger corrected likelihood has been selected, "In this example, O", in the row corresponding to the selected corrected likelihood Pnew in Pnew selection 824a.

[0089] The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for the third row of row number 2 of the corrected likelihood information using formula (1) from the information in the other rows of row number 2 of the weight sensor information and the information in the first row of row number 1 of the TOF sensor information. The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for the fourth row of row number 2 of the corrected likelihood information using formula (1) from the information in the other rows of row number 2 of the weight sensor information and the information in the other rows of row number 1 of the TOF sensor information. Then, the product position determination unit (for integration) 430 selects the larger of the corrected likelihood Pnew for the third row of row number 2 of the corrected likelihood information and the corrected likelihood Pnew for the fourth row of row number 2 of the corrected likelihood information, and stores information indicating that the larger corrected likelihood Pnew has been selected, "In this example, O", in the row corresponding to the selected corrected likelihood Pnew in Pnew selection 824a.

[0090] Furthermore, the product position determination unit (for integration) 430 calculates the average Pnew' of the corrected likelihoods Pnew of the inferred product C and the inferred product A that are inferred to have been picked up at the same time. The product position determination unit (for integration) 430 calculates the average Pnew' of the corrected likelihoods Pnew of the inferred product C and the inferred product A by finding the average Pnew' of the selected corrected likelihoods Pnew, and stores it in the rows corresponding to the first to fourth rows of row number 2 of Pnew' 824a.

[0091] Furthermore, the product position determination unit (for integration) 430 calculates the sum Pnew_sum of the corrected likelihoods Pnew of the inferred product C and the inferred product A that are inferred to have been picked up by the same hand at the same time. The product position determination unit (for integration) 430 calculates the sum Pnew_sum of the corrected likelihoods Pnew of the inferred product C and the inferred product A that are inferred to have been picked up by the left hand at the same time by determining the sum Pnew_sum of the corrected likelihoods Pnew of the information in the first row of row number 2 and the corrected likelihoods Pnew of the information in the third row of row number 2, and stores this in Pnew_sum 826 in the fifth row of row number 2.

[0092] The product position determination unit (for integration) 430 calculates the sum Pnew_sum of the corrected likelihood Pnew of the information in the second row of row number 2 and the corrected likelihood Pnew of the information in the fourth row of row number 2, and stores the sum Pnew_sum of the corrected likelihoods Pnew of estimated product C and estimated product A, which are estimated to have been picked up by the right hand at the same time, in Pnew_sum 826 of the sixth row of row number 2.

[0093] In the fourth specific example, the calculated corrected likelihood information includes information on row number 1, which includes two rows, and information on row number 2, which includes six rows.

[0094] The estimated product to be picked up in the information in the first row of row number 1 is B, and the corrected likelihood Pnew is 0.3. The information in the first row of row number 1 corresponds to information when user Us1 picks up product B with his left hand. The estimated product to be picked up in the information in the other rows of row number 1 is B, and the corrected likelihood Pnew is 0.5. The information in the other rows of row number 1 corresponds to information when user Us1 picks up product B with his right hand.

[0095] The estimated picked-up product in the information in the first row of row number 2 is C, the corrected likelihood Pnew is 0.6, and the average corrected likelihood Pnew' is 0.6. The information in the first row of row number 2 corresponds to information when user Us1 picks up product C with his left hand.

[0096] The information in the second row of row number 2 indicates that the estimated picked-up product is C, the corrected likelihood Pnew is −0.2, and the average corrected likelihood Pnew′ is 0.6. The information in the second row of row number 2 corresponds to the information when user Us1 picks up product C with his right hand.

[0097] The information in the third row of row number 2 indicates that the estimated product to be picked up is A, the corrected likelihood Pnew is −0.2, and the average corrected likelihood Pnew′ is 0.6. The information in the third row of row number 2 corresponds to the information when user Us1 picks up product A with his left hand.

[0098] The information in the fourth row of row number 2 indicates that the estimated product to be picked up is A, the corrected likelihood Pnew is 0.6, and the average corrected likelihood Pnew' is 0.6. The information in the fourth row of row number 2 corresponds to the information when user Us1 picks up product A with his right hand.

[0099] The estimated products picked up in the information on the fifth row of row number 2 are C and A, and the sum Pnew_sum of the corrected likelihoods Pnew is 0.4 (= 0.6 + (-0.2)). The information on the fifth row of row number 2 corresponds to the information when user Us1 picked up product C and product A with his left hand.

[0100] The estimated products picked up in the information on the sixth row of row number 2 are C and A, and the sum Pnew_sum of the corrected likelihoods Pnew is 0.4 (= 0.6 + (-0.2)). The information on the sixth row of row number 2 corresponds to the information when user Us1 picked up product C and product A with his right hand.

[0101] The product position determination unit (for integration) 430 determines (identifies) the estimated product to be picked up that has the highest likelihood among the corrected likelihood Pnew, the average corrected likelihood Pnew', and the sum Pnew_sum of the corrected likelihoods Pnew, from among the estimated products to be picked up, as the product to be picked up from shelf SB1. In specific example 4, the product position determination unit (for integration) 430 determines product A and product C as the products to be picked up because the average corrected likelihood Pnew' of estimated product A and estimated product C is the highest.

[0102] <Example 5> Specific example 5 of the operation of item identification device 400 will be described using Figures 12A and 12B. Figures 12A and 12B are diagrams for explaining specific example 5 of the operation of item identification device 400. As shown in Figure 12A, distance measurement sensor 200a and distance measurement sensor 200b detect the position of user X's outstretched hand at position P91X1, and detect the position of user Y's outstretched hand at position P91Y1. Assume that weight sensor 300 detects weight changes (changes in total weight and weight balance), and product acquisition events occur at positions P101, P102a, and P102b. Based on the weight changes (changes in total weight and weight balance), it is detected that products have been acquired simultaneously at positions P102a and P102b.

[0103] The product position determination unit (for shelves) 410 and the hand reaching position determination unit 420 of the product identification device 400 acquire the weight sensor information and TOF sensor information shown in Fig. 12B, as in Example 1. Note that information about multiple products picked up at the same time (product C and product A in this example) is summarized into information about one row number including two rows.

[0104] In Example 5, the weight sensor information includes information on row number 1 and information on row number 2, which includes two rows.

[0105] The X-axis position Xw of the information in row number 1 is 0.50, the estimated product to be picked up is B, and the likelihood Pw is 0.8. The X-axis position Xw of the information in one row of row number 2 is 0.20, the estimated product to be picked up is C, and the likelihood Pw is 0.7. The X-axis position Xw of the information in another row of row number 2 is 0.70, the estimated product to be picked up is A, and the likelihood Pw is 0.7.

[0106] The TOF sensor information includes information on row number X and row number Y. The item pick-up position Xt for the information on row number X is 0.25, and the estimated picked-up item is C. The information on row number X is information obtained based on the distance data (point cloud) of user X. The information on row number Y is information obtained based on the distance data (point cloud) of user Y.

[0107] The item identification device 400 applies the weight sensor information and the TOF sensor information to the calculation formula (1) to calculate the corrected likelihood information. Specifically, the product position determination unit (for integration) 430 of the product identification device 400 calculates the corrected likelihood Pnew for each estimated picked product using formula (1). The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for one row of row number 1 of the corrected likelihood information using formula (1) from the information on row number 1 of the weight sensor information and the information on row number X of the TOF sensor information. The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for the other rows of row number 1 of the corrected likelihood information using formula (1) from the information on row number 1 of the weight sensor information and row number Y of the TOF sensor information. Then, the product position determination unit (for integration) 430 selects the larger of the corrected likelihood Pnew in one row of row number 1 of the corrected likelihood information and the corrected likelihood Pnew in another row of row number 1 of the corrected likelihood information, and stores information indicating that the larger corrected likelihood Pnew was selected, "In this example, O", in the row corresponding to the selected corrected likelihood Pnew in Pnew selection 824a.

[0108] The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for the first row of row number 2 of the corrected likelihood information using formula (1) from the information in the first row of row number 2 of the weight sensor information and the information in row number X of the TOF sensor information. The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for the second row of row number 2 of the corrected likelihood information using formula (1) from the information in the first row of row number 2 of the weight sensor information and the information in row number Y of the TOF sensor information. Then, the product position determination unit (for integration) 430 selects the larger of the corrected likelihood for the first row of row number 2 of the corrected likelihood information and the corrected likelihood Pnew for the second row of row number 2 of the corrected likelihood information, and stores information indicating that the larger corrected likelihood Pnew has been selected, "In this example, O", in the row corresponding to the selected corrected likelihood Pnew in Pnew selection 824a.

[0109] The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for the third row of row number 2 of the corrected likelihood information using formula (1) from the information in the other rows of row number 2 of the weight sensor information and the information in row number X of the TOF sensor information. The product position determination unit (for integration) 430 calculates the corrected likelihood Pnew for the fourth row of row number 2 of the corrected likelihood information using formula (1) from the information in the other rows of row number 2 of the weight sensor information and the information in row number Y of the TOF sensor information. Then, the product position determination unit (for integration) 430 selects the larger of the corrected likelihood Pnew for the third row of row number 2 of the corrected likelihood information and the corrected likelihood Pnew for the fourth row of row number 2 of the corrected likelihood information, and stores information indicating that the larger corrected likelihood has been selected, "In this example, O", in the row corresponding to the selected corrected likelihood Pnew in Pnew selection 824a.

[0110] Furthermore, the product position determination unit (for integration) 430 calculates the average Pnew' of the corrected likelihoods Pnew of the inferred product C and the inferred product A that are inferred to have been picked up at the same time. The product position determination unit (for integration) 430 calculates the average Pnew' of the corrected likelihoods Pnew of the inferred product C and the inferred product A by finding the average Pnew' of the selected corrected likelihoods Pnew, and stores it in the rows corresponding to the first to fourth rows of row number 2 of Pnew' 824a.

[0111] Furthermore, the product position determination unit (for integration) 430 calculates the sum Pnew_sum of the corrected likelihoods Pnew of the estimated product C and estimated product A that are presumed to have been picked up by user X at the same time or user Y at the same time. The product position determination unit (for integration) 430 calculates the sum Pnew_sum of the corrected likelihoods Pnew of the estimated product C and estimated product A that are presumed to have been picked up by user X at the same time by determining the sum Pnew_sum of the corrected likelihoods Pnew of the information in the first row of row number 2 and the corrected likelihoods Pnew of the information in the third row of row number 2, and stores this in Pnew_sum 826 in the fifth row of row number 2.

[0112] The product position determination unit (for integration) 430 calculates the sum Pnew_sum of the corrected likelihood Pnew of the information in the second row of row number 2 and the corrected likelihood Pnew of the information in the fourth row of row number 2, and simultaneously calculates the sum Pnew_sum of the corrected likelihoods Pnew of estimated product C and estimated product A that are estimated to have been picked up by user Y, and stores this in Pnew_sum 826 in the sixth row of row number 2.

[0113] In the fifth specific example, the calculated corrected likelihood information includes information on row number 1, which includes two rows, and information on row number 2, which includes six rows.

[0114] The estimated product picked up in the information in the first row of row number 1 is B, and the corrected likelihood Pnew is 0.3. The information in the first row of row number 1 corresponds to the information that user X picked up product B.

[0115] The estimated product picked up in the information in the other rows of row number 1 is B, and the corrected likelihood Pnew is 0.2. The information in the other rows of row number 1 corresponds to the information that user Y picked up product B.

[0116] The estimated product picked up in the information on the first line of line number 2 is C, the corrected likelihood is 0.6, and the average corrected likelihood Pnew' is 0.55. The information on the first line of line number 2 corresponds to the information that user X picked up product C at the same time as product A.

[0117] The estimated product picked up in the information on the second line of line number 2 is C, the corrected likelihood Pnew is -0.5, and the average corrected likelihood Pnew' is 0.55. The information on the second line of line number 2 corresponds to the information that user Y picked up product C at the same time as product A.

[0118] The estimated product picked up in the information on the third line of line number 2 is A, the corrected likelihood Pnew is -0.2, and the average corrected likelihood Pnew' is 0.55. The information on the third line of line number 2 corresponds to the information that user X picked up product A and product C at the same time.

[0119] The estimated product picked up in the information on the fourth line of row number 2 is A, the corrected likelihood Pnew is 0.5, and the average corrected likelihood Pnew' is 0.55. The information on the fourth line of row number 2 corresponds to the information that user Y picked up product A and product C at the same time.

[0120] The estimated products picked up in the information on the fifth row of row number 2 are C and A, and the sum Pnew_sum of the corrected likelihoods Pnew is 0.4 (= 0.6 + (-0.2)). The information on the fifth row of row number 2 corresponds to the information that user X picked up product C and product A.

[0121] The estimated products picked up in the information on the sixth row of row number 2 are C and A, and the sum Pnew_sum of the corrected likelihoods Pnew is 0 (= 0.5 + (-0.5)). The information on the sixth row of row number 2 corresponds to the information that user X picked up product C and product A.

[0122] The product position determination unit (for integration) 430 determines (identifies) the estimated picked-up product having the highest likelihood among the corrected likelihood Pnew, the average corrected likelihood Pnew', and the sum of the corrected likelihoods from among the estimated picked-up products as the picked-up product from shelf SB1. In specific example 5, the average corrected likelihood Pnew' of estimated picked-up product A and estimated picked-up product C is the highest, so the product position determination unit (for integration) 430 determines product A and product C as the picked-up products. Furthermore, based on the row for which the corrected likelihood Pnew was selected, the product position determination unit (for integration) 430 determines product C as the picked-up product of user X and product A as the picked-up product of user Y.

[0123] <Specific operation> 13 is a flowchart showing the processing flow executed by the item identification system. The system starts processing from S1300, and executes the processing of S1305 and S1310 and the processing of S1315 and S1320 described below in parallel.

[0124] S1305: The weight sensor 300 detects a change in weight.

[0125] S1310: The product position determination unit (for shelves) 410 of the product identification device 400 calculates several candidates for the position of the product where it is estimated that the product was picked up (i.e., the product acquisition event occurrence position (X-direction position Xw)) based on the measurement values ​​of the weight sensors 300 of each weight sensor 300 and the product management information 400. That is, the product position determination unit (for shelves) 410 calculates the weight sensor information described above.

[0126] S1315: The distance measurement sensor 200a and the distance measurement sensor 200b detect the hand outstretched position.

[0127] S1320: The hand reaching position determination unit 420 of the item identification device 400 calculates several candidates for the estimated product pick-up position (i.e., item pick-up position Xt) where the product is estimated to have been picked up, based on the detected position information and the product management information 400. That is, the hand reaching position determination unit 420 calculates the above-mentioned TOF sensor information.

[0128] Thereafter, the item identification system executes S1325 described below, and then proceeds to S1395 to temporarily end this processing flow.

[0129] S1325: The commodity position determination unit (for integration) 430 of the commodity identification device 400 calculates corrected likelihood information from the two pieces of sensor information (weight sensor information and TOF sensor information), and determines (identifies) the extracted commodity based on the corrected likelihood information. Details of the process of S1325 will be described later.

[0130] <s1325> The details of the processing of S1325 mentioned above will now be described. Fig. 14 is a flowchart showing the processing flow executed by the product position determination unit (for integration) 430 of the product identification device 400. The product position determination unit (for integration) 430 starts processing from step 1400 and proceeds to step 1405, where it determines whether or not there is an estimated picked-up product that was simultaneously acquired in the weight sensor information.

[0131] If there is no estimated product to be picked up that was simultaneously acquired in the weight sensor information, the product position determination unit (for integration) 430 judges "NO" in step 1405 and executes the processes of steps 1410 and 1415 described below in order, then proceeds to step 1495 and temporarily ends this processing flow.

[0132] Step 1410: The product position determination unit (for integration) 430 calculates the corrected likelihood information using the calculation method described above with reference to FIG. 8B (FIG. 9B).

[0133] Step 1415: The commodity position determination unit (for integration) 430 specifies the estimated picked commodity having the highest corrected likelihood Pnew as the picked commodity based on the corrected likelihood information.

[0134] If there is an estimated picked-up product that was acquired simultaneously in the weight sensor information, the product position determination unit (for integration) 430 judges "YES" in step 1405 and proceeds to step 1420, where it determines whether the number of users is one or not based on the TOF sensor information.

[0135] If there is one user, the product position determination unit (for integration) 430 judges "YES" in step 1420 and proceeds to step 1425 to determine whether the TOF sensor information contains only one hand reaching detection position (item pick-up position Xt).

[0136] If the TOF sensor information contains only one hand reaching detection position (item pick-up position Xt), the product position determination unit (for integration) 430 judges "YES" in step 1425 and executes the processes of steps 1430 and 1435 described below in order, then proceeds to step 1495 and temporarily ends this processing flow.

[0137] Step 1430: The product position determination unit (for integration) 430 calculates the corrected likelihood information by the calculation method described above with reference to FIG. 10B.

[0138] Step 1435: The product position determination unit (for integration) 430 identifies the picked product based on the corrected likelihood Pnew and the average likelihood, as described above.

[0139] If the TOF sensor information contains two or more hand reaching detection positions (item pick-up positions Xt), the product position determination unit (for integration) 430 judges "NO" in step 1425 and executes the processes of steps 1440 and 1445 described below in order, then proceeds to step 1495 and temporarily ends this processing flow.

[0140] Step 1440: The product position determination unit (for integration) 430 calculates the corrected likelihood information using the calculation method described above with reference to FIG. 11B.

[0141] Step 1445: As described above, based on the corrected likelihood information, the product position determination unit (for integration) 430 identifies the predicted product to be picked as the product to be picked, which has the highest likelihood among the corrected likelihood Pnew, the average corrected likelihood Pnew', and the sum of the corrected likelihoods Pnew_sum.

[0142] In step 1420 described above, if the number of users is two or more, the product position determination unit (for integration) 430 judges "NO" in step 1420 and executes the processing of steps 1450 and 1455 described below in order, then proceeds to step 1495 and temporarily ends this processing flow.

[0143] Step 1450: The product position determination unit (for integration) 430 calculates the corrected likelihood information using the calculation method described above with reference to FIG. 12B.

[0144] Step 1455: As described above, the product position determination unit (for integration) 430 identifies, as the picked product, the estimated picked product having the highest likelihood among the corrected likelihood Pnew, the average likelihood, and the sum Pnew_sum of the corrected likelihood Pnew based on the corrected likelihood information. Furthermore, the product position determination unit (for integration) 430 identifies the user of the identified picked product.

[0145] <Effects> As described above, the item identification system according to the first embodiment of the present invention calculates a corrected likelihood Pnew by correcting the likelihood Pw of an estimated picked-up item based on the measurement value of weight sensor 300 using the X-direction position Xw based on the measurement value of weight sensor 300 and the picked-up position Xt detected by distance measurement sensors 200a and 200b. The item identification system determines (identifies) the picked-up item based on the corrected likelihood Pnew. This allows the item identification system to accurately identify items that were picked up simultaneously from different positions on the same shelf SB.

[0146] <<Variation 1>> In the above first embodiment, the item identification device 400 may weight each likelihood Pw of the weight sensor information by multiplying it by a weighting coefficient, and calculate the corrected likelihood Pnew using the likelihood after weighting (by substituting it into calculation formula (1)).

[0147] <<Variation 2>> In the above first embodiment, if the highest likelihood Pw among the likelihoods Pw of the weight sensor information is greater than or equal to a predetermined threshold likelihood, the item identification device 400 may determine (identify) the item with the highest likelihood Pw as the item picked up from the shelf SB without taking into account the corrected likelihood Pnew.

[0148] <<Variation 3>> In the first embodiment, if there is an estimated taken product for which the occurrence position of a product acquisition event could not be obtained, the product identification device 400 may consider the X-axis position Xw of the location where the product acquisition event occurred for that estimated taken product to be a predetermined position based on the location where that product was placed. For example, if the estimated taken product for which the occurrence position of a product acquisition event could not be obtained is product C, product C is located in the X-axis position range of 0 to less than 0.3, so based on this range, the X-axis position Xw of the location where the product acquisition event occurred for that estimated taken product may be calculated to be 0.15.

[0149] <<Variation 4>> In the above first embodiment, if the item identification device 400 is unable to obtain the item pick-up position Xt because the distance measurement sensor 200a and the distance measurement sensor 200b are unable to detect the hand-reaching position, it may estimate the item picked up from the shelf SB based only on the likelihood Pw of the weight sensor information without calculating the corrected likelihood Pnew.

[0150] <<Variation 5>> In the first embodiment, when the distance measurement sensor 200a and the distance measurement sensor 200b cannot detect the outstretched hand position but can detect the position where the head of the user US1 appears on the virtual screen, the item identification device 400 may use the detected head position as the item pick-up position Xt to calculate the corrected likelihood Pnew. In this case, the item identification device 400 may calculate the corrected likelihood Pnew by using a smaller weighting coefficient in the calculation formula (1) than usual (for example, smaller than when the outstretched hand position can be detected).

[0151] <<Variation 6>> In the first embodiment, the item identification device 400 may determine the value of the weighting coefficient in formula (1) by machine learning. For example, an optimal weighting coefficient may be determined by machine learning using a large amount of weight sensor information and TOF sensor information (learning data) when the correct answer (correct picked item) is known.

[0152] <<Variation 7>> In the above first embodiment, the item identification device 400 may set the weighting coefficient in calculation formula (1) to a larger value depending on the degree to which the results of the distance measurement sensors 200a and 200b are to be given importance when identifying the picked-up product.

[0153] <<Second embodiment>> An item identification system according to a second embodiment of the present invention will now be described. The item identification system according to the second embodiment of the present invention differs from the item identification system according to the first embodiment only in the following points. The item identification device 400 calculates the range in which the item can be picked up in the X direction based on the item pick-up position Xt detected using the distance measurement sensor 200a and the distance measurement sensor 200b. The item identification device 400 identifies the item to be picked up based on the range in which the item can be picked up and the likelihood Pw of the weight sensor information.

[0154] The following description will focus on this difference.

[0155] FIG. 15 is a diagram illustrating the operation of the item identification device 400. As shown in FIG. 15, when the item identification device 400 acquires a position P111 as the item pickup position Xt, it sets a possible item pickup range R111 based on the position P111. Based on the possible item pickup range R111, the item identification device 400 excludes from the candidate items for pickup those items that are not likely to be picked up. For example, in this example, while items A and B have the potential to have been picked up, item C has no potential to have been picked up, so item C is excluded from the candidate items for pickup. Based on the likelihood Pw of each estimated item to have been picked up from the weight sensor information, the item identification device 400 determines the item with the highest likelihood Pw as the item picked up from shelf SB1.

[0156] <Effects> As described above, the article identification system according to the second embodiment of the present invention, like the first embodiment, can accurately identify products that are simultaneously picked up from different positions on the same shelf SB.

[0157] <<Third Embodiment>> An item identification system according to a third embodiment of the present invention will be described. The item identification system according to the third embodiment of the present invention differs from the item identification system according to the first embodiment only in the following points. The item identification device 400 calculates the range in which an item can be picked up in the X direction based on the item pickup position Xt detected using the distance measurement sensor 200a and the distance measurement sensor 200b and the X direction position Xw of the product acquisition event occurrence position detected using the weight sensor 300. The item identification device 400 identifies the item to be picked up based on the range in which an item can be picked up and the likelihood Pw of the weight sensor information.

[0158] The following description will focus on this difference.

[0159] Figure 16 is a diagram for explaining the operation of the item identification device 400. As shown in Figure 16, when the item identification device 400 acquires position P211 as the item pickup position Xt, it sets a first item pickup possibility candidate range R211 based on position P211. When the item identification device 400 acquires positions P311, P312, and P313 as the X-direction positions Xw of the product acquisition event occurrence position, it sets a second item pickup possibility candidate range R311 based on position P311, sets a second item pickup possibility candidate range R312 based on position P312, and sets a second item pickup possibility candidate range R313 based on position P313.

[0160] The item identification device 400 sets the range in which the first item pick-up possibility candidate range R211 in the X direction and the second item pick-up possibility candidate ranges R311 to R313 overlap in the Y direction as the item pick-up possibility range R400.

[0161] Based on the item pick-up possibility range R400, the item identification device 400 excludes items within a range that are unlikely to be picked up from the item pick-up candidate. For example, in this example, item A may have been picked up, while items B and C are unlikely to be picked up, so items B and C are excluded from the item pick-up candidate. The item identification device 400 determines item A, from which items B and C have been excluded, as the item to be picked up from shelf SB1. Note that if there are multiple items remaining after the items have been removed, the item with the highest likelihood Pw is identified as the item to be picked up from shelf SB1 based on the likelihood Pw of each estimated picked-up item from the weight sensor information.

[0162] <Effects> As described above, the item identification system according to the third embodiment of the present invention can accurately identify items that are picked up simultaneously from different positions on the same shelf SB, similar to the first embodiment.

[0163] <<Other variations>> The present invention is not limited to the above-described embodiments and modifications, and various modifications can be adopted within the scope of the present invention. Furthermore, the above-described embodiments and modifications can be combined with each other without departing from the scope of the present invention.

[0164] In each of the above embodiments and modifications, when calculating the likelihood Pw based on the measurement value measured by the weight sensor 300, the purchase rate of the product and whether the product is on sale or discounted may be taken into consideration. In each of the above embodiments and modifications, when calculating the likelihood Pw based on the measurement value detected by the weight sensor 300, an assumption that the target product has the same or similar attributes as other products placed close to each other may be taken into consideration. Furthermore, in the above embodiments, when calculating the likelihood Pw, a history of past likelihoods Pw and their correct / incorrect results may be taken into consideration. Furthermore, in the above embodiments, when calculating the likelihood Pw based on the weight change of the shelf SB detected by the weight sensor 300, the likelihood Pw may be calculated using techniques such as machine learning, linear regression, logistic regression, decision tree analysis, and support vector machine.

[0165] The present invention can also have the following configuration.

[0166] [1] a weight sensor installed on a shelf on which an item is placed to measure the weight of the item; a distance measuring sensor that includes the shelf and a predetermined area in front of the shelf in a measurement range and measures a distance to a measurement object that exists in the measurement range; an item identification device including an information processing device that acquires weight measurement data, which is the weight measured by the weight sensor, and acquires distance data, which is the distance measured by the distance measuring sensor, representing a person present within the measurement range; An article identification method using By the information processing device, Based on the weight measurement data when the item is picked up from the shelf, an item picked up from the shelf is estimated, and for each of the estimated items, a first item pick-up position which is an estimated pick-up position of the estimated picked-up item and a degree of accuracy indicating the degree of accuracy of the estimation of the estimated picked-up item are calculated; acquiring a second item pick-up position based on the position of a predetermined part of the person when the person picks up the item from the shelf based on the distance data of the person; For each of the plurality of estimated picked-up items, a corrected accuracy is calculated by correcting the probability based on the first item picked-up position and the second item picked-up position, and based on the corrected accuracy, the picked-up item that is the one or more items picked up from the shelf is identified from the plurality of estimated picked-up items. Article identification method.

[0167] [2] On the computer, Weight measurement data is the weight measured by a weight sensor that is installed on a shelf on which the item is placed and that is acquired from the weight sensor that measures the weight of the item; and an item identification program that executes processing using distance data that is a distance that represents a person present within a measurement range measured by a distance measuring sensor, the distance data being acquired from a distance measuring sensor that measures a distance to a measurement object present within the measurement range, the program including the shelf and a predetermined area in front of the shelf, The computer, Based on the weight measurement data when the item is picked up from the shelf, an item picked up from the shelf is estimated, and for each of the estimated items, a first item pick-up position which is an estimated pick-up position of the estimated picked-up item and a degree of accuracy indicating the degree of accuracy of the estimation of the estimated picked-up item are calculated; acquiring a second item pick-up position based on the position of a predetermined part of the person when the person picks up the item from the shelf based on the distance data of the person; For each of the plurality of estimated picked-up items, a corrected accuracy is calculated by correcting the probability based on the first item picked-up position and the second item picked-up position, and a process is executed to identify picked-up items that are one or more of the items picked up from the shelf from among the plurality of estimated picked-up items based on the corrected accuracy. Item Identification Program.

[0168] The following items [3] to [8] of the item acquisition determination system will be described with reference to FIG.

[0169] [3] a plurality of weight sensors provided spaced apart from one another on a shelf capable of placing a plurality of articles thereon, the weight sensors detecting weights of the articles; a first detection unit that detects information including a weight distribution on the shelf on which the item is placed, using the detection result of the weight sensor; a first determination unit that determines the validity of the item candidates acquired from the shelf using information detected by the first detection unit, the first determination unit determining the validity of a plurality of the item candidates; a distance measuring sensor that detects the distance to an article or person near the shelf; a second detection unit that detects an article, a person, and a movement of the person near the shelf using a detection result of the distance measurement sensor; a second determination unit that determines which person has made a motion toward which item using the detection result of the second detection unit; a third determination unit that determines which person has acquired which item using the determination results of the first determination unit and the determination results of the second determination unit; Equipped with In a first situation (see row Rw1 in FIG. 17 ), a first item (B in FIG. 17 ) is placed in a first area in the approximate center of the shelf in one direction of the shelf, and second and third areas sandwiching the first area on both sides in the one direction, a second item (A in FIG. 17 ) is placed in the second area, and a third item (C in FIG. 17 ) is placed in the third area, As shown in row Rw1, the first determination unit determines that the plurality of candidate items include one second item (A) and one third item (C), and that the most appropriate candidate item is one second item (A) and the next most appropriate candidate item is one third item (C), and, When the second determination unit determines that the first person (X in FIG. 17) has made a motion toward the third item (C), the third determination unit determines that the first person (X) has acquired the third item (C). An item acquisition determination system.

[0170] [4] a plurality of weight sensors provided spaced apart from one another on a shelf capable of placing a plurality of articles thereon, the weight sensors detecting weights of the articles; a first detection unit that detects information including a weight distribution on the shelf on which the item is placed, using the detection result of the weight sensor; a first determination unit that determines the validity of the item candidates acquired from the shelf using information detected by the first detection unit, the first determination unit determining the validity of a plurality of the item candidates; a distance measuring sensor that detects the distance to an article or person near the shelf; a second detection unit that detects an article, a person, and a movement of the person near the shelf using a detection result of the distance measurement sensor; a second determination unit that determines which person has made a motion toward which item using the detection result of the second detection unit; a third determination unit that determines which person has acquired which item using the determination results of the first determination unit and the determination results of the second determination unit; Equipped with In a second situation (see row Rw2 in Figure 17), in which a first item (B) is placed in a first area in approximately the center of the shelf in one direction of the shelf, and second and third areas sandwiching the first area on both sides in the one direction, a second item is placed in the second area, and a third item (C) is placed in the third area, the weights of the second item and the third item (C) are approximately equal, and the weight of the first item (B) is approximately twice the weight of the second item, As shown in row Rw2, the first determination unit determines that the multiple candidate items are one of the first items (B), one of the second items (A), and one of the third items (C), and that the most appropriate candidate item is one of the first items (B), and the next most appropriate candidate items are one of the second items (A) and one of the third items (C), and, When the second determination unit determines that the first person (X) has made a motion toward the second object (A) and the second person (Y in FIG. 17) has made a motion toward the third object (C), the third determination unit determines that the first person (X) has acquired the second item (A) and the second person (Y) has acquired the third item (C); An item acquisition determination system.

[0171] Note that "approximately equal" includes "same," and "approximately twice" includes "twice."

[0172] [5] a plurality of weight sensors provided spaced apart from one another on a shelf capable of placing a plurality of articles thereon, the weight sensors detecting weights of the articles; a first detection unit that detects information including a weight distribution on the shelf on which the item is placed, using the detection result of the weight sensor; a first determination unit that determines the validity of the item candidates acquired from the shelf using information detected by the first detection unit, the first determination unit determining the validity of a plurality of the item candidates; a distance measuring sensor that detects the distance to an article or person near the shelf; a second detection unit that detects an article, a person, and a movement of the person near the shelf using a detection result of the distance measurement sensor; a second determination unit that determines which person has made a motion toward which item using the detection result of the second detection unit; a third determination unit that determines which person has acquired which item using the determination results of the first determination unit and the determination results of the second determination unit; Equipped with In a second situation (see row Rw3 in Figure 17), in which a first item (B) is placed in a first area in approximately the center of the shelf in one direction of the shelf, and second and third areas sandwiching the first area on both sides in the one direction, a second item (A) is placed in the second area, and a third item (C) is placed in the third area, the weights of the second item (A) and the third item (C) are approximately equal, and the weight of the first item (B) is approximately twice the weight of the second item (A), As shown in row Rw3, the first judgment unit judges that the plurality of candidate items are one of the first items (B), one of the second items (A), and one of the third items (C), and that the most appropriate candidate items are one of the second items (A) and one of the third items (C), and that the next most appropriate candidate item is one of the first items (B), and, When the second determination unit determines that the first person (X) has made a motion toward the first item (B), the third determination unit determines that the first person (X) has acquired the first item (B); An item acquisition determination system.

[0173] [6] a plurality of weight sensors provided spaced apart from one another on a shelf capable of placing a plurality of articles thereon, the weight sensors detecting weights of the articles; a first detection unit that detects information including a weight distribution on the shelf on which the item is placed, using the detection result of the weight sensor; a first determination unit that determines the validity of the item candidates acquired from the shelf using information detected by the first detection unit, the first determination unit determining the validity of a plurality of the item candidates; a distance measuring sensor that detects the distance to an article or person near the shelf; a second detection unit that detects an article, a person, and a movement of the person near the shelf using a detection result of the distance measurement sensor; a second determination unit that determines which person has made a motion toward which item using the detection result of the second detection unit; a third determination unit that determines which person has acquired which item using the determination results of the first determination unit and the determination results of the second determination unit; Equipped with In a second situation (see row Rw4 in Figure 17), in which a first item (B) is placed in a first area in approximately the center of the shelf in one direction of the shelf, and second and third areas sandwiching the first area on both sides in the one direction, a second item (A) is placed in the second area, and a third item (C) is placed in the third area, the weights of the second item (A) and the third item (C) are approximately equal, and the weight of the first item (B) is approximately twice the weight of the second item (A), As shown in row Rw4, the first judgment unit judges that the plurality of candidate items are one of the first items (B), one of the second items (A), and one of the third items (C), and that the most appropriate candidate item is one of the first items (B), and the next most appropriate candidate item is one of the second items (A) and one of the third items (C), and, When the second determination unit determines that the first person (X) has made a motion toward the second item (A) and the third item (C), the third determination unit determines that the first person (X) has acquired one of the second items (A) and one of the third items (C); An item acquisition determination system.

[0174] [7] a plurality of weight sensors provided spaced apart from one another on a shelf capable of placing a plurality of articles thereon, the weight sensors detecting weights of the articles; a first detection unit that detects information including a weight distribution on the shelf on which the item is placed, using the detection result of the weight sensor; a first determination unit that determines the validity of the item candidates acquired from the shelf using information detected by the first detection unit, the first determination unit determining the validity of a plurality of the item candidates; a distance measuring sensor that detects the distance to an article or person near the shelf; a second detection unit that detects an article, a person, and a movement of the person near the shelf using a detection result of the distance measurement sensor; a second determination unit that determines which person has made a motion toward which item using the detection result of the second detection unit; a third determination unit that determines which person has acquired which item using the determination results of the first determination unit and the determination results of the second determination unit; Equipped with As shown in FIG. 17 , when the item determined to have the highest validity by the first determination unit does not match the item determined by the second determination unit, and the item determined to have the second highest validity by the first determination unit matches the item determined by the second determination unit, the third determination unit determines that the item determined by the second determination unit has been acquired. An item acquisition determination system.

[0175] In the item acquisition determination systems [3] to [7], the first determination unit, the second determination unit, and the third determination unit may be configured by various programs stored in the ROM 2002 and / or the storage device 2004 and executed by the CPU 2001 of the item identification device 400. In the item acquisition determination systems [3] to [7], the validity or validity may be determined by the above-mentioned accuracy. [Explanation of symbols]

[0176] 200a... distance measurement sensor, 300... weight sensor, 400... item identification device, 410... product position determination unit (for shelves), 420... hand reach position determination unit, 430... product position determination unit (for integration)

Claims

1. a weight sensor installed on a shelf on which an item is placed to measure the weight of the item; a distance measuring sensor that includes the shelf and a predetermined area in front of the shelf in a measurement range and measures a distance to a measurement object that exists in the measurement range; an item identification device including an information processing device that acquires weight measurement data, which is the weight measured by the weight sensor, and acquires distance data, which is the distance measured by the distance measuring sensor, representing a person present within the measurement range; An item identification system comprising: The information processing device includes: Based on the weight measurement data when the item is picked up from the shelf, an item picked up from the shelf is estimated, and for each of the estimated items, a first item pick-up position which is an estimated pick-up position of the estimated picked-up item and a degree of accuracy indicating the degree of accuracy of the estimation of the estimated picked-up item are calculated; acquiring a second item pick-up position based on the position of a predetermined part of the person when the person picks up the item from the shelf based on the distance data of the person; calculating a corrected accuracy by correcting the probability based on the first item pick-up position and the second item pick-up position for each of the plurality of estimated pick-up items; If there are no simultaneously picked up estimated items among the plurality of picked up estimated items, the one or more picked up items are identified from the plurality of picked up estimated items based on the correction accuracy, If there are multiple presumed picked-up items that were picked up at the same time among the multiple presumed picked-up items, it is determined whether there is only one person present within the measurement range; When there is only one person present within the measurement range, if only one second item pick-up position is acquired, Further, an average of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Based on the correction accuracy of the estimated picked-up item that is not the estimated picked-up item at the same time and the average of the correction accuracy, Identifying the estimated picked item or the estimated picked item that is not a simultaneously picked item with the highest probability as the picked item; When there is one person present within the measurement range, if two second item pick-up positions are acquired, Further, an average of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Further, a sum of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Based on the correction accuracy of the estimated picked-up items that are not the estimated picked-up items at the same time, the average of the correction accuracy, and the sum of the correction accuracy, If the correction accuracy is the highest, the estimated picked item that is not an estimated simultaneously picked item and has the highest probability is identified as the picked item; If the average of the correction accuracy is the highest or the sum of the correction accuracy is the highest, the simultaneously picked up estimated item is identified as the picked up item; When there is more than one person present within the measurement range, if two second article pick-up positions based on the predetermined parts of two different people are acquired, Further, an average of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Further, a sum of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Based on the correction accuracy of the estimated picked-up items that are not the estimated picked-up items at the same time, the average of the correction accuracy, and the sum of the correction accuracy, If the correction accuracy is the highest, the estimated picked item that has the highest probability and is not an estimated simultaneously picked item is identified as the picked item, and further, the person who picked up the identified picked item is identified; If the average of the correction accuracy rates is the highest, or if the sum of the correction accuracy rates is the highest, the simultaneously picked up estimated items are identified as the picked up items, and further, the person who picked up the identified picked up items is identified. It was configured as follows: Item identification system.

2. 2. The item identification system according to claim 1, The information processing device includes: correcting the accuracy by subtracting from the accuracy a value obtained by multiplying an absolute value of a difference between the first item pick-up position and the second item pick-up position by a weighting coefficient; It was configured as follows: Item identification system.

3. 2. The item identification system according to claim 1, The information processing device includes: weighting the likelihood before calculating the corrected accuracy; It was configured as follows: Item identification system.

4. 2. The item identification system according to claim 1, The information processing device includes: If the certainty is equal to or greater than a predetermined threshold certainty, the estimated picked item having the certainty equal to or greater than the threshold certainty is identified as the picked item without calculating the correction certainty. It was configured as follows: Item identification system.

5. 2. The item identification system according to claim 1, The information processing device includes: acquiring, as the position of the predetermined part of the person, the position of the person's hand when the person picks up the item from the shelf; It was configured as follows: Item identification system.

6. 6. The item identification system according to claim 5, The information processing device includes: If the position of the person's hand cannot be acquired, and if a predetermined position other than the position of the person's hand can be acquired as the position of the predetermined part of the person, the predetermined position when the person picks up the item from the shelf is acquired as the second item pick-up position. It was configured as follows: Item identification system.

7. 2. The item identification system according to claim 1, The information processing device includes: The first item pick-up location can be obtained; If the second item pickup position cannot be acquired, the picked-up item is identified based on the first item pickup position. It was configured as follows: Item identification system.

8. 2. The item identification system according to claim 1, The information processing device includes: setting a predetermined range of possibility of picking up an item based on the second picking up position; Excluding the items that are estimated to be picked up that exist outside the range of possible pick-up of the items from the identified items to be picked up. It was configured as follows: Item identification system.

9. 2. The item identification system according to claim 1, The information processing device includes: setting a predetermined first item pick-up possibility candidate range based on the first item pick-up position; setting a predetermined second item pick-up possibility candidate range based on the second item pick-up position; A range based on the first item pick-up possibility candidate range and the second item pick-up possibility candidate range is set as an item pick-up possibility range; Excluding the items that are estimated to be picked up that exist outside the range of possible pick-up of the items from the identified items to be picked up. It was configured as follows: Item identification system.

10. a weight sensor installed on a shelf on which an item is placed to measure the weight of the item; a distance measuring sensor that includes the shelf and a predetermined area in front of the shelf in a measurement range and measures a distance to a measurement object that exists in the measurement range; an item identification device including an information processing device that acquires weight measurement data, which is the weight measured by the weight sensor, and acquires distance data, which is the distance measured by the distance measuring sensor, representing a person present within the measurement range; An article identification method using By the information processing device, Based on the weight measurement data when the item is picked up from the shelf, an item picked up from the shelf is estimated, and for each of the estimated items, a first item pick-up position which is an estimated pick-up position of the estimated picked-up item and a degree of accuracy indicating the degree of accuracy of the estimation of the estimated picked-up item are calculated; acquiring a second item pick-up position based on the position of a predetermined part of the person when the person picks up the item from the shelf based on the distance data of the person; For each of the plurality of estimated picked-up items, a correction accuracy is calculated by correcting the probability based on the first item picked-up position and the second item picked-up position, and if there are no estimated simultaneous picked-up items, which are multiple estimated picked-up items that were picked up simultaneously, a picked-up item that is one or more of the items picked up from the shelf is identified from the plurality of estimated picked-up items based on the correction accuracy; If there are multiple presumed picked-up items that were picked up at the same time among the multiple presumed picked-up items, it is determined whether there is only one person present within the measurement range; When there is only one person present within the measurement range, if only one second item pick-up position is acquired, Further, an average of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Based on the correction accuracy of the estimated picked-up item that is not the estimated picked-up item at the same time and the average of the correction accuracy, Identifying the estimated picked item or the estimated picked item that is not a simultaneously picked item with the highest probability as the picked item; When there is one person present within the measurement range, if two second item pick-up positions are acquired, Further, an average of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Further, a sum of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Based on the correction accuracy of the estimated picked-up items that are not the estimated picked-up items at the same time, the average of the correction accuracy, and the sum of the correction accuracy, If the correction accuracy is the highest, the estimated picked item that is not an estimated simultaneously picked item and has the highest probability is identified as the picked item; If the average of the correction accuracy is the highest or the sum of the correction accuracy is the highest, the simultaneously picked up estimated item is identified as the picked up item; When there is more than one person present within the measurement range, if two second article pick-up positions based on the predetermined parts of two different people are acquired, Further, an average of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Further, a sum of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Based on the correction accuracy of the estimated picked-up items that are not the estimated picked-up items at the same time, the average of the correction accuracy, and the sum of the correction accuracy, If the correction accuracy is the highest, the estimated picked item that has the highest probability and is not an estimated simultaneously picked item is identified as the picked item, and further, the person who picked up the identified picked item is identified; If the average of the correction accuracy rates is the highest, or if the sum of the correction accuracy rates is the highest, the simultaneously picked up estimated items are identified as the picked up items, and further, the person who picked up the identified picked up items is identified. Article identification method.

11. On the computer, Weight measurement data is the weight measured by a weight sensor that is installed on a shelf on which the item is placed and that is acquired from the weight sensor that measures the weight of the item; and an item identification program that executes processing using distance data that is a distance that represents a person present within a measurement range measured by a distance measuring sensor, the distance data being acquired from a distance measuring sensor that measures a distance to a measurement object present within the measurement range, the program including the shelf and a predetermined area in front of the shelf, The computer, Based on the weight measurement data when the item is picked up from the shelf, an item picked up from the shelf is estimated, and for each of the estimated items, a first item pick-up position which is an estimated pick-up position of the estimated picked-up item and a degree of accuracy indicating the degree of accuracy of the estimation of the estimated picked-up item are calculated; acquiring a second item pick-up position based on the position of a predetermined part of the person when the person picks up the item from the shelf based on the distance data of the person; For each of the plurality of estimated picked-up items, a correction accuracy is calculated by correcting the probability based on the first item picked-up position and the second item picked-up position, and if there are no estimated simultaneous picked-up items, which are multiple estimated picked-up items that were picked up simultaneously, a picked-up item that is one or more of the items picked up from the shelf is identified from the plurality of estimated picked-up items based on the correction accuracy; If there are multiple presumed picked-up items that were picked up at the same time among the multiple presumed picked-up items, it is determined whether there is only one person present within the measurement range; When there is only one person present within the measurement range, if only one second item pick-up position is acquired, Further, an average of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Based on the correction accuracy of the estimated picked-up item that is not the estimated picked-up item at the same time and the average of the correction accuracy, Identifying the estimated picked item or the estimated picked item that is not a simultaneously picked item with the highest probability as the picked item; When there is one person present within the measurement range, if two second item pick-up positions are acquired, Further, an average of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Further, a sum of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Based on the correction accuracy of the estimated picked-up items that are not the estimated picked-up items at the same time, the average of the correction accuracy, and the sum of the correction accuracy, If the correction accuracy is the highest, the estimated picked item that is not an estimated simultaneously picked item and has the highest probability is identified as the picked item; If the average of the correction accuracy is the highest or the sum of the correction accuracy is the highest, the simultaneously picked up estimated item is identified as the picked up item; When there is more than one person present within the measurement range, if two second article pick-up positions based on the predetermined parts of two different people are acquired, Further, an average of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Further, a sum of the correction accuracy of the plurality of estimated picked-up items constituting the estimated picked-up items at the same time is calculated; Based on the correction accuracy of the estimated picked-up items that are not the estimated picked-up items at the same time, the average of the correction accuracy, and the sum of the correction accuracy, If the correction accuracy is the highest, the estimated picked item that has the highest probability and is not an estimated simultaneously picked item is identified as the picked item, and further, the person who picked up the identified picked item is identified; If the average of the correction accuracy rates is the highest, or if the sum of the correction accuracy rates is the highest, the simultaneously picked up estimated items are identified as the picked up items, and further a process is executed to identify the person who picked up the identified picked up items. Item Identification Program.

Images