Management server and management method for managing products in unmanned stores

Abstract

To facilitate previous learning processing and the actual application by minimizing an application object when applying a skeleton detection model to an image captured by a camera and by obtaining an estimation point of the hand by calculation.SOLUTION: A management system for managing commodity products comprises: means which acquires weight information transmitted from a weight sensor installed in a commodity shelf; means which acquires position information of a customer from an image captured by a first camera sensor; means which acquires a feature point of a human body of the customer by applying a skeleton detection model from an image captured by a second camera sensor and calculates an estimation point of the hand by using the feature point of the human body; and means which determines pick-up of a commodity product on the basis of the position information of the customer, the interrelation between the calculated estimation point of the hand and the position of the commodity shelf and the change in output of the weight sensor.SELECTED DRAWING: Figure 38

Description

[Technical field]

[0001] The present invention relates to a management server and a management method for managing products in an unmanned store. [Background technology]

[0002] Background art of this technical field is JP 2020-173815 A (Patent Document 1). This publication states that "In an autonomous store tracking system, products 111, 112, 113, and 114 are arranged in a product storage area 102 of a store 101. Cameras 121 and 122 monitor the product storage area 102 and one or more areas of the store 101 through which people move. A person 103 is monitored standing near the product storage area 102 at time 141" (see Abstract). Furthermore, WO 2015 / 173869 (Patent Document 2) is a related background technique. This publication states that "when the detection unit (52) detects a human hand within a first area set according to the position of the product, and then detects no human hand within the same area as the first area or a second area including the first area, and no product is detected within the product detection area set according to the position, the detection unit (52) detects that the product has been picked up" (see abstract). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-173815 A [Patent Document 2] International Publication No. 2015 / 173869 Summary of the Invention [Problem to be solved by the invention]

[0004] In the prior art, in order to monitor the behavior of customers who purchase products in unmanned stores, image analysis is performed on images captured by cameras installed in the store. For example, in the disclosures of Patent Documents 1 and 2, image analysis based on a human skeletal model is performed on the captured images to detect multiple feature points from the entire body of the customer (for example, see Figures 6A-E, 7A-B, 11, and 14 of Patent Document 1, and Figure 3 of Patent Document 2, etc.). However, in these patent documents, it is not possible to perform highly accurate product pick-up determination by only using the calculated feature points. [Means for solving the problem]

[0005] In order to solve the above problem, for example, the configuration described in the claims is adopted. The present application includes a plurality of means for solving the above problem, and an example of the system is a management system for managing products, characterized by having a means for acquiring weight information transmitted from a weight sensor installed on a product shelf, a means for acquiring customer position information from an image captured by a first camera sensor, a means for acquiring body feature points of the customer by applying a skeleton detection model from an image captured by a second camera sensor and calculating an estimated hand point using the body feature points, and a means for determining picking up a product based on the customer position information, the correlation between the calculated estimated hand point and the product shelf position, and a change in the output of the weight sensor. Effect of the Invention

[0006] According to the present invention, by applying a skeleton detection model to an image captured by a camera and calculating estimated hand points, it is possible to perform a more accurate product pick-up determination. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating an example of a configuration of an entire product management system 1. [Diagram 2] 2 is an example of a hardware configuration of a management server 101. [Diagram 3] 2 is an example of a hardware configuration of the mobile terminal 102. [Figure 4] 1 is a diagram illustrating an example of a hardware configuration of a signage terminal 103. [Diagram 5] 2 is an example of a hardware configuration of camera sensor units 104 and 104a. [Figure 6] 2 is an example of a hardware configuration of the weight sensor unit 105. [Figure 7] This is an example of a connection configuration for communicating with a microcomputer 710 connected to a weight sensor device via wireless connection. [Figure 8] This is an example of a connection configuration for communicating with a microcomputer 710 connected to a weight sensor device 608 via a wired connection. [Figure 9] 1 is a diagram showing an example of a hardware configuration of the entrance management terminal 106. [Figure 10] 2 is an example of a hardware configuration of a payment processing server 107. [Figure 11] FIG. 2 is a diagram illustrating an example of a functional configuration of a product management system 1. [Figure 12] FIG. 2 is a diagram illustrating an example of a functional configuration of a signage display. [Figure 13] 13 is an example of product master information 1300. [Figure 14] 14 is an example of user master information 1400. [Figure 15] 15 is an example of segment information 1500. [Figure 16] 16 is an example of acquisition determination range information 1600. [Figure 17] 17 is an example of location tracking information 1700. [Figure 18] 18 is an example of shelf event information 1800. [Figure 19] 19 is an example of quotation management information 1900. [Figure 20] 2 is an example of weight information 2000 transmitted from the microcomputer 710 to the data aggregation transmitter 630. [Figure 21]1 is a configuration example of a frame for installing a product shelf. [Figure 22] FIG. 2 is a diagram illustrating the configuration of a product shelf. [Diagram 23] FIG. 13 is a schematic diagram illustrating a product acquisition determination range. [Figure 24] 2 is an example of a weight information accumulation process flow 2400. [Diagram 25] 2 is an example of a location information accumulation process flow 2500. [Figure 26] 26 illustrates an example of an overall process flow 2600 for purchasing a product in a store. [Figure 27] 27 is an example of a store entry process flow 2700. [Figure 28] 2 is an example of a product evaluation process flow 2800. [Figure 29] 2 is an example of a customer determination process flow 2900. [Diagram 30] 3 is an example of an estimate display process flow 3000. [Diagram 31] 3 is an example of a payment processing flow 3100. [Diagram 32] 3 illustrates an example of a signage display process flow 3200. [Diagram 33] 3 is an example of a hardware reference model 3300. [Diagram 34] 13 is an example of an order management screen. [Diagram 35] 13 is an example of a product purchase screen. [Diagram 36] This is an example of the application of a product management system. [Figure 37] Illustrates skeletal features of the human body. [Figure 38] This is an example of detecting skeletal information of a customer from a captured image. [Figure 39] This is an example of detecting skeletal information of a customer from a captured image. [Diagram 40] This is an example of detecting skeletal information of a customer from a captured image. [Diagram 41] 13 is an example of a process for detecting skeletal information of a customer from a captured image. [Diagram 42] 4 is an example of a skeleton detection process flow 4200. [Diagram 43] This is an example of the inside of a store captured by two cameras. [Diagram 44] This is an example of detecting a customer's skeletal information from two captured images. [Diagram 45] This is an example of product pick-up determination. [Figure 46] 13 is an example of a criterion used to determine whether to pick up a product. [Figure 47] This is an example of product pick-up determination. [Figure 48] This is an example of product pick-up determination. [Figure 49] This is an example of product pick-up determination. [Figure 50] This is an example of product pick-up determination. [Figure 51] This is an example of product pick-up determination. [Figure 52] This is an example of product pick-up determination. [Figure 53] This is an example of a floor in a store. [Figure 54] This is an example of a store floor being divided into multiple areas. [Figure 55] This is an example of an actual photograph of a product shelf in a store. [Figure 56] This is an example of a store floor being divided into multiple areas. [Figure 57] An example is tracking customers in a store using multiple cameras. [Figure 58] 1 is an example of customer location tracking. [Figure 59] This is an example of extracting information about a customer's clothing. [Figure 60] 1 is an example of customer location tracking. [Figure 61] 1 is an example of customer location tracking. [Figure 62] 13 is an example of a process for detecting customer skeletal information, position information, and product weight data from a captured image. [Figure 63] This is an example of product pick-up determination. [Figure 64]This is an example of product pick-up determination. [Figure 65] 65 is an example of output from the flow in FIG. 64. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS EXAMPLES

[0008] Hereinafter, the embodiments will be described with reference to the drawings. This embodiment assumes a product management system 1 that manages products in an unmanned store. When a customer enters a store, the customer records their entry by holding a two-dimensional barcode or a contactless IC chip displayed on the customer's mobile device over a store entry management terminal. When the customer picks up a product displayed on a shelf in the store, information and the number of the product picked up are displayed on the customer's mobile device, and payment for the product is completed when the customer leaves the store. Although we envision providing a system that can be implemented even in unmanned stores, stores can also be staffed.

[0009] In addition, this embodiment installs cameras and product shelves in several product shelf installation frames with predetermined dimensions, and then automatically executes several pre-registered initial system settings, thereby reducing the labor, time, and cost involved in setting up each store when setting up an unmanned store.

[0010] FIG. 1 is a diagram showing an example of the configuration of an entire product management system 1. The product management system 1 comprises a signage terminal 103 installed in or around a product shelf installation frame 2100 in a store, a position detection camera sensor unit 104, a skeleton detection camera sensor unit 104a, a weight sensor unit 105, and a store entry management terminal 106, and is connected to a management server 101 and a payment processing server 107 via a network. This product management system 1 is accessed from a mobile terminal 102 via the network. The network may be wired or wireless, and each terminal can send and receive information via the network.

[0011] The management server 101 accumulates various events that occur in the store, analyzes them, and performs processes such as product determination, customer determination, and quotation display. The mobile terminal 102 is a mobile terminal such as a smartphone, tablet, or wearable device owned by a customer, and displays information about a product that the customer has picked up. The camera sensor unit 104 has a camera management terminal 530 to which a plurality of camera devices 506 are connected, and acquires the movement trajectory (for example, walking trajectory) of a customer in a store. The camera sensor unit 104a has a camera management terminal 530a connected to multiple camera devices 506a, and acquires the characteristic parts of the bone structure of customers in the store. Although the camera management terminal 530 and the camera management terminal 530a are separate terminals, a single terminal may have both functions.

[0012] The weight sensor unit 105 has a plurality of weight sensor devices 608 installed in a product basket on a product shelf, a microcomputer 710 and a data aggregation transmitter 630 , and detects a change in weight and transmits the weight information to the management server 101 . The entrance management terminal 106 reads the two-dimensional barcode or contactless IC chip displayed on the mobile terminal held by the customer, and identifies the customer's user information. The payment processing server 107 receives a payment processing request for the product acquired by the customer from the management server 101, and executes the payment processing using the payment method associated with the user information of the customer.

[0013] Each terminal of the product management system 1 and the management server 101 may be, for example, a portable terminal (mobile terminal) such as a smartphone, tablet, mobile phone, or personal digital assistant (PDA), or a wearable terminal such as glasses, wristwatch, or clothing. They may also be stationary or portable computers, or servers located on the cloud or network. In terms of function, they may be a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, or an MR (Mixed Reality) terminal. Or they may be a combination of a plurality of these terminals. For example, a combination of one smartphone and one wearable terminal may function logically as one terminal. They may also be other information processing terminals.

[0014] Each terminal and management server 101 of the product management system 1 includes a processor that executes an operating system, applications, programs, etc., a main storage device such as a RAM (Random Access Memory), an auxiliary storage device such as an IC card, a hard disk drive, an SSD (Solid State Drive), a flash memory, etc., a communication control unit such as a network card, a wireless communication module, a mobile communication module, etc., input devices such as a touch panel, a keyboard, a mouse, voice input, and input by motion detection by imaging of a camera unit, and an output device such as a monitor or display. Note that the output device may be a device or terminal that transmits information to be output to an external monitor, display, printer, device, etc.

[0015] The main memory stores various programs and applications (modules), and the processor executes these programs and applications to realize the various functional elements of the overall system. Each of these modules may be implemented in hardware by integration or the like. Each module may be an independent program or application, or may be implemented as a subprogram or function within a single integrated program or application. In this specification, each module is described as an entity (subject) that performs processing, but in reality, a processor that processes various programs, applications, etc. (modules) executes the processing.

[0016] Various databases (DB) are stored in the auxiliary storage device. A "database" is a functional element (storage unit) that stores a data set so that it can handle any data operation (e.g., extraction, addition, deletion, overwriting, etc.) from a processor or an external computer. The method of implementing the database is not limited, and may be, for example, a database management system, spreadsheet software, or text files such as XML and JSON. When implemented in a database management system, it may be a relational database (RDBMS) or a non-relational database (non-RDBMS).

[0017] FIG. 2 shows an example of the hardware configuration of the management server 101. The management server 101 is configured, for example, as a server placed on a cloud. Various modules (programs and applications) 210 are stored in the main memory device 201, and the various functional elements of the management server 101 are realized by the processor 203 executing these programs and applications. Each module and its processing will be described later.

[0018] The auxiliary storage device 202 includes a master DB 220 and a various information storage DB 221 . The master DB 220 stores user master information 1400, product master information 1300, and the like. The various information storage DB 221 stores information transmitted from the camera sensor unit 104, the camera sensor unit 104a, the weight sensor unit 105, and the entrance management terminal 106, or stores various information processed and altered within the management server 101. The information stored in each area will be described later.

[0019] FIG. 3 shows an example of the hardware configuration of the mobile terminal 102. The mobile terminal 102 is, for example, a smartphone, a tablet, or the like. The main memory device 301 stores programs and applications such as a product purchase management module 310, and the processor 303 executes these programs and applications to realize the various functional elements of the mobile terminal 102.

[0020] The product purchase management module 310 cooperates with the estimate display module 1106 and the mobile purchase management module 1108 of the management server 101 to display a product purchase screen and a purchase history. The auxiliary storage device 302 stores user management information 320 stored in the mobile terminal 102 .

[0021] FIG. 4 shows an example of the hardware configuration of signage terminal 103. The signage terminal 103 is configured with a terminal such as a smartphone, a tablet, a notebook PC, a desktop PC, a microcomputer, a single board computer, etc. As a single board computer, for example, a Raspberry Pi can be used. Programs and applications such as a signage information display module 410 are stored in main memory device 401, and the respective functional elements of signage terminal 103 are realized by processor 403 executing these programs and applications.

[0022] The auxiliary storage device 402 stores signage information 420 such as advertisements for products to be displayed. Signage information display module 410 receives a display instruction from signage processing module 1202 of management server 101, and displays signage information 420 corresponding to the display instruction on output device 405 such as a display or electronic paper. Alternatively, the signage information transmitted from management server 101 may be received and displayed.

[0023] FIG. 5A shows an example of the hardware configuration of the position detection camera sensor unit 104. FIG. A plurality of camera devices 506 are connected via a network to a camera management terminal 530. The camera management terminal 530 is configured, for example, by a desktop PC, a microcomputer, a single board computer, a server on the cloud, or the like. The main memory device 501 stores programs and applications such as a track management module 510, and the processor 503 executes these programs and applications to realize the various functional elements of the camera management terminal 530. FIG. 5B shows an example of the hardware configuration of the skeleton detection camera sensor unit 104a. A plurality of camera devices 506a are connected to a camera management terminal 530a via a network. The camera management terminal 530a is configured, for example, by a desktop PC, a microcomputer, a single board computer, a server on the cloud, or the like. The main memory device 501a stores programs and applications such as a skeleton detection module 510a, and the processor 503a executes these programs and applications to realize the various functional elements of the camera management terminal 530a.

[0024] The camera device 506 includes a TOF (Time of Flight) sensor. The TOF sensor measures the distance to an object by detecting the time of flight (time difference) of light emitted from a light source, reflected by the object, and returning to the sensor. The ToF image sensor can also detect distance information for each pixel and obtain a highly accurate distance image. In addition, by using multiple ToF sensors and cameras, it is possible to determine the shape of an object and its three-dimensional overlap with higher accuracy. For example, when multiple customers are in a store at the same time, it is possible to determine where each person overlaps and detect hand movements.

[0025] In the case of measurement using multiple camera devices 506, in order to correctly obtain a movement trajectory and identify the shape of the target object, it is necessary to correctly measure the installation position and installation angle of each camera and to adjust the settings (calibration). However, in this embodiment, the camera devices 506 are installed at predetermined positions on a standardized product shelf installation frame 2100, and the settings of these cameras are configured collectively from the management server 101 side, thereby realizing simplification and efficiency of the initial setting and improvement of the accuracy of obtaining trajectory information (see FIG. 21). The multiple camera devices 506a can be configured in the same way.

[0026] The trajectory management module 510 collects and analyzes image information acquired from multiple camera devices 506, distance information to the target object, and the like, to identify the distance to the target object, the shape of the target object, and the position of the target object. It is also called the position detection module 510. Moreover, by accumulating this distance information in time series, the movement trajectory of the customer within the sensor acquisition range is acquired and accumulated in trajectory information 520. Specifically, when the multiple camera devices 506 start to detect a trajectory, the trajectory management module 510 assigns a person ID to the detected person and stores the position information of this person at each time in chronological order in trajectory information 520. By linking these pieces of time series information together, the movement trajectory of a person corresponding to a specific person ID can be acquired.

[0027] The trajectory management module 510 transmits to the management server 101 a person ID, time information, and position information at that time for reproducing a movement trajectory, or transmits trajectory information that is a compilation of these to the management server 101. The skeleton detection module 510a performs image analysis on the image information acquired from the multiple camera devices 506a, extracts features based on a human body skeleton model, and stores the features in the feature information 520a. Furthermore, the skeleton detection module 510a may calculate other features based on the detected features and store the other features in the feature information 520a. The skeleton detection module 510a may directly transmit the information on the characteristic parts of the customer to the management server 101. Alternatively, the skeleton detection module 510a may transmit the information on the characteristic parts of the customer to the trajectory management module 510 of the camera management terminal 530, and after integrating the information detected on the trajectory management module 510 side with the information on the characteristic parts of the customer, the trajectory management module 510 of the camera management terminal 530 may transmit it to the management module management server 101.

[0028] FIG. 6 shows an example of the hardware configuration of the weight sensor unit 105. A plurality of weight sensor devices 608 are connected to a data aggregation transmitter 630 via a microcomputer (microcontroller) 710. The data aggregation transmitter 630 is configured, for example, with a desktop PC, a microcomputer, a single board computer, a server on the cloud, etc. As the single board computer, for example, a Raspberry Pi can be used. The main memory device 601 stores programs and applications such as various modules 610 for processing the weight information received from the microcontroller 710 and transmitting it to the management server 101, and each functional element of the data aggregation transmitter 630 is realized by the processor 603 executing these programs and applications. The auxiliary storage device 602 stores IoT device management information 620 such as setting information and management information of the data aggregation transmitter 630, the microcomputer 710, and the weight sensor device 608.

[0029] FIG. 7 shows an example of a connection configuration for communicating with a microcomputer 710 connected to a weight sensor device 608 via wireless connection. A plurality of weight sensor devices 608 are installed on the product shelves, and a product basket is placed on each weight sensor device 608. The weight sensor device 608 is a device that measures the weight of products and the like placed in the product basket and detects changes in that weight. The acquired weight data is converted into digital information by an A / D converter 720 and transmitted to a microcomputer 710. The microcomputer 710 transmits the received weight information to the data aggregation transmitter 630 by using wireless communication such as Bluetooth (registered trademark).

[0030] FIG. 20 is an example of weight information 2000 transmitted from the microcomputer 710 to the data aggregation transmitter 630. Weight information 2000 has information on each item shown in item 2020, and has a value such as sample value 2040 as field name 2030. Weight information 2000 includes items such as shopping cart ID 2001, data type, weight 2002, number of transmissions, status 2003, time elapsed since startup, previous weight 2004, delay time 2005, and status change 2006.

[0031] The shopping basket ID 2001 is an ID set for each weight sensor device 608. It is associated with the product ID 1502 of the product placed in the shopping basket, and it is possible to calculate how many products have been removed from the shopping basket based on the change in weight 2002 and the product ID 1502 associated with the shopping basket ID 2001. Weight 2002 displays the weight of the contents of the shopping basket in grams. The weight of the shopping basket is excluded from the measured value, and the weight of the contents is displayed, and if there is no content, it shows 0. However, it may be configured to include the weight of the shopping basket. The number of transmissions indicates the number of times data has been transmitted since the start of the microcomputer 710. The number of transmissions is counted up successively, and the value is reset to 0 when the memory overflows.

[0032] A state 2003 indicates the state of the weight sensor device 608. If the measured weight data is in an unstable state, that is, the state is unstable, then 0 is indicated, and if the weight data is in a stable state, that is, the state is stable, then 1 is indicated. The time since startup is displayed in milliseconds as the time that has elapsed since the startup of the microcomputer 710. It counts up every millisecond, and if the memory overflows, the value is reset to 0. The previous weight 2004 displays the weight in grams at the time of the previous measurement. The delay time 2004 displays the time in milliseconds until the state 2003 changes from UNSTABLE to STABLE. The state change presence / absence 2006 is a flag indicating whether the state 2003 has changed since the previous measurement. If the state 2003 has changed since the previous measurement, the flag indicates 1, and if not, the flag indicates 0.

[0033] When a change is detected in the weight received from the weight sensor device 608, the microcontroller 710 sets the status 2003 to 0 indicating an “unstable state”, and transmits weight information 2000 including the weight 2002 in which the status change presence / absence 2006 is set to 1 indicating “present” to the data aggregation transmitter 630. The microcomputer 710 constantly receives weight data from the weight sensor device 608, but if there is no change in weight (the amount of change does not exceed the threshold), it periodically transmits the weight data, for example every 10 seconds, to the data aggregating transmitter 630. In this case, it sets status 2003 to 1 indicating a "STABLE state" and transmits weight information 2000 including weight 2002 in which status change presence / absence 2006 is set to 0 indicating "none" to the data aggregating transmitter 630.

[0034] Whether or not there is a change in weight is determined from the deviation between the average value of weight 2002 (for example, the average value of the past 10 values) and a newly acquired value. For example, if a change occurs from a stable state (state 2003 is STABLE) with a deviation greater than a threshold value (for example, σ=±1.0), it is determined that there has been a change, and state 2003 is changed to UNSTABLE. At this time, state change 2006 becomes 1 indicating "yes." Conversely, if the state becomes UNSTABLE once, such as when a product is picked up, the deviation between the average value and the newly acquired value is calculated in the same manner, and if the deviation is equal to or less than the threshold value and the number of times the average value has been acquired is three or more times, the state is changed to STABLE. When state 2003 changes from "UNSTABLE" to "STABLE", microcomputer 710 transmits weight information 2000 including the weight 2002 to data aggregation transmitter 630.

[0035] If the state 2003 between the previous measurement and the new measurement changes from "STABLE" to "UNSTABLE" or from "UNSTABLE" to "STABLE", then the state change presence / absence 2006 is set to 1, which indicates that a change has occurred. Here, in addition to using the deviation from the average value, a change in state may be determined simply when the difference between the previous weight and the latest weight exceeds a predetermined threshold, for example, 5 g. Similarly, it may be determined that the weight data is stable and in a STABLE state simply when the difference between the previous weight and the latest weight is equal to or less than a predetermined threshold, for example 5 g.

[0036] In this embodiment, the microcontroller 710 constantly receives weight data from the weight sensor device 608, but as described above, if there is no change in weight, it transmits weight information 2000 to the data aggregation transmitter 630, for example, every 10 seconds. In addition, if the weight changes, for example when a product is picked up, the first weight information 2000 is sent at that time, and the second weight information is sent after the weight has stabilized.

[0037] For example, when a customer picks up an item from a shopping cart, the moment the customer picks up the item, a force is applied that pushes the item down, momentarily increasing its weight. When the customer then picks up the item, the weight decreases by the weight of the item. Also, the weight does not stabilize the moment the product is picked up; rather, the weight value oscillates and then stabilizes after a few milliseconds to a few seconds. In this embodiment, taking into account the delay that occurs between the time a change in weight occurs and the time the value stabilizes, it is possible to carry out two-stage processing by sending the weight information twice.

[0038] For example, by executing the signage display process 3200 on the signage terminal 103 in response to the first transmission of the weight information 2000, when a product is picked up, an explanation of the product and advertising information for products that the customer should purchase together can be displayed on the signage terminal 103 immediately and without delay. On the other hand, by executing the product determination process 2800 in response to the second transmission of weight information 2000 based on the numerical value after the weight has stabilized, the number of products picked up can be counted correctly.

[0039] In addition, based on the weight information 2000 every 10 seconds when the weight is stable, the number of each product can be calculated and used for inventory management. For example, in this embodiment, the location and quantity of products are known, so if a product is about to run out, a shortage forecast is sent to prompt the back-of-store staff to replenish the product, and if a product is not in the correct location, a layout alert is sent to prompt the back-of-store staff to rearrange it. Inventory management such as the above becomes possible. In addition, a sales may begin by placing a predetermined number of products, such as 10, in a shopping basket, and when the number falls below a predetermined number, such as 3, the product ID (or shopping basket ID) and the difference between the current number of products (3) and the initial number of products (10), which is 7, may be notified, and based on this notification, 7 of the products may be automatically replenished using a mechanism such as a robot or conveyor.

[0040] The data aggregation transmitter 630 that receives the weight information 2000 converts the received information into a predetermined protocol such as the MQTT protocol, encrypts it, and transmits it to the management server 101 . Time information can also be added to each piece of weight information 2000. In this case, the time information when the data aggregating transmitter 630 transmits the data may be included in the weight information 2000 and transmitted, or the time information may be included when the microcomputer 710 transmits the weight information 2000 to the data aggregating transmitter 630. Here, weight time series data can be generated by arranging the data according to the time information included in the weight information 2000.

[0041] The data aggregation transmitter 630 can control and initialize the microcomputer 710 and the weight sensor device 608 based on the control information received from the management server 101. For example, it is possible to switch the power on and off, and add or update the setting information.

[0042] FIG. 8 shows an example of a connection configuration for communicating with a microcomputer 710 connected to a weight sensor device 608 via a wired connection. The microcomputer 710 is connected by wire to a plurality of weight sensor devices 608 via an A / D converter 720. The microcomputer is also connected by priority to a data aggregation transmitter 630, and communicates with the data aggregation transmitter 630 via a controller area network (CAN). The transceivers 730 and 731 transmit and receive data to and from each other using the CAN system, and a controller 740 connected in a daisy chain controls communication using the CAN system.

[0043] The microcontroller 710 can be powered via USB or CAN. As in FIG. 7, the microcomputer transmits weight information 2000 to the data aggregation transmitter 630 , and the data aggregation transmitter 630 transmits control information to the microcomputer 710 . A wired connection can avoid communication instability and delays that occur with a wireless connection.

[0044] FIG. 9 shows an example of the hardware configuration of the entrance management terminal 106. As shown in FIG. The entrance management terminal 106 is configured, for example, as a terminal such as a smartphone or tablet, or a dedicated terminal for reading two-dimensional barcodes and contactless IC chips. The main memory device 901 stores programs and applications such as an entrance management module 910, and the processor 903 executes these programs and applications to realize the various functional elements of the entrance management terminal 106.

[0045] The entrance management module 910 receives user identification information by reading a two-dimensional barcode displayed on the customer's mobile terminal 102 using the camera unit 906, or by communicating with a contactless IC chip in the customer's mobile terminal 102 using the input device 904. The store entry management module 910 transmits user identification information as information that the user has entered the store to the management server 101. At this time, the date and time when the user identification information was received may be transmitted to the management server 101 as the store entry time. The auxiliary storage device 902 stores store entry management terminal management information 920. This may store user identification information and time information acquired when a store entry process is performed.

[0046] In addition, information input when entering a store may be performed by reading a barcode such as a two-dimensional barcode, communicating with a contactless IC chip in the mobile terminal 102, or reading electronic money, a contactless IC card, or magnetic information. In this case, it is possible to change the behavior at the exit gate when leaving the store depending on how the customer entered the store. For example, if the customer enters the store using a two-dimensional barcode or a contactless IC chip on an app on the mobile terminal 102, the exit gate opens automatically and credit card payment is made on the app. A two-dimensional barcode may be printed on the product picked up when leaving the store, and the printed code may be read by an automatic payment machine, allowing guest users to make payments by credit card, electronic money, or cash.

[0047] FIG. 10 shows an example of the hardware configuration of the payment processing server 107. The payment processing server 107 is configured, for example, as a server located on the cloud. The main memory device 1001 stores a payment processing execution module 1010, and the processor 1003 executes these programs and applications to realize each functional element of the payment processing server 107.

[0048] The payment processing execution module 1010 executes payment processing for product purchases made by the mobile terminal 102 or at a store. The executed payment result is sent to the mobile terminal 102 and displayed on the screen of the output device 305 in cooperation with the product purchase management module 310. The auxiliary storage device 1002 stores payment processing information 1020 such as personal information, user information, credit card information, and payment history required for payment processing.

[0049] FIG. 11 is a diagram illustrating an example of the functional configuration of the product management system 1. As shown in FIG. The management server 101 stores various modules 210, which are programs and applications such as a weight information processing module 1101, a position information processing module 1102, a skeleton detection processing module 1102a, an entry / exit management module 1103, a product determination module 1104, a customer number determination module 1105, an estimate display module 1106, a payment management module 1107, and a mobile purchase management module 1108, and executes various processes. In addition, the management server 101 has segment information 1500, which stores information received from various sensors and information generated by the management server 101, acquisition determination range information 1600, position tracking information 1700, skeleton detection information 1700a, shelf event information 1800, estimate management information 1900, store entry information 1111, etc.

[0050] Although the various modules 210 are described as being placed on one management server 101, they may each be configured to be placed on a different virtual server on the cloud, or each group of these may be configured to be placed on a virtual server. In addition, the various types of information may be stored in separate storage devices on the cloud, or may be arranged on the cloud for each group of the various types of information.

[0051] The weight information processing module 1101 receives weight information including a shopping basket ID from the weight sensor unit 105, and accumulates the weight information together with time information corresponding to the weight information. The weight information accumulation process flow will be described later with reference to FIG.

[0052] The position information processing module 1102 receives from the camera sensor unit 104 a person ID, time information, and position information at that time for reconstructing a movement trajectory. The location information processing module 1102 identifies the area in which the person stayed based on the received location information, and accumulates information on the identified area in which the person stayed together with time information. The location information accumulation process flow will be described later with reference to FIG.

[0053] The skeleton detection processing module 1102a receives captured images of people in a store and time information from the camera sensor unit 104a. The skeleton detection processing module 1102a can obtain predetermined feature points (estimated hand points) of the person's upper body by calculation. By using this information, the management server 101 can determine the correspondence between the estimated hand points and the positions of the product shelves. The skeleton detection processing flow will be described later with reference to FIG.

[0054] The store entry / exit management module 1103 receives the customer's user identification information from the store entry management module 910 of the store entry management terminal 106, identifies which user ID the received user identification information corresponds to, and stores the information in store entry information 1111 together with the time information at which the information was received. In addition, the time when the barcode displayed on the customer's mobile terminal 102 is read or the time when information is obtained from the contactless IC chip by contactless communication may be stored instead of or in addition to the time information when the information was received. The store entry process flow will be described later with reference to FIG.

[0055] The product determination module 1104 determines the products picked up by the customer and the number of products based on the information accumulated in the shelf event information 1800. The product evaluation process flow will be described later with reference to FIG.

[0056] The customer number determination module 1105 identifies customers who were near the shelf where the weight change occurred at the time when the change occurred (the event occurred) based on the information accumulated in the shelf event information 1800, the position tracking information 1700, and the skeleton detection information 1700a. At that time, it determines which customer picked up how many products. The flow of the customer number determination process will be described later with reference to FIG.

[0057] The estimate display module 1106 compares the results of the store entry process, the product determination process, and the customer determination process, and displays the contents and quantity of the picked up products and their prices on the customer's mobile terminal 102. The estimate display process flow will be described later with reference to FIG.

[0058] After the quotation process is completed and the customer leaves the store, the payment management module 1107 requests the payment processing server 107 to execute the payment process. The payment processing flow will be described later in FIG.

[0059] The mobile purchase management module 1108 accepts orders when an online order is placed directly from the product purchase management module 310 of the mobile terminal 102 . When the mobile purchase management module 1108 receives a selection of products and quantities from the user, it calculates the total amount based on the product master information 1300 and displays it on the mobile terminal 102 . Thereafter, when an order request is accepted, the payment management module 1107 transmits the total amount and user information such as the corresponding user ID to the payment processing server 107, and the payment processing server 107 executes the payment process.

[0060] FIG. 12 is a diagram illustrating an example of a functional configuration of a signage display. The IoT information processing module 1201 of the management server 101 receives the weight information transmitted in accordance with the MQTT format. The IoT information processing module 1201 is disposed before the weight information processing module 1101 in Fig. 11. When the weight changes from a stable state, the data aggregating transmitter 630 transmits the weight information twice: at a first time when the weight changes and at a second time when the weight value stabilizes.

[0061] When the IoT information processing module 1201 receives the first weight information, it transfers the weight information to the signage processing module 1202. Note that instead of transferring the weight information directly, the weight information may be configured to notify the shopping basket ID included in the weight information. When the IoT information processing module 1201 receives the second weight information, it transfers the weight information to the weight information processing module 1101 in FIG. 11 to identify the products and the number of products. Although the IoT information processing module 1201 is configured to distribute the respective weight information, the first weight information may be transmitted to both the signage processing module 1202 and the weight information processing module 1101, and both weight information may be stored in the shelf event information 1800.

[0062] In this embodiment, the IoT information processing module 1201 receives weight information transmitted in MQTT format as an MQTT broker and distributes it to the signage processing module 1202 or weight information processing module 1101 that subscribes to it, but the weight sensor unit 105 may be configured to distribute the first weight information and the second weight information to different destinations in advance.

[0063] In addition, the allocation of the IoT information processing module 1201 may be performed depending on whether the number of receptions after stabilization is the first or second, or the allocation may be performed based on the status change 2006 included in the weight information or the flag of the status 2003. In this case, for example, if status change 2006 is "changed" and status 2003 has become "unstable", it is determined that this is the first weight information. If status change 2006 is "changed" and status 2003 has become "stable", it is determined that this is the second weight information. Alternatively, the determination may be made based only on status change 2006, which is a flag indicating that a change in weight has occurred. For example, when status change 2006 becomes "changed", indicating that a change in weight has occurred, the weight information may be sent to signage processing module 1202 as the first piece of information.

[0064] 13 to 19 show information stored in the various information storage DB 221 of the auxiliary storage device 202 of the management server 101. FIG. FIG. 13 is an example of the product master information 1300. The product master information 1300 has information on each item shown in item 1320, and has a value such as sample value 1340 as field name 1330. The product master information 1300 includes items such as stock availability, product image, product ID 1301, product name 1302, price 1303, priority, cost price, tax amount, sales type, weight 1304, and signage information 1305. The product ID 1301 is used as a key, and can be used to identify the product name 1302 or to display signage information 1305 corresponding to the product ID 1301 . The weight 1304 is used to calculate how many items have been taken out of the shopping basket based on the amount of change in weight acquired by the weight sensor unit 105.

[0065] FIG. 14 is an example of the user master information 1400. The user master information 1400 has information on each item shown in item 1420, and has a value such as sample value 1440 as field name 1430. The user master information 1400 includes items such as a user ID 1401, a user display ID, a name, an email address, a date of birth, a gender, and user identification information 1402. User ID 1401 is information that identifies a user, and each user's mobile terminal 102 is managed in association with this user ID 1401. Based on the user ID 1401, quotation information for purchased items, etc. can be displayed on the user's mobile terminal 102. The user identification information 1402 is information used to identify a user, and is, for example, a value such as a token (hash value) that is periodically updated for security reasons.

[0066] FIG. 15 is an example of segment information 1500. The segment information 1500 has information about each item shown in item 1520, and has a value such as sample value 1540 as field name 1530. The segment information 1500 is information that associates a shopping cart ID 1501 with a product ID 1502 . This segment information 1500 makes it possible to identify the product ID from the product basket ID included in the weight information received from the weight sensor unit 105, and to ascertain which product was picked up causing a change in weight.

[0067] In this embodiment, the setting for registering products in the product basket of the shelf can be easily performed. Initially, the setting is accepted with no products placed in it. Next, an instruction such as "Please place one product in the product basket" is displayed on the signage terminal 103, and the user is asked to place one product in the product basket. The weight sensor unit 105 obtains the weight of one product and transmits it to the management server 101. The product determination module 1104 of the management server 101 displays a list of candidates whose weight 1304 is close to the measured weight from among multiple products registered in the product master information 1300 on the signage terminal 103 located near the product shelf in order of proximity, and displays an instruction such as "Please select the product you put in your shopping cart from the list."

[0068] The administrator selects from the displayed list the same information as the product actually placed in the shopping basket. In response to this selection, the product determination module 1104 associates the shopping basket ID 1501 in which the shopping basket containing the product is placed with the product ID 1502, and stores the association in the segment information 1500. Instead of selecting from a list, the product may be automatically associated with the product having the closest weight. Next, an instruction such as "Please put three products in the shopping basket" is displayed on the signage terminal 103. The average weight is calculated from the weight information of the three products, and the product determination module 1104 automatically sets the weight error when the number of products is increased or decreased.

[0069] FIG. 16 is an example of acquisition determination range information 1600. The acquisition determination range information 1600 has information on each item shown in item 1620 , and has a value such as sample value 1640 as field name 1630 . The acquisition determination range information 1600 is information that defines the range within which products can be taken from each shopping basket. The acquired determination range information 1600 includes a shopping cart ID 1601 , a determination area 1602 , and a determination stay range 1603 . The shopping basket ID 1601 is associated with the weight sensor device 608 of the weight sensor unit 105, and it is possible to identify from which shopping basket a product was taken and as a result, a change in weight has occurred. The judgment area 1602 indicates the range within which the product associated with the shopping basket ID 1601 can be taken, using coordinates within the product shelf installation frame 2100 (see FIG. 21). The determined stay range 1603 indicates the range in which the product associated with the shopping basket ID 1601 can be picked up, in units of pre-defined sections.

[0070] The acquisition determination range will be described with reference to FIGS. FIG. 21 shows an example of the configuration of a frame for installing a product shelf. In this embodiment, cameras and product shelves are installed in a product shelf installation frame with predetermined dimensions, and several pre-registered initial system settings are automatically executed, thereby reducing the effort, time, and cost involved in setting up an unmanned store and customizing it for each store. The product shelf installation frame 2100 is, for example, a tower-shaped frame measuring 2.5 m in length and width and 3 m in height. A camera device 506 is installed at a predetermined position on four pillars 2101 or in the vicinity thereof (e.g., beams 2102). Alternatively, the frame 2100 may have an attachment portion for the camera device 506 so that the camera device 506 can be installed at this position.

[0071] One of the frame's pillars 2101 is set to coordinate (0,0), and a coordinate plane is defined in millimeters in the x and y directions from there. In the example shown in the figure, the coordinates of the lower left pillar are (0,0), the coordinates of the lower right pillar are (2500,0), and the coordinates of the pillar at the far right are (2500,2500). By installing a product shelf at a predetermined position within this, the settings of the camera sensor unit 104 and the weight sensor unit 105 are semi-automated.

[0072] FIG. 22 is a diagram for explaining the configuration of product shelves. In the front view of a product shelf shown in FIG. 2200, 3×3 product baskets 2202 are placed on one product shelf 2201. A weight sensor device 608 is placed under each product basket to measure the weight of the contents of each product basket. The product shelf is assumed to have a size of 450 mm (width x depth x height) x 500 mm (500 x 1500 mm). In this embodiment, several other product shelf sizes are prepared in advance in addition to this size, and by selecting a product shelf from these specified sizes, the setting information is sent from the management server 101 to the camera sensor unit 104 and the weight sensor unit 105, and the setting is completed semi-automatically. In 2250 of FIG. 22, which shows a product shelf viewed from the left side, a virtual acquisition area 2252 from which products can be acquired is set in front of a product shelf 2251. The virtual acquisition area is set to 760 mm from the end of the product shelf, which is the same range as a customer's reach for a product.

[0073] FIG. 23 is a schematic diagram for explaining the acquisition determination range of a product. In FIG. 23, 2300 showing a product shelf installation frame 2100 as viewed from above, the same coordinates as in FIG. 21 are displayed, with the coordinates of the lower left support pillar 2101 being (0,0) and the upper right support pillar 2101 being (2500,2500). Four product shelves 2301, AS01 to AS04, are installed in the front part of Fig. 23. If an area extending 760 mm in the y-axis direction from the tip of the product shelf and each 450 mm wide product shelf 2301 in the x-axis direction is defined as one area, then an area including one area in front of a product shelf, two areas on the left side, and two areas on the right side are defined as the area in which customers can acquire products, that is, the acquisition determination area.

[0074] For example, the acquisition judgment range 2352 of the product shelf 2351 of AS02 is the range of A0, A1, A2, A3, and A4. Similarly, the acquisition judgment range of the product shelf of AS01 is the range of A0, A1, A2, and A3, and the acquisition judgment range of the product shelf of AS04 is the range of A2, A3, A4, and A5. The acquisition judgment range for each product basket is stored in the acquisition judgment range information 1600 of FIG. 16. For example, AS02_B is stored as the product basket ID 1501 in the second product basket from the top of the product shelf AS02, and the judgment area 1602 corresponding to this product basket is the area surrounded by coordinates (0,500) (0,1260) (2150,1260) (2150,500).

[0075] In addition to coordinates, the inside of a store or the inside of the product shelf installation frame 2100 can also be expressed by pre-divided sections, and in the case of the product shelf AS02, the determined stay range 1603 corresponds to the A0 to A4 parts in Fig. 23. Here, one section is an area of ​​450 mm x 760 mm. In this embodiment, the product acquisition range is within the planar range of the x and y axes, but the product basket ID can also identify the product basket in the z-axis direction, for example, AS02_A for the top shelf, AS02_B for the second shelf, AS03_C for the third shelf, and accordingly, the judgment area can also be set to a value in the z-axis direction.

[0076] FIG. 17A is an example of location tracking information 1700. The location tracking information 1700 includes information on each item shown in item 1720, and includes a value such as sample value 1740 as a field name 1730. The position tracking information 1700 is information that chronologically stores where in the store and who was there, and is also called a position information storage unit or a position information storage means. The location tracking information 1700 includes items such as a time limit 1705, a stay range 1701, a person ID 1702, a store ID 1703, and a time stamp 1704. The stay range 1701 stores pre-divided sections within a store or within the product shelf installation frame 2100, as in FIG. 16, and stores the locations where customers stayed that were identified from the location information transmitted from the camera sensor unit 104.

[0077] FIG. 17B is an example of skeleton detection information 1700a. The skeleton detection information 1700a has information on each item shown in item 1720a, and has a value such as sample value 1740a as field name 1730a. The skeleton detection information 1700a is information on characteristic parts detected by applying a skeleton detection module to a captured image of a customer in a store. This information relates to the position coordinates of certain features of the upper half of the human body (e.g., head, left and right shoulders, left and right elbows, left and right wrists) detected by the skeleton detection module 510a of the skeleton detection camera sensor unit 104a. Furthermore, using the position coordinates of these features, the position coordinates of features of other parts of the human body (e.g., estimated points of left and right hands, etc.) calculated by the calculation can be included in the same way.

[0078] In addition, in the figure, skeleton detection information 1700a of a customer is illustrated when one customer is detected on the photographed image. When multiple customers are detected on the photographed image, skeleton detection information 1700a is created separately for each customer. In that case, the table in FIG. 17(B) may be created separately for each customer. Alternatively, the table in FIG. 17(B) may be created collectively for multiple customers on the same photographed image. Information about predefined features of the upper body and estimated hand positions can be integrated with detected information from position tracking information 1700. Furthermore, the skeleton detection information 1700a can include a deadline, a time stamp, and the like, just as in the case of the position tracking information 1700.

[0079] The table in Fig. 17(B) can include tracking information within the image. For example, the table in Fig. 17(A) and the table in Fig. 17(B) can be created separately, or can be created together. For example, the data in Fig. 17(B) created based on the image captured by the camera sensor unit 104a can be inserted into the table in Fig. 17(A) created based on the image captured by the camera sensor unit 104. Or, the reverse can be done.

[0080] The person ID 1702 is information for identifying a person transmitted from the camera sensor unit 104, and is not associated with a user ID at this stage. Therefore, although it is not specified which user it is, the fact that a certain person was in the stay range 1701 at the time indicated by the time stamp 1704 is stored. The store ID 1703 stores which store the event occurred at. In the time stamp, data sent from the camera sensor unit 104 is stored, for example, as a UNIX (registered trademark) time stamp (in seconds). Moreover, these records are deleted or migrated to another inexpensive storage area when the period specified in the period 1705 is exceeded.

[0081] FIG. 18 is an example of shelf event information 1800. The shelf event information 1800 has information on each item shown in item 1820 , and has a value such as sample value 1840 as a field name 1830 . The shelf event information 1800 stores events such as weight changes that occur on the product shelf based on weight information transmitted from the weight sensor unit 105. It may also be called a weight information storage unit or weight information storage means. The shelf event information 1800 has items such as an event ID, a shopping cart ID 1801, a previous weight 1802, a store ID, a time stamp 1803, a type, a weight 1804, and a weight change amount 1805. As the shopping basket ID 1801, a value corresponding to the shopping basket ID 2001 whose weight has changed and which is transmitted from the weight sensor unit 105 is stored.

[0082] The previous weight 1802 and the weight 1804 store the values ​​of the previous weight 2004 and the weight 2002, respectively, transmitted from the weight sensor unit 105. Note that the previous weight 1802 may be configured to store the previous weight stored in the management server 101, instead of the transmitted value. As the weight change amount 1805, the difference between the current weight 1804 and the previous weight 1802 is stored. The time when the event occurred is stored in the time stamp 1803. The time when the weight information processing module 1101 received the weight information is stored, but the time when the data aggregation transmitter 630 transmitted the weight information may be used, or the time when the microcomputer 710 transmitted the weight information, the time when the microcomputer 710 acquired the weight value, or the time when the weight sensor device 608 acquired the weight value may be used.

[0083] FIG. 19 is an example of quotation management information 1900. The quotation management information 1900 has information on each item shown in item 1920 , and has a value such as sample value 1940 as a field name 1930 . The estimate management information 1900 stores a user ID 1901, a person ID 1902, a product ID 1903, a number of products 1904, and a time stamp 1905, all of which are identified by the estimate display module 1106. In other words, this information records the results of determining who picked up what and how many products from the product shelf, when they were picked up, and so on.

[0084] 24 to 32 show examples of processes executed by each terminal of the product management system 1 and each module of the management server 101. FIG. FIG. 24 shows an example of a weight information accumulation process flow 2400. The weight information processing module 1101 receives the weight information transmitted from the weight sensor unit 105 (step 2410). The weight information processing module 1101 acquires the time information of the reception time (step 2420). In addition, other information stored in the shelf event information 1800 is acquired or calculated.

[0085] The weight information processing module 1101 associates the weight information and other information with the acquired time information and accumulates them in the shelf event information 1800 (step 2430). Although the time information is the time when the weight information processing module 1101 received the weight information, the time when the data aggregation transmitter 630 transmitted the weight information may also be used, or the time when the microcontroller 710 transmitted the weight information, the time when the microcontroller 710 acquired the weight value, or the time when the weight sensor device 608 acquired the weight value may also be used.

[0086] FIG. 25 shows an example of a location information accumulation process flow 2500. The position information processing module 1102 receives the person ID, the time information, and the position information at that time transmitted from the camera sensor unit 104 (step 2510). The position information processing module 1102 identifies the stay range in which the person indicated by the person ID stayed based on the position information (step 2520). The stay range is associated with information on sections that have been previously divided within the store or the product shelf installation frame 2100, and identifies which section the person belongs to. In addition, other information of the items 1720 indicated in the position tracking information 1700 is obtained or calculated. The position information processing module 1102 stores the identified stay range information, the person ID, the time information, and other information in association with each other in the position tracking information 1700 (step 2530). The location where a person stayed is stored as information on a predefined area such as A1, A2, etc. of sample value 1740 of stay range 1701, but the configuration may be such that coordinate information received as location information is stored, or that trajectory information that combines multiple pieces of location information is stored.

[0087] FIG. 42 is an example of a skeleton detection process flow 4200. The skeleton detection processing module 1102a receives feature points of the upper half of the human body (e.g., head, right shoulder, left shoulder, right elbow, left elbow, right wrist, left wrist) detected by applying the skeleton detection module on the captured image (step 4210). These feature points are obtained by the skeleton detection module 510a of the skeleton detection camera sensor unit 104a. Next, the hand estimation point is calculated by using the feature points of the upper half of the human body (step 4220). The hand estimation point is calculated by the skeleton detection module 510a of the skeleton detection camera sensor unit 104a. At this time, the position coordinates of other parts of the human body, such as the neck, may also be calculated.

[0088] Furthermore, information on the estimated hand positions is stored together with time information (step 4230). The position information acquired by the position information processing module 1102 and the position information acquired by the skeleton detection processing module 1102a can be associated with each other in terms of position coordinates. The skeleton detection camera sensor unit 104a may execute this skeleton detection processing flow 4200 and transmit the feature point information and the estimated position coordinates of the hand calculated from the feature point information to the management server 101. Alternatively, the management server 101 may execute each of the flows of steps 4210, 4220, and 4230 based on the captured image.

[0089] FIG. 26 shows an example of an overall process flow 2600 for purchasing a product in a store. When a customer enters a store and performs store entry processing on the store entry management terminal 106, the store entry / exit management module 1103 executes store entry processing 2700 (step 2610). Information about the movement trajectory of the customer as he moves through the store is stored in sequential position tracking information 1700, and information about the acquisition of products from product shelves is stored in shelf event information 1800. The product evaluation module 1104 executes a product evaluation process 2800 for identifying the type and number of products picked up by the customer based on the stored shelf event information 1800 (step 2620).

[0090] The customer number determination module 1105 executes a customer determination process 2900 (step 2630) for identifying a customer who was within reach of the product basket where a weight change occurred, based on the stored position tracking information 1700, skeleton detection information 1700a, and shelf event information 1800. At this time, it is preferable to determine how many products were picked up from the product shelf and by which customer, based on the result of determining the correlation between the estimated hand point obtained by calculation and the product shelf position, and the result of determining the change in output of the sensor that detected the weight change. The estimate display module 1106 compares the results of the store entry process 2700, the product determination process 2800, and the customer determination process 2900, calculates which user picked up which product and how many of them, and executes the estimate display process 3000 to display the estimate information on the user's mobile terminal 102 (step 2640).

[0091] Upon receiving the information indicating that the customer has left the store, the payment management module 1107 executes the payment process 3100 to request the payment process server 107 to process the payment for the quoted product (step 2650).

[0092] Each process flow will be explained in detail below. FIG. 27 shows an example of a store entry process flow 2700. The store entry / exit management module 1103 receives the customer's user identification information from the store entry management module 910 of the store entry management terminal 106 (step 2710), and obtains the time information of the received time as the store entry time (step 2720). The store entry / exit management module 1103 compares the received user identification information with the user identification information 1402 in the stored user master information 1400, and obtains the corresponding user ID 1401, thereby identifying which user has entered the store (step 2730). The store entry / exit management module 1103 stores the identified user ID 1401 and the store entry time in the store entry information 1111 (step 2740).

[0093] The store entry time is defined as the time when the store entry / exit management module 1103 of the management server 101 receives the customer identification information. However, the store entry time may be generated when the store entry management module 910 of the store entry management terminal 106 reads the barcode displayed on the customer's mobile terminal 102 or when information is obtained from the contactless IC chip via contactless communication, and may be transmitted to the management server 101 together with the customer identification information.

[0094] FIG. 28 shows an example of a product evaluation process flow 2800. The product evaluation module 1104 acquires information on the time when the change in weight occurred (event occurrence time) from the shelf event information 1800 (step 2810). The product evaluation module 1104 acquires the product basket ID in which a change in weight has occurred and the amount of weight change from the shelf event information 1800 (step 2820).

[0095] Based on the segment information 1500, the product determination module 1104 acquires information on the product ID 1502 associated with the product basket ID 1501, and product information corresponding to the product basket ID in which a change in weight has occurred from the product master information 1300 (step 2830). The product evaluation module 1104 calculates the number of products picked up by the customer based on the amount of weight change and the product weight 1304 stored in the product master information 1300 (step 2840). The product evaluation module 1104 outputs the acquired product information and the calculated product quantity information (step 2850). It is preferable to prevent a decrease in the accuracy of this determination by comparing the result of a determination of the correlation between the estimated hand position obtained by calculation and the position of the product shelf.

[0096] FIG. 29 shows an example of a customer determination process flow 2900. The customer number determination module 1105 acquires the time when the event occurred from the shelf event information 1800 (step 2910). The customer number determination module 1105 identifies the person ID of a person who stayed in the vicinity of the place where the event occurred from the shelf event information 1800, the store entry information 1111, and the position tracking information 1700 (step 2920). The customer number determination module 1105 outputs the identified person ID (step 2930).

[0097] For example, when an event occurs in which the weight of a shopping basket changes, the customer number determination module 1105 acquires from the shelf event information 1800 a timestamp 1803 corresponding to the time when the weight change occurred. Next, by comparing the judgment stay range 1603 corresponding to the shopping basket ID 1601 in the obtained judgment range information 1600 with the stay range 1701 corresponding to the person ID 1702 in the position tracking information 1700, the person ID of the customer who was staying within a range where the product could be obtained from the shopping basket where the weight change occurred at the time the event occurred is identified.

[0098] Furthermore, when an event occurs in which a customer enters a store, the customer number determination module 1105 obtains a timestamp corresponding to the time of entry from the store entry information 1111. Next, the person ID of the store entry information 1111 is compared with the location information (stay range 1701) and person ID 1702 of the location tracking information 1700 to identify the person ID of the customer who stayed near the store entry management terminal 106 at the time of entering the store.

[0099] The weight sensor unit 105 transmits weight information twice, once at the time when a change in weight acquired by the weight sensor device 608 occurs and once when the weight change has stabilized. The shelf event information 1800 can be configured to store both pieces of information, or can be configured to store only the weight information after the second weight has stabilized. The shelf event information 1800 can also store the difference time between the first time when the weight change corresponding to the first weight information occurred and the second time when the weight change became stable, in which case, even in a configuration in which the first weight information is not stored, the person ID of that time can be identified based on the time obtained by subtracting the difference time from the second time corresponding to the second weight information. In other words, based on the second time when the weight change became stable and the difference time, the number of products can be associated with the customer, and the number of products can be displayed on the customer's mobile terminal 102.

[0100] In addition, since there may be a slight time lag between the timestamp 1803 of the shelf event information 1800 and the timestamp 1704 of the position tracking information 1700 due to sensor errors, etc., it is also preferable to specify the person ID by setting a gradually wider range of times for identifying the customer's position. For example, the corresponding person ID is identified from the timestamp 1803 of the time when the weight change occurred and the position tracking information 1700 for a total of three seconds, one second before and one second after. If the corresponding person ID cannot be identified here, the range is expanded to the timestamp 1803 and two seconds before and after it, for a total of five seconds, to check whether the person ID can be identified. In this way, the judgment time is gradually expanded and the process is repeated until the person can be identified.

[0101] Furthermore, when multiple person IDs are obtained as candidates, the person with the highest probability is identified as the target. For example, the person closest to the shopping basket or the person who has reached for the shopping basket is identified as the person with the highest probability. This information can be acquired from the customer's position information and hand movement information acquired by the camera sensor unit 104. Furthermore, this information can be compared with the result of the determination of the correlation between the estimated hand points and the positions of the product shelves, which is calculated using the skeletal features extracted on the captured screen from the camera device 506a.

[0102] Also, the determination time range used for identifying a person may be changed according to the number of people staying in the vicinity of the shopping basket (e.g., inside the product shelf installation frame 2100) where the event occurred at the time before and after (e.g., one minute before and after) the time when the weight change occurred. In other words, if there is only one customer in the product shelf installation frame 2100 or the store, it is highly likely that the weight change that occurred there is due to that one customer, so in such a case, the determination time range is set to be wider. For example, the person IDs of customers staying within the product basket determination stay range 1603 for a total of 11 seconds, including the time when the weight change occurred and 5 seconds before and after that time, are identified.

[0103] Conversely, when there are many customers in the product shelf installation frame 2100 or in the store, the determination time range is shortened to perform processing to improve accuracy. For example, the personal ID of a customer who is in the product basket determination stay range 1603 for one second at the time when the weight change occurs is identified, and if the ID cannot be identified, the determination is repeated for a range of three seconds, which is expanded by one second before and after. In this way, the determination time range is expanded in stages. In this way, by dynamically changing the judgment time range for identifying a person depending on the product shelf installation frame 2100 and the number of customers present in the store at the time of an event, such as the time a change in weight occurs, it is possible to improve the speed and accuracy of judgment.

[0104] FIG. 30 shows an example of a quote display process flow 3000. The estimate display module 1106 acquires the results of the store entry process 2700, the product determination process 2800, and the customer determination process 2900, and compares these results (step 3010). The estimate display module 1106 acquires and stores the user ID, product information, and product quantity information corresponding to the event occurrence time (step 3020). For example, the product information and product quantity information at the time when the weight change identified by the product determination process 2800 occurred are associated with the person ID at the time when the weight change identified by the customer determination process 2900 occurred. Also, the user ID of a person who entered the store around the time when the person with the person ID identified by the customer determination process 2900 stayed near the store entry management terminal 106 is associated. The estimate display module 1106 stores the associated user ID, person ID, product ID, quantity, and timestamp in the estimate management information 1900.

[0105] In other words, weight information transmitted from the weight sensor unit 105 installed on the product shelf and first time information corresponding to this weight information are accumulated in chronological order in the shelf event information 1800. The customer's location information transmitted from the camera sensor unit 104 and second time information corresponding to this location information are stored in time series in the location tracking information 1700, The customer's skeleton position information transmitted from the camera sensor unit 104a and the third time information corresponding to this position information are stored in chronological order in the skeleton detection information 1700a. The fourth time information of the customer entering the store is stored in the store entry information 1111 in chronological order. By comparing the accumulated information based on the first time information, the second time information, the third time information, and the fourth time information, the number of products can be associated with the customer and user information, and product quantity information can be displayed on the mobile terminal 102 of the customer identified by the user information. The shelf event information 1800, the position tracking information 1700, the skeleton detection information 1700a, and the store entry information 1111 may each be stored in one or a part of different storage means, or all of them may be stored in the same storage means.

[0106] The quotation display module 1106 transmits the product information and the product quantity information to the mobile terminal 102 of the customer corresponding to the user ID 1901 based on the quotation management information 1900 and displays them (step 3030). The product purchase management module 310 of the mobile terminal 102 displays the received product information and product quantity information on the output device 305 such as a display. The estimate display module 1106 calculates the total price of the products that the customer has removed from the shopping cart before leaving the store, and displays the total price on the customer's mobile terminal 102 (step 3040).

[0107] If the products and the number of products purchased by the customer cannot be determined from the information from weight sensor unit 105 or the images from camera sensor unit 104 and camera sensor unit 104a, the information is stored in a check-required list. For example, if an irregular event occurs, such as when the detection of the amount of change in weight is suspicious, when the detection of position information from the camera device 506 or the calculation of the estimated hand position based on the image captured by the camera device 506a does not go well, or when the image is obscured and cannot be distinguished, the information is stored in a checklist for final human check. In this way, when the management server 101 is unable to fully distinguish, a human can assist later, thereby achieving a highly accurate shopping experience. The payment process itself is performed after leaving the store, so the customer experience of the user is not impaired.

[0108] In this embodiment, by accumulating in chronological order the information relating to the customer's location (movement trajectory) obtained from the camera sensor unit 104, the information relating to the customer's skeletal detection obtained from the camera sensor unit 104a (such as the estimated position of the hands), the information relating to weight changes obtained from the weight sensor unit 105, and the information relating to the customer's entry obtained from the entry management terminal 106, all information relating to what happened in the store, who did what, when, and where is stored in the management server 101 on the cloud in chronological order, making it possible to reproduce this information in chronological order. In addition, since all sensors operate independently, by accumulating each piece of information in chronological order, it becomes possible to improve various analyses and customer assessment accuracy, so it is easy to add sensors that estimate the user's gender and age at a later date, for example.

[0109] FIG. 31 is an example of a payment processing flow 3100. The payment management module 1107 receives information indicating that the customer has been outside the exit area for a certain period of time, for example, 10 seconds or more (step 3110). Whether or not the customer has left the exit area is determined by the camera sensor unit 104 detecting whether or not the customer has left a range of predetermined coordinates. In the example using the product shelf installation frame 2100 in Fig. 21 and Fig. 23, the area surrounded by the coordinates (0,0), (0,2500), (2500,2500), and (2500,0) is the exit area. In a configuration that does not use the product shelf installation frame 2100, the coordinates of the store exit area may be arbitrarily set inside or near the store.

[0110] In response to the customer being outside the store exit area for a certain period of time, the payment management module 1107 calculates the total price of all the products stored in the estimate display module 1106 (step 3120). The payment management module 1107 transmits the user ID of the user who picked up the item from the shopping cart and the total amount to the payment processing server 107, and the payment processing server 107 executes the payment processing based on the payment processing information 1020 corresponding to the user ID (step 3130).

[0111] In this embodiment, if the product or the number of products finally picked up is incorrect, a refund can be made from the product purchase management module 310 (for example, an app on a smartphone) of the mobile terminal 102. In the payment process, a payment reservation is made on the payment processing server 107 after leaving the store, and the payment confirmation process is performed after a certain time. This eliminates the need to pay fees when returning items.

[0112] FIG. 32 illustrates an example of a signage display process flow 3200. The signage processing module 1202 receives the first weight information at the timing when the weight value measured by the weight sensor unit 105 changes from a stable state to an unstable state from the IoT information processing module 1201 (step 3210). Note that the weight information may be received from the data aggregation transmitter 630 instead of being distributed and received by the IoT information processing module 1201.

[0113] The signage processing module 1202 acquires the shopping basket ID in which a change in weight has occurred from the weight information (step 3220). The signage processing module 1202 acquires product information, such as the product name 1302 and price 1303 of the product ID corresponding to the shopping cart ID, and signage information 1305 from the segment information 1500 and the product master information 1300 (step 3230).

[0114] The signage processing module 1202 transmits the acquired signage information to the signage terminal 103 and displays it (step 3240). For example, the signage information stored may include information such as the product name, price, and advertisement of the product ID corresponding to the shopping basket ID, and information such as the product name, price, and advertisement of the product associated with the corresponding product ID (for example, when an onion is picked up, information about curry roux, a dish that uses this onion).

[0115] Although the signage information is configured to be transmitted by signage processing module 1202, it may also be configured to read out signage information 420 stored in signage terminal 103 in response to the transmission of information identifying signage information, such as a product ID, from signage processing module 1202. The signage terminal 103 may be, for example, a tablet terminal configured to display signage information on its own display, or may be a single-board computer such as a Raspberry Pi or a computer configured to display signage information on an external display.

[0116] In addition, the signage processing module 1202 may be configured to transmit signage information or information identifying the signage information, such as a product ID, in a push format, or may be configured to obtain information in a pull format from the signage processing module 1202 to which the signage terminal 103 subscribes based on the MQTT format.

[0117] FIG. 33 is an example of a hardware reference model 3300. Typically, when trying to create an unmanned store, the positions and number of multiple camera sensors must be individually set for each store depending on the store's structure and product layout, which requires effort, time, and cost for setup.

[0118] On the other hand, in this embodiment, a product shelf installation frame 2100 of a predetermined size is prepared, and a camera device 506 is installed in a predetermined position as shown in Figure 21. Furthermore, by installing a product shelf of a predetermined size in a predetermined position within the product shelf installation frame 2100, the initial setup when setting up an unmanned store can be simplified and the labor, time, and cost required for setup can be reduced. Furthermore, by arranging the product shelf installation frames 2100, it is possible to easily adjust the size of the product management system 1 according to the size of the store.

[0119] In this embodiment, the configuration of the walk-through (for indoor use) 3310 using the product shelf installation frame 2100 has been described, but it is also possible to adopt a configuration of a walk-through (for outdoor use) 3320 using a larger container. In this case, too, it is possible to install the camera device 506 at a predetermined position in the container, as with the product shelf installation frame 2100, and simplify various initial settings.

[0120] The hardware reference model 3300 in FIG. 33 provides a customer with an interface to select from a website predetermined models such as the configuration of an unmanned store and the number of camera devices 506 and camera devices 506a, just like selecting a cloud service. The instance family 3301 indicates the configuration of an unmanned store or a mobile order. The instance type 3302 indicates a type indicating each configuration, such as IaaS (Infrastructure as a Service) in a cloud service. Image 3303 shows the store format assumed when each instance type is installed.

[0121] Camera 3304 indicates the presence or absence of a camera. The camera can include camera device 506 and camera device 506a. The multiple name correspondence 3305 indicates whether multiple customers can be distinguished. Indoor 3306 and outdoor 3307 indicate indoor and outdoor stores. The size estimate indicates the size of this product management system 1. For example, in a simplified product management system 1 where multiple user support 3305 is set to "None" and there is only one customer, the system can be constructed in a space-saving and inexpensive manner by using only the weight sensor unit 105 and not installing the camera sensor unit 104 and / or camera sensor unit 104a.

[0122] The management server 101 accepts the selection of instance type 3302 from the user, and delivers the necessary hardware such as the product shelf installation frame 2100, camera device 506, camera device 506a, etc., from a warehouse or the like to the store, as the case may be, and after installing these, transmits initial setting information and initial setting procedures from the management server 101 to the camera management terminal 530, camera management terminal 530a, and data aggregation transmitter 630, and these terminals and devices execute these, completing the initial setting for the camera sensor unit 104, camera sensor unit 104a, and weight sensor unit 105.

[0123] For example, when the power is turned on for a weight sensor unit 105 connected to a weight sensor device 608, a microcontroller 710, and a data aggregation transmitter 630 in a store, the weight sensor unit 105 automatically connects to the management server 101 on the cloud via a WiFi (or LTE) modem, and the ID and equipment information assigned to each device or equipment are sent to the management server 101. The management server 101 transmits setup data, setup programs, etc. corresponding to each device based on the configuration specified by the selected instance type 3302 to each device or instrument identified by the transmitted ID and equipment information, and executes the setup procedure.

[0124] By such a procedure, the management server 101 on the cloud can grasp the setting status and the operating status. Furthermore, when a failure occurs, the management server 101 can perform remote operation such as restarting or re-setting up each of the equipment and devices of the camera sensor unit 104, the camera sensor unit 104a, and the weight sensor unit 105.

[0125] 21, the camera device 506 is installed at a predetermined position near at least four posts, and after arranging a product shelf with a weight sensor device 608 installed at a predetermined position in the product shelf installation frame 2100, the management server sends information (each product basket ID) for identifying the multiple microcomputers 710 of the weight sensor (or the weight sensor device 608) and setting information to the data aggregating transmitter 630, and the data aggregating transmitter 630 sets the multiple microcomputers 710 in the management server based on the product basket IDs and setting information of the multiple microcomputers 710. The camera device 506a may be installed at a position different from the camera device 506.

[0126] The camera sensor unit 104 stores a shopping basket ID that identifies the microcomputer 710 of the weight sensor unit 105 in association with coordinate information within a product shelf installation frame 2100 as shown in Fig. 21 and Fig. 23. In the example of Fig. 23, the range surrounded by coordinates (350,0)(350,500)(800,500)(800,0) is associated with the shopping basket IDs AS01_A, AS01_B, and AS01_C present on the product shelf AS01.

[0127] FIG. 34 is an example of an order management screen 3400. The estimate display module 1106 of the management server 101 manages the order history of users. The order management screen 3400 displays a history 3401 of picking up an item and a history 3402 of putting that item back into the shopping cart, showing that one chocolate chip cookie was picked up and one was put back (decreased by 1). There is also a history 3403 of picking up a caramel waffle, and the total price 3404 of those items is displayed as 162 yen.

[0128] FIG. 35 is an example of a product purchase screen. 13 is an example of a screen displayed on a mobile terminal 102 identified by a user ID associated with a customer who has picked up a product from a shopping cart. Screen 3500 is an example of a screen displayed when it is detected that two candies have been removed from the shopping cart. One white chocolate and two candies are displayed as product 3501. Screen 3550 is an example of a screen displayed when it is detected that one candy has subsequently been returned to the shopping cart. One white chocolate and one candy are displayed as product 3502.

[0129] A typical POS system does not know the order in which purchased items were placed in the shopping cart. On the other hand, the product management system 1 of this embodiment keeps track of all in-store actions in chronological order, such as which product shelf the customer visited in what order, what was picked up and put back, and what was ultimately purchased in what order. This makes it possible to obtain more detailed behavior and purchase data, and it is possible to issue coupons to encourage customers to return to the store to purchase products that were picked up but not purchased or products that were placed on a product shelf where the customer stayed for a long time, and to display the customer's interest history on the app.

[0130] In addition, since authentication is performed using user identification information when entering the store, attribute information for that user can be obtained from the user master information 1400. By using this attribute information and tracking of behavior within the store, when a customer comes in front of a product shelf, it is possible to display signage information that matches the customer's past purchasing history or to display coupons on the signage to encourage upselling. In addition, by combining electronic price tags, it becomes possible to perform dynamic pricing and dynamic offers, such as displaying the discount amount with the coupon price applied on the electronic price tag for each customer, or displaying the amount displayed by the estimate display module 1106 as the discounted price.

[0131] In the product management system 1 of this embodiment, a mechanism for easily managing the sales of various products can be realized using the camera sensor unit 104, the camera sensor unit 104a, and the weight sensor unit 105. Also, by using a product shelf installation frame 2100, a container, and a product shelf with the positions for incorporating these sensors specified in advance, and by transmitting setting information and setting procedures in a push format from the management server 101 to the data aggregation transmitter 630, the camera management terminal 530, and the camera management terminal 530a to perform initial settings, the product management system 1 can be easily introduced as an unmanned terminal or the like. EXAMPLES

[0132] Next, a second embodiment of the product management system 1 according to the present invention will be described. In order to avoid repetition, differences from the first embodiment will be mainly described. The second embodiment may be implemented in combination with the first embodiment, or they may be implemented separately.

[0133] Referring to FIG. 36, the state of the inside of a store to which the second embodiment can be applied is shown in two parts, (A) and (B). In these figures, the state of the inside of the same store is photographed from the same viewpoint at different times. As can be seen from these figures, a plurality of product shelves are installed in the store. On each product shelf, a plurality of products are displayed for sale. Between each product shelf, there are passages where customers can walk around freely.

[0134] A "store" refers to a facility that provides goods or services for economic activities. A store only needs to have a space for installing product shelves, and does not necessarily need walls or ceilings surrounding the store. For example, a store may be set up as a section of a building, such as a shopping center, department store, or outlet mall. Alternatively, a store may be set up as an entire building, such as a small supermarket or a privately owned store. Alternatively, a store may be set up as an outdoor section without a building.

[0135] "Goods" refers to goods traded in a store as objects of economic activity. Goods are tangible objects such as food, beverages, stationery, or clothing. These tangible objects include any type of storage medium that temporarily stores data, programs, or the like. These tangible objects can also include any medium related to the provision of services.

[0136] A "product shelf" refers to any component for displaying products. For example, the product shelf may have one or more vertical stages of product display rods. One end of the rod may be fixed to the product shelf body, and the other end may be a free end for hanging products in a hook-like manner. Alternatively, the product shelf may have one or more vertical stages of product display boards. The boards may have a substantially horizontal upper surface, one end of which may be fixed to the product shelf body, and one or more products may be placed on the upper surface. Alternatively, the product shelf may have one or more vertical stages of product display baskets (see FIG. 22, etc.).

[0137] In this embodiment, a sensor for detecting or calculating the weight of a displayed product can be installed on each level of the product shelf. For example, a sensor for detecting the weight of a product hung on a rod of the product shelf like a hook is installed in association with the rod. Or, a sensor for detecting the weight of a product placed on a board of the product shelf or a shopping basket is installed in association with the board or shopping basket. The sensor can convert the detected weight of the rod, board, or shopping basket into a numerical value, output the value, and transmit the value to the management server 101.

[0138] Preferably, the sensor is a weight sensor. For example, the sensor is a load cell or an electromagnetic balance sensor. However, the sensor may include not only currently available sensors but also sensors that may be available in the future. Furthermore, the sensor is not limited to a form that directly detects weight. For example, the sensor may detect a displacement amount other than weight and calculate the change in weight based on the displacement amount.

[0139] Hereinafter, it is assumed that a weight sensor is installed on each shelf of the product shelf to directly detect the weight of the product placed on the bar or board of the product shelf or the product basket. This weight sensor may be the weight sensor device 608 (see Figs. 7 and 8) of the weight sensor unit 105 used in the first embodiment. The weight sensor device 608 detects the weight of a bar or board on a product shelf or a product basket when nothing is placed thereon as an initial value, and transmits the weight to the management server 101. However, the initial value is not limited to 0 g.

[0140] Next, the weight sensor device 608 periodically detects the weight of one or more products placed on each level (e.g., a rod, board, or shopping basket) of the product shelf at predetermined intervals and transmits the detected weight to the management server 101. Therefore, the amount of change in weight of each level of the product shelf based on the detected value of the weight sensor device 608 is periodically transmitted to the management server 101 at predetermined intervals. This information can be stored in any storage device associated with the management server 101.

[0141] Preferably, two types of cameras are provided inside the store to capture images of the interior of the store. One is a camera (first camera sensor) that functions as a camera that detects the position of a customer in the store. This camera may be the camera device 506 of the position detection camera sensor unit 104 in the first embodiment. The camera device 506 periodically acquires the movement trajectory (e.g., walking trajectory) of the customer in the store at a predetermined interval and transmits it to the management server 101.

[0142] One is a camera (second camera sensor) that captures images of customers in the store and applies a human body skeletal model to the captured images to extract feature points. This camera may be the camera device 506a of the skeleton detection camera sensor unit 104a in the first embodiment. The camera device 506a periodically extracts feature points of customers in the store at a predetermined interval and transmits them to the management server 101. Furthermore, the skeleton detection module 510a of the camera sensor unit 104a may calculate an estimated hand point or the like based on the above-mentioned feature points and transmit the value to the management server 101. Alternatively, the skeleton detection processing module 1102a of the management server 101 may calculate an estimated hand point or the like based on the above-mentioned captured image or feature points transmitted.

[0143] The position coordinates of the camera device 506 and the camera device 506a are known to the management server 101 that receives these outputs. Therefore, the management server 101 can mutually register or convert the position coordinates of the respective captured images. It should be noted that the above-mentioned "position coordinates" can include both the position (x, y, z) in a three-dimensional space and rotation information (θ, φ, ψ) as camera parameters. Therefore, based on the images captured by these cameras, the movement trajectory of each customer and the feature points based on the skeletal model of each customer are periodically transmitted to the management server 101 so that they are associated with each other (see FIG. 11). This information can be stored in any storage device associated with the management server 101.

[0144] As a result, the management server 101 can associate a person ID, time-series position information, and time-series skeletal information with each customer in the store. The skeletal information includes position information of characteristic parts (head, neck, shoulders, elbows, wrists, etc.) obtained by applying a skeletal detection module on a captured image, and position information of characteristic parts (hands, etc.) obtained by calculation using these characteristic parts.

[0145] In FIG. 11, the output from the position detection camera sensor unit 104 and the output from the skeleton detection camera sensor unit 104a are each transmitted separately to the management server 101. However, in other embodiments, the output from the position detection camera sensor unit 104 and the output from the skeleton detection camera sensor unit 104a may be integrated and then transmitted together to the management server 101. For example, the skeleton detection camera sensor unit 104a transmits information on the estimated position coordinates of the skeleton features and the hands to the position detection camera sensor unit 104. After that, the position detection camera sensor unit 104 may convert the information into coordinates, associate it with the coordinate information of the person, and transmit the information to the management server 101.

[0146] Preferably, the position detection camera 104 (camera device 506) and the skeleton detection camera 104a (camera device 506a) are installed in the store as separate cameras (see FIG. 36(A)). Each camera device 506, 506a may capture a subject as a still image (photograph) and acquire, record, or output the still image data. Alternatively, each camera may capture a subject as a video and acquire, record, or output the still image from a file in the video format. A video can be understood as a chronological sequence of multiple still images, so hereinafter, it is assumed that still images are electronically acquired by each camera.

[0147] Each camera device 506, 506a can be modified depending on the embodiment with respect to differences in camera size (small, large, etc.), differences in color information of still images captured by the camera (number of RGB bits, full color, true color, etc.), differences in camera shooting method (normal shooting, infrared shooting, depth camera that obtains depth information, etc.), differences in camera installation position (above, to the side or below within the store, etc.), etc.

[0148] Each camera device 506, 506a has a POV (Point-Of-View) or "viewpoint," and electronically captures a subject within a FOV (Field-Of-View) or "field of view" through the respective POV. Preferably, each camera device 506, 506a has a fixed POV / FOV in advance for each of a plurality of product shelves in order to optimally capture the movement of people around each product shelf. However, each camera device 506, 506a can be configured to have a variable POV / FOV.

[0149] Each camera device 506, 506a may be installed so as to capture a person from the front at approximately the same height. In this case, a skeleton detection model can be applied to the upper and lower halves of a person on the capture screen in a two-dimensional manner approximately evenly. However, the locations where cameras can be installed are limited. Also, as the number of people in the store increases, the POV / FOV of each camera is likely to be obstructed. As a result, there is a risk that the expected accuracy will be significantly reduced. In particular, the impact of people overlapping each other on the captured image (occlusion) and reducing the detection accuracy is significant. It is possible to address this issue by installing multiple cameras in the same space, but in that case, the cost is likely to be high.

[0150] Each camera device 506, 506a may be installed so as to photograph a person from above. In this case, the upper and lower halves of the person appear to have different depths on the photographed screen, so a 3D skeleton detection model is applied. Therefore, when detecting feature points in the upper and lower halves of the person, the difficulty of image analysis may increase. However, in this case, the POV / FOV of each camera above the person is less likely to be obstructed. In addition, the installation location of the cameras can be secured over a wider range. In particular, since it is possible to avoid people overlapping on the photographed image, the detection accuracy can be improved.

[0151] In this embodiment, preferably, the camera devices 506, 506a installed on the ceiling or above the store capture images of the inside of the store looking down (see FIG. 36(A)). Furthermore, when applying the skeleton detection model based on the captured image, the image captured from above and the skeleton coordinates (ground truth data) are given and the model is trained. In this case, a technique called deep learning transfer learning and fine tuning can be used. In addition, as described later, a method is added to make the application of the skeleton detection model as simple as possible. This allows skeleton detection to be performed with a relatively high degree of accuracy based on images captured by each camera device 506, 506a looking down from above. However, each camera device 506, 506a is capable of capturing images of customers from the side, below, or other positions within the store.

[0152] For example, a three-dimensional sensor or a TOF (Time of Flight) sensor that measures the distance to the object may be used as the position detection camera device 506. In this case, the three-dimensional sensor can obtain the distance to the object based on the time it takes for a pulsed light to be projected and for the pulsed light to be reflected by the object and return. When a TOF sensor is used, distance information of the object can be recorded for each pixel (picture element) or for each group of pixels on the shooting screen.

[0153] For example, a depth camera or a three-dimensional camera having a built-in depth sensor for acquiring depth information can be used as the skeleton detection camera device 506a. In particular, it enables three-dimensional recognition including depth on an image captured using a 3D camera. In this case, the 3D camera may be configured with an RGB camera and an infrared camera to acquire depth information in addition to color information, enabling three-dimensional recognition. It is possible to estimate a person's skeleton by applying a skeleton detection module to the captured image.

[0154] In conventional technology, when performing image analysis on a captured image to detect feature points of the human body, multiple feature points are detected over the entire body of a customer (see, for example, Figs. 6A-E, 7A-B, 11, and 14 of Patent Document 1, and Fig. 3 of Patent Document 2, etc.). However, when detecting multiple feature points over the entire body (upper and lower body) of a customer, there are problems in that advance learning processing is difficult, the actual work takes time, and it is prone to errors.

[0155] In particular, when it comes to the legs of the lower body, parts of the legs can be hidden in the image due to changes in posture, such as crossing the front and back legs. For parts that are not visible, there is a risk that there will be a discrepancy between the detection result and the correct answer, making it difficult to progress in learning. Also, when photographing a customer below from a camera installed on the ceiling, unlike when photographing from the front, the lower half of the body is likely to be hidden by the upper half of the body, making it difficult to ensure that the output quality is consistent between the upper and lower half of the body.

[0156] Furthermore, in conventional technology, feature points of the human hand / fingers have been extracted based on a captured image. This is because the movements of the hands and fingers are important in the process of picking up products. For example, Patent Document 2 explains that "the management server 10120 detects the human hand by skeletal analysis performed on the image of the person included in the captured image (see paragraph 0079)." However, because hands and fingers are relatively small and perform complex movements, there have been problems in that advance learning processing is difficult, actual work takes time, and errors are likely to occur.

[0157] In particular, the wrist is a part of the human body that connects the arm to the palm of the hand, and is associated with a joint (radiocarpal joint). In addition, the hand at the end of the wrist usually has five fingers (first to fifth fingers), and each finger is associated with multiple joints (first to fourth joints). Thus, the wrist, hand, and fingers perform complex movements that change their positions in various ways. As a result, there are particular difficulties in applying a skeleton detection module to the captured images to track the fine movements of the hand and fingers.

[0158] Furthermore, when a customer reaches out his / her arm toward a shelf to pick up a product, the movement of the hand or fingers trying to grab the product may be hidden by the shape of other products or the shelf around them. Also, the customer may put their hand or fingers in a pocket of their clothes, which may hide their position. Hands and fingers that are relatively easily hidden are prone to misalignment between the detection result and the correct answer, making it difficult to progress with learning.

[0159] In this embodiment, a control device (for example, the skeleton detection module 510a of the skeleton detection camera sensor unit 104a or the skeleton detection processing module 1102a of the management server 101) applies a skeleton detection module based on a human skeleton model on the screen captured by the camera device 506a. At this time, feature points are detected only on the upper half of the customer's body. The lower half of the customer's body is excluded from the application of the skeleton detection module. In this embodiment, the movement of the human body that contributes to picking up a product is limited to the upper body, and the movement of the lower body is practically negligible. As a result, in this embodiment, the number of targets to which the skeleton detection module is applied is halved, making the pre-learning process easier, speeding up the actual work, reducing the occurrence of errors, and increasing the reliability of the output.

[0160] Furthermore, in this embodiment, when a control device (for example, the skeleton detection module 510a of the skeleton detection camera sensor unit 104a or the skeleton detection processing module 1102a of the management server 101) applies a skeleton detection module on the screen captured by the camera device 506a, the parts of the human body from the wrist forward (hands and fingers) are excluded from the detection target. Instead, the position of the parts from the wrist forward is calculated based on other feature points (for example, shoulders, elbows, wrists) of the customer's upper body extracted by applying the skeleton detection module. In this embodiment, it is possible to estimate the positions of the hands based on the skeletal structure of the human body if some feature points of the upper body (for example, shoulders, elbows, and wrists) can be acquired.

[0161] In this embodiment, a depth camera or a 3D camera is used to capture images of the inside of the store from the ceiling, and it is determined which product on which shelf the user picked up. If the position of the hand or fingers is used to determine whether the user has picked up the product, the hand or fingers will overlap with the shelf when viewed from the camera position, making it impossible to determine the height (the product will be determined to have been picked up at the wrong height). Therefore, the position information of the hand or fingers overlapping the shelf is not used, and the hand position is calculated and found from the position coordinate information of the wrist, elbow or any part of the arm that does not overlap the shelf. As a result, the advance learning process can be made easier, the actual work can be completed more quickly, and the occurrence of errors can be reduced, resulting in increased reliability of the output.

[0162] For example, by applying a skeletal detection module to the screen captured by the camera device 506a of the camera sensor unit 104a, feature points are detected at two positions on the customer's upper body, the elbow and wrist of one of the arms used to pick up the product, and other feature points such as the hands and fingers are determined by calculation based on these positions. Preferably, a skeleton detection module is applied to the screen captured by the camera device 506a of the camera sensor unit 104a to detect feature points at four positions on the customer's upper body, namely the left and right elbows and the left and right wrists, and other feature points of the hands, fingers, etc. are found by calculation based on these positions.

[0163] More preferably, a skeletal detection module is applied to the screen captured by the camera device 506a of the camera sensor unit 104a to detect feature points at six positions on the customer's upper body, namely the left and right shoulders, the left and right elbows, and the left and right wrists, and other feature points of the hands, fingers, etc. are found by calculation based on these positions. More preferably, by applying a skeletal detection module to the screen captured by the camera device 506a of the camera sensor unit 104a, feature points are detected at seven positions on the customer's upper body, namely the head, left and right shoulders, left and right elbows, and left and right wrists, and other feature points such as the hands and fingers are found by calculation based on these positions. More preferably, by applying a skeletal detection module to the screen captured by the camera device 506a of the camera sensor unit 104a, feature points are detected limited to a total of eight positions on the customer's upper body, namely the head, neck, left and right shoulders, left and right elbows, and left and right wrists, and other feature points such as the hands and fingers are found by calculation based on these positions.

[0164] Referring to (A) of Fig. 37, the feature points of the human body when the skeleton detection module applied in the conventional technology is applied are illustrated. In this example, 18 feature points are detected from the upper and lower halves of the human body. There are particular difficulties in detecting these feature points from actual photographed images, and errors often occur when actually applied. 37B shows an example of feature points of the upper half of the human body in this embodiment. In this example, it is preferable to detect a maximum of eight positions (thick lines) of the human body, including the head (1), neck (2), left and right shoulders (3, 6), left and right elbows (4, 7), and left and right wrists (5, 8), and calculate the positions of other parts of the human body (hands, neck, etc.). Preferably, each feature point is based on pixel coordinates in the captured image.

[0165] Preferably, when detecting the position of the head (1), small facial features such as the eyes, ears, nose, etc. are excluded to make it easier to detect the head (1). Preferably, the position of the neck (2) is calculated based on the positions of the extracted features (head (1), left and right shoulders (3, 6), left and right elbows (4, 7), left and right wrists (5, 8)). Preferably, based on the positions of the extracted feature parts (head (1), left and right shoulders (3, 6), left and right elbows (4, 7), left and right wrists (5, 8)), the positions of the left and right hands beyond the left and right wrists (5, 8) (estimated hand positions) are calculated. Preferably, the positions of the chest and back torso are excluded from the detection target, but it is possible to detect or calculate the front-to-back direction of the human body.

[0166] In particular, by extracting the position of the elbow (4, 7) and the position of the wrist (5, 8), the positions of the arm, forearm, and upper arm can be calculated. In this case, the "arm" refers to the part of the human body from the shoulder (3, 6) to the wrist (5, 8). Moreover, the term "forearm" refers to the part of the human body closer to the wrist (5, 8) when the arm is divided into two sections by the elbow (4, 7). Moreover, the term "upper arm" refers to the part of the human body closer to the shoulder (3, 6) when the arm is divided into two sections by the elbow (4, 7). The "forearm" and "upper arm" can be determined as a straight line based on the position coordinates in three-dimensional space of the shoulder (3, 6), elbow (4, 7), and wrist (5, 8). Preferably, the "forearm" and "upper arm" are calculated as simple straight lines based on the skeletal structure of the human body. However, it is possible to calculate the "forearm" and "upper arm" as arcs or curves based on the skeletal structure of the human body.

[0167] Furthermore, the "forearm center point" can be calculated as the center position that divides the length of the forearm from the elbow (4, 7) to the wrist (5, 8) into two equal parts. Similarly, it is possible to calculate the "upper arm center point" as the center position that divides the length of the upper arm from the elbow (4, 7) to the shoulder (3, 6) into two equal parts. However, since the parts of the hand and fingers play an important role in the product purchasing process, in this embodiment, the "forearm" and the "center point of the forearm" are calculated. Instead of finding the "forearm center point," the length of the forearm from the elbow (4, 7) to the wrist (5, 8) may be divided into three equal parts, and the part closest to the wrist (5, 8) may be found by calculation. Also, instead of finding the "forearm center point," the length of the forearm from the elbow (4, 7) to the wrist (5, 8) may be divided into three equal parts, and the part closest to the shoulder (3, 6) may be found by calculation. Also, instead of finding the "forearm center point", the length of the forearm from the elbow (4, 7) to the wrist (5, 8) may be divided into a number of intervals, and any interval may be found by calculation.

[0168] Furthermore, the "neck (2)" can be determined as a part of the human body that connects the head and the trunk. In this embodiment 2, the position of the neck (2) may be calculated based on the positions of the left and right shoulders (3, 6). Since the length (shoulder width) of the left and right shoulders (3, 6) usually does not change significantly due to the movement of the human body, the position of the neck (2) in the horizontal direction can be determined by determining the intermediate position coordinates in three-dimensional space. Furthermore, the position of the neck (2) in the vertical direction can be determined based on the positions of the left and right shoulders (3, 6). For example, the position may be the horizontal center position of the positions of the left and right shoulders (3, 6) and a position lifted a predetermined distance upward. In this case, the position of the neck (2) may be determined using the position of the head (1) and the positions of the left and right shoulders (3, 6).

[0169] The head (1) may be determined as the part of the human body that is located at the highest position and has a predetermined size. This may allow the position of the head (1) to be determined uniformly regardless of, for example, changes in hairstyle or differences such as the presence or absence of a hat. Furthermore, since details such as the face are excluded from the detection target, the position coordinates of the head (1) can be obtained relatively easily. When determining the position of the head (1), it is possible to distinguish between the front (the direction of the face) and the rear (the direction of the non-face).

[0170] During the product pick-up process, the position of the hand at the tip of the wrist is particularly important. "Wrist (5, 8)" refers to the part of the human body that connects the arm to the palm of the hand. "Hand" refers to the part of the human body from the wrist (5, 8) to the fingertips. The hand, which is an organ at the end of the arm, is not included in the arm. In this embodiment, the estimated positions of the hands can be calculated based on the positions of the elbows (4, 7) and wrists (5, 8) that are directly determined by image analysis performed on the captured images.

[0171] The size of a hand can be changed by opening and closing it. Therefore, the estimated point of the hand changes depending on which position of the hand is used as a reference. In this embodiment, the estimated point of the hand is obtained based on an actual product purchasing process. For example, the estimated point of the hand may be obtained as the center position of the palm, focusing on the action of grabbing the product. Alternatively, the estimated point of the hand may be obtained as the tip of the fingers when the hand is opened, focusing on the maximum reach of the product. Alternatively, the estimated point of the hand may be obtained at another position, focusing on the action of picking up the product.

[0172] For example, the "estimated hand point" may be determined as a position obtained by extending the length of the forearm from the elbow (4, 7) to the wrist (5, 8) by a specified multiple when a straight line is applied along the direction of the forearm, based on the position of the elbow (4, 7). By determining the "estimated hand point" at a position that is extended from the wrist by a specified multiple along the forearm, the approximate center point (or tip) of the hand when the customer actually stretches his / her arm toward the shelf and is about to grab a product is obtained. Therefore, regardless of various movements of the wrist, fingers, etc., the reference position of the hand trying to grab a product is obtained in a unified manner.

[0173] The above-mentioned "predetermined multiple" is a value arrived at by the applicant through trial and error after various tests. For example, the estimated hand point is a position that is 4 / 3 times the length of the forearm from the elbow (4, 7) to the wrist (5, 8) in the direction of the forearm, based on the elbow (4, 7). For example, the estimated hand point is a position that is 1.33 times the length of the forearm from the elbow (4, 7) to the wrist (5, 8) in the direction of the forearm, based on the elbow (4, 7). For example, the estimated hand point is a position that is 1.3 times the length of the forearm from the elbow (4, 7) to the wrist (5, 8) in the direction of the forearm, based on the elbow (4, 7).

[0174] As described above, when picking up a product, the size of the hand varies. Therefore, the predetermined multiple can change depending on how the reference point of the hand is set. For example, the predetermined multiple can change depending on whether the reference point is the center position of the palm or the tips of the fingers when the hand is opened. In addition, the proportion of the hand that is hidden when purchasing a product can change depending on the size and shape of the product shelves in the store in the embodiment. Therefore, the above "predetermined multiple" can be modified depending on the embodiment.

[0175] For example, the estimated hand point may be modified within a range of about ±5%, ±10%, ±15%, ±20%, or ±25% with respect to a position obtained by extending the forearm length from the elbow to the wrist by "4 / 3", "1.33", or "1.3" times along the direction of the forearm, based on the elbow. In addition, it is possible to modify the estimated hand point by a smaller or larger percentage than the "predetermined multiple" depending on the embodiment.

[0176] The estimated point of the hand may also be found by extending the length from the elbow to the "forearm center point" by a predetermined multiple (for example, 8 / 3, 2.67, 2.7, etc.). Additionally, the estimated point of the hand may be found by extending the length from the "forearm center point" to the wrist by a predetermined multiple (eg, 8 / 3, 2.67, 2.7, etc.). In other words, for example, if the distance from the elbow to the center of the forearm is 1, the distance from the elbow to the wrist is 2, and the distance from the wrist to the hand is 0.67 (elbow to wrist / 3), then the distance from the elbow to the hand is 2.67. In these cases, the predetermined multiple and the percentage of correction can be determined in the same manner as for increasing the length of the forearm.

[0177] Similarly, an estimated point of the hand may be found by extending the length from "any position on the arm" to the wrist by a predetermined multiple. Furthermore, the estimated point of the hand may be obtained as a single point in a three-dimensional space, or may be obtained as an arbitrary extent (width / size) in the three-dimensional space. For example, the estimated point of the hand may be obtained as an extent from the fingertips to the center of the palm.

[0178] In this embodiment, the "estimated hand points" of a customer can be determined in a unified manner, without being affected by differences in gender, height, or age. That is, anatomically, the difference between men and women is considered to be small with respect to bone structure. If people are the same height, the shoulder width and hand size are considered to be almost the same regardless of gender. Therefore, this embodiment can obtain the "estimated hand position" regardless of the gender of the customer.

[0179] Anatomically, the size of the hand tends to be proportional to the height. Generally, taller people tend to have longer forearms and larger hands. Similarly, shorter people tend to have shorter arms and smaller hands. This can be similarly applied to the relationship between adults and children. Therefore, this embodiment can find the "estimated hand point" regardless of differences in height and age of the customer.

[0180] Furthermore, in this embodiment, as described later, some effort is made to enable the height position coordinates associated with the "estimated hand point" to be acquired with relatively high accuracy. Therefore, in this embodiment, since the "estimated hand point" is calculated based on feature points detected only on the upper body of the human body, the "estimated hand point" can be calculated for a customer who comes to the store in a wheelchair, in the same way as for other customers who are walking normally. Therefore, this embodiment can obtain the "estimated hand points" of a customer without being affected by the difference between an able-bodied person who walks normally and a disabled person who moves around in a wheelchair.

[0181] Furthermore, in this embodiment, as described above, the "estimated hand point" can be obtained regardless of the state of the lower body of the human body. Therefore, this embodiment can obtain the "estimated hand point" of the customer without being affected by the difference between a healthy person who walks normally and a disabled person who moves around using crutches. Therefore, this embodiment can uniformly determine the "estimated hand point" of the customer under substantially most circumstances.

[0182] Furthermore, in this embodiment, by extracting feature points (head, shoulders, elbows, wrists) limited to the upper body of a person, perspective deviations on the image are minimized compared to the case where multiple feature points are extracted from the whole body. Furthermore, in this embodiment, parts that are easily hidden are excluded from the detection target, and image analysis is performed by focusing on feature points that are easy to detect, thereby minimizing errors in image analysis.

[0183] For example, in the conventional technology, when a person below is photographed from a camera installed near the ceiling of a store, the perspective difference between the upper and lower halves of the body is different from that when the person is photographed from the front, and the lower half of the body tends to appear smaller than the upper half. Also, the lower half of the body is sometimes hidden by the upper half. In this embodiment, such inconvenience is minimized.

[0184] 38(A) and (B) show an example of applying a skeleton detection model to a captured image. In this example, an example of applying a skeleton detection model to each person is shown based on an image of a person below captured by a camera (e.g., camera device 506a of camera sensor unit 104a) installed near the ceiling of a store. In FIG. 38(A), when a skeleton detection model is applied to a captured image, multiple feature points (e.g., 18 points) are detected from the entire body based on the skeletal features of the human body (see FIG. 37(A)). In this example, it can be seen that there is a large fluctuation in detection accuracy at the feet in particular. It can also be seen that detection is difficult for the finer parts of the hand and fingers from the wrist down.

[0185] In FIG. 38(B), when applying a skeleton detection model to a captured image according to this embodiment, a more limited number of feature points (e.g., 8) are detected, limited to the upper body (see FIG. 37(B)). In this embodiment, the detection of human body feature points is limited to parts that are relatively easy to detect, thereby avoiding a decrease in the accuracy of application of the skeleton detection model. Furthermore, the positions of the hands and fingers are calculated based on the detected feature points, making it possible to obtain positions that cannot be obtained by simply applying the skeleton detection model.

[0186] An example of a skeleton detection model applied to a captured image is shown in Fig. 39. In this way, when a normal skeleton detection model is applied, the wrist and beyond may not be output. If the forearm direction is detected only up to the wrist position, the distance between the product and the product is too large. As a result, in the conventional technology, when the product pick-up decision is made based on the wrist position, detection errors and setting errors are likely to occur. As a result, there is a high risk of making a wrong product pick-up decision between other customers nearby. In this embodiment, the estimated hand points are found by calculation, minimizing the occurrence of such a risk.

[0187] Furthermore, in this embodiment, some effort is made to enable the height position coordinates related to the "estimated hand point" to be acquired with relatively high accuracy. Referring to FIG. 40(A), there is illustrated a case where the body part beyond the wrist is hidden by the top board of a product shelf when the skeleton detection model is applied. If an attempt is made to detect the position of the wrist in this state, the wrist may be mistakenly detected on a part of the product shelf (such as the top board) as shown in (B) of Fig. 40. As a result, when an attempt is made to obtain the position coordinate of the wrist in the height direction (Z-axis direction) at the pixel position on the image, a deviation in the detection target may occur. In this way, when the camera captures an image diagonally downward from the ceiling in a three-dimensional space, particularly in the Z-axis coordinate (depth information), the position information of a part of the product shelf (such as the top plate) may be erroneously obtained instead of the position coordinate of the wrist. In such a case, there is a risk that the position coordinate of the wrist cannot be obtained accurately.

[0188] In this embodiment, when a customer approaches a product shelf during the product purchasing process, the position of the hand or wrist can be obtained by using the position between the elbow and wrist instead of the position of the wrist. For example, a "forearm center point" may be obtained as the center position that divides the length of the forearm from the elbow to the wrist in half, and the depth may be obtained at that position.

[0189] In the actual product purchasing process, when trying to grab a product on a shelf, not only the hand and fingers but also part of the wrist or forearm may be hidden. However, even in such cases, it is rare for the "center point of the forearm" to be hidden by the shelf. Therefore, in this embodiment, for example, when a customer approaches a product shelf, the position of the "center point of the forearm" may be obtained as wrist position information (particularly the depth).

[0190] However, the position from which the depth is obtained may vary based on the size of the boards on each shelf of the product shelves, etc. Therefore, in this embodiment, the position of the wrist in the Z-axis direction is not limited to the position of the "center point of the forearm." For example, the length of the forearm from the elbow to the wrist may be divided into thirds to determine the position on the side closer to the wrist or the side closer to the shoulder, and the position in the Z-axis direction may be obtained at that position. Alternatively, an arbitrary position along the length of the forearm from the elbow to the wrist may be found, and the position in the Z-axis direction at that position may be obtained.

[0191] Furthermore, with regard to the position in the Z-axis direction, position information of the "center point of the forearm" may be obtained, and position information of other parts of the upper body, such as the shoulder, the center point of the upper arm, and the elbow, may be obtained, and these values ​​may be used in combination. Therefore, in this embodiment, when a hand is placed inside any shelf of a product shelf, even if the part beyond the wrist that is attempting to grasp the product is hidden by a top plate or the like above, the position in the Z-axis direction can be determined relatively accurately.

[0192] 41, a process for determining the estimated hand points in this embodiment is illustrated. This process can be performed by the skeleton detection module 510a of the skeleton detection camera sensor unit 104a (see FIG. 5B). First, the skeleton detection module 510a receives an image captured by the camera device 506a as an input (step S1). This captured image is, for example, an RGB image.

[0193] Next, the skeleton detection module 510a acquires the coordinates (X, Y coordinates) of the skeleton for a predetermined part based on the captured image (step S2). For example, the (X, Y coordinates) of the elbow and wrist are acquired. Next, the skeleton detection module 510a calculates the (X, Y coordinates) of the "center point of the forearm" based on the (X, Y coordinates) of the elbow and wrist (step S3). Next, the skeleton detection module 510a obtains the depth (Z coordinate, etc.) at the "forearm center point" and / or the position of the elbow (step S4).

[0194] Next, the skeleton detection module 510a extends the length of the forearm by a predetermined multiple using the elbow as a reference (step S5). For example, the skeleton detection module 510a obtains the length from the elbow to the "forearm center point" (or from the "forearm center point" to the wrist) and extends that length by 2.67 times. As a result, the skeleton detection module 510a calculates the coordinates (X, Y coordinates) of the "estimated hand point". Next, the skeleton detection module 510a obtains the Z coordinate (depth) of the elbow and the "center point of the forearm" (step S6), and similarly multiplies this by 2.67 to calculate the Z coordinate of the "estimated point of the hand". In this way, the skeleton detection module 510a calculates the position coordinates (X, Y, Z coordinates) in three-dimensional space of the "estimated hand point" from the coordinates of the elbow and the "forearm center point".

[0195] In addition, multiplying the length from the elbow to the "center point of the forearm" by 2.67 (2 and 2 / 3) means that the "estimated point of the hand" is located 1.67 (1 and 2 / 3) times the length from the elbow to the wrist. It is possible to calculate the estimated position of the hand from the X, Y, and Z coordinates, including the Z coordinate, from the beginning, but if the hand is inside the shelf, the Z coordinate (or depth) of the hand cannot be accurately calculated from a camera taken from the ceiling, resulting in errors. Therefore, in this embodiment, even if the hand is inside the shelf, the Z coordinate (or depth) of the photographed elbow and the "center point of the forearm" are obtained, and the X, Y, and Z coordinates of the hand are calculated by extending the length between them.

[0196] As described above, in this embodiment, by applying a skeleton detection model limited to the upper body of a human body based on an image actually captured by the camera device 506a of the skeleton detection camera sensor unit 104a installed in a store, the skeleton detection module 510a can detect eight feature points of the head, neck, shoulders, elbows, and wrists. Various learning or training processes are performed in advance for the skeleton detection model to be applied to the captured images.

[0197] In this embodiment, image analysis is performed by focusing on easily detectable feature points (head, neck, shoulders, elbows, wrists, etc.) based on people's movements in a store (walking, etc.) and the actual product purchasing process, making the pre-learning process easier. At this time, in this embodiment, the movement of the human body is taken into consideration. For example, the human body can perform various movements such as bending the waist, leaning the body, moving the arms up and down, moving the arms back and forth, and folding the arms. In this embodiment, the learning work is performed by concentrating on the movements of people expected in the store, making the operation easy. For example, normal walking postures and postures with a hunched back in the store are mainly learned, and special movements such as somersaults are excluded from the learning content in advance.

[0198] During the learning stage, the skeleton detection module 510a of the camera management terminal 530a may perform image analysis of still images based on characteristic shapes, figures, contours, lines, vertices, colors, etc., particularly of the head, shoulders, elbows, and wrists of the human body. In this case, the skeleton detection module 510a may classify the received still image for each pixel or each pixel group, and may identify the shape, figure, outline, line, vertex, color, etc. for each pixel or each pixel group to identify characteristic targets related to the head, shoulders, elbows, and wrists.

[0199] The skeleton detection module 510a is preferably trained in image analysis on various images in advance by machine learning using AI. In particular, based on various concrete examples, image analysis is actually performed to train the module to identify and extract targets (shapes, figures, contours, lines, vertices, colors, etc.) from images. The training is performed, for example, on images of hundreds, thousands, or more different targets, and various results are accumulated. At that time, statistical data may be calculated and generated for the results. After a high identification rate, for example, an identification rate of about 99% or about 99.9% or more, is obtained by training, image processing may actually be performed to identify targets from images.

[0200] In the training carried out by the AI ​​machine learning, various image information may be used as teacher data. For example, a judgment model may be generated by machine learning in which a still image is used as input and information on the shape, figure, outline, line, vertex, color, etc. of a characteristic object is used as output. Training may take into account the state of the store at various times of the day, for example, learning the difference in how shadows are cast during the day when light is shining into the store and at night when only the store's interior lights are on, or learning about various states of the store when there are no people in the store, when there are only a few people in the store, when there are many people in the store, etc. A new still image may be input to this determination model so that a characteristic target object can be obtained as the output.

[0201] During actual skeleton detection, skeleton detection module 510a may generate a confidence score for each feature point based on data accumulated during a prior training session. For example, when the detected position matches the previously learned position at an almost equal rate (about 100% or about 99%), a high reliability may be assigned.Alternatively, the greater the deviation of the detected position compared to the previously learned position, the lower the reliability may be assigned.

[0202] For example, an image may be used as input, and the positions (pixel coordinates in the image) and reliability (a value between 0 and 1, or between 1 and 10, where the higher the value, the higher the expected accuracy of the position) of feature points (head, neck, shoulders, elbows, wrists, etc.) of each person appearing in the image may be displayed. The latter may be displayed by changing the color of the joint positions in the image. For example, green may indicate high reliability, yellow may indicate medium reliability, and orange may indicate low reliability. By displaying the reliability using color coding or color gradation, it may be possible to quickly visually determine which joint positions are highly reliable.

[0203] In this embodiment, the camera device 506a of the skeleton detection camera sensor unit 104a installed on the ceiling or above the store captures the customer from above, minimizing obstruction between the subject and the camera (see FIG. 36(A)). However, even in such a case, depending on the customer's position and how the customer overlaps with other customers, there may be cases where parts of the customer's body, such as the arms or shoulders, are not completely captured.

[0204] On the other hand, in this embodiment, preferably, multiple camera devices 506a are installed so that each product shelf installed in the store can be photographed in the best possible way (see FIG. 36(A)). Therefore, it is possible that at least two camera devices 506a may photograph customers near the same product shelf from different POVs / FOVs at almost the same time. Therefore, if the image captured from only one camera device 506a is not sufficient to detect a specific bone structure of a customer's upper body, the bone structure detection module 510a may use the image captured from another camera device 506a to fill in the missing parts.

[0205] Referring to (A) and (B) of FIG. 43, images of the same customer in a store taken from two different POV / FOVs of the camera device 506a are illustrated. In FIG. 43(A), a customer is photographed from the front facing a product shelf with one arm down. FIG. 43(B) shows a customer facing a shelf from the side, with one arm down and the other bent. Ideally, it is preferable to photograph a customer facing the product shelf from the front as shown in FIG. 42(A), but depending on the customer's standing position, orientation, etc., it may happen that the entire upper half of the customer's body is not necessarily photographed. In such a case, an image photographed by another camera device 506a that photographed the same customer at approximately the same time may be used as a supplementary image.

[0206] For example, assume that the same person is photographed at approximately the same time by two camera devices 506a. The position coordinates of each camera device 506a are known to the camera management terminal 530a that receives each photographed image, and the position coordinates on each photographed image can be registered or converted to each other. For example, the skeletal features of the person photographed in (A) of Fig. 43 are used as a reference, and for missing parts (for example, when the feature parts of one arm side can be detected but the feature parts of the other arm side cannot be detected), the skeleton detection module 510a uses the skeletal features of the person photographed in (B) of Fig. 43. At that time, the position coordinates of the feature parts in (B) of Fig. 43 may be converted to position coordinates in (A) of Fig. 43, and the positions of the feature parts of the two photographed images may be superimposed in the three-dimensional space of (A) of Fig. 43.

[0207] 44(A) and (B) show an example in which the positions of the skeletons detected based on the images captured by the two camera devices 506a are grouped together on the same coordinate system. For example, when the position coordinates of each joint detected from each image can be overlapped with each other, each joint is linked to the same joint.

[0208] 44(A), the skeleton detection module 510a groups the position coordinates of the other joints based on the coordinates of the head among the joints detected by each camera device 506a. The skeleton detection module 510a then overlaps the coordinates of the head, neck, shoulders, elbows, and wrists detected by each camera device 506a to associate the position coordinates of the joints detected by each camera with each other. In this case, when the head position coordinates acquired from the two camera devices 506a overlap each other within a predetermined threshold, for example, when the overlap is within a range of 100% to 99%, or 100% to 95%, or 100% to 90%, the two are linked to each other. The same applies to the neck, shoulders, elbows, wrists, etc.

[0209] FIG. 44B illustrates an example in which the skeleton detection module 510a consolidates the position coordinates of the respective feature parts acquired from the two camera devices 506a into one position coordinate. At this time, the skeleton detection module 510a determines a priority order in advance when combining the position coordinates in the two three-dimensional spaces into one. For example, a feature based on a still image taken from the front of the product shelf may be given priority over a feature based on a still image taken at an angle to the product shelf (for example, from the side). In this way, in this embodiment, if a portion that could not be completely captured from the POV / FOV of one camera device 506a can be captured from the POV / FOV of the other camera device 506a, the skeleton detection module 510a can complement the missing portion with the captured portion.

[0210] Therefore, in this embodiment, in the process illustrated in FIG. 41, the skeleton detection module 510a may detect features of the customer's upper body by applying the skeleton detection module based on one captured image obtained from one camera device 506a, and calculate an "estimated hand point." Alternatively, in this embodiment, in the process illustrated in FIG. 41, the skeleton detection module 510a may detect features of the customer's upper body by applying the skeleton detection module based on a combination of two or more captured images obtained from two or more camera devices 506a, and calculate an "estimated hand point." Therefore, in this embodiment, it is possible to obtain an "estimated hand point" that serves as a reference when a customer picks up a product under various circumstances.

[0211] Next, the determination of an event of picking up a product, which is performed using the "estimated hand point", will be described. "Picking up" refers to the act of picking up one or more products displayed on a shelf in a store with the customer's hand. The shape of the hand and the position of the fingers at that time are optional, and the direction or height from which the products are picked up are also optional. The position coordinates of the "estimated hand point" acquired by the skeleton detection module 510a of the camera management terminal 530a are transmitted to the management server 101. Alternatively, the information may be integrated with the information acquired by the position detection module 510 of the camera management terminal 530 and then transmitted to the management server 101. By acquiring the position of the "estimated hand point," the management server 101 can determine the relative relationship between the already known positions of the product shelves (product positions) and the position of the "estimated hand point" in three-dimensional space.

[0212] The above determination is made based on the degree of proximity of the two three-dimensional coordinates within the same three-dimensional space. For example, if it is determined that the position of the "estimated hand point" overlaps with the position of the product on the shelf (product position) within a range of 100% to 90%, or within a range of 100% to 95%, or within a range of 100% to 99%, it can be determined that a hand is present near a product on the shelf. Or, for example, a predefined threshold associated with each item on a shelf may be used to determine that a hand is present near the item if an "estimated hand point" is detected above that threshold. In these cases, the customer number determination module 1105 of the management server 101 can estimate / assume the occurrence of an event in which a product is picked up. As described above, the output of the weight sensor device 608 associated with the product shelf is transmitted to the customer number determination module 1105 of the management server 101. Therefore, the management server 101 can determine with high accuracy whether or not an event of actually picking up a product has occurred by further combining and using the output value from the weight sensor device 608.

[0213] That is, in this embodiment, a weight sensor device 608 is incorporated in each level of the product shelf to detect the weight of the product placed on each rod, board, or shopping basket (see Fig. 7, Fig. 8, etc.). The customer number determination module 1105 of the management server 101 periodically receives the output of the weight sensor device 608 at a predetermined interval. Normally, the weight of the product itself does not change over time. Therefore, as long as the weight sensor device 608 is operating according to the specifications, the customer number determination module 1105 of the management server 101 can constantly track whether or not a product is placed on that product shelf.

[0214] For example, if there are two products weighing 50g each on a shelf, and the initial value is set to 0g (assuming that a stick, board, or shopping basket weighs 0g), the weight of the shelf will be detected as 100g. Suppose that the weight of the relevant product shelf changes at or around the time when the "estimated hand point" overlaps with the position of the product on the product shelf (for example, a value of 100g changes to 50g). In this case, based on these two pieces of data, the customer number determination module 1105 of the management server 101 can determine that the product whose weight has changed (50g=100g-50g) has been picked up by the customer associated with the "estimated hand point."

[0215] The customer number determination module 1105 of the management server 101 can refer to the weight data periodically (by loop processing) acquired by the weight sensor device 608 on each shelf of the product shelf as “weight time-series data (part of weight information).” For example, the weight sensor may acquire the weight data on each shelf of the product shelf at 100 ms intervals. As described above, the camera devices 506, 506a are provided near the product shelves, and can periodically (by loop processing) capture images of the surroundings of the product shelves. The cycle in which the weight sensor device 608 detects weight and the cycle in which the camera devices 506, 506a capture images do not need to match, but should be short enough to sequentially track the movement of people within the store.

[0216] The camera device 506a may constantly capture images of people near the product shelves periodically at predetermined intervals. Alternatively, when the camera device 506a detects that a person has approached near a product shelf, the camera device 506a may take an image of the person using the output as a trigger. Alternatively, when the camera device 506a detects that a person has approached near a product shelf and has stretched out his / her arm toward the product shelf, the camera device 506a may take an image of the person using the output as a trigger. In each case, the camera management terminal 530a or the management server 101 may calculate the position coordinates (X, Y, Z coordinates) of the above-mentioned "estimated hand point" in three-dimensional space, triggered by receiving an input of a captured image transmitted from the camera device 506a.

[0217] In this embodiment, the process (flow) of determining the number of customers performed by the customer number determination module 1105 of the management server 101 has a flow consisting of the following steps. The customer number determination module 1105 measures or calculates in advance the distance between the average customer's hand position and a product segment or product on the shelf (or a threshold value at which it can be determined that a hand has approached a product) and stores the value (pre-processing). The customer number determination module 1105 receives images captured by the camera device 506 of the position detection camera sensor unit 104, tracking the position of each customer within the store. The customer number determination module 1105 receives images of the appearance of each customer in the store captured by the camera device 506a of the skeleton detection camera sensor unit 104a. The customer number determination module 1105 applies a skeleton detection processing module to the captured image from the camera device 506a and obtains the position coordinates of the estimated hand points calculated based on specific features of the detected upper body of the customer.

[0218] Furthermore, the customer number determination module 1105 may combine the output from the position detection camera sensor unit 104 and the output from the skeleton detection camera sensor unit 104a in one three-dimensional space. This information is aggregated into one person ID for each customer and accumulated in chronological order. For example, the customer number determination module 1105 may use the calibration information to convert the position of each customer's hand (in the skeleton detection information) from pixel coordinates in the image to coordinates in the position tracking space. Furthermore, the customer number determination module 1105 may match the position of each customer in the position tracking information with the position of each customer in the skeleton detection information to determine the person ID of the customer in the skeleton detection information.

[0219] Furthermore, the customer number determination module 1105 knows the position coordinates of each product shelf in the store. Therefore, it is possible to determine which customer in the store is reaching out to which product shelf. For example, the management server 101 calculates the distance between the estimated hand point and the product segment or product position (hereinafter, hand segment distance). Furthermore, the customer number determination module 1105 receives weight time series data for each product shelf displaying products, and continuously analyzes the time series changes in the weight of the products.

[0220] Referring to Figure 62, another example is shown in which the output of the skeleton detection camera sensor unit 104a, the output of the position detection camera sensor unit 104, and the output of the weight sensor unit 105 are transmitted to the customer number determination module 1105 of the management server 101. The skeleton detection camera sensor unit 104a captures an RGB image and a depth image by capturing images of the inside of the store. These images can be processed by the skeleton detection module 510a on the edge device side.

[0221] The following process may be configured so that the image is transmitted from the edge device to the management server 101 (cloud side) and then the process is performed on the server side. First, in step X1, a skeleton detection model is applied to the RGB image to obtain the X and Y coordinates of predetermined features (all or any part of the head, neck, shoulders, elbows, and wrists, etc.) based on a skeleton model of the human body.

[0222] On the other hand, in step X2, the depth image is subjected to coordinate conversion distortion correction, so that the depth image can be associated with the RGB image. In step X3, the two are combined to integrate the X, Y coordinates of a specific feature with the depth information (Z coordinate). Furthermore, in step X4, by applying coordinate conversion distortion correction to this result, the distortion of the X and Y coordinates on the RGB image can be corrected. As a result, the X and Y coordinates of a predetermined feature and depth information (Z coordinate) can be obtained as camera coordinates. Furthermore, in step X5, by performing a predetermined calculation based on this position information, the coordinates of the estimated position of the hand can be obtained. Therefore, finally, the X, Y, and Z camera coordinates of a predetermined feature of the skeleton and the estimated point of the hand can be obtained.

[0223] As shown in step Y1, the position detection camera sensor unit 104 can obtain an ID and X, Y, and Z coordinates as ToF coordinates for a person by capturing an image inside the store. This image is processed by the position detection module 510 on the edge device side such as a PC. Meanwhile, in step Y2, the position detection camera sensor unit 104 can receive an output from the skeleton detection camera sensor unit 104a and obtain the X, Y, and Z camera coordinates of a predetermined characteristic part of the skeleton and an estimated point of the hand.

[0224] Furthermore, in step Y3, by applying coordinate conversion correction to this result, the camera coordinates can be associated with the ToF coordinates. Furthermore, in step Y4, by combining the outputs of steps Y1 and Y3 in the same space, the skeleton and the person can be associated. Therefore, finally, the X, Y, and Z coordinates as the ToF coordinates of the estimated points of the ID, head, and hand of the person (customer) can be obtained.

[0225] Furthermore, the weight sensor device 608 of the weight sensor unit 105 acquires weight data in a time series manner. As shown in step Z1, this data is processed by a module 610 on the edge device side, such as a Raspberry Pi (registered trademark). The customer number determination module 1105 of the management server 101 receives the previously acquired product positions on the product shelves in the store, the person (customer) ID, the X, Y, and Z coordinates as the ToF coordinates of the estimated head and hand points acquired from the position detection camera sensor unit 104, and the time-series weight data acquired from the weight sensor device 608. Therefore, by using these data in combination, it is possible to determine whether a customer has picked up a product.

[0226] Therefore, when a change occurs in which products are picked up consecutively within a short period of time, the customer number determination module 1105 can determine how many products were picked up at what time and by whom. Therefore, in this embodiment, when multiple customers simultaneously or nearly simultaneously pick up one or more items that are close to each other, it is possible to determine which customer picked up which item and how many of them. Hereinafter, the determination of whether or not a product has been picked up will be described in more detail using the output of the weight sensor device 608 and the estimated hand point.

[0227] Referring to FIG. 45, an example is shown in which the change in output of the weight sensor device 608, which is sent to the customer number determination module 1105, is overlaid with the change in distance between the estimated point of the customer's hand and the shelf. An example of weight time-series data for each product shelf is shown in a graph at the top of Fig. 45. The horizontal axis of this graph represents time (ms), and the vertical axis represents the value (g) recorded by the weight sensor device 608.

[0228] At the bottom of Fig. 45, a graph is shown showing the change in distance between the position of the estimated hand point of each of two customers (users) near a product shelf equipped with the weight sensor device 608 and the shelf. The horizontal axis of this graph shows time (ms), and the vertical axis shows the change in distance (mm) between the estimated hand point of each user and the shelf (or product) (correlation between the estimated hand point and the product shelf position). When the distance between the estimated hand point and the shelf falls below a threshold, the customer number determination module 1105 determines that there is a possibility that a product has been picked up in that area. Each user is assigned a person ID (identification number) 1 or 2.

[0229] 45, it can be seen that there is no change in the output of the weight sensor device 608 between t0 and t2 (ms). Therefore, the customer number determination module 1105 can infer that there is no change in the products on the product shelves during this period. Between t2 and t4 (ms), a drop in the output of the weight sensor device 608 is recorded. During this period, a drop of 50 g in the output of the weight sensor device 608 is finally recorded. Therefore, the customer number determination module 1105 can infer that one or more items equivalent to this difference in weight (50 g) have been picked up.

[0230] It can be seen that the output of the weight sensor device 608 maintains a significantly reduced value between t4 and t7 (ms). Therefore, during this period, the customer number determination module 1105 can infer that no changes have occurred to the products on the shelves. In the graph, the output of the weight sensor device 608 is shown as a straight line for simplification. In reality, when a product that is hooked on a bar on a store shelf is picked up, a load is applied to the bar, so the output of the weight sensor does not form a straight line. The same is true when a product is placed on a shelf board or a shopping basket.

[0231] In this embodiment, the customer number determination module 1105 makes the determination by combining the hand segment distances of the users 1 and 2 and the time of the weight time-series data. In this example, it can be seen that the estimated hand points of user 1 were away from the product shelves until around time t1, but the estimated hand points got closer to the product shelves between around time t1 and t5. Time t1 is the time when, after User 1 begins to extend his / her hand towards the product shelf, it is determined that the distance between the hand and the shelf falls below (or exceeds) a predetermined threshold and the hand approaches the product segment.

[0232] Time t5 is the time when, after the hand has been brought close to the product segment and the hand begins to move back away from the product shelf, the distance between the hand and the shelf exceeds a predetermined threshold value, and it is determined that the hand has been moved away from the product segment. Note that the graph shows the output of the hand estimate as a straight line for simplification, but it should be understood that in reality, the trajectory will not be a straight line.

[0233] Similarly, it can be seen that the estimated hand points of User 2 were away from the product shelves until around time t3, but the estimated hand points approached the product shelves between around time t3 and t6. Time t3 is the time when, after User 1 begins to extend his / her hand towards the product shelf, it is determined that the distance between the hand and the shelf falls below (or exceeds) a predetermined threshold and the hand approaches the product segment.

[0234] Time t6 is the time when, after the hand has been brought close to the product segment and the hand begins to move back away from the product shelf, the distance between the hand and the shelf exceeds a predetermined threshold value, and it is determined that the hand has been moved away from the product segment. The difference in the size of the vertical axis on the graph between User 1 and User 2 is based on the difference in the distance between each customer and the product shelf. The threshold value can be set to any value taking into consideration the configuration of the product shelf, the size of the product, etc.

[0235] Based on the graph, the customer number determination module 1105 can see that a change occurs in the output of the weight sensor device 608 between times t2 and t4. From this, the customer number determination module 1105 can estimate that a product pickup event has occurred during this period in particular. However, the customer number determination module 1105 can understand that during this period, the estimated points of the hands of user 1 and user 2 were present near the product. Therefore, the customer number determination module 1105 performs processing to determine whether the item was picked up by user 1 or user 2.

[0236] In this embodiment, the customer number determination module 1105 performs calculations to determine who picked up which product and how many of each product. In particular, the customer number determination module 1105 divides the period during which the weight change occurred (from time t2 to time t4) for each user, calculates the difference in weight between both ends of each time period, and associates it as the weight picked by that user. That is, the customer number determination module 1105 estimates the hand stretching section for each user and obtains the weight change at both ends of the section. For example, the customer number determination module 1105 acquires time t2 and time t4 as the period in which a change in weight occurred.

[0237] Furthermore, for user 1, times t1 and t5 are acquired as periods during which hand movement occurred that would enable the user to pick up the product (section of reaching out). Furthermore, for user 2, times t3 and t6 are acquired as periods during which the hand moved to enable picking up the product (section of reaching out). The management server 101 chronologically organizes and compares this information, and divides the period (from time t2 to time t4) so ​​that the movement of each user becomes clearer.

[0238] First, the customer number determination module 1105 determines that in the period from time t1 to time t3, the hand movements of user 1 are mainly of concern, and the hand movements of user 2 are not of concern. Furthermore, the customer number determination module 1105 determines that the hand movements of users 1 and 2 are of primary concern during the period from time t3 to time t4. Therefore, the customer number determination module 1105 divides the period during which the weight change occurred (from t2 to t4) into a first interval (from t2 to t3) and a second interval (from t3 to t4) based on the hand movements of each user. Then, the customer number determination module 1105 estimates that in the first period (from t2 to t3), the hand movement of user 1 mainly contributes to the weight change. Furthermore, the customer number determination module 1105 estimates that in the second period (from t3 to t4), the hand movements of User 1 and User 2 mainly contribute to the weight change.

[0239] Furthermore, the customer number determination module 1105 divides the weight in accordance with the division of the above-mentioned period (from t2 to t4). That is, the customer number determination module 1105 assumes that the change in the product weight (50 g in total) occurs linearly, divides this change in weight into a first interval and a second interval, and performs a proportional calculation. For example, the customer number determination module 1105 calculates that a change in weight of 35 g has occurred for the first section when the total weight is 50 g. Furthermore, this change in weight (35 g) is associated with only the user 1. Furthermore, the customer number determination module 1105 calculates that a change in weight equivalent to 15 g has occurred for the second section, assuming a total weight of 50 g. Furthermore, this change in weight (15 g) is linked to both user 1 and user 2.

[0240] Next, the customer number determination module 1105 sums up the possible weights for each user, compares them with the weights of the products (registered in advance), and converts them into the number of products. For example, in the first period (from t2 to t3), the customer number determination module 1105 associates the change in the weight sensor (35 g) with only user 1. For this period, the management server 101 does not need to distinguish between user 1 and user 2. However, in the second period (from t3 to t4), the change (15 g) in the weight sensor is associated with user 1 and user 2. During this period, the customer number determination module 1105 needs to consider whether the weight is caused by user 1 or user 2 by making a case-by-case determination.

[0241] The customer number determination module 1105 knows that 50g worth of products were picked up in the end. Therefore, for the second period (from t3 to t4), the customer number determination module 1105 associates the amount (50g-30g=15g) obtained by subtracting the fractional weight associated only with user 1 with user 1 and compares the results.

[0242] For example, the customer number determination module 1105 assumes, as hypothesis 1, that in the second period (from t3 to t4), user 1 picks up an item weighing 15g. In this case, it is assumed that user 1 ultimately picks up an item weighing (50g=35g+15g), and user 2 finally picks up an item weighing (0g=0g+0g). Additionally, the customer number determination module 1105 assumes, as assumption 2, that in the second period (from t3 to t4), user 2 picks up an item weighing 15g. In this case, it is assumed that user 1 ultimately picks up an item weighing (35g=35g+0g), and user 2 finally picks up an item weighing (15g=0g+15g).

[0243] Next, the customer number determination module 1105 divides the final estimated change in weight for each of hypotheses 1 and 2 by the weight per item (50 g). For assumption 1, the customer quantity determination module 1105 determines that user 1 picked up 1.0 item (=50g / 50g) and user 2 picked up 0.0 item (=0g / 50g). For hypothesis 2, the customer quantity determination module 1105 determines that user 1 picked up 0.7 units (=35g / 50g) of products and user 2 picked up 0.3 units (=15g / 50g) of products.

[0244] When picking up an item, load is applied to the bars or boards of the product shelves or the shopping basket, so the readings of the weight sensor may not be perfectly linear and may fluctuate. Therefore, it is preferable for the customer number determination module 1105 to determine in advance an allowable error per item. For example, for a 50g item, an error of ±0.2 items (10g) is allowed.

[0245] Next, the customer number determination module 1105 determines whether each of hypotheses 1 and 2 is within the range of the allowable error per product. Assumption 1 can be determined to be within the acceptable error range for both users 1 and 2. As for hypothesis 2, it can be determined that the errors for both users 1 and 2 are outside the allowable error range. Therefore, the customer number determination module 1105 can ultimately determine that User 1 picked up products weighing 50 g each, and User 2 did not pick up any products.

[0246] The allowable error used in the determination by the customer number determination module 1105 can be set in various ways. For example, if the weight change is 100 g and is within the allowable error of 2 units (50×2±10×2) (2.0 units), the number of picked up products may be determined as 2 units. In addition, if the weight change is 105 g, which is within the allowable error of two items (50×2±10×2) (2.1 items), the number of items picked up may be determined as two. In addition, if the weight change is 125 g and is not within the allowable error of 2 items (50 x 2 ± 10 x 2) (2.5 items), the number of picked items may not be determined as 2 items, but may be left as indeterminate. Also, if the weight change is 160 g, which is within the allowable error of 3 units (50×3±10×3) (3.2 units), the number of picked up products may be determined as 3 units.

[0247] In this embodiment, more detailed control can be applied to the weight / number conversion process. Generally, as a characteristic of the weight sensor device 608, the error tends to become relatively smaller as the range of weight change becomes larger. Therefore, in this embodiment, when the weight change from the weight sensor device 608 is large, the allowable error may be relatively small. For example, referring to FIG. 46, when the weight of each product is 100g, the change in each value from 1 to 9 is illustrated in the case where the weight change is negative (product is taken). This value is used in the determination process of the customer number determination module 1105.

[0248] In Fig. 46, from the left, the number of products picked up, the target value (positive number) of the weight change, the measured value of the actual weight change, the allowable overall difference, and the allowable difference per product are shown. As can be seen from the figure, for example, when one product is picked up, the allowable difference per product is 15g. As the number of products picked up increases, the allowable difference per product decreases, and for example, when nine products are picked up, the allowable difference per product is 6.22g.

[0249] In this way, the customer number determination module 1105 can set the numerical value that is the basis for the determination at the stage of determining how many products have been picked up, depending on the weight (number) of the products picked up. This allows the accuracy of the determination to be improved as the number of products picked up increases, based on the characteristics of the weight sensor. Note that in FIG. 46, the number of products picked up is limited to a range of 1 to 9. If the number exceeds this range, a determination may be made by an appraiser or the like. Alternatively, the range shown in FIG. 46 may be expanded to accommodate a wider range of numbers.

[0250] In addition, depending on the weight (number) of the items picked up, the heavier the item is, the more accurate the criteria applied, and as a result, the applied criteria may overlap within a certain range. For example, there is a partial overlap (746g to 752g) between the allowable total difference of 7 items (648g to 752g) and the allowable total difference of 8 items (746g to 854g). In this case, the customer number determination module 1105 applies the determination preferentially starting from the item with the smallest number, thereby avoiding overlapping (mixed) results. For example, in this case, the number of 7 items is applied preferentially.

[0251] 63, there is shown a simplified diagram of a determination flow 6300 of the customer number determination module 1105. The flow of FIG. 63 will be described with reference again to the graph shown in FIG. First, in step 6310, the customer number determination module 1105 obtains the hand stretching interval for each user. For example, for user 1, the section between time t1 and time t5 is acquired as the section in which the hand approaches the product, and for user 2, the section between time t3 and time t6 is acquired as the section in which the hand approaches the product. Next, in step 6320, the hand stretching intervals associated with each user are classified into cases. For example, section 1 between time t1 and time t3 can be associated only with user 1. Section 2 between time t3 and time t5 can be associated with both user 1 and user 2. Section 3 between time t5 and time t6 can be associated only with user 2.

[0252] Next, in step 6330, the weight changes at both ends of each section divided into cases are obtained. For example, for section 1, a weight change of 35 g is obtained. For section 2, a weight change of 15 g is obtained. For section 3, a weight change of 0 g is obtained. Next, in step 6340, a hypothesis is made as to what would happen if the weight change for a section associated with multiple users were allocated to each user. For example, for section 1, a weight change of 35 g is obtained only for user 1. For section 2, a weight change of 15 g is obtained for users 1 and 2, so it is assumed that a weight change of 15 g is allocated to each of users 1 and 2 (see assumptions 1 and 2 above).

[0253] Next, in step 6350, the validity of the above assumption regarding the weight of each product is determined. For example, hypothesis 1 is determined to be valid when compared to the weight of each product, 50g, because a 50g product is allocated only to user 1. Assumption 2 is determined to be invalid when compared to the weight of each product, 50g, because a 35g product is allocated to user 1 and a 15g product is allocated to user 2.

[0254] The determination made by the customer number determination module 1105 regarding which user picked up which number of products is not limited to the flow of FIG. For example, referring to FIG. 64, another example of the determination flow 6400 of the customer number determination module 1105 is shown in a simplified manner. In this example, first, in step 6410, the customer number determination module 1105 obtains the hand stretching interval for each user.

[0255] Next, in step 6420, the customer number determination module 1105 obtains the weight change at both ends of the hand reaching section for each user. Next, in step 6430, the customer number determination module 1105 obtains the percentage of weight change per product for each user. Next, in step 6440, the customer number determination module 1105 compares the percentage of weight change for all users.

[0256] Referring to FIG. 65, an example output of the flow illustrated in FIG. 64 is shown. In this example, there are three customers near a certain product (worth 150g) in the store, and they are assigned IDs #1, #2, and #3, respectively. The period during which each customer reached near the product was calculated, and the percentage of weight change corresponding to that period was calculated. For example, for a product weighing 150 g each, a weight change of 20 g is calculated for customer #1, a weight change of 120 g is calculated for customer #2, and a weight change of 10 g is calculated for customer #3.

[0257] Therefore, the customer number determination module 1105 can estimate that the product was picked up by customer #2 by comparing the ratio of these three weight changes. As shown in Fig. 65, the weight change in the hand stretching section for each user may be displayed in a graph form to facilitate visual comparison. This output may be displayed on the screen of a user terminal or the like. The display format is not limited to Fig. 65, and may be a pie chart or the like. In this embodiment, the determination made by the customer number determination module 1105 regarding which user picked up which number of products is not limited to the flows shown in Figures 63 and 64. The flows shown in Figures 63 and 64 can be modified and changed in various ways.

[0258] In addition, if the customer quantity determination module 1105 is unable to determine the items and the number of items purchased by the customer based on information from the weight sensor unit 105 or images from the camera sensor unit 104 and the camera sensor unit 104a, it is preferable to store the information in a checklist. For example, when an irregular event occurs, such as when the detection of the change in weight is suspicious, when the detection of the position information from the camera device 506 or the calculation of the estimated hand position based on the image captured by the camera device 506a does not go well, or when the image is obscured and cannot be distinguished, the customer number determination module 1105 stores the information in a checklist that requires a final human check. In this way, when the customer number determination module 1105 is unable to make a complete determination, a human can assist later, thereby realizing a highly accurate shopping experience. The payment process itself is performed after the customer leaves the store, minimizing any damage to the customer experience.

[0259] If the management server 101 is unable to make a sufficient determination in the flow of the customer number determination module 1105, it is preferable that the determination is impossible and that appropriate instructions are displayed on the terminal of each user. For example, for items that the customer quantity determination module 1105 was unable to adequately determine, a store clerk may contact the user experiencing the problem and display instructions to inquire about which items the user picked up.

[0260] In either a manned or unmanned store, in a post-processing step, each user may select which products they picked up by self-reporting. This post-processing step is preferably performed before the users leave the store, but can also be performed after the users leave the store. In this way, when the customer number determination module 1105 is unable to make a satisfactory determination regarding the picking up of an item, it is preferable to ask the user for cooperation in resolving the problem. The display of instructions to ask the user for cooperation can be carried out in various ways. For example, for an item that could not be determined, a message such as "Checking" may be displayed on the terminal to ask the user to check it.

[0261] Referring to FIG. 47, another example of the determination of product pickup performed by the management server 101 is shown. In the example of Figure 45, one product is picked up. In an actual product purchasing process, multiple products may be picked up consecutively. Even in such a case, this embodiment makes it possible to calculate the number of products picked up. It should be noted that the example of FIG. 47 can include all of the determination contents described above with reference to FIGS. 45, 46, 63, and 64.

[0262] Referring to FIG. 47, the change in the output of the weight sensor is shown overlaid with the change in the estimated hand position of two users. The figure shows an example of weight time series data for each product shelf in a graph. The horizontal axis of the graph shows time (ms), and the vertical axis on the left shows the value (g) recorded by the weight sensor.

[0263] In the figure, a graph is shown showing an example of changes in the estimated hand points of two users standing near a product shelf equipped with the weight sensor device 608. The horizontal axis of the graph represents time (ms), and the vertical axis on the right represents the distance (mm) between each customer's hand and the product. Each user is assigned a person ID (identification number), A and B.

[0264] When an item that is hooked onto a bar on a shelf is picked up, a load is applied to the bar, and the output of the weight sensor device 608 is not uniform. The same is true when an item is placed on a shelf board or a shopping basket. Referring to FIG. 47, there is no change in the output of the weight sensor device 608 between approximately 1500 ms and 2500 ms, and therefore the customer number determination module 1105 can infer that no change has occurred in the products on the product shelves. Between 2500 ms and 3800 ms, a drop in the output of the weight sensor device 608 is recorded. Therefore, the customer number determination module 1105 can infer that a product has been picked up (or attempted to be picked up) from the product shelf.

[0265] Between 3800 ms and 5000 ms, the output of the weight sensor device 608 records a large drop with an instantaneous bounce, and between 5000 ms and 6000 ms, the output of the weight sensor device 608 is stable. Therefore, the customer number determination module 1105 can infer that a product has been picked up from the product shelf. During this period, the output of the weight sensor device 608 finally drops from about 300 g to 80 g. Therefore, the customer number determination module 1105 can estimate that one or more products with this difference in weight (about 220 g) have been picked up.

[0266] During the period when the change in the output of the weight sensor occurred, the customer quantity determination module 1105 can understand that there were hand movements of two users near the product in question. Therefore, the customer number determination module 1105 aligns the time of the hand segment distance and weight time series data of each user A and B. Then, from the hand segment distance information, it determines the period (hand reaching section) during which each customer is reaching out to the corresponding segment. At that time, it is determined for each user A and B whether there was a hand movement exceeding a predetermined threshold.

[0267] In particular, the customer number determination module 1105 estimates the section of the hand stretching and obtains the weight change at both ends of the section. For example, the customer number determination module 1105 finds that, for user A, the estimated hand point is located near the product on the product shelf during the period from approximately 2200 ms to 3100 ms. Furthermore, the customer number determination module 1105 finds that, for customer B, the estimated hand point is located near the product on the product shelf within the period from approximately 3600 ms to 4500 ms. Next, the customer number determination module 1105 calculates the difference between the weights at both ends for each period, and associates it as the weight taken by the related user. At this time, the customer number determination module 1105 sums up the weights for each user, compares them with the weights of the products (registered in advance), and converts them into the number of products.

[0268] For example, the customer number determination module 1105 performs processing including the following steps. The customer number determination module 1105 sets the weight of each product and the allowable error rate in advance and stores the corresponding data in a storage device. For example, if the weight of each product is 50g, the allowable error rate is set to 0.2 units (10g). Next, the customer number determination module 1105 receives a numerical value (g) indicating a change in weight of an item when the item is picked up by a weight sensor device 608 associated with the item shelf.

[0269] Next, the customer number determination module 1105 performs an operation of dividing the changed weight value (g) by the weight (g) of one item of the product that is stored in advance. Next, the customer number determination module 1105 compares the result of the above calculation with the error rate of the product that is stored in advance. If the comparison results in the error rate being within the allowable error rate, the customer number determination module 1105 determines the number.

[0270] For example, the customer number determination module 1105 mainly focuses on the hand movement of user A during the period from time t11 to time t12. During this period, the hand movement of user B is recognized, but since the distance from the product segment is far, the hand movement can be ignored. Furthermore, the customer number determination module 1105 learns that there was a change in weight of about 100 g (300 g - 200 g) during this period. Also, the customer number determination module 1105 mainly focuses on the hand movement of user B during the period from time t13 to time t14. Furthermore, it is found that there was a change in weight of about 120 g (200 g-80 g) during this period.

[0271] From these, the customer number determination module 1105 can assume that user A picked up an item weighing approximately 100 g, and then user B picked up an item weighing approximately 120 g. Therefore, the customer quantity determination module 1105 verifies the above assumption based on the stored weights of the products in the product segment. If the above verification results in a deviation from the allowable error, the customer number determination module 1105 determines the number to be indefinite, and otherwise determines the number to be definite.

[0272] As configured as described above, in this embodiment, when a product pickup event occurs and there are multiple users in the vicinity, it is possible to determine which user has picked up how many products. Furthermore, in this embodiment, the customer number determination module 1105 of the management server 101 is devised to enable the determination of product pickup even if the weight change based on the output of the weight sensor device 608 is not ideal.

[0273] For example, referring to Figures 48 and 49, further examples of the output of the weight sensor are shown. In the examples shown in these figures, similar to Figures 45 and 47, the changes in the output of the weight sensor and the changes in the estimated points of the hands of two users are shown superimposed. In addition, the examples of Figures 48 and 49 can include all of the judgment contents described above with reference to Figures 45, 46, 47, 63, and 64, but in order to avoid duplication, explanations of the overlapping parts will be omitted.

[0274] In the example of FIG. 48, it takes a relatively long time (about 2 seconds) for the weight to fall completely. In the example of Fig. 49, the weight goes up and down even though only one product is picked up at a time. As shown in Fig. 48 and Fig. 49, the output of the weight sensor device 608 does not necessarily indicate an ideal weight change. When a sensor (such as weight sensor device 608) is installed on a bar, board, shopping basket, etc. on a product shelf to measure or calculate its weight, a load is applied to the bar, board, shopping basket, etc. when it is picked up, and the weight value may fluctuate. As a result, it may take some time for the weight sensor device 608 to output a change in weight. Also, the change in weight may fluctuate up and down. In these cases, in the prior art, it may be difficult to determine how many items were picked up and at what time using only the weight information from the weight sensor device 608.

[0275] In this embodiment, in the above case, the customer number determination module 1105 of the management server 101 tracks the estimated hand point of each user near the product shelf in question. The estimated hand point is compared with a predetermined threshold to determine whether the hand is extended close to the product. Thus, the management server 101 estimates the section of each user's hand extension. The customer number determination module 1105 can refer to the output from the weight sensor device 608 or the like for the time at both ends of each section. Thus, the customer number determination module 1105 can obtain the weight change at both ends of the section of each user's hand extension.

[0276] The customer number determination module 1105 can perform case classification to convert weight changes to numbers for each user. For example, if a change in the estimated hand point is detected for only one user, weight changes are converted to numbers for only that user. Also, if a change in the estimated hand point is detected for only two or more users, cases are classified for each user, and each weight change is converted to a number. The customer number determination module 1105 determines whether each case classification is valid with the finally obtained weight change.

[0277] For example, as shown in FIG. 48, even if it takes time for the weight to change, By estimating the stretching section and obtaining the weight change at both ends, it is possible to respond to such weight changes. In addition, as shown in FIG. 49, even if there are fluctuations in the weight change, By estimating the stretching section and obtaining the weight change at both ends, it is possible to respond to such weight changes. Thus, the customer quantity determination module 1105 can determine the number of items picked up even if the output of the weight sensor device 608 is not ideal.

[0278] In reality, the output of the weight sensor device 608 and the position of the estimated hand point determined by calculation are not always obtained accurately. Therefore, in this embodiment, further improvements can be made so that the customer number determination module 1105 of the management server 101 can more effectively determine whether or not a product has been picked up. For example, there may be a slight difference between the time when the user actually reaches out to grab a product from a product shelf and the time when the weight sensor device 608 actually outputs the change in the weight of the product. This is because a change in the output of the weight sensor device 608 may be obtained around the time when the product is actually picked up.

[0279] Referring to FIG. 50A, the upper part shows the change in output of the weight sensor device 608, and the lower part shows the change in distance of the estimated point of the user's hand in relation to the item. As can be seen from (A) of Figure 50, at time T1, the user reaches out his / her hand toward the product, and the distance between the hand and the product exceeds the threshold value, approaching the product. Furthermore, at time T2, the user moves his / her hand back away from the product, and the distance between the hand and the product exceeds the threshold value. Somewhat delayed from this movement of the user's hand, a drop in the output of the weight sensor device 608 is detected between times Ta and Tb. In this way, if there is a discrepancy between the time based on the movement of the user's hand and the time based on the output of the weight sensor device 608, the management server 101 may not be able to obtain an accurate correspondence between the movement of the estimated point of the user's hand and the change in the output of the weight sensor device 608.

[0280] Referring to FIG. 50(B), similarly, the upper part shows the change in the output of the weight sensor, and the lower part shows the change in the distance between the item and the estimated point of the user's hand. As can be seen from (B) of Figure 50, when the customer number determination module 1105 of the management server 101 detects a discrepancy between the time based on the user's hand movement and the time based on the output of the weight sensor device 608, it makes a correction by shifting one of the times forward or backward so that the correspondence between the two becomes clearer. For example, in this figure, the customer number determination module 1105 shifts times Ta and Tb, which are based on the output of the actual weight sensor device 608, slightly forward (see times Ta' and Tb').

[0281] This correction time (Ta-Ta' or Tb-Tb') can be preset as a fixed value. In this case, if a deviation is detected between the time based on the user's hand movement and the time based on the output of the weight sensor device 608, one of the times is shifted forward or backward by a predetermined amount. Alternatively, this correction time (Ta-Ta' or Tb-Tb') can be preset as a variable value. In this case, if a difference is detected between the time based on the user's hand movement and the time based on the output of the weight sensor device 608, one of the times is shifted forward or backward by a magnitude corresponding to the magnitude of the difference.

[0282] In this way, the customer number determination module 1105 can correct either the time based on the movement of the user's hand or the time based on the output of the weight sensor device 608. By setting the correction time in this way, the customer number determination module 1105 can obtain a more accurate correspondence between the movement of the estimated point of the customer's hand and the change in the output of the weight sensor device 608.

[0283] For example, during image analysis based on a captured image, a delay may occur in the movement of the estimated hand point. Or, when a product is picked up, the weight sensor device 608 may detect the event with a delay. Even in such a case, this embodiment introduces a correction time to make it possible to more accurately determine whether the product has actually been picked up. As a precondition for introducing the correction time, it is preferable to determine whether or not the difference between the time based on the movement of the user's hand and the time based on the output of the weight sensor device 608 is smaller than a predetermined value.

[0284] In this embodiment, further improvements can be made so that the customer number determination module 1105 can more effectively determine whether or not a product has been picked up. For example, when a customer actually reaches out his / her hand to a product shelf to grab a product, there may be a slight deviation between the actual position of the characteristic parts and the calculated position of the characteristic parts when detecting the skeleton based on the captured image. If this deviation is large, there is a risk that the hand is not detected as being stretched out even though the customer is actually reaching out to pick up the product.

[0285] Referring to FIG. 51(A), the upper part shows the change in output of the weight sensor device 608, and the lower part shows the change in distance of the estimated point of the user's hand relative to the item. As can be seen from (A) of Figure 51, at time T1, the user reaches out his / her hand toward the product, and the distance between the hand and the product exceeds the threshold value, approaching the product. Furthermore, at time T2, the user moves his / her hand back away from the product, and the distance between the hand and the product exceeds the threshold value. Furthermore, during this period, it is detected that the user puts his / her hand back and then extends it again in a short period between times Ta and Tb. Therefore, the customer number determination module 1105 cannot associate the movement of the user's hand with the change in the output of the weight sensor device 608 between times Ta and Tb (determines that the user has not picked up any products during that period).

[0286] In this embodiment, the length and direction of the forearm are obtained from the positions of the elbow and wrist, and the position of the estimated point of the hand can be calculated by extending the length. If the position of the elbow or wrist, which is the reference for this calculation, is shifted, the estimated point of the hand may be shifted. For example, even if the user actually reaches his or her hand into the shelf, it may be determined that the hand has not reached the specified position. For example, in (A) of Fig. 51, although the user actually extends his / her hand toward the product continuously between times T1 and T2, a break in the movement of the hand is detected between times Ta and Tb. If this break in the movement is caused by an inaccurate calculation of the estimated position of the hand, the customer number determination module 1105 cannot correctly determine whether the product has been picked up.

[0287] Referring to FIG. 51(B), similarly, the upper part shows the change in the output of the weight sensor, and the lower part shows the change in the distance between the product and the estimated point of the user's hand. That is, the customer number determination module 1105 constantly monitors the positions of the estimated hand points linked to the person ID. When the estimated point of a hand linked to a person ID exceeds a predetermined threshold, it is estimated that the hand has reached a position on a shelf where a product can be picked up (for example, time T1).

[0288] Thereafter, when the estimated hand point associated with the same person ID falls below a predetermined threshold, it is estimated that the hand has moved away from a position on the shelf where a product can be picked up (for example, at time Ta). Thereafter, when the estimated point of the hand associated with the same person ID exceeds a predetermined threshold again, it is estimated that the hand has reached a position on the shelf where a product can be picked up (for example, at time Tb). Thereafter, when the estimated hand point associated with the same person ID falls below a predetermined threshold, it is estimated that the hand has moved away from a position on the shelf where a product can be picked up (for example, at time T2).

[0289] In this manner, when a break is detected in the period during which the customer's hand is extended (eg, between times Ta and Tb), the customer number determination module 1105 makes the following determination. A determination is then made as to whether the person ID of the hand in a position to pick up the product is the same before and after the detected gap. If the above determination shows that the person IDs before and after the break are the same, then it is further determined whether or not the time interval in which the break occurred (between times Ta and Tb) is below a predetermined threshold value.

[0290] If the above determination shows that a gap occurred in a very short time, the customer number determination module 1105 determines that the gap can be ignored and applies corrections so that the gap does not actually exist, thereby avoiding or minimizing the adverse effect of the gap on the product pickup determination. For example, the customer number determination module 1105 minimizes or zeros the interval between times Ta and Tb as illustrated at time Tc. This prevents a discontinuity in the user's hand movement from causing a malfunction in the product pick-up determination.

[0291] For example, the quality of the captured image of the camera device 506a of the camera sensor unit 104a is not uniform, and the position of the elbow or wrist on the captured image may be shifted or blurred due to a sudden movement of the customer being photographed. However, even in such a case, the person ID linked to the characteristic part can be continuously recognized. For example, the person ID may be verified by the camera device 506 of another camera sensor unit 104.

[0292] Therefore, the customer number determination module 1105 determines whether or not a position shift has occurred for each person ID, and if the interval is short (for example, between times Ta and Tb), it makes a correction to ignore the shift (for example, time Tc). Thus, in this embodiment, when a user is present alone during the determination of actual product pick-up, if the gap that occurs during the period of reaching out is short, and the personal IDs of the users reaching out on both sides of that period are the same, it is determined that the user was reaching out during the gap as well, and this is overwritten.

[0293] In this embodiment, further improvements can be made so that the customer number determination module 1105 can more effectively determine whether or not a product has been picked up. Referring to (A) of FIG. 52, the upper part shows the change in output of the weight sensor device 608, and the lower part shows the change in distance between the estimated points of the hands of user 1 and user 2 and the product. As can be seen from (A) of Figure 52, at time T1, user 1 reaches out his / her hand toward the product, and the distance between his / her hand and the product exceeds the threshold value, approaching the product. Furthermore, at time T2, user 1 moves his / her hand back away from the product, and the distance between his / her hand and the product exceeds the threshold value.

[0294] At time T3, user 2 reaches out his / her hand toward the product, and the distance between his / her hand and the product exceeds the threshold value, approaching the product. At time T4, user 2 moves his / her hand back away from the product, and the distance between his / her hand and the product exceeds the threshold value. There is a gap between the movements of these two users 1 and 2, and it is detected that from time T2 to T3, neither user 1 nor 2 reaches out their hand toward the product. The output of the weight sensor device 608 is detected with some delay (time difference) in relation to the hand movements of the users 1 and 2.

[0295] In this way, when a user reaches out towards a product shelf to grab a product, there may be a slight difference between the time of skeleton detection and the time of output from the weight sensor device 608. The estimated hand points are detected based on the captured image, but the position of the skeleton detection and the time of the weight sensor device 608 may be blurred. As a result, it may not be possible to correctly associate the change in output of the weight sensor device 608 with the users 1 and 2. As shown in (A) of FIG. 52, if there is a section where no one extends their hand due to a slight delay in the weight change or a time difference, it may not be possible to link the weight change in that section with the user.

[0296] For example, at times T2 and T3, the hands of users 1 and 2 are detected as moving away from the product, so any weight changes that occur during this period cannot be associated with either user 1 or 2. Furthermore, after time T4, the hands of users 1 and 2 are detected as moving away from the product, so the weight change that occurs during this period cannot be associated with either user 1 or 2. In these cases, the customer number determination module 1105 cannot correctly determine whether or not a product has been picked up.

[0297] Referring to FIG. 52(B), similarly, the upper part shows the change in output of the weight sensor device 608, and the lower part shows the change in distance between the item and the estimated point of the user's hand. For example, the customer number determination module 1105 detects that there was hand movement of two or more users 1 and 2 before and after a change in the output of the weight sensor device 608, and that there was a gap between the movements of each user 1 and 2 (for example, between times T2 and T3, or a specified period after time T4).

[0298] In this case, the customer number determination module 1105 compares the size of the gap (for example, between times T2 and T3, or a predetermined period after time T4) with a predetermined threshold value to determine whether the size is negligible. If it is determined that the movement cannot be ignored, the customer number determination module 1105 performs a correction such that the movement of each user 1, 2 is expanded forward and backward by a predetermined period of time to reduce the size of the gap (for example, between times T2 and T3, or a predetermined period after time T4). This allows for a better correspondence between the movements of each user 1, 2 and changes in the output of the weight sensor.

[0299] For example, the customer count determination module 1105 may do any one or more or all of the following: For example, the time T1 at which the user 1 reaches out his / her hand towards the product shelf is shifted forward by a predetermined time amount (for example, time Ta). For example, the time T2 at which the user 1 removes his / her hand from the product shelf is shifted back by a predetermined time amount (for example, time Tb). For example, the time T3 at which the user 2 reaches out his / her hand towards the product shelf is shifted forward by a predetermined time amount (for example, time Tc). For example, the time T4 at which the user 2 removes his / her hand from the product shelf is shifted back by a predetermined time amount (for example, time Td).

[0300] In this case, the time lengths Ta, Tb, Tc, and Td introduced by the correction may be the same, or may vary depending on the implementation situation. Therefore, the customer number determination module 1105 zooms in on the hand movements of each user 1, 2 back and forth to shorten or minimize the intervals during which no one is reaching out, making it easier to link the changes in weight to users 1, 2.

[0301] The customer number determination module 1105 may perform the control illustrated in Fig. 50(B), Fig. 51(B), and Fig. 52(B) alone or in combination with each other. Furthermore, in these determination controls, the controls illustrated in Fig. 45, Fig. 46, Fig. 47, Fig. 48, Fig. 49, Fig. 63, and Fig. 64 may be appropriately selected and combined. For example, if a change in the weight of a certain product is detected in a section where multiple users are simultaneously reaching out around the product, there is a risk that it will not be possible to adequately determine which user picked up how many products. In this embodiment, in cases where the number of users / items is likely to be uncertain, a determination logic for when the hands overlap may be introduced to deal with the situation. Therefore, in this embodiment, the “estimated hand point” is used to determine whether an event of actually picking up a product has occurred, and efforts are made to make this operation easier and to improve the accuracy of the determination.

[0302] With the above configuration, the customer number determination module 1105 can perform the following control when multiple customers are near a certain product in the store. For each customer, we determine the correlation between the computed hand estimate and the location of the shelf (product). In particular, we determine the time interval during which the customer's hand is near the product. A change in the output of the weight sensor is determined. In particular, the amount of change in the weight of the product corresponding to the above-mentioned time interval is obtained for each customer from the time-series data of the weight obtained by the weight sensor. For each customer, it is determined whether or not an item has been picked up from the shelf. In particular, the ratio of the weight change of the item corresponding to the above time interval for each customer is calculated with respect to the weight of each item.

[0303] Therefore, this embodiment makes it possible to determine the movement of a user picking up an item under various circumstances. For example, in this embodiment, it is possible to determine which user has taken which product and how many of each product. In particular, this embodiment makes it possible to correctly determine the user who picked up the product and the number of products. In particular, the picking up of products is determined for each user's person ID. Furthermore, this embodiment determines the pick-up of products, including the number and type, for each user's personal ID. In particular, it can handle cases where products are picked up continuously at time intervals that are inseparable from the weight sensor.

[0304] For example, in this embodiment, it is possible to output information such as "user A and user B each took one item a." In addition, in this embodiment, it is possible to output information such as "User A took two items a, and User B took one item a." In addition, in this embodiment, it is possible to output information such as "user A took one product a, and user B took one product b." Therefore, in this embodiment, it is possible to determine which user picked up which product and how many of them they picked up. This makes it possible to perform product pick-up determination with higher accuracy than in the prior art.

[0305] In contrast, in conventional technology, when determining the movements of a customer picking up a product using images captured by a camera and the output of a weight sensor, it can be difficult to make a determination if multiple customers are simultaneously within reach of a certain product (multiple hands are simultaneously near the product). Furthermore, in the conventional technology, when multiple products are picked up consecutively from the same position (when the products are picked up at close times), it tends to be difficult to make a judgment. Furthermore, in the conventional technology, when multiple adjacent products are picked up at the same time (when the products being picked up are close to each other), it tends to be difficult to make a judgment. Furthermore, in conventional technology, when multiple products are picked up at the same time, it tends to be difficult to make a judgment.

[0306] In particular, in conventional technology, when determining the movement of a customer picking up an item, the "weight information" of the item only includes the difference in weight change before and after the state stabilizes. Furthermore, in the conventional technology, when determining the movement of a customer picking up an item, it is assumed that only one customer has picked up an item at the time when the weight information changes. In addition, in the conventional technology, the weight sensor would determine that the user closest to the weight sensor had picked up the product and output the result. Therefore, when there are multiple users in the vicinity, the weight sensor would sometimes output a result that could not be determined sufficiently.

[0307] In this way, in the conventional technology, when multiple customers who are close to each other take products, it is not possible to determine who took them. Also, when products are taken consecutively (without a time interval) from a shelf with the same weight sensor, it is not possible to adequately determine which customer took how many products. The present embodiment provides a suitable means for solving such problems of the conventional technology.

[0308] In the conventional technology, when determining whether or not a product has been picked up based on image analysis of a captured image, a deep neural network (DNN) was sometimes used to analyze the image of the product or the image of the hand trying to grab the product. A DNN is a neural network that is adapted to deep learning and has four or more layers. A DNN can be applied to image recognition, and in particular, when two-dimensional image data is converted into an appropriate numeric string such as one-dimensional and input into a DNN, it can output the results of recognizing what is in the image and where it is located.

[0309] When using DNN for image recognition, the following problems arose: In DNN, the types of products that can be judged are limited. In contrast, as is well known, new products are developed almost continuously for products sold in general stores. For example, new products are constantly being introduced in supermarkets and convenience stores. With DNN, if you want to increase the number of products to be judged, you need to change the model configuration and re-train the model. If new products are introduced frequently, it is not possible to ensure a sufficient learning period, which poses operational challenges.

[0310] Furthermore, when applying DNN to captured images, only display methods that show the presence or absence of products can be adopted. In the case of product shelves with multiple levels (such as flat stacking), DNN can be applied because only products placed on the top shelf without a top can be photographed in their entirety. However, it may be difficult to apply DNN to the second and subsequent levels of product shelves, because it is not possible to photograph the entire product. Therefore, operational issues remain for DNN-based image recognition in stores with multiple levels of product shelves.

[0311] In contrast, in this embodiment, the customer number determination module 1105 of the management server 101 determines whether or not a product has been picked up by combining the customer's movement trajectory obtained based on the image captured by the camera device 506 of the camera sensor unit 104, the estimated point of the customer's hand obtained based on the image captured by the camera device 506a of the camera sensor unit 104a, and the read value of the weight sensor device 608 of the product shelf. Therefore, this embodiment can be applied without selecting the type of product as long as the weight of each product can be known. Therefore, this embodiment makes it easier to increase the type of product.

[0312] Furthermore, in this embodiment, it is not necessary that the entire product is photographed by the camera device 506 of the camera sensor unit 104 or the camera device 506a of the camera sensor unit 104a. Therefore, this embodiment can be applied regardless of the way the products are displayed. For example, this embodiment can be applied to product shelves of any shape (such as a flat stacking method).

[0313] Furthermore, in this embodiment, some improvements are made to better determine whether or not a product has been picked up. Typically, multiple product shelves are installed in a store (see (A) and (B) in FIG. 36). Therefore, it is possible that the same customer's actions are captured by multiple cameras (e.g., camera device 506, camera device 506a, etc.) at approximately the same time. In order to maintain good quality of the captured images, it is preferable to set the priority order of each camera in advance based on the expected usage situation.

[0314] For example, to capture a customer picking up a product from a shelf, it is preferable to capture the front view of the customer facing the shelf from above. In this case, it is preferable to capture the overall movement of the customer's arms as they try to grab the product. It is even more preferable to capture the left and right arms of the customer relatively evenly. In contrast, when capturing a photo of a customer from the side, the movement of the customer's arms as they try to grab the product may be hidden by the customer's body. It is also difficult to capture the left and right arms of the customer relatively evenly.

[0315] Thus, the content of the images captured by each camera is not uniform. As described above, when both an image of the customer from the front and an image of the customer from the side are available, it is preferable to give a higher priority to the former than to the latter. This ensures that image analysis is performed based on the image of the preferred quality.

[0316] Referring to FIG. 53, a floor plan of an actual store in which this embodiment can be used is shown. As can be seen from the figure, the store is provided with an entrance (entrance area / exit area), followed by a floor on which multiple product shelves are installed. Note that while the figure shows the floor as being substantially rectangular, the actual shape of the floor can be configured in various ways. Usually, in order to make effective use of the space in the store, each product shelf installed on the floor is divided into multiple tiers, one above the other, with multiple products displayed on each tier. The space between these product shelves is free space in which customers can walk around freely.

[0317] Referring to FIG. 54, the floor is divided mainly based on the installation position of the product shelves in the floor plan of the store illustrated in FIG. 53. In this way, a plurality of areas are set on one floor. Each area is illustrated as a substantially rectangular frame, and one or more product shelves can be installed within the frame. The shape and size of this area can be configured in various ways. It is preferably associated with a range that can be well photographed by a camera installed nearby. Adjacent areas may have a gap between them. Adjacent areas may also be partially overlapped with each other.

[0318] Referring to Fig. 55, an example of an actual camera capture image is shown for the product shelves and / or area in the store illustrated in Fig. 54. This capture image is applicable to both camera device 506 of camera sensor unit 104 and camera device 506a of camera sensor unit 104a. As can be seen from FIG. 55, the area may be set to surround the front of the shelf (the position where the products are placed and a little in front of it).

[0319] As can be seen from FIG. 55, for example, the POV / FOV of the camera is set so that the product shelf is photographed from almost directly above, and the arms of a customer attempting to pick up an item displayed on the shelf can be photographed from the front. Usually, a customer faces the shelf directly in order to check the products displayed therein. In this case, the upper half of the customer's body can be photographed relatively well. It is also possible to photograph at least the outstretched arms of a customer standing at an angle to the shelf and trying to pick up an item from the side.

[0320] The position of each product shelf and / or area and the POV / FOV of each camera that captures the surroundings are known in advance. Therefore, it is known which camera can capture which product shelf optimally. The management server 101 determines the priority order of the preferred cameras for each product shelf and / or area, and can perform image analysis of the captured images according to that order. For example, if it can be assumed in advance that images from multiple cameras are available, the accuracy of the position will be reduced if the image is taken from the side, so if an image taken from the front is available, the latter will be used preferentially.

[0321] Priorities can be set in a variety of ways based on the situation in the store. For example, the priority may be based on the distance between the location of each camera 506, 506a and the location of the shelves and / or areas. For example, the priority may be determined based on the correspondence between the POV / FOV of each camera 506, 506a and the people who may be present around the product shelves and / or the area. For example, the priority may be determined based on the correspondence between the images captured by each camera 506, 506a and the orientation of the subject. For example, the priority may be determined based on the presence or absence of an obstacle between the image captured by each camera 506, 506a and the subject. For example, the priority may be determined based on the images captured by each camera 506, 506a and the accuracy of skeleton detection based on the images.

[0322] Additionally, various priorities can be set based on the size, appearance, quality, etc. of the images captured by each camera 506, 506a. Furthermore, the size, shape, etc. of each area can be set in various ways in relation to the priority. In this way, the management server 101 may set in advance a priority for which camera's captured image is to be used, so that the quality of the image analysis satisfies a predetermined standard.

[0323] 56, eight areas are defined in a floor plan (floor) of a certain store, and the photographing conditions of each area are illustrated. These photographed images can be applied to both or either of the cameras 506 and 506a. 56, the state inside the store can be captured as multiple (e.g., eight) captured images. Depending on which of these captured images is used, tracking of customers moving inside the store can change.

[0324] For example, even if the movements of a certain customer can be captured well based on images captured by a certain camera 506, 506a, the captured images may no longer be able to capture the movements of the customer adequately due to a change in the customer's position. The management server 101 may determine in advance a priority order for which of a plurality of possible captured images (e.g., eight) is preferably used, depending on changes in the customer's position or movements.

[0325] Referring to FIG. 57, in the case illustrated in FIG. 56, an example is shown in which the behavior of a customer moving around the store is tracked from two different cameras. In the figure, from the upper right to the lower left, eight cameras are illustrated, denoted by reference numerals 0236, 0108, 0227, 0357, 0570, 0390, 0391, and 0138. These cameras may be either or both of camera device 506 of camera sensor unit 104 and camera device 506a of camera sensor unit 104a.

[0326] In this example, a user with a person ID of 837 is detected by two cameras (0227 and 0391) at almost the same time. The person with the person ID "837" displayed in small letters corresponds to the person ID output based on the image (ToF) captured by the camera device 506 of the camera sensor unit 104. The person with the person ID "837" displayed in large letters corresponds to the person ID of the user who is estimated to be present on the floor as a result of linking the skeleton and position based on the image captured by the camera device 506a of the camera sensor unit 104a.

[0327] The customer's location information detected by each camera may change depending on the relationship between the POV / FOV of each camera and the customer's location within the store. The management server 101 continuously tracks the customer's location, and can determine in advance a priority for which camera's captured image is to be used according to changes in the customer's location. As a result, the management server 101 can always track the location of a customer moving around the store with high accuracy. This can be applied to both the camera device 506 of the camera sensor unit 104 for detecting a customer's position and the camera device 506a of the camera sensor unit 104a for detecting a customer's bone structure.

[0328] The POV / FOV of each camera 506, 506a is different from each other so as to appropriately cover the vicinity of the corresponding product shelf. Preferably, the POV / FOV of each camera 506, 506a is fixed in advance so as to capture the state of the nearby product shelf well. However, if the pre-set POV / FOV is not suitable for capturing a good image of the surroundings of a certain product shelf, the management server 101 may change the POV / FOV of the cameras 506 and 506a.

[0329] For example, if a customer is in a relatively corner of a store and cannot be photographed clearly, the direction or angle of the cameras 506 and 506a may be adjusted. Therefore, the management server 101 can set a priority for each of the cameras 506, 506a in advance, and can also modify the POV / FOV of each of the cameras 506, 506a according to the actual operational situation. In this manner, this embodiment is devised so that the movement of the customer can be continuously tracked.

[0330] In this embodiment, further improvements can be made so that the management server 101 can better track customers. This control may be performed, for example, by the position information processing module 1102. Also, it may be performed by the skeleton detection processing module 1102a, the customer number determination module 1105, etc. For example, in the conventional technology, depending on the way a customer moves in a store, the camera device 506 of the camera sensor unit 104 or the camera device 506a of the camera sensor unit 104a may not be able to completely track the customer. If the customer's position information cannot be accurately tracked, it becomes problematic because it is not possible to adequately determine whether the customer should pick up the product afterwards.

[0331] For example, referring to FIG. 58, there is shown a simplified example of a tracking loss / swap of the camera device (ToF) 506 of the camera sensor unit 104 in the prior art. As can be seen from FIG. 58, there are two users in a store. In this example, they are assigned person IDs 0 and 1, respectively. Each user 0 and 1 is tracked by the same or different cameras. As shown in (1) to (2) in the figure, when the positions of two customers approach each other, a situation may occur in which the two customers cannot be clearly identified. In the figure, after the two customers suddenly approach each other, the person IDs 0 and 1 are swapped, as shown in (2) to (3) in the figure. In this way, if the customer's personal ID is swapped in the tracking information (personal information), there is a risk that subsequent tracking of the customer, skeletal detection, and so on will all be erroneous in a chain reaction.

[0332] In this embodiment, a means for preventing the above-mentioned situation from occurring is provided. In particular, in this embodiment, when extracting the customer's position based on the image captured by the camera device 506 of the camera sensor unit 104 or the camera device 506a of the camera sensor unit 104a, information on the customer's clothing is extracted at the same time. Each customer can be tracked by combining position information and clothing information (color information). Therefore, even if two customers are close to each other and a situation occurs in which they cannot be clearly identified based on position information alone, it is possible to distinguish them from each other based on the clothing information of both customers.

[0333] For example, in this embodiment, the color or shape of the customer's clothes is extracted based on the image captured by the camera device 506 of the camera sensor unit 104 or the camera device 506a of the camera sensor unit 104a. For example, the color of the customer's shoulders is extracted. The consistency of color and shape information of the customer's head may be lost due to putting on and taking off a hat or the like. Elbows, wrists, arms, and the like can move relatively freely and may be hidden depending on the viewing angle.

[0334] For example, in the case of an elbow, if the detection position shifts or if the sleeve is rolled up, it can be either bare skin or clothing. In addition, in the case of a wrist, if the hand is put in or taken out of a pocket, it can be either clothing or a hand. In contrast, the position of the shoulder can be relatively continuously visible in many situations. Therefore, by using shoulder color information, it is possible to continuously track the customer's clothing information.

[0335] In addition, in this embodiment, other color information of the customer may be used in addition to the shoulder color. For example, shoulder color information may be combined with color information of the chest or back. Furthermore, information on the shape of the customer's clothes on the shoulders, chest and / or back may be used. Furthermore, instead of simple color, the shape, pattern, color and / or feature amount of the entire clothes (for example, the feature amount can be extracted by a separately created model / feature amount calculation method) may be used.

[0336] Referring to FIG. 59, an example is shown of obtaining shoulder color information for a customer. In this embodiment, the color of the customer's clothes may be determined by acquiring the color of the coordinates of the customer's shoulder from the RGB image captured by the camera device 506 of the camera sensor unit 104 or the camera device 506a of the camera sensor unit 104a. In addition, as the head moves left and right, it is possible that the position of the head and the position of the shoulders may overlap partially. Therefore, it is preferable to adopt the coordinates of the shoulder that is farthest from the head out of the coordinates of both shoulders. This makes it possible to prevent the head from overlapping the shoulders depending on the angle at which the user is photographed. In FIG. 59, the position of the head is compared with the positions of the left and right shoulders, and the right shoulder, which is farther from the head, may be adopted, and the left shoulder may not be adopted.

[0337] In this way, for each customer who enters the store from a specific area (near the entrance) in the store, the color of the shoulder clothing is recorded together with the position information for a certain period of time immediately after entering the store. Alternatively, the clothing information may be acquired at a position and timing that allows a stable determination of the user's clothing, not limited to immediately after entering the store. For example, the customer number determination module 1105 starts acquiring clothing information at an arbitrary timing, and then periodically records the shoulder color for several tens of seconds, one minute, several minutes, or any other arbitrary period, calculates the average value, and registers the value as the color of the customer's clothing.

[0338] Thereafter, for example, the customer number determination module 1105 continues to acquire the shoulder color of each customer, and when multiple customers approach each other (see (2) in FIG. 58), the customer's position information and clothing color information are combined and used to distinguish between them. If there is a temporary exchange of position information and / or clothing color information between multiple customers, the identification numbers are swapped, etc., to prevent or correct the assignment of erroneous identification numbers (person IDs).

[0339] When tracking of customer location information is operating normally, for example, the customer number determination module 1105 functions as follows. For example, when a customer comes to a preset appearance area (near the entrance) in a store, a unique person ID is assigned to each customer and tracking begins. The person ID may be configured using numbers and / or letters, etc. For example, the person ID may be assigned as a serial number to each customer who visits the store. Also, for example, the person ID may be assigned as a date / time combination. For example, a customer who visits a store at 7:10:30 a.m. on May 30, 2021 may be assigned the identification number 20210530071030. When multiple customers appear at the same date / time, they may be distinguished from each other by adding a code such as a, b, or c to the end.

[0340] While the customer is in the store, the person ID assigned to each customer is maintained. Then, when the customer enters a preset disappearance area (near the exit) in the store, the person ID assigned to each customer is erased and tracking ends. If the same person visits the same store again, a different person ID is assigned each time. In the conventional technology, the customer's ID may disappear even if the customer does not come to the designated disappearance area in the store (lost person ID). In such a case, there was a problem that the tracking information of the user would not be output thereafter. In addition, if the customer came to the designated appearance area (near the entrance) in the store again, a different person ID would be assigned to the customer (unintended reappearance of a person ID). In that case, there was a problem that the customer would be judged as a different customer (different person) from then on.

[0341] In this embodiment, a means for preventing the occurrence of the above-mentioned situation of a person ID being lost / reappearing is further provided. For example, in this embodiment, even if a customer's personal ID is temporarily lost, the customer can be tracked based on bone structure information and clothing color information detected from an image captured by the camera device 506 of the camera sensor unit 104 or the camera device 506a of the camera sensor unit 104a. Therefore, for a customer who is tracked based on skeletal information and / or clothing color information, the person ID can be continuously maintained. Also, even if the customer comes to a specified appearance area (near the entrance) in the store again, it is possible to avoid / correct the assignment of a new person ID to the customer.

[0342] Referring to FIG. 60, a simplified diagram of how customer tracking can be achieved is shown. In this embodiment, for example, based on the position information, bone structure information, and / or clothing color information, the customer number determination module 1105 holds the person IDs of the customers tracked by the camera device 506 of the camera sensor unit 104. Therefore, as illustrated in (1) to (2) of FIG. 60, customers who are close to each other can be identified. Furthermore, as illustrated in (3) of Figure 60, if tracking temporarily ends without the customer's position entering the disappearance area (see dotted line A), for example, the customer number determination module 1105 can assume that the customer remains in the store and retain the person ID.

[0343] For example, the customer number determination module 1105 allows data on each currently tracked customer to be constantly referenced. For example, each currently tracked customer is associated with bone structure information and / or clothing color information. For example, when remaining skeletal information and / or clothing color information is found among all customers in the store, the customer number determination module 1105 may assume that a customer who is assumed to remain in the store (a customer whose tracking has temporarily disappeared) is at the location where the skeletal information and / or clothing color information was found, and resume tracking from that location.

[0344] Referring to (4) to (5) in Figure 60, an example is shown in which a customer who has lost his / her location information and only has bone structure information and / or clothing color information re-enters the appearance area and is assigned a different person ID (2). In this way, in the conventional technology, once tracking of a person's ID is lost, a different person ID is assigned to that person.

[0345] In contrast, referring to (1) to (3) in Fig. 61, in this embodiment, when a customer who has lost his / her position information and only has bone structure information and / or clothing color information re-enters the appearance area and is assigned a different person ID (2), the newly assigned person ID (2) is replaced with the original person ID (0). As a result, in this embodiment, it is possible to prevent a customer from being mistakenly identified as a different person.

[0346] As described above, in this embodiment, when extracting feature points (head, shoulders, elbows, wrists) limited to the upper body of a person based on an image captured by a camera, not only position information but also clothing information (color, shape, pattern, and / or feature amount of the clothing) may be extracted. For example, the customer number determination module 1105 may further detect color information around the shoulders of the human body by applying a skeleton detection model to the image captured by the camera. For example, the customer number determination module 1105 may assign a person ID to the owner of the shoulder based on the detected shoulder color information. Therefore, in this embodiment, it is possible to more accurately track the movement of customers in a store, which in turn makes it possible to accurately determine whether customers have picked up products in the store.

[0347] In this embodiment, a camera capable of acquiring depth information can be used as the camera sensor unit 104a. However, in other embodiments, a normal camera (2D camera) can be used as the camera sensor unit 104a. In addition, in this embodiment, the camera sensor unit 104 can track the position using a ToF sensor module. However, in other embodiments, the position tracking module itself using the camera sensor unit 104 can be omitted. In other embodiments, user identification and matching may be performed between images captured by multiple camera sensor units 104a. In this way, it is possible to track the customer's position and obtain skeletal information using only the multiple camera sensor units 104a.

[0348] In this embodiment, the criterion (threshold) for determining the degree of approach when a customer reaches out toward a product can be set by manually acquiring the product position. However, in other embodiments, it is possible to apply automatic correction (calibration) to the setting of this criterion. In other embodiments, the coordinates of the customer's hand when picking up a product can be acquired, and the position of each product can be overwritten to automatically correct the product position.

[0349] In this embodiment, if the judgment of which customer picked up the product is insufficient, the judge can make a manual judgment by the judge. In this case, the judge can directly play back the video captured by the camera sensor units 104, 104a. However, in other embodiments, a means for assisting the judge's judgment can be provided. In another embodiment, the position and orientation of each camera (camera parameters) can be calculated by calibrating the camera that captures the video for judgment. Furthermore, by using these parameters, the position of the product or the position of the user on the screen can be calculated and annotated in the video to make it easier for the judge to understand.

[0350] The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0351] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0352] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. The above-described embodiments disclose at least the configurations described in the claims. [Explanation of symbols]

[0353] 1...Product management system, 101...Management server, 102...Mobile terminal, 103...Signage terminal, 104...Camera sensor unit for position tracking, 104a...Camera sensor unit for bone structure detection, 105...Weight sensor unit, 106...Store entry management terminal, 107...Payment processing server, 506...Camera device, 506a...Camera device, 608...Weight sensor device, 630...Data aggregation transmitter, 710...Microcomputer, 2100...Frame for installing on product shelf

Claims

1. A management system for managing products, A means for acquiring weight information transmitted from a weight sensor installed on a product shelf; A means for acquiring customer position information from an image captured by a first camera sensor; a means for acquiring body feature points of the customer from an image captured by a second camera sensor by applying a skeleton detection model, and calculating an estimated hand point using the body feature points; and a means for determining whether or not a product has been picked up based on the customer's position information, a correlation between the calculated estimated hand point and the product shelf position, and a change in the output of the weight sensor; The means for determining the pickup of the product includes: determining a period of reaching based on a correlation between the estimated hand point and the shelf location, the period during which a hand movement occurs that allows the customer to pick up the product; determining an amount of weight change at both ends of the hand-stretching period for each of the plurality of customers based on the change in output of the weight sensor; determining the customer who picked up the product based on the weight change amount calculated for each of the plurality of customers; Management system.

2. the second camera sensor is a camera sensor unit for detecting a skeleton, and detects characteristic points of an elbow and a wrist of the human body by applying the skeleton detection model to a captured image; Using the positions of the detected feature points, a position obtained by extending the length of the forearm, which is the length between the elbow and the wrist, by a predetermined multiple is calculated as an estimated point of the hand. The management system according to claim 1 .

3. The camera sensor unit for detecting skeleton calculates a position where the length of the forearm is extended within any one of a range of 4 / 3 times ±25%, a range of 1.33 times ±25%, or a range of 1.3 times ±25% as an estimated point of the hand. The management system according to claim 2.

4. The skeleton detection camera sensor unit includes: The position of the center that bisects the length of the forearm is calculated as the forearm center point; A position obtained by extending a length from the elbow to the center point of the forearm by a predetermined multiple along the direction of the forearm is calculated as an estimated point of the hand. The management system according to claim 2.

5. The skeleton detection camera sensor unit includes: Calculating the forearm center point from a two-dimensional coordinate in a horizontal direction between the elbow and the wrist; A position obtained by extending the length from the elbow to the center point of the forearm in three-dimensional coordinates by a predetermined multiple is calculated as an estimated point of the hand. The management system according to claim 4.

6. The camera sensor unit for detecting bone structure detects seven feature points of the human body, including the head, the left and right shoulders, the left and right elbows, and the left and right wrists. The management system according to any one of claims 2 to 5.

7. The first camera sensor is a position tracking camera sensor unit that tracks the position of the customer and is a camera sensor that measures a distance to the customer. The management system according to any one of claims 1 to 6.

8. The first camera sensor is a position tracking camera sensor unit that tracks the position of the customer, and converts the information of the feature points and the estimated hand point transmitted from the second camera sensor into a coordinate space identical to the position information of the customer and integrates the information. The management system according to any one of claims 1 to 7.

9. At least one of the first camera sensor and the second camera sensor further detects color, shape, pattern, and / or feature information of a shoulder area of ​​the human body of the customer; assigning identification information to the customer corresponding to the detected shoulder using information on the color, shape, pattern, and / or feature amount of the shoulder; The management system according to any one of claims 1 to 8.

10. The means for determining the pickup of the product includes: Weight information for each product is acquired, and a customer for whom the weight change amount calculated for each of the plurality of customers is close to an integer multiple of the weight information for each product is determined to be the customer who picked up the product. The management system according to claim 1 .

11. The means for determining the pickup of the product includes: a correction is performed by shifting the time of the change in the output of the weight sensor or the time of the change in the position of the estimated point of the hand forward or backward; The management system according to any one of claims 1 to 10.

12. The means for determining the pickup of the product includes: When it is determined that the number of customers near the product is one, if there is a gap in the period of the customer's reaching, a correction is made to eliminate or minimize the gap.

12. The management system of claim 1, 10 or 11.

13. The means for determining the pickup of the product includes: When it is determined that there are a plurality of customers near the product, if there is a gap between the periods of the plurality of customers stretching their hands, a correction is performed to enlarge the hand movements of the customers before and after the gap so as to eliminate or minimize the gap. The management system according to any one of claims 1 to 10.

14. A management method for managing products, which is executed by a management server, comprising the steps of: Acquires weight information transmitted from weight sensors installed on product shelves, Acquire customer location information from an image captured by the first camera sensor; acquiring body feature points of the customer from an image captured by a second camera sensor by applying a skeleton detection model, and calculating an estimated hand point using the body feature points; determining a period of reaching, which is a period during which the customer moves their hands to pick up the product, based on the correlation between the customer's position information, the estimated hand point, and the product shelf position; determining an amount of weight change at both ends of the hand-stretching period for each of the plurality of customers based on the change in output of the weight sensor; determining the customer who picked up the product based on the weight change amount calculated for each of the plurality of customers; A management method comprising: executing a process.

15. A program for causing a management server to execute each step of the method according to claim 14.

Images