Smart Grip

Abstract

We provide technology for easily monitoring user purchasing behavior. [Solution] A smart grip according to one aspect of the present disclosure is configured to be usable as a handle for a shopping basket. The smart grip comprises a detection sensor, a positioning sensor, and a control device. The detection sensor is configured to detect when an item is placed into the basket. The positioning sensor is configured to measure a position. The control device is configured to, when the detection sensor detects that an item has been placed into the basket, generate an item placement record that includes the event location where the item was placed, which is measured by the positioning sensor, and to save the generated item placement record.

Description

Technical Field

[0001] The present disclosure relates to a smart grip, a shopping basket, and an information processing device.

Background Art

[0002] There is a system that monitors the purchasing behavior of users in a store by installing a plurality of sensors such as imaging devices and load sensors in the store and measuring the behavior of users with the installed plurality of sensors. For example, in Patent Document 1, a system is proposed that receives an indicator located within a facility by a user, receives an order for a product via an input device, and determines whether the order is associated with the user based on sensing data generated by one or more sensors in the facility.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When monitoring the purchasing behavior of users only with sensors installed in the store, the monitoring process may become complicated, such as the sensor arrangement being different for each store, the number of sensors becoming huge, and the amount of sensing data to be processed becoming huge. That is, it may become difficult to simply monitor the purchasing behavior of users.

[0005] In one aspect, the present disclosure has been made in view of such circumstances, and one of its purposes is to provide a technique for simply monitoring the purchasing behavior of users. Another one of the purposes is to provide a technique for utilizing the product input record generated by monitoring the purchasing behavior.

Means for Solving the Problems

[0006] This disclosure adopts the following configuration to solve the aforementioned problems. Note that the following configurations can be combined as appropriate.

[0007] A smart grip relating to one aspect of this disclosure is configured to be used as a handle for a shopping basket. The smart grip comprises a detection sensor, a positioning sensor, and a control device. The detection sensor is configured to detect when an item (target item) is placed into the basket. The positioning sensor is configured to measure a location. When the detection sensor detects the placement of an item, the control device is configured to generate an item placement record that includes the event location where the item was placed, which is measured by the positioning sensor, and to store the generated item placement record.

[0008] According to this configuration, by generating product placement records using detection and positioning sensors provided in the smart grip, user purchasing behavior can be monitored without installing sensors in the store. This simplifies monitoring. The smart grip according to this configuration may also be used in conjunction with sensors installed in the store to monitor user purchasing behavior. In this case, by using the product placement records to improve the efficiency of processing sensing data obtained from sensors installed in the store, monitoring can be further simplified.

[0009] The smart grip relating to the above aspect is a housing that accommodates a detection sensor, a positioning sensor, and a control device, and may further include a housing that has one or more grooves for hooking onto a handle portion provided on a basket. With this configuration, a smart grip that can be used as a basket handle can be provided by replacing the existing basket handle portion.

[0010] The smart grip relating to the above aspect may further include an output device. The control device may be further configured to output relevant information related to the insertion of a product from the output device when the detection sensor detects the insertion of a product. With this configuration, by providing relevant information when a product is inserted, it is possible to stimulate purchasing activities, such as further stimulating the desire to purchase the product.

[0011] In the smart grip relating to the above aspect, the relevant information may include at least one of advertising information and coupon information for other products placed around the event location. With this configuration, it is possible to expect to stimulate the desire to purchase other products by providing at least one of the advertising information and coupon information.

[0012] The smart grip relating to the above aspect may further include an imaging device. The control device may be configured to further perform the following actions: when the detection sensor detects the insertion of a product, it acquires an image captured by the imaging device, and identifies the product in the image by performing image analysis on the acquired image. With this configuration, the inserted product can be identified on the smart grip.

[0013] The form of this disclosure is not limited to the smart grip described above. One aspect of this disclosure may be a component of a shopping basket. Another aspect of this disclosure may be the basket (including the cart) itself. Furthermore, one aspect of this disclosure may be an information processing method that implements all or part of the control device (computer) described above, or a program, or a machine-readable storage medium that stores such a program. Here, a machine-readable storage medium may be a non-temporary medium that stores information such as programs by electrical, magnetic, optical, mechanical, or chemical action. A non-temporary storage medium may include storage media (CDs, DVDs, semiconductor memory, etc.), auxiliary storage devices of a computer, external storage devices connected to a computer, etc.

[0014] For example, a shopping cart relating to one aspect of this disclosure may comprise a cart body, a detection sensor, a positioning sensor, and a control device. The detection sensor may be configured to detect when an item is placed into the cart body. The positioning sensor may be configured to measure a position. The control device may be configured to, when the detection sensor detects that an item has been placed in, generate an item entry record that includes the event position where the item was placed, which is measured by the positioning sensor, and to save the generated item entry record.

[0015] The shopping basket relating to the above aspect may be configured as a cart by further equipping it with one or more wheels. With this configuration, it is expected that monitoring of user purchasing behavior can be simplified in the form of a cart.

[0016] Furthermore, an information processing device relating to another aspect of this disclosure includes a control unit. The control unit is configured to acquire product input records generated by a shopping device, which include the event location where the target product was input, identify the input target product according to the acquired product input records, and output information regarding the results of the target product identification. This configuration makes it possible to provide a technology for identifying input products using product input records.

[0017] In the information processing device relating to the above aspect, identifying a target product in accordance with the product input record may be performed by comparing the product input record with map information that includes identification information of the product in association with the location where the product is displayed in the store, thereby identifying the target product that was input at the event location. With this configuration, the target product can be appropriately identified by using the map information.

[0018] In the information processing device relating to the above aspect, identifying a target product in accordance with the product input record may be comprised of acquiring an image captured by an imaging device that includes the event location of the product input record within its imaging range, and then identifying the target product in the image by performing image analysis on the acquired image. Performing image analysis on all images obtained from all imaging devices installed in the store could result in an enormous amount of processing load. In contrast, with this configuration, the range of images subject to image analysis can be narrowed down to the range of event detection by the product input record. This makes it possible to improve the efficiency of product identification processing (image analysis).

[0019] In the information processing device relating to the above aspect, load sensors may be installed on each product shelf in the store. Identifying a target product in accordance with the product loading record may be performed by acquiring measurement data from load sensors installed near the event location of the product loading record, and identifying the product shelf whose weight has decreased according to the acquired measurement data. Constantly monitoring all load sensors installed in the store could result in an enormous amount of processing load. In contrast, with this configuration, the range of load sensors to be monitored can be narrowed to the range of event detection by the product loading record. This makes it possible to improve the efficiency of the product identification process.

[0020] Furthermore, an information processing device relating to another aspect of this disclosure includes a control unit. The control unit is configured to acquire product entry records generated by a shopping device, including the event location where the target product was entered, generate the target user's movement history from the event location of the acquired product entry record, and output the generated movement history. This configuration makes it possible to provide a technology for monitoring the movement history of a target user using product entry records.

[0021] In addition, an information processing apparatus according to another aspect of the present disclosure includes a control unit. The control unit acquires a product input record of a target user generated by a shopping device, the product input record including the event position where the product was input, acquires a purchase record of the purchased products of the target user, and collates the acquired product input record and purchase record with map information including identification information of the product in association with the position where the product is displayed in the store, thereby identifying the location where the target user put the purchased product in the shopping cart, and outputs information regarding the identified location. The same product may be placed at a plurality of locations (for example, permanent shelves, special shelves, etc.) in the store. According to this configuration, it is possible to provide a technique for identifying the location where the target user put the purchased product in the shopping cart by using the product input record.

[0022] In the information processing apparatus according to each of the above aspects, the shopping device may be at least one of the smart grip and the shopping cart according to any of the above aspects. That is, the smart grip and the shopping cart according to one of the above aspects are an example of a shopping device. However, the shopping device is not limited to the smart grip and the shopping cart according to one of the above aspects. As long as it is possible to generate a product input record including the event position, the configuration and type of the shopping device are not particularly limited and may be appropriately selected according to the embodiment.

[0023] Note that the form of the present disclosure is not limited to the above information processing apparatus (computer). One aspect of the present disclosure may be an information processing method for realizing all or a part of the above information processing apparatus, an information processing system, a program, or a machine-readable storage medium such as a computer that stores such a program. The information processing system may be configured by, for example, a plurality of computers such as a server device and a store terminal.

Effects of the Invention

[0024] ​According to one aspect of the present disclosure, it is possible to provide a technique for simply monitoring a user's purchasing behavior. Further, according to another aspect of the present disclosure, it is to provide a technique for using a product input record generated by monitoring the purchasing behavior.

Brief Description of the Drawings

[0025] [Figure 1] FIG. 1 schematically shows an example of a scene to which the present disclosure is applied. [Figure 2] FIG. 2 schematically shows an example of a usage scene of a product input record. [Figure 3] FIG. 3 schematically shows an example of a method of using a product input record. [Figure 4] FIG. 4 schematically shows an example of a method of identifying a product using a product input record. [Figure 5] FIG. 5 schematically shows an example of a method of identifying a product using a product input record. [Figure 6] FIG. 6 schematically shows an example of a method of identifying a product using a product input record. [Figure 7] FIG. 7 schematically shows another example of a method of using a product input record. [Figure 8] FIG. 8 schematically shows another example of a method of using a product input record. [Figure 9] FIG. 9 is a side view schematically showing an example of the structure of a smart grip. [Figure 10] FIG. 10 is a plan view schematically showing an example of the structure of a smart grip. [Figure 11] FIG. 11 schematically shows an example of the hardware configuration of an information processing apparatus. [Figure 12] FIG. 12 schematically shows an example of the software configuration of a smart grip (control device). [Figure 13] FIG. 13 schematically shows an example of the software configuration of an information processing apparatus. [Figure 14] FIG. 14 is a flowchart showing an example of a processing procedure related to the generation of a product input record by a control device. [Figure 15] Figure 15 is a flowchart showing an example of a processing procedure related to product identification by a control device. [Figure 16] Figure 16 is a flowchart showing an example of a processing procedure for outputting related information by the control device. [Figure 17] Figure 17 is a flowchart showing an example of a processing procedure for product identification using product input records from an information processing device. [Figure 18] Figure 18 is a flowchart showing an example of a processing procedure for generating a movement history using product input records from an information processing device. [Figure 19] Figure 19 is a flowchart showing an example of a processing procedure for location identification using product input records from an information processing device. [Figure 20] Figure 20 schematically illustrates another example of a scenario in which this disclosure applies. [Figure 21] Figure 21 schematically illustrates another example of a scenario in which this disclosure applies. [Modes for carrying out the invention]

[0026] Hereinafter, embodiments relating to one aspect of this disclosure will be described with reference to the drawings. However, the embodiments described below are merely illustrative in all respects of this disclosure. Various improvements or modifications may be made without departing from the scope of this disclosure. In implementing this disclosure, specific configurations may be adopted as appropriate depending on the embodiment. In this embodiment, the data appearing is described in natural language, but more specifically, it is specified in pseudo-language, commands, parameters, machine code, electrical signals, etc., that can be recognized by machines such as computers.

[0027] §1 Examples of Application Figure 1 schematically shows an example of a scenario to which this disclosure applies. The smart grip 1 according to this embodiment is configured to be usable as a handle for a basket B1. The smart grip 1 comprises a control device 11, a detection sensor 13, and a positioning sensor 14. The detection sensor 13 is configured to detect when a product TP is placed into the basket B1. The positioning sensor 14 is configured to measure a position. When the detection sensor 13 detects the placement of a product TP, the control device 11 is configured to generate a product placement record 20 that includes the event position 201 in which the product TP was placed, as measured by the positioning sensor 14, and to save the generated product placement record 20.

[0028] According to this embodiment, by generating product input records 20 using the detection sensor 13 and positioning sensor 14 provided on the smart grip 1, it is possible to monitor user purchasing behavior without installing sensors in the store. This is expected to simplify the monitoring of purchasing behavior. The smart grip 1 according to this embodiment may also be used in conjunction with sensors installed in the store to monitor user purchasing behavior. In this case, by using the product input records 20 to improve the efficiency of processing sensing data obtained from sensors installed in the store, it is expected that the monitoring of purchasing behavior will be simplified.

[0029] [Basket] As shown in Figure 1, in a typical example, basket B1 may be a shopping basket comprising a basket body B10 and a pair of handles B11. If basket B1 comprises a basket body B10, then placing goods TP into the basket body B10 may be equivalent to placing goods TP into basket B1.

[0030] The configuration of the basket body B10 and each handle portion B11 may be determined as appropriate depending on the embodiment. For example, the basket body B10 may be configured to receive goods TP from the upper end, with the upper end open and the lower end closed at the bottom. The basket body B10 may have any shape, such as a roughly rectangular parallelepiped or a roughly truncated square pyramid. The pair of handle portions B11 may be arranged separately along the longitudinal direction of the basket body B10. Each end of the handle portion B11 may be connected to each side of the basket body B10 in the short direction so that the handle portion B11 can rotate around an axis in the short direction. In this way, each handle portion B11 may be configured to change its angle by rotating at the connection point with the basket body B10 and to be gripped along the short direction.

[0031] The configuration of the basket B1 is not limited to the example in Figure 1 and may be modified as appropriate depending on the embodiment. For example, the number of handles B11 is not limited to two. There may be one handle or three or more. Furthermore, each end of one handle B11 may be connected to one side of the basket body B10 in the short direction in the longitudinal direction, and each end of the other handle B11 may be connected to the other side of the basket body B10 in the short direction in the longitudinal direction. In this way, each handle B11 may be configured to be gripped along the longitudinal direction.

[0032] Furthermore, the type of basket B1 is not limited to the example in Figure 1 and may be appropriately changed depending on the embodiment. The type of basket B1 is not particularly limited as long as it can be used to carry the product TP, and may be appropriately selected depending on the embodiment. In another example, basket B1 may include bags that can be used for shopping, such as plastic bags or paper bags. Basket B1 may also include devices that can hold products by hanging them, such as hangers.

[0033] [Must be usable as a basket handle] The fact that it can be used as a handle for a basket (basket B1, etc.) means that when carrying a product (product TP), the Smart Grip 1 is positioned as at least part of the handle that the user grasps. It may be configured in any form in which it is placed. In one example, it may be usable as a basket handle by replacing an existing handle provided on the basket by hanging the basket handle portion (handle portion B11, etc.) on the smart grip 1. The basket handle portion may include any component that can be hung on the smart grip 1, such as a bag handle portion or a hanger hook. In another example, it may be usable as a basket handle by being attached to the basket handle portion. In yet another example, it may be usable as a basket handle by being attached to the basket body (basket body B10, etc.) as a handle portion. That is, the smart grip 1 may be configured as the basket handle portion itself.

[0034] [Product / Store] Product TP may include any type of item that is carried. Product TP may also be called the target product. Smart Grip 1 may be used in any store. The store configuration is not particularly limited and may be determined as appropriate depending on the embodiment. The store's sales format may be either unmanned or manned. Sensors such as imaging devices and load sensors may or may not be installed in the store. If sensors are installed in the store, the purchasing behavior of the target user may be monitored through the cooperation of the sensors of Smart Grip 1 (detection sensor 13, positioning sensor 14, etc.) and the sensors installed in the store. Alternatively, Smart Grip 1 may monitor the purchasing behavior of the target user independently of the sensors installed in the store (i.e., without coordinating with the store's sensors).

[0035] [Product launch] Placing a product (product TP) into a basket (basket B1) may include any method of carrying the product from the sales floor. As illustrated in Figure 1, a typical example of placing a product into a basket may be placing the product TP into the basket body B10 of basket B1, which includes the smart grip 1. In addition, placing a product into a basket may include any action similar to placing a product into a basket, such as hanging the handle of the basket containing the product onto the smart grip 1. Hanging the handle of the basket containing the product may include hanging the handle of the plastic bag containing the product, or hanging the hook of the hanger from which the product is hanging, etc.

[0036] [Product Input Record] The product input record 20 is configured to include an event location 201. The event location 201 is derived from the measurement values ​​of the positioning sensor 14. For example, the measurement values ​​of the positioning sensor 14 may be used directly as the event location 201. Alternatively, the event location 201 may be calculated from the measurement values ​​of the positioning sensor 14 by applying arbitrary calculation processing to one or more measurement values ​​of the positioning sensor 14. The event location 201 may be derived from values ​​measured in real time by the positioning sensor 14 in response to the detection of product input TP by the detection sensor 13. The event location 201 may be configured as appropriate to indicate the position where the product TP was input (input position). In a typical example, the event location 201 may be configured to indicate the input position using two-dimensional or three-dimensional coordinate values. However, the configuration of the event location 201 is not limited to such examples and may be modified as appropriate depending on the embodiment. As long as this event location 201 is included, the configuration of the product input record 20 is not particularly limited and may be defined as appropriate depending on the embodiment. The product input record 20 may consist only of the event location 201, or it may also include other information besides the event location 201.

[0037] (Identification information) As an example of other information, the product input record 20 may further include identification information 203 for identifying the user (target user) using the smart grip 1. The source of the identification information 203 may be determined as appropriate depending on the embodiment. In one example, the smart grip 1 stores A further unit 12 may be provided. Identification information 125 may be stored in the storage unit 12 in advance. The identification information 125 may be configured arbitrarily as long as it is possible to identify the target user. For example, the identification information 125 may be configured to directly indicate the target user, such as the target user's account or user identifier. The identification information 125 may also be configured to indirectly indicate the target user, such as the identifier of the smart grip 1 or the identifier of the control device 11. The identification information 203 of the product input record 20 may be appropriately derived from the identification information 125 in the storage unit 12. In one example, the identification information 125 in the storage unit 12 may be used as is as the identification information 203 of each product input record 20. In another example, the identification information 203 may be calculated from the identification information 125 by applying arbitrary arithmetic operations (deletion of values, addition of values, arithmetic operations, conversion, etc.) to the identification information 125.

[0038] According to one example of this embodiment, since the product input record 20 includes identification information 203, the user corresponding to the product input record 20 (i.e., the user who input the product TP at the event location 201 indicated in the product input record 20) can be easily identified. When product input records 20 from multiple users are collected, the product input record 20 of a specific user can be easily extracted from the collected product input records 20 by searching using the identification information 203.

[0039] Furthermore, if the system uses equipment installed in the store to identify users corresponding to product placement events, it may increase the burden of installing sensors and the processing load for tracking people. For example, this could involve installing imaging devices in multiple locations within the store and periodically performing image analysis (such as face recognition) on the images obtained from these devices. In contrast, according to one example of this embodiment, the user corresponding to a product placement event can be identified by linking the identification information 125 of the smart grip 1 to the product placement record 20 as identification information 203. This can reduce at least one of the burdens of installing sensors in the store or the processing load for tracking people.

[0040] Furthermore, the identification information 125 held in the smart grip 1 may be used for purposes other than deriving the identification information 203 of the product input record 20. For example, the identification information 125 of the smart grip 1 may be used to authenticate the user at least when entering the store or leaving the store. For example, the smart grip 1 may exchange data with the store's system (store terminal, etc.) when entering the store or leaving the store. The method of data exchange may be arbitrarily selected.

[0041] As an example of a data exchange method, the smart grip 1 may further include an output device 15. The output device 15 may include a display. The smart grip 1 (control device 11) may convert the identification information 125 into a code and output the obtained code from the output device 15. The code may further include any information other than the identification information 125. For example, the code may be generated as a one-time code by further including temporary information such as time and random numbers. The type of code may be arbitrarily selected. For example, the code may consist of a one-dimensional code or a two-dimensional code. Code readers may be provided at the entrance and exit of the store. The store's system may perform data exchange with the smart grip 1 by reading the information (including the code) from the smart grip 1 using the reader.

[0042] As another example of a data exchange method, the smart grip 1 may further include a communication module 17. The communication module 17 may include a proximity wireless communication module. The proximity wireless communication standard is not particularly limited and may be appropriately selected depending on the embodiment. Accordingly, modules (such as readers) corresponding to the proximity wireless communication module of the smart grip 1 may be provided at the entrance and exit of the store. Accordingly, the smart grip 1 and the store system may perform data exchange via the proximity wireless communication module.

[0043] The store's system may obtain identification information 125 from Smart Grip 1 during data exchange. The store's system may authenticate the user according to the obtained identification information 125. Good. The store system may be configured to control the unlocking and locking of the entrance and exit doors in response to user authentication. For example, the store system may keep the entrance and exit doors locked until user authentication is performed. The store system may unlock the corresponding entrance or exit door in response to successful user authentication. After the user has passed through the entrance or exit (for example, after opening and then closing the door), the store system may relock the corresponding entrance or exit door. According to this example embodiment, users present in the store can be easily identified. Furthermore, the control device 11 of the smart grip 1 may start executing processes to monitor the user's purchasing behavior, such as generating a product entry record 20, after successful authentication at the entrance. After successful authentication at the exit, the control device 11 may stop executing the monitoring process. This allows for appropriate control of the operation of the control device 11 so that the monitoring process is performed within the store.

[0044] (time) Furthermore, as an example of other information, the product input record 20 may further include the time 205 when the product input record 20 was generated (i.e., when the product TP was input). The source of time 205 may be determined as appropriate depending on the embodiment. In one example, the smart grip 1 may further include a timer. The control device 11 may refer to the timer when it detects the input of product TP by the detection sensor 13 (generating the product input record 20) and obtain the time indicated by the timer as time 205. According to one example of this embodiment, by using time 205, it is possible to extract product input records 20 (input events) corresponding to a specific time period from among multiple product input records 20. Also, by using time 205, it is possible to classify the product input records 20 by time period and analyze the characteristics of the input events in each time period (for example, which product TPs located in which positions are more likely to be purchased).

[0045] (others) In the example shown in Figure 1, the product input record 20 includes the event location 201, identification information 203, and time 205. However, the configuration of the product input record 20 is not limited to this example and may be modified as appropriate depending on the embodiment. For example, at least one of the identification information 203 and time 205 may be omitted. Furthermore, the product input record 20 may include other information besides the identification information 203 and time 205. In one example, the product input record 20 may further include an identifier for identifying the product input record 20.

[0046] [Saving records of product input] The product input record 20 may be stored in any memory area. In one example, the smart grip 1 may further include a storage unit 12. The generated product input record 20 may be stored in the storage unit 12. In another example, the smart grip 1 may further include a communication module 17. In this case, the smart grip 1 may transmit the product input record 20 to an external computer (including the information processing device 2 described later), such as a terminal or server device, via the communication module 17. As a result, the product input record 20 may be stored in the memory area of ​​the external computer. The terminal may include a user terminal, a store terminal (a terminal installed in a store), etc. The server device may include an external storage device such as a NAS (Network Attached Storage). In either the case where the product input record 20 is stored in the storage unit 12 within the smart grip 1 or in an external computer, storing the product input record 20 may involve using cache memory, RAM (Random This may include temporarily storing the product entry record 20 in memory such as Access Memory.

[0047] Furthermore, the process of saving the product input record 20 may be executed at any time after the process of generating the product input record 20 has been executed once or more. In one example, Smart Grip 1 may sequentially save the generated product input records 20 in response to the generation of the product input records 20. In another example, Smart Grip 1 may repeatedly execute the generation of product input records 20 to form a group of product input records, and save at least a portion of the formed group of product input records. (Multiple product input records 20) may be saved all at once.

[0048] [Product identification using smart grip] In one example of this embodiment, the control device 11 may be configured to identify product TP according to the observation results of product TP by sensors provided on the smart grip 1. The sensors used for identification may be appropriately selected depending on the embodiment. The detection sensor 13 may be used for identification, or sensors other than the detection sensor 13 may be used for identification. The identification method may be appropriately determined depending on the sensors used.

[0049] In one example, the smart grip 1 may further include an imaging device 16. The imaging device 16 may be positioned appropriately so as to capture the product TP being placed in the basket B1 (basket body B10) when the product TP is placed in the basket. For example, the imaging device 16 may be positioned so that the basket B1 (basket body B10, etc.) is within its imaging range. The type of imaging device 16 is not particularly limited as long as it can generate an image in which the product TP can be identified, and may be appropriately selected depending on the embodiment. The imaging device 16 may include any sensor that acquires data in the form of an image or image representation, such as an RGB camera, depth sensor, infrared sensor, radar, LiDAR (light detection and ranging), etc.

[0050] The control device 11 may acquire an image 165 captured by the imaging device 16 when it detects the insertion of a product TP using the detection sensor 13. The timing of imaging by the imaging device 16 (generation of the image 165) may be appropriately defined depending on the embodiment. For example, if the imaging device 16 is positioned to capture the product TP after insertion detection, the control device 11 may activate the imaging device 16 in response to the detection of product TP insertion by the detection sensor 13 and control the operation of the imaging device 16 to capture the inserted product TP. The control device 11 may control the imaging device 16 to capture the product TP at any timing after detecting the insertion of the product TP. For example, the control device 11 may control the imaging device 16 to capture the product TP immediately after detecting the insertion of the product TP. The control device 11 may appropriately acquire one or more images 165 generated by this imaging from the imaging device 16. After generating one or more images 165, the control device 11 may stop the operation of the imaging device 16 or switch to a standby state. In this example, the period during which the imaging device 16 is activated can be narrowed, which is expected to reduce the processing load and power consumption in the smart grip 1. In another example, the imaging device 16 may be controlled to continuously generate captured images. The generated captured images may be stored in a storage area such as the storage unit 12 for at least a certain period of time. Captured images generated at or around the time when the detection sensor 13 detects the insertion of the product TP are likely to show the inserted product TP. Therefore, the control device 11 may acquire one or more captured images generated at or around the time when the detection sensor 13 detects the insertion of the product TP from the captured images stored in the storage area as captured image 165.

[0051] After acquiring the captured image 165, the control device 11 may identify the product TPs (product products) in the captured image 165 by performing image analysis on the acquired image 165. The identification method by image analysis may be determined as appropriate depending on the embodiment. In one example, if a code generated by encoding the identification information of the product TP is assigned to the product TP, analyzing the captured image 165 may include decoding the code assigned to the product TP. The type of code may be arbitrarily selected. The code may be, for example, a one-dimensional code, a two-dimensional code, etc. In another example, the control device 11 may identify the product TPs in the image analysis based on their characteristics (appearance, etc.) rather than on the code.

[0052] The image analysis method is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the control device 11 may analyze the captured image 165 using general image analysis techniques such as edge detection and pattern matching. In another example, the control device 11 analyzes the image The captured image 165 may be analyzed using a trained machine learning model that has acquired capabilities. The machine learning model is configured to include one or more computational parameters that can be adjusted by machine learning. One or more computational parameters are used for the calculation of the desired inference (such as image analysis). The machine learning model may consist of, for example, a neural network, a support vector machine, or other functional equations (computational models). The machine learning method may be appropriately selected depending on the machine learning model to be adopted (for example, backpropagation). Training a machine learning model involves adjusting (optimizing) the values ​​of the computational parameters using training samples. The machine learning model may be appropriately trained to derive the true value of the corresponding analysis result when given images of training samples. A large-scale model such as a Vision Language Model (VLM) may be used as the trained machine learning model.

[0053] The control device 11 may output the product TP identification result as appropriate. The product TP identification result may be used for any purpose. For example, the product TP identification result may be added to the product input record 20. That is, generating the product input record 20 including the event location 201 may be configured by generating the product input record 20 including the event location 201 and the product TP identification result. According to this example, by referring to the product TP identification result, the product TP that was input in the event indicated by the product input record 20 can be identified. If an output device 15 is provided, the control device 11 may output the product TP identification result from the output device 15. Also, if a communication module 17 is provided, the control device 11 may output the product TP identification result to an external computer (store terminal, user terminal, server device, etc.).

[0054] In one example of this embodiment, the inserted product TP can be identified on the Smart Grip 1. The computer that performs the image analysis of the captured image 165 is not limited to the control device 11. In another example, the control device 11 may send the captured image 165 along with a request to perform image analysis on the captured image 165 to an external computer such as a store terminal, user terminal, or server device. The control device 11 may then obtain the product TP identification result by receiving a reply from the external computer containing the image analysis result, including the product TP identification result. The method for identifying the product TP is not limited to the image analysis method described above and may be modified as appropriate depending on the embodiment. In another example, the Smart Grip 1 may further be equipped with a code reader. The control device 11 may identify the product TP by reading the code assigned to the product TP with the reader. Furthermore, the identification process on the Smart Grip 1 may be omitted. The product insertion record 20 may be generated without identifying the product TP. The process of identifying the product TP may be performed by an external computer (including the information processing device 2 described later) instead of the control device 11.

[0055] [Output related information] In one example of this embodiment, the smart grip 1 may further include an output device 15. The output device 15 may be configured to output information. The output device 15 may consist of, for example, a display, a speaker, an indicator light, etc. The format of the information output may be appropriately selected from text, images, sound, etc. If the output device 15 includes a display, it is preferable that the display be placed in a location where the output information can be seen while the user is holding the smart grip 1. This allows the user to check the information displayed on the display while using the smart grip 1.

[0056] The conditions for outputting information and the information to be output may be determined as appropriate depending on the embodiment. In one example, the control device 11 may be further configured to output related information 155 related to the input of product TP from the output device 15 when the detection sensor 13 detects the input of product TP. According to this example embodiment, by providing related information 155 when product TP is input, it is possible to stimulate purchasing activities, such as further stimulating the desire to purchase products.

[0057] The related information 155 may be output at any time after the detection sensor 13 detects the insertion of the product TP. For example, the control device 11 may output the related information 155 in real time from the output device 15 after the detection sensor 13 detects the insertion of the product TP. The source of the related information 155 may be determined as appropriate depending on the embodiment. For example, if the smart grip 1 is equipped with a storage unit 12, the related information 155 may be stored in the storage unit 12 in advance. The control device 11 may acquire the related information 155 from the storage unit 12. Alternatively, for example, the related information 155 may be stored in the storage area of ​​an external computer (store terminal, user terminal, server device, etc.). The control device 11 may acquire the related information 155 from the external computer at any time before outputting it.

[0058] Furthermore, the content of the related information 155 is not particularly limited as long as it relates to purchasing activities such as the event location 201, the product TP to be inserted, and the act of inserting the product TP, and may be appropriately selected depending on the embodiment. If there are multiple candidates for related information 155, the related information 155 to be output may be appropriately selected from among the multiple candidates. For example, related information 155 may be selected from among multiple candidates depending on at least one of the event location 201, product TP, and user information. Also, for example, related information 155 may be selected randomly. The selection of related information 155 may be performed by either the control device 11 or an external computer (store terminal, user terminal, server device, etc.). When selecting related information 155 from multiple candidates, any information may be referenced. The information used for selecting related information 155 may be held by at least one of the control device 11 or an external computer and may be appropriately referenced when selecting related information 155.

[0059] In one example of this embodiment, the content of the related information 155 may consist of at least one of the following 1 to 7 examples. For the sake of explanation, in each of the following examples, it is assumed that the control device 11 selects the related information 155. However, as mentioned above, the computer that performs the process of selecting the related information 155 is not limited to the control device 11. At least a part of the processing of the control device 11 described below may be performed by an external computer. The information that the control device 11 refers to when selecting the related information 155 may be held in either the control device 11 or the external computer.

[0060] (1) Case 1 In one example, related information 155 may include at least one of the advertising information 1550 and coupon information 1555 for other products placed around event location 201. Other products are products other than the product TP (target product) that was introduced. Advertising information 1550 includes advertisements for other products. The content of the advertisements may be determined as appropriate. In a simple example, advertising information 1550 may consist of the names of other products. Coupon information 1555 includes information indicating discounts on other products.

[0061] Other products may be identified as appropriate. For example, candidate related information 155 may be predetermined for each area within the store. Accordingly, when the placement of product TP is detected, the control device 11 may identify the area to which the event location 201 belongs, and select related information 155 according to the identified area. When identifying product TP, the control device 11 may exclude candidates corresponding to the product TP indicated by the identification result and select related information 155 from the remaining candidates. Alternatively, for example, the control device 11 may refer to information indicating the correspondence between the display location of products within the store and the identification information of the products (such as map information 30 described later). The control device 11 may determine other products by selecting products displayed in the vicinity of event location 201 but at a different location from event location 201, from among the products within the store indicated by the referenced information. The scope of "periphery" may be determined as appropriate. For example, whether or not it belongs to the "periphery" may be determined by a method such as threshold determination. The control device 11 may obtain related information 155 by selecting a candidate linked to other products from among multiple candidates.

[0062] In one example of this embodiment, by providing at least one of the advertising information 1550 and the coupon information 1555, it is possible to stimulate the desire to purchase other products. In this example of this embodiment, it is preferable that the control device 11 outputs related information 155, including at least one of the advertising information 1550 and the coupon information 1555 of other products, in real time immediately after detecting the insertion of product TP by the detection sensor 13. This makes it possible to provide the target user with at least one of the advertising information 1550 and the coupon information 1555 of other products while the target user is located near those products. In other words, it is possible to provide the target user with related information 155 of products located near them. As a result, since less effort is required to pick up other products, it is possible to expect an increase in the probability that the target user will purchase other products.

[0063] (2) Case 2 For example, the related information 155 may also relate to product TP. For instance, it may include at least one of the advertising information, coupon information, and attribute information of product TP. The attribute information may include all kinds of information relating to the characteristics of product TP. For example, the attribute information may consist of the product TP's expiration date, release date, allergy information, reviews, related products, recommended products, or a combination thereof. The allergy information indicates the allergens contained in product TP. Related products may be other products related to product TP. For example, other products related to product TP may be products used together with product TP. Recommended products may be other products recommended to people who purchase product TP. Related products and recommended products may partially overlap or not overlap. The related information 155 relating to product TP may be specified as appropriate. For example, when identifying product TP, the control device 11 may obtain the related information 155 relating to product TP according to the product TP identification result. According to one example of this embodiment, information about product TP can be provided to the user.

[0064] (3) Third case For example, the content of related information 155 may be determined according to past purchase history. The control device 11 may identify products that have not been purchased for a certain period of time by referring to the purchase history. Related information 155 may consist of a notification informing the user of the identified product. Also, when identifying a product TP, the control device 11 may extract the history related to the product TP from the purchase history according to the product TP identification result. Related information 155 may consist of information regarding the purchase history of the extracted product TP. For example, if the purchase history of the product TP determines that the product TP has been purchased multiple times in the past, related information 155 may consist of a notification recommending a change to a subscription. The threshold number of purchases for recommending a change to a subscription may be arbitrarily defined. Also, for example, if the product TP has been purchased within a predetermined period in the past, related information 155 may consist of a notification indicating that the product TP has been purchased in the past. The predetermined period may be arbitrarily defined. According to this example, information related to past purchase history can be provided to the user.

[0065] (4) Case Study 4 For example, related information 155 may consist of information about the budget. For example, related information 155 may consist of a notification (warning) indicating that the budget limit is approaching. The control device 11 may acquire the amount (price) of the product TP as appropriate. For example, the information indicating the amount may be linked to the product TP (identification information, etc.). When identifying a product TP, the control device 11 may acquire information indicating the amount of the product TP according to the identification result of the product TP. Each time the detection sensor 13 detects the input of a product TP, the control device 11 may calculate the total amount of one or more product TPs placed in basket B1 by adding the acquired amounts. The control device 11 may then determine whether the calculated total amount is close to the budget limit. If it is determined that the total amount is approaching the budget limit, the control device 11 will The control device 11 may output related information 155 consisting of a notification indicating that the budget limit has been reached. Otherwise, the control device 11 may omit outputting related information 155 consisting of a notification indicating that the budget limit has been approached. The budget limit may be set arbitrarily. For example, the budget limit may be input by operating the control device 11 or via an external computer. Furthermore, whether or not the budget limit has been approached may be determined by any method such as threshold determination. The threshold may be defined arbitrarily. According to one example of this embodiment, information regarding the budget can be provided to the user, such as a warning that the budget limit has been approached. Note that the information regarding the budget may be changed as appropriate. For example, whether or not the budget limit has been approached may be replaced with whether or not the budget limit has been exceeded. The stage for notifying a warning is not limited to the budget limit. Furthermore, multiple stages for notifying a warning may be set, and the notification content may be changed for each stage.

[0066] (5) Case 5 For example, related information 155 may consist of information about the product TP and any information derived from the user information of the target user. User information may include any information about the target user. User information may also include information about the target user's family. For example, if the product TP contains an allergen that is present in at least one of the target user's family members, related information 155 may consist of a warning indicating that the product TP contains an allergen. In this case, user information may include allergy information indicating the allergen that is present in at least one of the target user's family members. Allergy information for the product TP may be acquired as appropriate. For example, attribute information including allergy information may be linked to the product TP (identification information, etc.). When identifying a product, the control device 11 may acquire the allergy information for the product TP according to the product identification result. The control device 11 may refer to user information as appropriate. The control device 11 may compare the allergy information for the product TP with the allergy information included in the user information. The control device 11 may determine, according to the result of the comparison, whether or not the product TP contains an allergen that is present in at least one of the target user's family members. If it is determined that an allergen is contained in product TP, the control device 11 may output a warning indicating that the product TP contains an allergen as related information 155. According to one example of this embodiment, information about the product TP concerning the target user, such as a warning indicating that the product TP contains an allergen, can be provided to the user. Note that the information derived from product TP and user information is not limited to the allergy-related information described above and may be appropriately modified depending on the embodiment.

[0067] (6) Case 6 For example, related information 155 may consist of information obtained by searching for the product TP to be introduced on the web. For example, related information 155 may consist of information showing a comparison between the in-store selling price and the selling price on an e-commerce (electronic commerce) site of the product TP to be introduced. The configuration may be as follows. Store and e-commerce site sales prices may be acquired as appropriate. For example, information showing the store sales price may be linked to product TP (identification information, etc.). When identifying product TP, the control device 11 may acquire information showing the store sales price of product TP according to the product TP identification result. The control device 11 may also acquire information showing the e-commerce site sales price by performing a web search according to the product TP identification result. The e-commerce site for searching sales prices may be arbitrarily selected. For example, one or more e-commerce sites may be predetermined. The control device 11 may generate information showing a comparison between the store sales price and the e-commerce site sales price based on the acquired information. The configuration of the comparison information may be determined as appropriate according to the embodiment. In a simple example, the control device 11 may output the store sales price and the e-commerce site sales price as they are in the related information 155. In another example, the control device 11 may determine whether the store sales price is cheaper than the e-commerce site sales price. If the selling price at the store is lower than the selling price on the e-commerce site, the control device 11 will include information indicating that the selling price at the store is lower than the selling price on the e-commerce site (for example, the message "NNN yen cheaper than the AAA site") as related information 155. It may be output in this way. According to one example of this embodiment, information about product TP obtained on the web, such as the result of a comparison with the selling price on an e-commerce site, can be provided to the user.

[0068] Furthermore, the information obtained through web searches is not limited to comparisons with sales prices on e-commerce sites and may be appropriately selected depending on the embodiment. Also, information retrieval may be performed using any method. For example, a publicly known search engine may be used for information retrieval. In another example, information retrieval may be performed using general methods such as extracting information containing specific words from specific sites, without using a publicly known search engine. A trained machine learning model, such as a Large Language Model (LLM), may also be used for information retrieval.

[0069] (7) Case 7 In one example, related information 155 may consist of coupon information with set usage conditions. The coupon information may be related to the product TP entered, as in the first example, the second example, etc. Alternatively, the coupon information may be unrelated to the product TP entered. The usage conditions are the conditions for receiving the discount shown in the coupon information. The usage conditions may be set as appropriate. The usage conditions may include, for example, the number of products to be purchased, the products to be purchased, and the expiration date. According to this example, by providing coupon information with set usage conditions, it is possible to stimulate the purchasing intent of target users.

[0070] [Use of product input records] The product input record 20 may be used for any purpose, such as identifying product TP or analyzing purchasing behavior within the store. Information processing regarding the product input record 20 may be performed in real time in response to the generation of the product input record 20, or it may be performed retrospectively after the product input record 20 has been generated. Information processing regarding the product input record 20 may be performed on the control device 11, or it may be performed on an external computer (store terminal, user terminal, server device, etc.).

[0071] Figure 2 schematically shows an example of how the product input record 20 according to this embodiment can be used. In one example, the smart grip 1 (control device 11) may directly or indirectly provide the generated product input record 20 to the information processing device 2 via a network. Providing it indirectly means providing it via another computer. The network standard is not particularly limited and may be appropriately selected depending on the embodiment.

[0072] Information processing device 2 is one or more computers configured to perform information processing related to the product input record 20. Information processing device 2 is an example of the above-mentioned external computer. For example, information processing device 2 may be a store terminal deployed in a store, a user terminal owned by a target user, a server device, etc. Information processing related to the product input record 20 may be appropriately selected depending on the embodiment. For example, information processing related to the product input record 20 may include identification of product TP, analysis of purchasing behavior (including generation of movement history), provision of information, etc. In one example of this embodiment, the information processing performed by information processing device 2 may include at least one of the following first to third examples.

[0073] Note that Smart Grip 1 is an example of a shopping device. In the following examples, for the sake of explanation, we will assume that the information processing device 2 performs information processing on the product input record 20 generated by Smart Grip 1. However, the product input record processed by the information processing device 2 is not limited to the product input record 20 generated by Smart Grip 1. Product input records may be generated by other shopping devices other than Smart Grip 1, as long as they include the event location. Other shopping devices may be used during shopping and configured as appropriate to generate product input records that include the event location. The information processing device 2 acquires product input records generated by other shopping devices other than Smart Grip 1 and performs information processing on the acquired product input records. The system may be configured in such a way. Furthermore, a portion of the product input records processed by the information processing device 2 may be collected from the smart grips 1 of one or more target users, and another portion of the product input records may be collected from other shopping devices.

[0074] (A) Case 1 Figure 3 schematically shows an example of how to use the product input record 20 according to this embodiment (product identification). In one example, the information processing device 2 may acquire the product input record 20 of the target user generated by the shopping device (Smart Grip 1). The product input record 20 may include the event location 201 where the target product TP1 was input. The target product TP1 corresponds to product TP. The information processing device 2 may identify the input target product TP1 according to the acquired product input record 20. The information processing device 2 may then output information regarding the result of identifying the target product TP1.

[0075] According to one example of this embodiment, a technology can be provided to identify an inserted product (target product TP1) using the product input record 20. The method for identifying the target product TP1 according to the product input record 20 (event position 201) may be appropriately defined depending on the embodiment. In one example of this embodiment, at least one of the following three methods may be adopted as a method for identifying the target product TP1 from the product input record 20.

[0076] (A-1) First identification method Figure 4 schematically shows an example of a first product identification method using the product input record 20 according to this embodiment. In one example, identifying the target product TP1 according to the product input record 20 may be performed by comparing the product input record 20 with map information 30 to identify the target product TP1 that was input at event location 201.

[0077] The map information 30 may be configured to include product identification information in association with the location where the products are displayed within the store. That is, the map information 30 may be configured to show the correspondence between the display location of products within the store and the product identification information. The data format of the map information 30 may be appropriately selected depending on the embodiment. For example, the map information 30 may consist of multiple records (such as a table-format database) that each hold various information such as display location and identification information for each product.

[0078] Furthermore, the method for comparing the product input record 20 and the map information 30 may be arbitrarily defined. For example, the information processing device 2 may compare the display position of each product shown in the map information 30 with the event position 201 included in the product input record 20. Depending on the result of the comparison, the information processing device 2 may extract products from the products included in the map information 30 that are displayed at a position that matches the event position 201. A position that matches the event position 201 may be, for example, the same position as event position 201, or the position closest to event position 201. The information processing device 2 may identify the target product TP1 by recognizing that the extracted product is the target product TP1.

[0079] According to one example of this embodiment, the target product TP1 can be appropriately identified by using the map information 30. However, there are cases where it is difficult to uniquely identify the target product TP1 using only the event location 201, such as when multiple types of products are placed together at the same location. In this case, the information processing device 2 may further use information other than the event location 201 to identify the target product TP1. For example, the other information may include the weight of the product. As an example, the smart grip 1 may be configured to measure the weight of the inserted target product TP1 by having the detection sensor 13 include a load sensor (weight sensor), or by further providing a load sensor in addition to the detection sensor 13. The amount of change (increase) in weight when the insertion of the target product TP1 is detected corresponds to the weight of the target product TP1. Product insertion record 2 0 may include the measurement result of the weight of this target product TP1. Map information 30 may also include information indicating the weight per unit of each product. If multiple types of products are displayed at the same location, multiple products will be extracted by the search using the event location 201. The information processing device 2 may extract from among the multiple products displayed at the location corresponding to event location 201 a product whose weight matches the weight of target product TP1 included in the product input record 20. Matching the weight of target product TP1 means, for example, that the weight (increase) included in the product input record 20 is an integer multiple (including 1) of the weight of the product shown in map information 30, or that the weight included in the product input record 20 is approximated to an integer multiple of the weight of the product shown in map information 30. The information processing device 2 may identify target product TP1 by recognizing that the extracted product is target product TP1. This makes it possible to appropriately identify target product TP1 even in cases where it is difficult to identify target product TP1 using only event location 201, such as when multiple types of products are displayed at the same location.

[0080] (A-2) Second identification method Figure 5 schematically shows an example of a second product identification method using the product input record 20 according to this embodiment. In one example, identifying the target product TP1 according to the product input record 20 may be configured by acquiring an image 400 of the imaging device 40 that includes the event position 201 of the product input record 20 in its imaging range, and then identifying the target product TP1 that appears in the image 400 by performing image analysis on the acquired image 400. The imaging device 40 may directly or indirectly provide the image 400 to the information processing device 2. The information processing device 2 may actively or passively acquire the image 400 from the imaging device 40. The second identification method is an example of a method for identifying product TP in cooperation with store sensors.

[0081] Similar to the imaging device 16 described above, the imaging device 40 includes any sensor that acquires data in the form of an image or image representation, such as an RGB camera, depth sensor, infrared sensor, radar, LiDAR, etc. It may include the following. Because it has fewer physical constraints than the imaging device 16, the imaging device 40 may use a sensor with higher performance than the imaging device 16. The method for image analysis of the captured image 400 may be the same as the method for image analysis of the captured image 165. In addition, the captured image 400 used for identifying the target product TP1 may be appropriately acquired from captured images generated at the time the input of the target product TP1 is detected and at a time close to that time.

[0082] One or more imaging devices 40 may be installed within the store as appropriate. In the example shown in Figure 5, at least three imaging devices 40 are installed within the store. The number of imaging devices 40 to be installed is not limited to the example in Figure 5 and may be determined as appropriate depending on the embodiment of the store, etc. The imaging device 40 that acquires the captured image 400 may be appropriately selected from one or more imaging devices 40 installed within the store according to the event location 201.

[0083] In the example shown in Figure 5, the event location 201 is within the imaging range of one imaging device 40. However, when two or more imaging devices 40 are installed in a store, it is possible that multiple imaging devices 40 are arranged so that the event location 201 is within their imaging range. When multiple imaging devices 40 are arranged so that the event location 201 is within their imaging range, the information processing device 2 may acquire captured images 400 from each of the multiple imaging devices 40 and perform image analysis on each acquired captured image 400. Alternatively, the information processing device 2 may select one or more imaging devices 40 from the multiple imaging devices 40, acquire captured images 400 from each of the selected one or more imaging devices 40, and perform image analysis on each acquired captured image 400. The method for selecting the imaging devices 40 is not particularly limited and may be defined as appropriate depending on the embodiment. For example, each imaging device 40 may be assigned a priority in advance, and the information processing device 2 may select one or more imaging devices 40 in order from the highest priority. Furthermore, for example, the information processing device 2 may select one or more imaging devices 40 based on any criterion, such as the event position 201 being close to the center of the imaging range.

[0084] The selection of an imaging device 40 may be replaced with the selection of an image 400. For example, the information processing device 2 may acquire an image 400 from each of the multiple imaging devices 40 and select one or more images 400 that satisfy predetermined conditions from among the multiple acquired images 400. The predetermined conditions may be appropriately defined to select an image suitable for identifying the target product TP1. For example, the predetermined conditions may include criteria such as high clarity of the image of the target product TP1, large size of the image of the target product TP1, and the entire target product TP1 being visible. The information processing device 2 may perform image analysis on the one or more selected images 400.

[0085] Performing image analysis on all images captured by all imaging devices installed in the store could result in an enormous amount of processing load. In contrast, according to this embodiment, the range of images subject to image analysis (captured images 400) can be narrowed down to the range of event detection by the product entry record 20. This makes it possible to improve the efficiency of product identification processing (target product TP1) by image analysis. In this example, since the product entry record 20 includes identification information 203, it is possible to identify the target user who placed the identified target product TP1 into basket B1. This makes it possible to identify the target user corresponding to the obtained identification result without tracking the target user with store sensors.

[0086] (A-3) Third identification method Figure 6 schematically shows an example of a third product identification method using the product input record 20 according to this embodiment. In one example, a load sensor 45 (weight sensor) may be installed on each product shelf PS in the store. Identifying the target product TP1 in accordance with the product input record 20 may be configured by acquiring measurement data 450 from a load sensor 45 installed near the event location 201 of the product input record 20, and identifying the product shelf PS whose weight has decreased according to the acquired measurement data 450, thereby identifying the target product TP1. The load sensor 45 may be appropriately positioned to measure the weight of products displayed on the product shelf PS. The load sensor 45 may directly or indirectly provide the measurement data 450 to the information processing device 2. The information processing device 2 may actively or passively acquire the measurement data 450 from the load sensor 45. Similar to the second identification method, the third identification method is also an example of a method for identifying product TP in cooperation with store sensors.

[0087] The arrangement of the product shelves PS may be determined as appropriate depending on the embodiment. The product shelves PS may be arranged in any direction, such as horizontally or vertically. Whether or not the load sensor 45 is present near the event position 201 may be determined based on whether or not the load sensor 45 (product shelf PS) is within the range 2010 near the event position 201. The range 2010 near the event position 201 and the determination method may be defined as appropriate depending on the embodiment.

[0088] For example, the information processing device 2 may set a range 2010 based on the event location 201 and determine whether the position of the load sensor 45 (product shelf PS) falls within the set range 2010. The position of the load sensor 45 may be determined from the load sensor 45 itself. Alternatively, the position of the product shelf PS may be used as the position of the load sensor 45. That is, the presence of the load sensor 45 near the event location 201 may be constituted by the presence of the product shelf PS on which the load sensor 45 is installed near the event location 201. The information processing device 2 may refer to information indicating the position of the load sensor 45 (product shelf PS) as appropriate. Information indicating the position of the load sensor 45 may be held by the information processing device 2 or by another computer other than the information processing device 2. In addition, the attribute values ​​(shape, size, etc.) of the range 2010 may be arbitrarily defined. The event location 201 may be placed at any position relative to the range 2010. In one example, the event location 201 may be used as the center of the range 2010. However, the placement of event position 201 is not limited to this example. Event position 201 may be placed at a location other than the center of range 2010. 2 may determine whether the load sensor 45 is located near the event location 201 based on the result of determining whether the position of the load sensor 45 is within the range 2010. Specifically, the information processing device 2 may determine that the load sensor 45 is located near the event location 201 based on the determination that the load sensor 45 is within the set range 2010.

[0089] Furthermore, for example, the information processing device 2 may calculate the distance between the position of the load sensor 45 (product shelf PS) and the event position 201. The information processing device 2 may determine whether the calculated distance is less than a threshold. Depending on the result of determining whether the calculated distance is less than a threshold, the information processing device 2 may identify whether the load sensor 45 is located near the event position 201. Specifically, if the information processing device 2 determines that the calculated distance is less than a threshold, it may identify that the load sensor 45 is located near the event position 201. On the other hand, if the information processing device 2 determines that the calculated distance exceeds a threshold, it may identify that the load sensor 45 is not located near the event position 201. If the calculated distance is equal to the threshold, the information processing device 2 may identify whether the load sensor 45 is located near the event position 201 or not. The threshold may be arbitrarily defined. The range identified by the threshold is an example of range 2010.

[0090] Depending on the relative positions of each product shelf PS, multiple load sensors 45 (product shelf PS) may be located near the event location 201 (i.e., they may be located within the range 2010 near the event location 201). The information processing device 2 may acquire measurement data 450 from each of the multiple load sensors 45 located near the event location 201. In the example shown in Figure 6, three load sensors 45 are located near the event location 201. In this case, the information processing device 2 may acquire measurement data 450 from each of the three load sensors 45. The information processing device 2 may identify the target product TP1 by identifying the product shelf PS with decreasing weight from among the multiple product shelf PS based on the acquired measurement data 450.

[0091] Products corresponding to product shelves PS may be identified as appropriate. For example, the information processing device 2 may identify products (target products TP1) corresponding to product shelves PS (load sensor 45) by referring to information (such as the map information 30 mentioned above) that shows the correspondence between product shelves PS and products. The information showing the correspondence between product shelves PS and products may be stored in the information processing device 2 or in another computer other than the information processing device 2.

[0092] If multiple types of products are displayed on a single product shelf PS, the information processing device 2 may acquire the change in weight (decrease) in the measurement data 450 of the load sensor 45 as the weight of the target product TP1. The information processing device 2 may extract a product from among the multiple types of products displayed on the product shelf PS whose weight has decreased that has a weight that matches the acquired weight. Matching the acquired weight means, for example, that the acquired weight (decrease) is an integer multiple (including 1) of the product's weight, or that the acquired weight is approximated to an integer multiple of the product's weight. The information processing device 2 may identify the target product TP1 by recognizing that the extracted product is the target product TP1.

[0093] Furthermore, typically, the target product TP1 is placed in the basket B1 (basket body B10) after being picked up by the target user from the product shelf PS. Therefore, the start of the weight change due to taking the target product TP1 from the product shelf PS occurs before the time (time 205) when the detection sensor 13 detects the product TP being placed in the basket. On the other hand, due to the configuration of the product shelf PS, the loading speed of the target product TP1, etc., the end time of the weight change due to taking the target product TP1 may be before or after the time when the detection sensor 13 detects the product TP being placed in the basket. Therefore, the information processing device 2 may acquire partial data from the measurement data 450 accumulated by the load sensor 45, with a start time earlier than time 205 in the product loading record 20. The predetermined interval may be appropriately defined depending on the embodiment. Location 2 may identify the product shelf PS with reduced weight based on the acquired partial data.

[0094] Constantly monitoring all load sensors installed in a store could result in an enormous amount of processing load. In contrast, this embodiment allows the range of load sensors 45 to be monitored to be limited to the range of event detection by the product input record 20. This improves the efficiency of identifying products (target product TP1) based on weight changes. In this example as well, since the product input record 20 includes identification information 203, it is possible to identify the target user who placed the identified target product TP1 into basket B1. This makes it possible to identify the target user corresponding to the obtained identification result without having to track the target user with store sensors.

[0095] (A-4) Others In one example of this embodiment, the information processing device 2 may identify the target product TP1 by employing at least one of the first to third identification methods described above. The information processing device 2 may identify the target product TP1 in multiple ways. For example, the information processing device 2 may employ all of the first to third identification methods described above. In this case, the information processing device 2 may determine the final identification result of the target product TP1 by ensembling (for example, by majority vote) the identification results of each method.

[0096] The process of identifying the target product TP1 may be executed at any time. For example, the information processing device 2 may sequentially identify the target product TP1 corresponding to the generated product input record 20 in response to the generation of the product input record 20. The information processing device 2 may also identify the target product TP1 in real time when it detects the input of the target product TP1. Alternatively, for example, the information processing device 2 may perform the identification process for multiple product input records 20 at once. In this case, the identification process for at least some of the target product TP1 may be performed retrospectively.

[0097] Furthermore, the identification result of the target product TP1 may be used for any purpose, such as calculating the purchase amount of the target user or managing the inventory quantity of the product. For example, the information processing device 2 may use the identification result of the target product TP1 to calculate the total amount of the products purchased by the target user. Similar to the budget limit example above, the information processing device 2 may obtain information indicating the amount of the target product TP1 according to the identification result of the target product TP1. The information processing device 2 may calculate the total amount of all target product TP1s entered by adding the obtained amounts for each target product TP1. The calculated total amount is an example of information regarding the results of identifying the target product TP1. The calculated total amount may be used for the target user's payment. For example, the information processing device 2 may be a store terminal or a server device. The information processing device 2 may execute the payment processing for the target user based on the calculated total amount. Also, for example, the information processing device 2 may be independent of the store system, including a cash register. The information processing device 2 may notify the store system of the calculated total amount, causing the store system to execute the payment processing for the target user. Furthermore, whether or not the target user has completed their purchasing behavior (i.e., placed all items to be purchased into the shopping cart) can be determined by any method. For example, whether or not the target user's purchasing behavior has been completed can be determined based on whether or not any termination conditions are met, such as the target user reaching the store's cash register, the cash register being operated, or the target user performing an operation on the shopping device (Smart Grip 1) to complete the purchasing behavior.

[0098] In another example, the information processing device 2 may manage the inventory quantity of target product TP1 using the identification result of the target product TP1. For example, the information processing device 2 may reduce the inventory quantity of the identified target product TP1 in accordance with its identification. The amount to be reduced may be determined as appropriate. In one example, the information processing device 2 may obtain the change (increase) in the weight of basket B1 due to the input of target product TP1 via the product input record 20. The information processing device 2 divides the obtained change (increase) in weight by the weight per unit of target product TP1. The number of items to reduce may be determined by this. Information indicating the inventory quantity of each product may be stored in the information processing device 2, or it may be stored in another computer accessible to the information processing device 2. After reducing the inventory quantity of the target product TP1, the information processing device 2 may determine whether the inventory quantity of the target product TP1 is below a threshold. The threshold may be arbitrarily defined. If the inventory quantity of the target product TP1 is below the threshold, the information processing device 2 may output an instruction to replenish the target product TP1. An instruction to replenish the target product TP1 is an example of information related to the result of identifying the target product TP1. As an example of an instruction to replenish the target product TP1, the information processing device 2 may send a notification (message, etc.) to a terminal held by the store manager to prompt the replenishment of the target product TP1. In addition, if a robot device capable of automatically transporting the target product TP1 is deployed in the store, the information processing device 2 may give a command to the robot device to replenish the target product TP1. This makes it possible to automate the replenishment of products.

[0099] (B) Second case Figure 7 schematically shows an example of how to use the product input record 20 according to this embodiment (generation of movement history). In one example, the information processing device 2 may acquire the product input record 20 of the target user generated by the shopping device (Smart Grip 1). The product input record 20 may include the event location 201 where the target product TP1 was input. The information processing device 2 may generate the target user's movement history 50 from the event location 201 of the acquired product input record 20. The information processing device 2 may output the generated movement history 50.

[0100] The configuration of the movement history 50 is not particularly limited, and can be determined as appropriate depending on the embodiment, as long as it includes an event location 201. In one example, the movement history 50 may consist of the event location 201 of the product entry record 20 generated on the target user's shopping device (smart grip 1) from the time the user enters the store through the entrance / exit ET until payment is made at the register RS. In another example, the movement history 50 may further include a trajectory 500 within the store. The trajectory 500 may be configured as appropriate to show the progress of the target user's movement within the store.

[0101] The trajectory 500 may be generated in any way. For example, the information processing device 2 may generate the trajectory 500 by connecting feature points of movement, such as connecting the entrance / exit ET to the first event position 201, connecting the k-th event position 201 to the (k+1)-th event position 201 (where k is an integer greater than or equal to 1), and connecting the last event position 201 to register RS. In another example, the shopping device (smart grip 1) may measure the position using the positioning sensor (positioning sensor 14) while the target user is in the store, even at times other than when the product is placed in the store. The information processing device 2 may further use this position measurement result to generate the trajectory 500. For example, the shopping device (smart grip 1) may continue measuring the position using the positioning sensor (positioning sensor 14) for a predetermined period after measuring the event position 201. The predetermined period may be appropriately defined depending on the embodiment. The information processing device 2 may acquire these measurement results and, based on the acquired measurement results, identify the progress of movement (direction of movement, etc.) from each event position 201 and then generate a trajectory 500.

[0102] In the example shown in Figure 7, the store has one entrance / exit ET, eight product shelves PS, and three cash registers RS. The scenario assumes that the target user places items into a basket at six locations L1 to L6 (i.e., six product placement records 20 are generated, and the event location 201 of each product placement record 20 indicates each location L1 to L6). In one example, the information processing device 2 may generate a trajectory 500 by sequentially connecting the entrance / exit ET, locations L1 to L6 indicated by each event location 201, and the cash registers RS. Note that the store layout is not limited to the example in Figure 7 and may be modified as appropriate depending on the embodiment. The store may have multiple floors, such as a basement and floors above the second floor. The store may be provided with multiple entrances / exits ET. The entrance and exit may be provided separately.

[0103] In one example of this embodiment, a technology is provided to monitor the movement history 50 of a target user using product input records 20. The movement history 50 may be used for any purpose, such as analyzing customer movement within a store. The generated movement history 50 may be output as is. The information processing device 2 may also generate analysis results such as a heat map by analyzing the movement history 50 of multiple users. The heat map may be generated as appropriate to show customer dwell time (visit rate for each area, transition probability, etc.). The information processing device 2 may output the generated analysis results. These analysis results can be used to identify areas within the store that are frequently and infrequently visited by customers. In one example, these analysis results may be reflected in determining the store layout (arrangement and size of product shelves PS, etc.). For example, the store layout may be determined to optimize customer movement according to the analysis results, such as widening the width of frequently visited aisles or narrowing areas that are not frequently visited. Layout optimization may be performed manually or by computer information processing, such as using a trained machine learning model. Information processing related to layout optimization may be performed by either the information processing device 2 or another computer. The information processing device 2 may output the results of the layout optimization.

[0104] (C) Third case Figure 8 schematically shows an example of how to use the product entry record 20 according to this embodiment (location identification). In one example, the information processing device 2 may acquire the product entry record 20 of the target user generated by the shopping device (Smart Grip 1). The product entry record 20 may include the event location 201 where the product (target product TP1) was entered. The information processing device 2 may acquire the purchase record 35 of the target user's purchased products. The information processing device 2 may identify the location where the target user placed the purchased products in the basket (basket B1) by comparing the acquired product entry record 20 and purchase record 35 with map information 30. The information processing device 2 may output information regarding the identified location.

[0105] The configuration of the purchase record 35 is not particularly limited, as long as it includes identification information of the purchased items, and may be appropriately determined depending on the embodiment. The data format of the purchase record 35 may be appropriately selected depending on the embodiment. For example, the purchase record 35 may be configured to hold various information such as identification information for each item. The purchase record 35 may be generated appropriately when the purchased items are settled. For example, the purchase record 35 may be generated by a cash register. A POS (Point of Sale) register may be used as the cash register in the store. The purchase record 35 is generated by a POS register. It may be generated by the following: In other words, the purchase record 35 may be composed of POS information. A known configuration may be used for the POS information.

[0106] If the information processing device 2 is a store terminal, the POS register may be included in the information processing device 2 or connected to the information processing device 2. The information processing device 2 may appropriately acquire purchase records 35 from the POS register. If the information processing device 2 includes a POS register, the purchase records 35 may be generated by the information processing device 2 when the target user makes a payment. Also, if the information processing device 2 is independent of the store system (POS system, etc.) that includes the POS register, the store system may generate the purchase records 35 when the target user makes a payment. The store system may directly or indirectly provide the generated purchase records 35 to the information processing device 2.

[0107] In purchase record 35, the purchased items are already identified by the identification information. When comparing the item entry record 20 and purchase record 35 with the map information 30, the information processing device 2 may compare the location of the purchased items identified in purchase record 35 on the map information 30 with the event location 201. Based on this comparison result, the information processing device 2 may identify the location where the target user placed the purchased items in the shopping cart.

[0108] For reasons such as strengthening sales promotions, the same product may be placed in multiple locations within a store (e.g., permanent shelves, special shelves, etc.). In response to this, according to one example of this embodiment, it is possible to provide a technology that uses the product placement record 20 to identify the location where a target user placed a purchased product in their shopping cart. That is, after identifying the purchased product using the purchase record 35, the location where the purchased product was placed in the shopping cart can be identified using the product placement record 20 (event location 201) and map information 30. As a result, even when the same product is placed in multiple locations within the store, it is possible to analyze from which location each target user obtained the purchased product. Through this analysis, it is possible to measure the effect of each location within the store on stimulating purchases, such as the effect of special shelves on stimulating purchases.

[0109] (D) Other At least one of the above information processing steps—product identification (first example), movement history generation (second example), and location identification (third example)—may be performed on different computers. For example, a different computer may be used as the information processing device 2 for each information processing step. Alternatively, at least two of the information processing steps from the first to third examples may be performed on the same computer. The computers used to perform the information processing for each example may be selected as appropriate.

[0110] §2 Example Configuration [Hardware configuration] (Smart Grip) As shown in Figure 1, the smart grip 1 includes a computer to which a control device 11, a memory unit 12, a detection sensor 13, a positioning sensor 14, an output device 15, an imaging device 16, and a communication module 17 are electrically connected. In the smart grip 1, at least a part of this computer may be housed in a box.

[0111] The control device 11 is configured to perform information processing based on a program and various data, including a processor P. The processor P is a hardware processor and is an example of a processor resource. The type of processor P may be appropriately selected depending on the embodiment. For example, the processor P may be a CPU (Central Processing Unit), a microcontroller, etc. It may consist of a cross-processor, FPGA (field-programmable gate array), DSP (digital signal processor), GPU (graphics processing unit), ASIC (application-specific integrated circuit), etc.

[0112] The storage unit 12 may consist of a hard disk drive, a solid-state drive, a semiconductor memory, etc. The storage unit 12 is an example of a memory resource. In this embodiment, the storage unit 12 stores various information such as a program 81 and identification information 125. The program 81 is a program that causes the control device 11 to execute information processing related to the generation of the product input record 20 (Figure 14 described later). The program 81 includes a series of instructions for the said information processing. In one example, when the control device 11 performs information processing to identify product TP by image analysis of the captured image 165, the program 81 may further include a series of instructions for information processing related to the identification of product TP (Figure 15 described later). In another example, when the control device 11 performs information processing to output related information 155, the program 81 may further include a series of instructions for information processing related to the output of related information 155 (Figure 16 described later). The identification information 125 may be omitted.

[0113] At least one of the program 81 and the identification information 125 may be stored in the storage medium SR1 instead of or together with the storage unit 12. The storage medium SR1 is configured to store various types of information (stored programs, etc.) by electrical, magnetic, optical, mechanical, or chemical means so that a machine such as a computer can read the information. The storage unit 12 and the storage medium SR1 are examples of non-temporary storage media. The Smart Grip 1 (computer) stores at least one of the program 81 and the identification information 125 in the storage medium. The information may be obtained from storage medium SR1. Storage medium SR1 may be a disk-type storage medium (CD, DVD, etc.) or a non-disk-type storage medium such as semiconductor memory (flash memory, etc.). Any drive device may be used to read the information stored in storage medium SR1. The type of drive device may be selected according to storage medium SR1.

[0114] The detection sensor 13 is configured to detect when product TP is placed into basket B1. The type of detection sensor 13 is not particularly limited as long as it can detect the placement of product TP, and may be appropriately selected depending on the embodiment. For example, the detection sensor 13 may consist of one or more load sensors (weight sensors). In this case, the detection sensor 13 can detect the placement of product TP by detecting the change (increase) in weight caused by the placement of product TP. The detection sensor 13 may be placed in any location where a change in load occurs due to the placement of product TP (i.e., where the change in weight due to the placement of product TP can be observed).

[0115] Furthermore, for example, the detection sensor 13 may be composed of one or more imaging devices. The imaging devices include RGB cameras, depth sensors, infrared sensors, radar, LiDAR, etc., which capture images or image representations. The system may include any sensors that acquire data. In this case, the detection sensor 13 may be the same as the imaging device 16, or it may be provided separately from the imaging device 16. The detection sensor 13 can detect the placement of product TP by capturing product TP in the image captured by the imaging device. The detection sensor 13 may be placed at any location where product TP is within the imaging range at least at one of the following times: from when the user picks up product TP until they place it in the basket B1, and after it has been placed in the basket B1. When reducing processing load and power consumption by limiting the period during which the imaging device 16 is activated as described above, it is preferable to use a different type of sensor for the detection sensor 13 that has a lower processing load and power consumption than the imaging device 16.

[0116] Furthermore, for example, the detection sensor 13 may be composed of one or more contact sensors. In this case, the detection sensor 13 can detect the insertion of product TP by contact with the product being inserted or with other objects (such as a switch member) that have been displaced by the insertion of product TP. The detection sensor 13 may be placed at any location where the inserted product TP makes direct or indirect contact. Indirect contact may include contact with other objects that have been displaced by the insertion of product TP. The detection sensor 13 may be composed of multiple types of sensors.

[0117] The positioning sensor 14 is configured to measure position. The type of positioning sensor 14 is not particularly limited as long as it can measure position, and may be appropriately selected depending on the embodiment. For example, the positioning sensor 14 may consist of a GPS (Global Positioning System) sensor, a GNSS (Global Navigation Satellite System) sensor, etc. Also, for example, The positioning sensor 14 may consist of a beacon receiver. In this case, multiple beacon transmitters may be installed in the store. The position of the smart grip 1 may be appropriately measured (estimated) from the strength of the beacon signal received by the positioning sensor 14 (beacon receiver). For example, the positioning sensor 14 may estimate the distance to each beacon transmitter according to the strength of the signal received by the beacon receiver. The positioning sensor 14 may measure the position of the smart grip 1 from the estimated distance to each beacon transmitter by triangulation. At least a part of this position measurement process may be performed in the control device 11.

[0118] The communication module 17 is configured to perform wired or wireless communication over a network. The communication module 17 may consist of, for example, a wired LAN (Local Area Network) module, a wireless LAN module, etc. The communication module 17 may also include a proximity wireless communication module. The network standard is not particularly limited and may be appropriately selected depending on the embodiment. For example, the type of network may be appropriately selected from the Internet, wireless communication network, mobile communication network, telephone network, dedicated network, etc. Smart Grip 1 (control Your device 11) can exchange data with other computers (for example, information processing devices 2, etc.) via the communication module 17.

[0119] Regarding the specific hardware configuration of the Smart Grip 1 (computer), components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control device 11 may include a plurality of hardware processors (processor P). At least one of the output device 15, imaging device 16, and communication module 17 may be omitted. At least one of the program 81 and identification information 125 may be stored in an external storage device such as a NAS. An external storage device is also an example of a non-temporary storage medium. The Smart Grip 1 may be configured to be operable by further including an input device. The input device may be configured to accept information input. The input device may consist of, for example, a keyboard, touch panel, operator, microphone, etc. The input device and output device 15 may be integrally configured by a touch panel display, etc. The output device 15 is an example of a feedback module. The feedback module may be configured to provide feedback to the user through at least one of visual, auditory, and tactile means. The Smart Grip 1 may further include feedback modules other than the output device 15. For example, the output device 15 may further include a haptic feedback module configured to provide information feedback via touch. The haptic feedback module may consist of any device capable of transmitting information via touch. For example, the haptic feedback module may consist of a servo motor, a vibration motor, or a combination thereof. If a haptic feedback module is included, the smart grip 1 may output any information, such as notification of the output of related information 155, via haptic feedback. The smart grip 1 may include multiple computers. The hardware configurations of each computer may or may not be the same. In addition, the computers included in the smart grip 1 may be specially designed computers, general-purpose PCs (Personal Computers), tablet PCs, terminal devices, etc. It may consist of a smartphone, etc. For example, the computer included in Smart Grip 1 may consist of a smartphone. Smart Grip 1 may be constructed by mounting a smartphone on a housing (attachment).

[0120] (Smart Grip Structure) Figures 9 and 10 are schematic side and top views showing an example of the structure of the smart grip 1 according to this embodiment. In one example, the smart grip 1 may further include a housing 90 that houses a detection sensor 13, a positioning sensor 14, and a control device 11. The housing 90 may be provided with one or more grooves (grooves 915, grooves 935, etc.) for hooking the handle portion B11 provided on the basket B1. According to this example, a smart grip 1 that can be used as a basket handle can be provided by replacing the existing basket handle portion (handle portion B11).

[0121] The structure of the smart grip 1, including the shape of the housing 90, the size of the housing 90, the arrangement of each component (control device 11, etc.) within the housing 90, the number of grooves, the shape of the grooves, and the arrangement of the grooves, may be appropriately determined according to the embodiment. In the example shown in Figures 9 and 10, it is assumed that the detection sensor 13 is composed of a pair of load sensors 130, the output device 15 includes a display 151, and the imaging device 16 is provided separately from the detection sensor 13. It is also assumed that the housing 90 has a pair of grooves (915, 935). Furthermore, it is assumed that the smart grip 1 includes a vibration motor 180 and a servo motor 185 as a tactile feedback module.

[0122] For the sake of explanation, the left-right direction in Figures 9 and 10 will be referred to as the "front-back direction." The left direction in Figures 9 and 10 will be referred to as the "front direction," and the right direction in Figures 9 and 10 will be referred to as the "back direction." The direction perpendicular to the plane of the paper in Figure 9 and the up-down direction in Figure 10 will be referred to as the "left-right direction." The up direction in Figure 10 will be referred to as the "right direction," and the down direction in Figure 10 will be referred to as the "left direction." Up and down in Figure 9 The direction perpendicular to the plane of the paper in Figure 10 is referred to as the "up and down direction." The upward direction in Figure 9 is referred to as the "upward direction," and the downward direction in Figure 9 is referred to as the "downward direction."

[0123] In one example, the housing 90 may comprise a first part 91, a second part 92, and a third part 93. In the example shown in each figure, the first part 91, the second part 92, and the third part 93 are arranged in a line in the front-to-back direction from rear to front. Each part (91, 92, 93) is formed in the shape of a rectangular parallelepiped. The first part 91 and the third part 93 may have substantially the same shape. The width (length in the left-to-right direction) of the second part 92 may be shorter than that of the first part 91 and the third part 93. This allows the second part 92 to constitute the part that the user grasps. In the example shown in each figure, the second part 92 extends along the front-to-back direction. Therefore, the user can hold the smart grip 1 by grasping the second part 92 with their hand along the front-to-back direction. The total length of the circumference of the second part 92 around the front-to-back axis may be determined within a range of a length that can be grasped by hand.

[0124] Side walls may be provided at the horizontal (left-right and front-back) ends of each section (91, 92, 93). A bottom surface may also be provided at the lower end of each section (91, 92, 93). This allows each section (91, 92, 93) to have an internal space enclosed by its side walls, which houses the computer portion of the Smart Grip 1. The internal spaces of the first section 91 and the second section 92 may be at least partially continuous. The internal spaces of the second section 92 and the third section 93 may also be at least partially continuous. The upper ends of each section (91, 92, 93) may be closed with a lid or the like, or they may be open.

[0125] A groove 915 may be provided on the rear end side of the first portion 91. In the example shown in each figure, the groove 915 is formed by recesses provided on the left-right opposing side walls of the first portion 91. Each recess is rectangular in shape. The groove 915 is formed by aligning the recesses in the front-rear direction. Even when the upper end of the first portion 91 is closed, the portion of the groove 915 on the upper end surface of the first portion 91 may be left open.

[0126] Similarly, a groove 935 may be provided on the front end side of the third portion 93. In the example shown in each figure, the groove 935 is formed by recesses provided on the left-right opposing side walls of the third portion 93. Each recess is rectangular in shape. The groove 935 is formed by aligning the recesses in the front-rear direction. Even when the upper end of the third portion 93 is closed, the portion of the groove 935 on the upper end surface of the third portion 93 may be open. The width (length in the front-rear direction) of each groove (915, 935) may be the same as or slightly larger than the width of the handle portion B11 of the corresponding basket B1. This allows each groove (915, 935) to be configured to receive each handle portion B11 of the basket B1.

[0127] Each pair of load sensors 130 may be positioned in the internal space of the first part 91 and the third part 93, respectively, aligned with the positions of the pair of grooves (915, 935). If the portion of the handle B11 that hooks onto the smart grip 1 is formed in a straight line, in one example, the recesses constituting each groove (915, 935) may be formed such that the lower end of the recess is located slightly below the measuring portion (contact surface with the handle B11) of the load sensor 130. This ensures that when each handle B11 is hooked onto each groove (915, 935), each handle B11 contacts the measuring portion of each load sensor 130. As a result, each load sensor 130 can observe the weight change when product TP is placed in the basket B1 (basket body B10). The placement of the load sensors 130 is an example of a location where a change in load occurs due to the placement of product TP. However, the positional relationship between each groove (915, 935) and each load sensor 130 is not limited to this example and may be appropriately changed depending on the embodiment, such as the shape of the handle portion B11. For example, each handle portion B11 may be formed so that the portion located in the internal space of each part (91, 93) when placed on the smart grip 1 curves downward. In this case, the lower end of the recess constituting each groove (915, 935) is positioned above the measuring portion of the load sensor 130. It may also be formed to be placed.

[0128] In the example shown in each figure, the imaging device 16 is positioned within the internal space of the first part 91. For example, an opening may be provided in the bottom surface of the first part 91 to match the position of the imaging unit 161 (light-receiving unit, etc.) of the imaging device 16. The imaging device 16 may be positioned so that the imaging unit 161 is exposed from the bottom surface of the first part 91 by fitting the imaging unit 161 into the opening in its bottom surface. This allows the imaging device 16 to image the area below the smart grip 1. Therefore, when the smart grip 1 is attached to the basket B1 by hooking each handle B11 of the basket B1 into the respective grooves (915, 935), the imaging device 16 can include the storage area of ​​the basket body B10 in its imaging range. As a result, when product TP is placed into the basket body B10, the imaging device 16 can observe the product TP being placed. This placement of the imaging device 16 is just one example of a location where the product TP being placed can be observed.

[0129] In the example shown in each figure, the vibration motor 180 and the servo motor 185 are arranged in the internal space of the second part 92. In this example, the second part 92 constitutes the part that the user grips. Therefore, by arranging the vibration motor 180 and the servo motor 185 in the second part 92, tactile feedback can be provided to the user using the smart grip 1. The types of vibration motor 180 and servo motor 185 are not particularly limited and may be appropriately selected depending on the embodiment.

[0130] The vibration motor 180 is configured to provide tactile feedback to the user by vibrating. In other words, the vibration motor 180 can provide feedback to the user through vibration. The feedback through vibration may be used as desired. For example, when the smart grip 1 outputs related information 155 to the display 151, it may notify the user that the related information 155 has been output by vibrating the vibration motor 180.

[0131] The servo motor 185 may include a shaft 1851 and an arm 1852. The shaft 1851 constitutes the axis of rotation. The arm 1852 is attached to the shaft 1851 and rotates around the axis of the shaft 1851 in accordance with the rotation of the shaft 1851. The arm 1852 may be configured to rotate clockwise or counterclockwise so that a part of the arm 1852 (e.g., the tip) comes into contact with the user's hand grasping the second part 92. Alternatively, the arm 1852 may be configured to rotate clockwise or counterclockwise so that the tip of the arm 1852 strikes the side wall of the second part 92, and the resulting vibration is transmitted to the user's hand grasping the second part 92. Having such a configuration, the servo motor 185 can provide feedback in a specific direction. In the example shown in each figure, the arm 1852 can provide feedback to the left side of the user's hand grasping the second part 92 by rotating clockwise. On the other hand, by rotating the arm 1852 counterclockwise, feedback can be provided to the right side of the user's hand grasping the second part 92. This feedback indicating a specific direction may be used as is. For example, if the information output to the display 151 includes information about direction (e.g., the location of an item), the smart grip 1 may indicate the direction of the object to the user by outputting feedback indicating the direction of the object using the servo motor 185. In the example shown in each figure, the arm 1852 is formed in a flat shape. However, the shape of the arm 1852 is not limited to this example. As long as it is possible to return tactile feedback in a specific direction by rotating, the shape of the arm 1852 is not particularly limited and may be appropriately selected depending on the embodiment.

[0132] In addition, in the example shown in each figure, the control device 11, memory unit 12, positioning sensor 14, and communication module 17 are housed in a box 1100. The box 1100 is formed in the shape of a rectangular parallelepiped and is located in the internal space of the third section 93. A display 151 is located on the top surface of the box 1100. The third portion 93 is positioned such that, when the upper end of the third portion 93 is closed, the display 151 may be positioned so that its screen is exposed from the upper end surface of the third portion 93. In this example, when the user properly grasps the smart grip 1, the third portion 93 is positioned in front of the second portion 92. Therefore, while using the smart grip 1, the screen of the display 151 is positioned slightly in front of the user's gripping hand. This allows the user to easily view the screen of the display 151 without it being obstructed by body parts while using the smart grip 1.

[0133] The structure of the smart grip 1 is not limited to the examples in Figures 9 and 10, and may be modified as appropriate depending on the embodiment. For example, the shape of the housing 90 may be modified as appropriate. In another example, the housing of the smart grip 1 may be formed in the shape of a ring, and a part of the ring may be configured to open and close by attaching and detaching the ends together. In this way, the housing of the smart grip 1 may be configured to replace the handle by receiving the handle portion B11 of the basket B1 inside the ring formed by the housing. In yet another example, the housing of the smart grip 1 may be formed to receive a terminal device such as a smartphone. In this case, the smart grip 1 may be constructed by attaching the terminal device to the housing. Also, the positions of each groove (915, 935) may be changed as appropriate. The number of grooves provided in the housing (housing 90) is not limited to two, but may be one or three or more. Also, the arrangement of each component may be changed as appropriate. The arrangement of the load sensor 130 may be changed as appropriate depending on the structure of the housing (housing 90). When a sensor other than the load sensor 130 is used as the detection sensor 13, the detection sensor 13 may be appropriately placed in a location where it can detect the loading of product TP. The arrangement of the imaging device 16 may be appropriately changed depending on the embodiment. The imaging device 16 may be placed in a location other than the internal space of the first part 91. The components housed in the box 1100 may be appropriately changed. The box 1100 may be placed in a location other than the internal space of the third part 93. The box 1100 may be omitted.

[0134] (Information processing device) Figure 11 schematically shows an example of the hardware configuration of the information processing device 2 according to this embodiment. The information processing device 2 according to this embodiment is a computer in which a control unit 21, a storage unit 22, a communication module 23, an input device 24, an output device 25, and a drive 26 are electrically connected.

[0135] The control unit 21 is configured to perform information processing based on programs and various data, by including hardware processors such as a CPU, RAM (Random Access Memory), and ROM (Read Only Memory). The control unit 21 (CPU) is an example of processor resources.

[0136] The storage unit 22 may consist of a hard disk drive, a solid-state drive, semiconductor memory, etc. The storage unit 22, RAM, and ROM are examples of memory resources. In this embodiment, the storage unit 22 stores various information such as the program 82. The program 82 is a program that causes the information processing device 2 to perform information processing related to the product input record 20. The program 82 includes a series of instructions for the information processing. For example, when causing the information processing device 2 to perform product identification processing, the program 82 may include a series of instructions for information processing related to product identification (Figure 17 described later). For example, when causing the information processing device 2 to perform movement history generation processing, the program 82 may include a series of instructions for information processing related to movement history generation (Figure 18 described later). For example, when causing the information processing device 2 to perform location identification processing, the program 82 may include a series of instructions for information processing related to location identification (Figure 19 described later).

[0137] The communication module 23 is configured to perform wired or wireless communication over a network. The communication module 23 may consist of, for example, a wired LAN module, a wireless LAN module, etc. The information processing device 2 can exchange data with other computers (for example, the computer part of Smart Grip 1, etc.) via the communication module 23.

[0138] The input device 24 is configured to accept information input. The input device 24 may consist of, for example, a mouse, keyboard, or control panel. The output device 25 is configured to output information. The output device 25 may consist of, for example, a display or speaker. The operator can operate the information processing device 2 by using the input device 24 and the output device 25. The input device 24 and the output device 25 may be connected via an external interface. The external interface may be appropriately configured to connect to an external device via wired or wireless connection, for example, a USB (Universal Serial Bus) port or a dedicated port. The input device 24 and the output device 25 may be integrated into a single unit, such as a touch panel display.

[0139] Drive 26 is a device for reading various information, such as programs, stored in the storage medium SR2. Program 82 may be stored in the storage medium SR2 in place of or together with the storage unit 22. The information processing device 2 may retrieve program 82 from the storage medium SR2. The storage medium SR2 may be configured in the same way as the storage medium SR1. The storage unit 22 and the storage medium SR2 are examples of non-temporary storage media. The type of drive 26 may be selected according to the storage medium SR2. Drive 26 may be connected via an external interface.

[0140] Regarding the specific hardware configuration of the information processing device 2, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. Hardware processors may consist of microprocessors, FPGAs, DSPs, GPUs, ASICs, etc. At least one of the communication module 23, input device 24, output device 25, and drive 26 may be omitted. The program 82 may be stored in an external storage device such as a NAS. An external storage device is also an example of a non-temporary storage medium. The information processing device 2 may consist of multiple computers. In this case, the hardware configuration of each computer may or may not be the same. Furthermore, the information processing device 2 may be a computer specifically designed for this purpose, as well as a general-purpose server device, a general-purpose PC, a tablet PC, a terminal device, etc.

[0141] [Software Configuration] (Smart Grip) Figure 12 schematically shows an example of the software configuration of the smart grip 1 (control device 11) according to this embodiment. The control device 11 of the smart grip 1 executes instructions contained in the program 81 stored in the storage unit 12 using the processor P. As a result, the smart grip 1 (control device 11) operates as a computer equipped with a detection unit 111, a generation unit 112, and a storage processing unit 113 as software modules. In other words, in this embodiment, each software module of the smart grip 1 is realized by the control device 11 (processor P).

[0142] The detection unit 111 is configured to detect the insertion of product TP according to the measurement result of the detection sensor 13. The generation unit 112 is configured to generate a product insertion record 20 that includes the event position 201 measured by the positioning sensor 14, where the product TP was inserted, when the detection sensor 13 detects the insertion of product TP. The storage processing unit 113 is configured to save the generated product insertion record 20.

[0143] If the control device 11 is configured to perform output processing of related information 155, the smart grip 1 (control device 11) may further include an output processing unit 114 as a software module. The output processing unit 114 may be configured to output related information 155 related to the insertion of product TP from the output device 15 when the detection sensor 13 detects the insertion of product TP. If the output of related information 155 is not performed, the output processing unit 114 may be omitted from the software configuration of the smart grip 1.

[0144] If the control device 11 is configured to perform product TP identification processing by image analysis, the smart grip 1 (control device 11) may further include an image acquisition unit 115, an identification unit 116, and a result output unit 117 as software modules. The image acquisition unit 115 is configured to acquire an image 165 captured by the imaging device 16 when the detection sensor 13 detects the insertion of a product TP. The identification unit 116 is configured to identify the product TP captured in the image 165 by performing image analysis on the acquired image 165. The result output unit 117 is configured to output information regarding the product TP identification result. If product TP identification processing is not performed on the smart grip 1, the image acquisition unit 115, the identification unit 116, and the result output unit 117 may be omitted from the software configuration of the smart grip 1.

[0145] (Information processing device) Figure 13 schematically shows an example of the software configuration of the information processing device 2 (control unit 21) according to this embodiment. The control unit 21 of the information processing device 2 executes instructions contained in the program 82 stored in the storage unit 22 using the CPU. As a result, the information processing device 2 (control unit 21) operates as a computer equipped with an acquisition unit 211, an information processing unit 212, and an output processing unit 213 as software modules. In other words, in this embodiment, each software module of the information processing device 2 is realized by the control unit 21 (CPU).

[0146] The acquisition unit 211 is configured to acquire the product entry record 20 of the target user generated by the shopping device (Smart Grip 1). The product entry record 20 may include the event position 201 where the target product TP1 was entered. The information processing unit 212 is configured to perform information processing related to the product entry record 20. The output processing unit 213 is configured to output information related to the results of the information processing related to the product entry record 20.

[0147] If the control unit 21 is configured to perform product identification processing, the information processing unit 212 may include an identification unit 2121. The identification unit 2121 is configured to identify the target product TP1 that has been inserted, according to the acquired product insertion record 20. Any of the first to third identification methods described above may be used as the identification method. Accordingly, the output processing unit 213 may be configured to output information regarding the result of identifying the target product TP1.

[0148] If the control unit 21 is configured to perform the process of generating a movement history, the information processing unit 212 may include a history generation unit 2122. The history generation unit 2122 is configured to generate the movement history 50 of the target user from the event position 201 of the acquired product entry record 20. Accordingly, the output processing unit 213 may be configured to output the generated movement history 50.

[0149] If the control unit 21 is configured to perform location characteristic processing, the acquisition unit 211 may be further configured to acquire the purchase record 35 of the target user's purchased items. The information processing unit 212 may include a location identification unit 2123. The location identification unit 2123 is configured to identify the location where the target user placed the purchased items in the basket (basket B1) by comparing the acquired item input record 20 and purchase record 35 with map information 30. Accordingly, the output processing unit 213 may be configured to output information regarding the identified location. In the information processing unit 212, at least the identification unit 2121, the history generation unit 2122, and the location identification unit 2123 One of these can be omitted.

[0150] (others) In this embodiment, an example is described in which each software module of Smart Grip 1 and Information Processing Device 2 is implemented by a general-purpose processor (processor P, CPU). However, some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may also be implemented as a hardware module. Regarding the software configuration of Smart Grip 1 and Information Processing Device 2, modules may be omitted, replaced, and added as appropriate, depending on the embodiment.

[0151] §3 Example of Operation [Smart Grip] (Generating product input records) Figure 14 is a flowchart showing an example of a processing procedure for generating product input records 20 by the control device 11 of the Smart Grip 1 according to this embodiment. The following processing procedure for generating product input records 20 is an example of a generation method (information processing method) executed by a computer (control device 11). However, the following processing procedure for generating product input records 20 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure for generating product input records 20 can be omitted, replaced, and added as appropriate, depending on the embodiment.

[0152] In step S101, the control device 11 operates as a detection unit 111. The control device 11 acquires measurement results from the detection sensor 13. It detects the insertion of product TP according to the measurement results from the detection sensor 13. If the insertion of product TP is detected, the control device 11 proceeds to the next step S102. If the insertion of product TP is not detected, the control device 11 repeatedly executes the process in step S101 and waits until the insertion of product TP is detected.

[0153] In step S102, the control device 11 operates as a generation unit 112. The control device 11 obtains the event position 201 where the product TP was placed by referring to the measurement results of the positioning sensor 14. The control device 11 generates a product placement record 20 that includes the obtained event position 201. Once the product placement record 20 is generated, the control device 11 proceeds to the next step S103.

[0154] In step S103, the control device 11 operates as a storage processing unit 113. The control device 11 saves the generated product input record 20. In one example, the control device 11 may save the generated product input record 20 in the storage unit 12. In another example, the control device 11 may save the product input record 20 to an external computer such as an information processing device 2 by transmitting the product input record 20 to an external storage area. Once the product input record 20 is saved, the control device 11 proceeds to the next step S104.

[0155] In step S104, the control device 11 determines whether or not to terminate the process. The determination criteria can be set arbitrarily. For example, the control device 11 may decide not to terminate the process until a termination instruction is given. On the other hand, when a termination instruction is given, the control device 11 may decide to terminate the process. The termination instruction can be given in any way. In another example, if the target user is inside the store, the control device 11 may decide not to terminate the process. On the other hand, if the target user is not inside the store (has left the store), the control device 11 may decide to terminate the process. Whether or not the user is inside the store may be determined appropriately according to the measurement results of the positioning sensor 14, etc.

[0156] If the control device 11 determines that it is not time to terminate the process, it returns to step S101 and executes the process again from step S101. If it determines that it is time to terminate the process, the control device 11 terminates the processing procedure for generating the product input record 20 in this example. Note that the timing of terminating the process is not limited to this example. The control device 11 may terminate the processing procedure for generating the product input record 20 at any time.

[0157] In one example, the control device 11 may execute the series of processes from steps S101 to S104 in real time. The processing order of each step may also be changed as appropriate. In another example, the control device 11 may repeat the processes of steps S101 and S102 multiple times, and then execute the process of step S103 all at once.

[0158] According to this embodiment, by generating a product input record 20 using the detection sensor 13 and positioning sensor 14 through the processing in step S102, it is possible to monitor user purchasing behavior without installing sensors in the store. This is expected to simplify the monitoring of purchasing behavior. Furthermore, the smart grip 1 according to this embodiment may be used together with sensors installed in the store to monitor user purchasing behavior. In this case, by utilizing the product input record 20 as in the second identification method, third identification method, etc., the processing efficiency of sensing data obtained from sensors installed in the store can be improved. This is expected to simplify the monitoring of purchasing behavior.

[0159] (Product identification) Figure 15 is a flowchart showing an example of a processing procedure for identifying product TP by the control device 11 of the Smart Grip 1 according to this embodiment. The following processing procedure for identifying product TP is an example of an identification method (information processing method) executed by the computer (control device 11). However, the following processing procedure for identifying product TP is merely an example, and each step may be modified as much as possible. Furthermore, depending on the embodiment, steps in the following processing procedure for identifying product TP can be omitted, replaced, and added as appropriate.

[0160] In step S201, the control device 11 operates as a detection unit 111. The processing in step S201 may be the same as in step S101. If the input of product TP is detected, the control device 11 proceeds to the next step S202. If the input of product TP is not detected, the control device 11 repeatedly executes the processing in step S201 and waits until the input of product TP is detected.

[0161] In step S202, the control device 11 operates as an image acquisition unit 115. That is, the control device 11 acquires an image 165 from the imaging device 16. In step S203, the control device 11 operates as an identification unit 116. That is, the control device 11 identifies the product TP shown in the image 165 by performing image analysis on the acquired image 165. In step S204, the control device 11 operates as a result output unit 117. That is, the control device 11 outputs information regarding the identification result of the product TP. The output destination and content of the information may be appropriately selected according to the embodiment. After outputting the information regarding the identification result, the control device 11 proceeds to the next step S205.

[0162] In step S205, the control device 11 determines whether or not to terminate the process. The process in step S205 may be the same as in step S104 above. If it is determined not to terminate the process, the control device 11 returns to step S201 and executes the process again from step S201. If it is determined to terminate the process, the control device 11 terminates the processing procedure related to the identification of product TP in this example. Note that the timing of terminating the process is not limited to this example. The control device 11 may terminate the process related to the identification of product TP at any time. You may terminate the processing procedure.

[0163] In one example of this embodiment, the processing in steps S202 and S203 allows the Smart Grip 1 to identify the product TP that has been inserted. In one example, the control device 11 may execute the series of processes from steps S201 to S205 in real time. The processing order of each step may be changed as appropriate. In another example, the control device 11 may execute the processes from steps S201 to S203 multiple times and then execute the process of step S204 all at once. Also, when the control device 11 executes both the processes from steps S101 to S104 and steps S201 to S205, the process of step S201 may be executed in common with step S101. The process of step S205 may be executed in common with step S104. The processes of steps S202 to S204 may be executed at any timing after step S101 and before step S104.

[0164] (Output of related information) Figure 16 is a flowchart showing an example of a processing procedure for outputting related information 155 by the control device 11 of the smart grip 1 according to this embodiment. The following processing procedure for outputting related information 155 is an example of an output method (information processing method) executed by the computer (control device 11). However, the following processing procedure for outputting related information 155 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure for outputting related information 155 can be omitted, replaced, and added as appropriate, depending on the embodiment.

[0165] In step S301, the control device 11 operates as a detection unit 111. The processing in step S301 may be the same as in step S101. If the input of product TP is detected, the control device 11 proceeds to the next step S302. If the input of product TP is not detected, the control device 11 repeatedly executes the processing in step S301 and waits until the input of product TP is detected.

[0166] In step S302, the control device 11 operates as an output processing unit 114. That is, the control device 11 outputs related information 155 related to the input of product TP from the output device 15. In one example, the related information 155 may consist of at least one of the first to seventh examples above. For example, if the first example is adopted, the related information 155 may include at least one of the advertising information 1550 and coupon information 1555 of other products placed around the event location 201. After outputting the related information 155, the control device 11 proceeds to the next step S303.

[0167] In step S303, the control device 11 determines whether or not to terminate the process. The process in step S303 may be the same as in step S104 above. If it is determined not to terminate the process, the control device 11 returns to step S301 and executes the process again from step S301. If it is determined to terminate the process, the control device 11 terminates the processing procedure related to the output of related information 155 in this example of operation. Note that the timing of terminating the process is not limited to this example. The control device 11 may terminate the processing procedure related to the output of related information 155 at any timing.

[0168] In one example of this embodiment, by providing related information 155 when product TP is entered through the process in step S302, it is possible to stimulate purchasing activity, such as further stimulating the desire to purchase the product. In one example, the control device 11 may execute the series of processes from steps S301 to S303 in real time. The processing order of each step may be changed as appropriate. In another example, the control device 11 performs a predetermined process before the process in step S302. The control device 11 may determine whether the conditions are met. The predetermined conditions may be arbitrarily defined. For example, the predetermined conditions may include whether the output of related information 155 is permitted. Whether or not the output of related information 155 is permitted may be set as appropriate. The control device 11 may execute the process of step S302 only if the predetermined conditions are met. Also, if the control device 11 executes the processes of steps S101 to S104 and steps S301 to S303, the process of step S301 may be executed in common with step S101. The process of step S303 may be executed in common with step S104. The process of step S302 may be executed at any timing after step S101 and before step S104.

[0169] [Information Processing Device] (Product identification) Figure 17 is a flowchart showing an example of a processing procedure for product identification using the product input record 20 by the information processing device 2 according to this embodiment. The following product identification processing procedure is an example of an identification method (information processing method) executed by a computer. However, the following product identification processing procedure is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following product identification processing procedure can be omitted, replaced, and added as appropriate, depending on the embodiment.

[0170] In step S501, the control unit 21 operates as an acquisition unit 211. That is, the control unit 21 acquires the product input record 20 of the target user generated by the shopping device. In one example, the control unit 21 may acquire the product input record 20 from the smart grip 1. In step S502, the control unit 21 operates as an identification unit 2121. That is, the control unit 21 identifies the target product TP1 that was input according to the acquired product input record 20. In one example, the control unit 21 may identify the target product TP1 by at least one of the first to third identification methods described above. For example, if the first identification method is adopted, the control unit 21 may identify the target product TP1 that was input at the event location 201 by comparing the product input record 20 with the map information 30. Alternatively, for example, if the second identification method is adopted, the control unit 21 may acquire an image 400 of the imaging device 40 that includes the event location 201 of the product input record 20 in its imaging range. The control unit 21 may identify the target product TP1 in the captured image 400 by performing image analysis on the acquired image 400. Alternatively, for example, if a third identification method is adopted, the control unit 21 may acquire measurement data 450 from a load sensor 45 installed near the event location 201 of the product input record 20. The control unit 21 may identify the target product TP1 by identifying the product shelf PS with reduced weight according to the acquired measurement data 450. In one example, the control unit 21 may determine the final identification result of the target product TP1 by ensemble the identification results obtained by multiple methods. In step S503, the control unit 21 outputs information regarding the result of identifying the target product TP1.

[0171] The output destination and content of the information may be appropriately selected depending on the embodiment. In one example, the control unit 21 may output the identification result of the target product TP1 as is. In another example, the control unit 21 may perform arbitrary information processing (calculation of the purchase amount, inventory management, etc.) according to the obtained identification result. The control unit 21 may output the result of performing the information processing as information related to the identification result. The output of the result of performing this information processing may include, for example, outputting specific information according to the identification result, or controlling the operation of the controlled device according to the identification result. Outputting specific information may include notifying that the payment process is complete, notifying the store system of the calculated total amount, or notifying a message prompting the replenishment of the target product TP1. Controlling the operation of the controlled device may include giving a command to the robot device to replenish the target product TP1. The output destination may be, for example, RAM, storage unit 22, output device 25, another computer (including smart grip 1), controlled device, etc. Once the output of information is complete, the control unit 21 will perform the operation example described above. The processing procedure related to product identification is now complete.

[0172] In one example of this embodiment, the processing in steps S501 and S502 allows for the identification of the input product (target product TP1) using the product input record 20. The control unit 21 may execute the series of processes from steps S501 to S503 at any timing. In one example, the control unit 21 may execute the series of processes from steps S501 to S503 in real time in response to the generation of the product input record 20. In another example, the control unit 21 may acquire multiple product input records 20 in the process of step S501. The control unit 21 may execute the process of step S502 for each acquired product input record 20. In this case, the identification process of at least some of the target products TP1 (the process of step S502) may be executed retrospectively.

[0173] (Generating movement history) Figure 18 is a flowchart showing an example of a processing procedure for generating a movement history 50 using the product input record 20 by the information processing device 2 according to this embodiment. The following processing procedure for generating the movement history 50 is an example of a generation method (information processing method) executed by a computer. However, the following processing procedure for generating the movement history 50 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure for generating the movement history 50 can be omitted, replaced, and added as appropriate, depending on the embodiment.

[0174] In step S601, the control unit 21 operates as an acquisition unit 211. The processing in step S601 may be the same as in step S501. In step S602, the control unit 21 operates as a history generation unit 2122. That is, the control unit 21 generates the movement history 50 of the target user from the event position 201 of the acquired product entry record 20. In step S603, the control unit 21 operates as an output processing unit 213. That is, the control unit 21 outputs the generated movement history 50. As with step S503, the output destination and content of the information may be appropriately selected according to the embodiment. Once the output of the information is complete, the control unit 21 terminates the processing procedure for generating the movement history 50 according to this example of operation.

[0175] In one example of this embodiment, the processing in steps S601 and S602 allows for the generation of a target user's movement history 50 using the product input record 20. The generated movement history 50 allows for the analysis of the target user's purchasing behavior. The control unit 21 may execute the series of processes in steps S601 to S603 at any timing. The control unit 21 may also execute both the processes in steps S501 to S503 and steps S601 to S603. In this case, the processing in step S601 may be executed in common with step S501. The processing order of steps S502, S503, S602, and S603 is not particularly limited and may be determined as appropriate depending on the embodiment.

[0176] (Location identification) Figure 19 is a flowchart illustrating an example of a processing procedure for identifying the input location of a product (target product TP1) using the product input record 20 by the information processing device 2 according to this embodiment. The following processing procedure for identifying the input location is an example of an identification method (information processing method) executed by a computer. However, the following processing procedure for identifying the input location is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure for identifying the input location can be omitted, replaced, and added as appropriate, depending on the embodiment.

[0177] In steps S701 and S702, the control unit 21 operates as the acquisition unit 211. The processing in step S701 may be the same as in step S501. In step S702, the control unit 21 acquires the purchase record 35 of the target user's purchased items. In step S703, the control unit 21 operates as a location identification unit 2123. That is, the control unit 21 identifies the location where the target user placed the purchased items in the basket (basket B1) by comparing the acquired item input record 20 and purchase record 35 with map information 30. In step S704, the control unit 21 operates as an output processing unit 213. That is, the control unit 21 outputs information related to the identified location. As with step S503 above, the output destination and content of the information may be appropriately selected according to the embodiment. Once the output of the information is complete, the control unit 21 terminates the processing procedure related to identifying the input location in this operation example.

[0178] In one example of this embodiment, the processing in steps S701 to S703 makes it possible to identify the location where the purchased item was placed in the shopping cart. As a result, even if the same item is placed in multiple locations within the store, it is possible to analyze from which location each target user obtained the purchased item. The control unit 21 may execute the series of processes in steps S701 to S704 at any timing. The processing order of each step may be changed as appropriate. In another example, the processing in step S702 may be executed before step S701. The processing in step S702 may be executed at least partially in parallel with the processing in step S701. Furthermore, the control unit 21 may execute both the processing in steps S501 to S503 and steps S701 to S704. In this case, the processing in step S701 may be executed in common with step S501. The processing order of steps S502, S503, S702, S703 and S704 is not particularly limited and may be determined as appropriate depending on the embodiment. The control unit 21 may execute both steps S601 to S603 and steps S701 to S704. In this case, the process of step S701 may be executed in common with step S601. The processing order of steps S602, S603, S702, S703, and S704 is not particularly limited and may be determined as appropriate depending on the embodiment. When all processes of product identification, movement history generation, and input location identification are performed, the processes of steps S501, S601, and S701 may be executed in common.

[0179] §4 Variant While embodiments of this disclosure have been described in detail above, the above description is merely illustrative in all respects. The processes and means described in this disclosure can be freely combined and implemented as long as no technical inconsistencies arise. Furthermore, various improvements or modifications may be made to the above embodiments as appropriate. For example, the following modifications are possible. In the following, the same reference numerals are used for components similar to those in the above embodiments, and explanations of points similar to those in the above embodiments have been omitted as appropriate.

[0180] <4.1> The above embodiment is configured as a smart grip 1. However, the embodiments of this disclosure are not limited to the smart grip 1. The embodiments of this disclosure may be components of a shopping basket, or the basket (including a cart) itself. The type of basket is not particularly limited and may be appropriately selected depending on the embodiment.

[0181] (First variation) Figure 20 schematically illustrates another example of a scenario to which this disclosure applies. In the example in Figure 20, the embodiment of this disclosure is a shopping basket 1A. The shopping basket 1A comprises a basket body 102A, a detection sensor 13, a positioning sensor 14, and a control device 11. The basket body 102A may be configured in the same way as the basket body B10 of the basket B1 described above. Except for replacing the basket body B10 with the basket body 102A, the detection sensor 13, the positioning sensor 14, and the control device 11 may be configured in the same way as in the above embodiment. That is, the detection sensor 13 is connected to the basket body 102A. The control device may be configured to detect the insertion of product TP (target product). When the detection sensor 13 detects the insertion of product TP into the basket body 102A, the control device 11 may generate a product insertion record 20 that includes the event position 201 measured by the positioning sensor 14, where the product TP was inserted.

[0182] Other configurations of the shopping basket 1A are not particularly limited and may be determined as appropriate depending on the embodiment. In one example, the shopping basket 1A may include a pair of handles 101A attached to the basket body 102A, and a grip 100A that spans the pair of handles 101A. Each handle 101A may be configured in the same way as each handle B11 of the basket B1. When these configurations are adopted, in one example, the shopping basket 1A may be constructed by using the smart grip 1 for the grip 100A. However, the method of constructing the shopping basket 1A is not limited to such examples.

[0183] Similar to the Smart Grip 1 described above, the shopping basket 1A may further include a storage unit 12. The shopping basket 1A may also further include at least one of an output device 15, an imaging device 16, and a communication module 17. If the shopping basket 1A further includes an output device 15, the control device 11 may output relevant information 155 from the output device 15 when it detects the insertion of a product TP by the detection sensor 13. If the shopping basket 1A further includes an imaging device 16, the control device 11 may acquire an image 165 from the imaging device 16 when it detects the insertion of a product TP by the detection sensor 13, and identify the product TP by performing image analysis on the acquired image 165.

[0184] The location of the computer components of the shopping basket 1A (control device 11 to communication module 17) is not limited to the grip 100A. At least a portion of the computer components of the shopping basket 1A may be placed anywhere other than the grip 100A. In another example, if a load sensor is used for the detection sensor 13, the detection sensor 13 may be placed on the bottom surface of the storage compartment of the basket body 102A. The imaging device 16 may be placed on the edge of the opening of the basket body 102A with the imaging unit 161 facing the storage compartment of the basket body 102A.

[0185] The shopping basket 1A is an example of a shopping device. According to this modified example, when a basket form is adopted, it is expected that the user can easily monitor their purchasing behavior. The configuration of the shopping basket 1A may be changed as appropriate depending on the embodiment. In another example, the grip 100A may be omitted. In this case, the user may hold the shopping basket 1A by grasping the pair of handles 101A.

[0186] (Second variation) Figure 21 schematically shows another example of a scenario to which this disclosure applies. In the example in Figure 21, the form of this disclosure is a shopping basket 1B. Shopping basket 1B corresponds to a form in which the shopping basket 1A is further equipped with one or more wheels 103B to form a cart. That is, shopping basket 1B comprises a basket body 102B, a detection sensor 13, a positioning sensor 14, a control device 11, and one or more wheels 103B. The basket body 102B may be the same as the basket body 102A of the first modified example. The detection sensor 13 may be configured to detect when a product TP (target product) is placed into the basket body 102B. When the control device 11 detects that a product TP has been placed into the basket body 102B by the detection sensor 13, it may generate a product entry record 20 that includes the event position 201 in which the product TP was placed, which is measured by the positioning sensor 14. The number of wheels 103B may be appropriately selected depending on the embodiment. In a typical example, the shopping basket 1B may have four wheels 103B. The shopping basket 1B may also be called a shopping cart.

[0187] The arrangement of each wheel 103B and the configuration for supporting the basket body 102B are appropriate according to the embodiment. This may be determined as appropriate. For example, the shopping basket 1B may include a grip 100B and a base 101B. The base 101B may include a leg 1011B and a support column 1012B. The leg 1011B may be positioned below the basket body 102B. Each wheel 103B may be attached to the leg 1011B. The support column 1012B may be configured to extend vertically upward from one end of the leg 1011B. The basket body 102B may be fixed to the support column 1012B. The grip 100B may be attached to the upper end of the support column 1012B.

[0188] As a result, the shopping basket 1B may have a configuration similar to that of a typical shopping cart. However, the configuration of the shopping basket 1B is not limited to this example. The shopping basket 1B may have a configuration similar to that of other types of carts, such as a silver cart. Furthermore, in the case of adopting these configurations, one example is that the shopping basket 1B may be constructed by using the above-mentioned smart grip 1 for the grip 100B. However, the method of constructing the shopping basket 1B is not limited to this example.

[0189] Similar to the smart grip 1 described above, the shopping basket 1B may further include a storage unit 12. The shopping basket 1B may also further include at least one of an output device 15, an imaging device 16, and a communication module 17. Similar to the first modified example described above, if the shopping basket 1B further includes an output device 15, the control device 11 may perform output processing of related information 155. If the shopping basket 1B further includes an imaging device 16, the control device 11 may perform product TP identification processing by image analysis.

[0190] The location of the computer components (control devices 11 to communication module 17) of the shopping basket 1B is not limited to the grip 100B. At least a portion of the computer components of the shopping basket 1B may be placed anywhere other than the grip 100B. In another example, if a load sensor is used for the detection sensor 13, the detection sensor 13 may be placed on the bottom surface of the storage compartment of the basket body 102B. The imaging device 16 may be placed with its imaging unit 161 facing the storage compartment of the basket body 102B, in the area of ​​the support column 1012B from the edge of the opening of the basket body 102B or from the upper end of the basket body 102B to the grip 100B.

[0191] The shopping basket 1B is an example of a shopping device. According to this modification, when a cart form is adopted, it is expected that the monitoring of the user's purchasing behavior will be simplified. The basket body 102B may also be used as a basket holder. In this case, other baskets may be placed in the basket body 102B, and the products TP may be placed in the other baskets. The detection sensor 13 may be appropriately positioned to detect when products TP are placed in other baskets. Detecting when products TP are placed in the basket body 102B may include detecting when products TP are placed in other baskets placed in the basket body 102B. The legs 1011B may also be configured to receive luggage. In this case, the detection sensor 13 may also be placed in the legs 1011B. If product TP identification by image analysis is also performed in the legs 1011B, the imaging device 16 may also be placed in the legs 1011B. [Explanation of Symbols]

[0192] 1…Smart Grip, 1A・1B…Shopping cart, 102A・102B…Basket body, 11...Control device, 13...Detection sensor, 14...Positioning sensor, 15...Output device, 16...Imaging device, 90...Housing, 915・935...Grooves, 20...Product input record, 201...Event location, 2… Information processing equipment, 30...Map information, 35...Purchase record, 40...imaging device, 400...imaging image, 45...Load sensor, 450...Measurement data, 50... Movement history, TP...Product, TP1...Target product, PS…Product shelf

Claims

1. A smart grip that can be used as a basket handle, Detection sensor, Positioning sensor, and control device Equipped with, The detection sensor is configured to detect when an item is placed into the basket. The positioning sensor is configured to measure position, The control device is When the detection sensor detects the insertion of the product, a product insertion record is generated that includes the event location where the product was inserted, which is measured by the positioning sensor, and To save the generated product input record, Configured to perform, Smart Grip.

2. A housing for housing the detection sensor, the positioning sensor, and the control device, further comprising a housing having one or more grooves for hooking the handle portion provided on the basket, The smart grip according to claim 1.

3. It is further equipped with an output device, The control device is further configured to output relevant information related to the input of the product from the output device when the detection sensor detects the input of the product. The smart grip according to claim 1.

4. The aforementioned related information includes at least one of the advertising information and coupon information for other products placed around the event location. The smart grip according to claim 3.

5. It is further equipped with an imaging device, The control device is When the detection sensor detects the insertion of the product, the image captured by the imaging device is acquired, and By performing image analysis on the acquired image, the product depicted in the image is identified. It is configured to perform further actions. The smart grip according to claim 1.

6. Basket body Detection sensor, Positioning sensor, and control device Equipped with, The detection sensor is configured to detect when an item is placed into the basket body. The positioning sensor is configured to measure position, The control device is When the detection sensor detects the insertion of the product, a product insertion record is generated that includes the event location where the product was inserted, which is measured by the positioning sensor, and To save the generated product input record, Configured to perform, Shopping cart.

7. By adding one or more wheels, it is configured as a cart. The shopping cart according to claim 6.

8. To obtain product entry records generated by shopping devices, including the event location where the target product was entered, for the target user. Identifying the target product that was put in according to the acquired product input record, and To output information regarding the results of identifying the aforementioned target product, A control unit configured to perform the following: Information processing device.

9. The information processing device according to claim 8, wherein identifying the target product in accordance with the product input record is performed by comparing the product input record with map information including identification information of the product in association with the location where the product is displayed in the store, thereby identifying the target product that was input at the event location.

10. Identifying the target product according to the product input record is: To acquire an image captured by an imaging device that includes the event location of the product input record within its imaging range, and By performing image analysis on the acquired image, the target product shown in the image is identified. Composed of, The information processing apparatus according to claim 8.

11. Each product shelf in the store is equipped with a load sensor. Identifying the target product according to the product input record is: To acquire measurement data from a load sensor installed near the event location in the product input record, and Based on the acquired measurement data, the target product is identified by identifying the shelf with reduced weight. Composed of, The information processing apparatus according to claim 8.

12. To obtain product entry records generated by shopping devices, including the event location where the target product was entered, for the target user. To generate the movement history of the target user from the event location of the acquired product entry record, and Output the generated movement history, A control unit configured to perform the following actions: Information processing device.

13. To obtain product entry records generated by shopping devices, including the event location where the product was entered, for a given user. To obtain the purchase records of the products purchased by the aforementioned target user, By comparing the acquired product entry records and purchase records with map information that includes product identification information in association with the location where the product is displayed in the store, the location where the target user placed the purchased product in the shopping cart can be identified, and Outputting information about the identified location, A control unit configured to perform the following actions: Information processing device.

Images