Smart Grip

Abstract

To provide a technology for easily monitoring a user's purchasing behavior. [Solution] A smart grip according to one aspect of the present disclosure is configured to be usable as a handle for a basket. The smart grip includes a detection sensor, a positioning sensor, and a control device. The detection sensor is configured to detect the insertion of a product into the basket. The positioning sensor is configured to measure a position. The control device is configured, when the detection sensor detects the insertion of the product, to generate a product insertion record including an event position measured by the positioning sensor, where the product was inserted, and to store the generated product insertion record.

Description

[Technical field]

[0001] This disclosure relates to a smart grip P Regarding. [Background technology]

[0002] There is a system that monitors the purchasing behavior of users in a store by installing multiple sensors such as an imaging device and a load sensor in the store and measuring the user's behavior with the installed multiple sensors. For example, Patent Document 1 proposes a system that receives an indicator of a user's location in a facility, receives an order for a product via an input device, and determines whether or not the order is associated with the user based on sensing data generated by one or more sensors in the facility. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2023-509844 Summary of the Invention [Problem to be solved by the invention]

[0004] When monitoring user purchasing behavior using only sensors installed in stores, the monitoring process may become complicated due to differences in sensor placement in each store, the number of sensors becoming huge, the amount of sensing data to be processed becoming huge, etc. In other words, it may become difficult to simply monitor user purchasing behavior.

[0005] In one aspect, the present disclosure has been made in consideration of the above circumstances, and one of its objectives is to provide a technology for easily monitoring user purchasing behavior and another objective is to provide a technology for utilizing product input records generated by monitoring purchasing behavior. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present disclosure employs the following configurations. Note that the following configurations can be appropriately combined.

[0007] A smart grip according to one aspect of the present disclosure is configured to be usable as a handle for a basket. The smart grip includes a detection sensor, a positioning sensor, and a control device. The detection sensor is configured to detect the insertion of a product (target product) into the basket. The positioning sensor is configured to measure a position. When the detection sensor detects the insertion of the product, the control device is configured to generate a product insertion record including an event position where the product was inserted, which is an event position measured by the positioning sensor, and to store the generated product insertion record.

[0008] According to this configuration, by generating a product input record using the detection sensor and positioning sensor provided in the smart grip, it is possible to monitor the purchasing behavior of users without installing sensors in the store. This is expected to simplify monitoring. The smart grip according to this configuration may be used to monitor the purchasing behavior of users together with sensors installed in the store. In this case, it is expected to simplify monitoring by using the product input record to improve the efficiency of processing sensing data obtained by sensors installed in the store.

[0009] The smart grip according to the above aspect may further include a housing that houses the detection sensor, the positioning sensor, and the control device and has one or more grooves for hooking a handle provided on a basket. With this configuration, it is possible to provide a smart grip that can be used as a handle for a basket by replacing the handle of an existing basket.

[0010] The smart grip according to the above aspect may further include an output device. The control device may be further configured to output related information related to the insertion of the product from the output device when the detection sensor detects the insertion of the product. With this configuration, by providing related information when the product is inserted, it is expected that the activation of purchasing activity such as stimulating the desire to purchase further products can be expected.

[0011] In the smart grip according to the above aspect, the related information may include at least one of advertisement information and coupon information for other products arranged in the vicinity of the event location. According to this configuration, by providing at least one of the advertisement information and the coupon information, it is expected that a desire to purchase other products can be stimulated.

[0012] The smart grip according to the above aspect may further include an imaging device. The control device may be configured to acquire an image captured by the imaging device when the detection sensor detects the insertion of a commodity, and to identify the commodity shown in the image by performing image analysis on the acquired image. With this configuration, the inserted commodity can be identified on the smart grip.

[0013] The embodiment of the present disclosure may not be limited to the smart grip. An aspect of the present disclosure may be a component of a basket used for shopping. An aspect of the present disclosure may be a basket (including a cart) itself. In addition, an aspect of the present disclosure may be an information processing method for realizing all or part of the control device (computer), a program, or a storage medium readable by a machine such as a computer that stores such a program. Here, the machine-readable storage medium may be a non-transitory medium that accumulates information such as a program by electrical, magnetic, optical, mechanical, or chemical action. The non-transitory storage medium may include a storage medium (CD, DVD, semiconductor memory, etc.), an auxiliary storage device of a computer, an external storage device connected to a computer, etc.

[0014] For example, a shopping basket according to one aspect of the present disclosure may include a basket body, a detection sensor, a positioning sensor, and a control device. The detection sensor may be configured to detect the insertion of an item into the basket body. The positioning sensor may be configured to measure a position. The control device may be configured, when the detection sensor detects the insertion of the item, to generate a product insertion record including an event position measured by the positioning sensor, where the product was inserted, and to store the generated product insertion record.

[0015] The shopping basket according to the above aspect may be configured as a cart by further including one or more wheels. With this configuration, the cart form is expected to facilitate monitoring of purchasing behavior by users.

[0016] According to another aspect of the present disclosure, an information processing device includes a control unit. The control unit is configured to acquire a product input record of a target user generated by a shopping device, the product input record including an event location where a target product was input, identify the input target product according to the acquired product input record, and output information related to the result of identifying the target product. With this configuration, it is possible to provide a technology for identifying an input product by utilizing the product input record.

[0017] In the information processing device according to the above aspect, identifying the target product according to the product input record may be configured by identifying the target product input at the event location by correlating the product input record with map information including identification information of the product in association with the display position of the product in the store. According to this configuration, the target product can be appropriately identified by using the map information.

[0018] In the information processing device according to the above aspect, identifying the target product according to the product input record may be configured by acquiring an image captured by an imaging device whose imaging range includes the event position of the product input record, and performing image analysis on the acquired captured image to identify the target product shown in the captured image. If image analysis is performed on all captured images acquired by all imaging devices installed in the store, the amount of processing may become enormous. In contrast, according to this configuration, the range of captured images to be subjected to image analysis can be narrowed down to the range of event detection based on the product input record. This can improve the efficiency of the product identification process (image analysis).

[0019] In the information processing device according to the above aspect, a load sensor may be installed on each product shelf in the store. Identifying the target product according to the product input record may be configured to acquire measurement data of a load sensor installed near the event position of the product input record, and identify the product shelf where the weight has decreased according to the acquired measurement data, thereby identifying the target product. If all load sensors installed in the store are constantly monitored, the amount of processing may become enormous. In contrast, according to this configuration, the range of load sensors to be monitored can be narrowed down to the range of event detection based on the product input record. This can improve the efficiency of the product identification process.

[0020] According to another aspect of the present disclosure, an information processing device includes a control unit. The control unit is configured to acquire a product input record of a target user, the product input record being generated by a shopping device and including an event location where a target product was input, generate a movement history of the target user from the event location of the acquired product input record, and output the generated movement history. With this configuration, it is possible to provide a technique for monitoring the movement history of a target user by utilizing the product input record.

[0021] Also, an information processing device according to another aspect of the present disclosure includes a control unit. The control unit is configured to execute the following: acquiring a product input record of a target user, which is a product input record generated by a shopping device and includes an event location where the product was input; acquiring a purchase record of a purchased product of the target user; correlating the acquired product input record and purchase record with map information including identification information of the product in association with a display location of the product in the store, thereby identifying a location where the target user put the purchased product into a basket; and outputting information about the identified location. The same product may be arranged in multiple locations (e.g., permanent shelves, special shelves, etc.) in a store. According to this configuration, a technology can be provided that uses the product input record to identify a location where the target user put the purchased product into a basket.

[0022] In the information processing device according to each of the above aspects, the shopping device may be at least one of the smart grip and the shopping basket according to any of the above aspects. That is, the smart grip and the shopping basket according to the above aspect are examples of the shopping device. However, the shopping device does not have to be limited to the smart grip and the shopping basket according to the above aspect. As long as it is possible to generate a product insertion record including the event location, the configuration and type of the shopping device are not particularly limited and may be appropriately selected depending on the embodiment.

[0023] The embodiment of the present disclosure is not limited to the above information processing device (computer). One aspect of the present invention may be an information processing method for realizing all or a part of the information processing device, an information processing system, a program, or a storage medium that stores such a program and is readable by a machine such as a computer. The information processing system may be composed of a plurality of computers such as a server device and a store terminal. Effect of the Invention

[0024] According to one aspect of the present disclosure, it is possible to provide a technique for simply monitoring a user's purchasing behavior. Also, according to another aspect of the present disclosure, it is possible to provide a technique for utilizing a product input record generated by monitoring the purchasing behavior. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 illustrates an example of a situation to which the present disclosure is applied. [Diagram 2] FIG. 2 is a schematic diagram showing an example of a usage scenario of the product input record. [Diagram 3] FIG. 3 shows a schematic example of a method for using the product input record. [Figure 4] FIG. 4 is a schematic diagram showing an example of a method for identifying a product using a product input record. [Diagram 5] FIG. 5 is a schematic diagram showing an example of a method for identifying a product using a product input record. [Figure 6] FIG. 6 is a schematic diagram showing an example of a method for identifying a product using a product input record. [Figure 7] FIG. 7 is a schematic diagram showing another example of a method for using the product input record. [Figure 8] FIG. 8 is a schematic diagram showing another example of a method for using the product input record. [Figure 9] FIG. 9 is a side view that illustrates an example of the structure of a smart grip. [Figure 10] FIG. 10 is a plan view that illustrates an example of the structure of a smart grip. [Figure 11] FIG. 11 illustrates an example of a hardware configuration of an information processing device. [Figure 12] FIG. 12 illustrates an example of a software configuration of the smart grip (control device). [Figure 13] FIG. 13 illustrates an example of a software configuration of an information processing device. [Figure 14] FIG. 14 is a flowchart illustrating an example of a process for generating a product input record by the control device. [Figure 15] FIG. 15 is a flowchart illustrating an example of a processing procedure regarding product identification by the control device. [Figure 16] FIG. 16 is a flowchart illustrating an example of a processing procedure regarding output of related information by the control device. [Figure 17] FIG. 17 is a flowchart illustrating an example of a processing procedure for product identification using a product input record by an information processing device. [Figure 18] FIG. 18 is a flowchart illustrating an example of a processing procedure for generating a movement history using a product input record by an information processing device. [Figure 19] FIG. 19 is a flowchart illustrating an example of a process for identifying a location using a product insertion record by an information processing device. [Figure 20] FIG. 20 illustrates another example of a situation to which the present disclosure is applied. [Figure 21] FIG. 21 illustrates a schematic diagram of another example of a situation to which the present disclosure is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, an embodiment according to one aspect of the present disclosure will be described with reference to the drawings. However, the embodiment described below is merely an example of the present disclosure in all respects. Various improvements or modifications may be made without departing from the scope of the present disclosure. In implementing the present disclosure, a specific configuration according to the embodiment may be appropriately adopted. Note that, although the data appearing in this embodiment is described in natural language, more specifically, it is specified by pseudo-language, commands, parameters, machine language, electrical signals, etc. that can be recognized by a machine such as a computer.

[0027] §1 Examples of application FIG. 1 is a schematic diagram showing an example of a situation to which the present disclosure is applied. A smart grip 1 according to this embodiment is configured to be usable as a handle for a basket B1. The smart grip 1 includes a control device 11, a detection sensor 13, and a positioning sensor 14. The detection sensor 13 is configured to detect the insertion of a product TP into the basket B1. The positioning sensor 14 is configured to measure a position. When the detection sensor 13 detects the insertion of the product TP, the control device 11 is configured to execute the following: generating a product insertion record 20 including the event position 201 where the product TP was inserted, which is the event position 201 measured by the positioning sensor 14; and storing the generated product insertion record 20.

[0028] According to this embodiment, the product input record 20 is generated using the detection sensor 13 and the positioning sensor 14 provided in the smart grip 1, so that the purchasing behavior of the user can be monitored without installing sensors in the store. This is expected to simplify the monitoring of purchasing behavior. The smart grip 1 according to this embodiment may be used to monitor the purchasing behavior of the user together with sensors installed in the store. In this case, the product input record 20 is used to improve the efficiency of processing the sensing data obtained by the sensors installed in the store, so that the monitoring of purchasing behavior can be expected to be simplified.

[0029] [Basket] 1, in a typical example, the basket B1 may be a shopping basket including a basket body B10 and a pair of handles B11. When the basket B1 includes the basket body B10, putting a product TP into the basket body B10 may correspond to putting the product TP into the basket B1.

[0030] The configuration of the basket body B10 and each handle portion B11 may be appropriately determined according to the embodiment. For example, the basket body B10 may be configured to receive the product TP from the upper end side by having an open upper end and a closed lower end at the bottom. The basket body B10 may have any shape, such as a substantially rectangular parallelepiped shape or a substantially rectangular pyramid shape. The pair of handle portions B11 may be arranged separately in the longitudinal direction of the basket body B10. Each end of the handle portion B11 may be connected to each side surface in the short side direction of the basket body B10 so that the handle portion B11 can be rotated around an axis in the short side direction. As a result, each handle portion B11 may be configured to be rotated at the connection portion with the basket body B10 to change the angle and be gripped along the short side direction.

[0031] The configuration of the basket B1 is not limited to the example of FIG. 1, and may be changed as appropriate depending on the embodiment. For example, the number of handles B11 is not limited to two. The number of handles B11 may be one, or three or more. In addition, each end of one handle B11 may be separated and connected to one side surface in the short direction of the basket body B10 in the longitudinal direction, and each end of the other handle B11 may be separated and connected to the other side surface in the short direction of the basket body B10 in the longitudinal direction. In this way, each handle B11 may be configured to be gripped along the longitudinal direction.

[0032] The type of basket B1 is not limited to the example in FIG. 1 and may be changed as appropriate depending on the embodiment. As long as it can be used to carry the product TP, the type of basket B1 is not particularly limited and may be selected as appropriate depending on the embodiment. In another example, the basket B1 may include a bag that can be used for shopping, such as a plastic bag or a paper bag. The basket B1 may also include a device that can hold products by hanging them, such as a hanger.

[0033] [Can be used as a basket handle] The fact that it can be used as a handle of a basket (such as the basket B1) means that the smart grip 1 is disposed as at least a part of a handle that is held by a user when carrying a product (product TP). The smart grip 1 may be configured in any form in which it can be placed on a basket. In one example, the smart grip 1 can be used as a handle of a basket by replacing an existing handle provided on the basket by hanging a handle part of the basket (handle part B11, etc.) on the smart grip 1. The handle part of the basket may include any component that can be hung on the smart grip 1, such as a handle part of a bag or a hook of a hanger. In another example, the smart grip 1 can be used as a handle of a basket by being attached to the handle part of the basket. In yet another example, the smart grip 1 can be used as a handle of a basket by being attached to the basket body (basket body B10, etc.) as a handle part. In other words, the smart grip 1 may be configured as the handle part of the basket itself.

[0034] [Product / Store] The product TP may include any kind of item to be carried. The product TP may be referred to as a target product. The smart grip 1 may be used in any store. The configuration of the store may not be particularly limited and may be appropriately determined according to the embodiment. The sales format of the store may be either unmanned or manned. The store may or may not be equipped with sensors such as an imaging device and a load sensor. When sensors are installed in the store, the purchasing behavior of the target user may be monitored by cooperation between the sensors of the smart grip 1 (detection sensor 13, positioning sensor 14, etc.) and the sensors installed in the store. Alternatively, the smart grip 1 may monitor the purchasing behavior of the target user independently of the sensors installed in the store (i.e., without cooperation with the sensors of the store).

[0035] [Product introduction] Putting a product (product TP) into a basket (basket B1) may include any form of carrying a product from a sales floor. As illustrated in FIG. 1, in a typical example, putting a product into a basket may be putting a new product TP into a basket body B10 of a basket B1 including a smart grip 1. In addition, putting a product into a basket may include any action similar to putting a product into a basket, such as hanging a handle of a basket containing a product on a smart grip 1. Hanging the handle of a basket containing a product may include hanging the handle of a plastic bag containing the product, hanging the hook of a hanger that is hanging the product, etc.

[0036] [Product input record] The product insertion record 20 is configured to include an event position 201. The event position 201 is derived from a measurement value of the positioning sensor 14. For example, the measurement value of the positioning sensor 14 may be used as the event position 201 as it is. Alternatively, the event position 201 may be calculated from the measurement value of the positioning sensor 14 by applying any arithmetic processing to one or more measurement values ​​of the positioning sensor 14. The event position 201 may be derived from a value measured in real time by the positioning sensor 14 in response to the detection of the insertion of the product TP by the detection sensor 13. The event position 201 may be appropriately configured to indicate the position where the product TP is inserted (insertion position). In a typical example, the event position 201 may be configured to indicate the insertion position by two-dimensional or three-dimensional coordinate values. However, the configuration of the event position 201 is not limited to such an example and may be appropriately changed depending on the embodiment. As long as the product insertion record 20 includes the event position 201, the configuration of the product insertion record 20 is not particularly limited and may be appropriately specified depending on the embodiment. The product insertion record 20 may be composed of only the event location 201, or may further include other information other than the event location 201.

[0037] (Identification information) As an example of other information, the product input record 20 may further include identification information 203 for identifying a user (target user) who uses the smart grip 1. The source of the identification information 203 may be appropriately determined depending on the embodiment. In one example, the smart grip 1 has a memory The storage unit 12 may further include a storage unit 12. The identification information 125 may be stored in advance in the storage unit 12. As long as the target user can be identified, the identification information 125 may be configured arbitrarily. For example, the identification information 125 may be configured to directly indicate the target user, such as the account or user identifier of the target user. The identification information 125 may be configured to indirectly indicate the target user, such as the identifier of the smart grip 1 or the identifier of the control device 11. The identification information 203 of the product input record 20 may be appropriately derived from the identification information 125 of the storage unit 12. In one example, the identification information 125 of the storage unit 12 may be used as the identification information 203 of each product input record 20 as it is. In another example, the identification information 203 may be calculated from the identification information 125 by applying any arithmetic processing (deletion of a value, addition of a value, arithmetic operations, conversion, etc.) to the identification information 125.

[0038] According to one example of this embodiment, since the product input record 20 includes the identification information 203, it is possible to easily identify the user corresponding to the product input record 20 (i.e., the user who input the product TP at the event position 201 indicated in the product input record 20). When the product input records 20 of multiple users are collected, it is possible to easily extract the product input record 20 of a specific user from the collected product input records 20 by searching using the identification information 203.

[0039] Furthermore, when identifying a user corresponding to an input event using equipment provided in a store, for example, imaging devices are installed in multiple locations in the store, and image analysis (such as face recognition) is periodically performed on images captured by the imaging devices, which may increase the burden of installing sensors and the processing load of tracking people. In contrast, according to one example of the present embodiment, the identification information 125 of the smart grip 1 is linked to the product input record 20 as the identification information 203, so that the user corresponding to the input event can be identified. This makes it possible to reduce at least one of the burden of installing sensors in the store and the processing load of tracking people.

[0040] The identification information 125 held in the smart grip 1 may be used for purposes other than deriving the identification information 203 of the product insertion record 20. As an example, the identification information 125 of the smart grip 1 may be used to authenticate a user at least when entering or leaving a store. For example, the smart grip 1 may exchange data with a store system (such as a store terminal) at the time of entering or leaving the store. The method of data exchange may be selected arbitrarily.

[0041] As an example of a data exchange method, the smart grip 1 may further include an output device 15. The output device 15 may include a display. The smart grip 1 (control device 11) may convert the identification information 125 into a code and output the obtained code from the output device 15. The code may further include any information other than the identification information 125. For example, the code may be generated as a one-time code by further including temporary information such as a time or a random number. The type of code may be selected arbitrarily. For example, the code may be composed of a one-dimensional code or a two-dimensional code. A code reader may be provided at the entrance and exit of the store. The store's system may execute data exchange with the smart grip 1 by reading the information (including the code) of the smart grip 1 with the reader.

[0042] As another example of the data exchange method, the smart grip 1 may further include a communication module 17. The communication module 17 may include a close-proximity wireless communication module. The standard of close-proximity wireless communication is not particularly limited and may be appropriately selected depending on the embodiment. Accordingly, a module (reader, etc.) corresponding to the close-proximity wireless communication module of the smart grip 1 may be provided at the entrance and exit of the store. Accordingly, the smart grip 1 and the store system may exchange data via the close-proximity wireless communication module.

[0043] The store system may acquire identification information 125 from the smart grip 1 in the data exchange. The store system authenticates the user according to the acquired identification information 125. The store system may be configured to control the unlocking and locking of the entrance and exit doors in response to the authentication of the user. For example, the store system may keep the entrance and exit doors locked unless the authentication of the user is performed. The store system may unlock the corresponding entrance or exit door in response to the authentication of the user being established. After the user passes through the entrance or exit (for example, the door is opened once and then closed), the store system may lock the corresponding entrance or exit door again. According to one example of the present embodiment, the user present in the store can be easily identified. In addition, the control device 11 of the smart grip 1 may start executing a process of monitoring the purchasing behavior of the user, such as generating a product insertion record 20, after the authentication at the entrance is established. After the authentication at the exit is established, the control device 11 may stop executing the monitoring process. This allows the operation of the control device 11 to be appropriately controlled so that the monitoring process is executed in the store.

[0044] (time) As another example of information, the product input record 20 may further include a time 205 when the product input record 20 was generated (i.e., the product TP was input). The source of the time 205 may be appropriately determined according to the embodiment. In one example, the smart grip 1 may further include a timer. The control device 11 may refer to the timer at the time when the detection sensor 13 detects the input of the product TP (generates the product input record 20), and obtain the time indicated by the timer as the time 205. According to one example of the present embodiment, by using the time 205, it is possible to extract a product input record 20 (input event) corresponding to a specific time period from among a plurality of product input records 20. In addition, by using the time 205, it is possible to classify the product input records 20 by time period and analyze the characteristics of the input event in each time period (for example, which position the product TP is located in is likely to be purchased, etc.).

[0045] (others) In the example of FIG. 1, the product input record 20 includes the event location 201, the identification information 203, and the time 205. However, the configuration of the product input record 20 is not limited to this example, and may be changed as appropriate depending on the embodiment. For example, at least one of the identification information 203 and the time 205 may be omitted. The product input record 20 may further include information other than the identification information 203 and the time 205. In one example, the product input record 20 may further include an identifier for identifying the product input record 20.

[0046] [Keeping records of product input] The product input record 20 may be stored in any memory area. In one example, the smart grip 1 may further include a memory unit 12. The generated product input record 20 may be stored in the memory unit 12. In another example, the smart grip 1 may further include a communication module 17. In this case, the smart grip 1 may transmit the product input record 20 to an external computer (including the information processing device 2 described later) such as a terminal or a server device via the communication module 17. As a result, the product input record 20 may be stored in the memory area of ​​the external computer. The terminal may include a user terminal, a store terminal (a terminal installed in a store), etc. The server device may include an external storage device such as a NAS (Network Attached Storage). In either case of storing the product input record 20 in the memory unit 12 in the smart grip 1 or storing it in an external computer, storing the product input record 20 may be performed using a cache memory, a RAM (Random Access Memory), or the like. This may include temporarily storing the item input record 20 in a memory such as a universally accessible memory (UDMA).

[0047] Furthermore, the process of saving the product input record 20 may be executed at any timing after the process of generating the product input record 20 has been executed one or more times. In one example, the smart grip 1 may sequentially save the generated product input record 20 in response to generating the product input record 20. In another example, the smart grip 1 may repeatedly execute the generation of the product input record 20 to form a product input record group, and store at least a part of the formed product input record group. (Multiple product input records 20) may be stored together.

[0048] [Product identification using smart grips] In one example of this embodiment, the control device 11 may be configured to identify the product TP according to the observation result of the product TP by a sensor provided in the smart grip 1. The sensor used for the identification may be appropriately selected depending on the embodiment. The detection sensor 13 may be used for the identification, or a sensor other than the detection sensor 13 may be used for the identification. The identification method may be appropriately determined depending on the sensor to be used.

[0049] In one example, the smart grip 1 may further include an imaging device 16. The imaging device 16 may be appropriately arranged so as to capture the product TP when the product TP is inserted into the basket B1 (basket body B10). For example, the imaging device 16 may be arranged so that the basket B1 (basket body B10, etc.) is within the imaging range. As long as an imaged image capable of identifying the product TP can be generated, the type of the imaging device 16 is not particularly limited and may be appropriately selected according to the embodiment. The imaging device 16 may include any sensor that acquires data in the form of an image or an image representation, such as, for example, an RGB camera, a depth sensor, an infrared sensor, a radar, a LiDAR (light detection and ranging), etc.

[0050] The control device 11 may acquire the captured image 165 of the imaging device 16 when the detection sensor 13 detects the insertion of the commodity TP. The timing of imaging by the imaging device 16 (generation of the captured image 165) may be appropriately specified according to the embodiment. In one example, when the imaging device 16 is disposed in a position where it can capture an image of the commodity TP after the insertion is detected, the control device 11 may start the imaging device 16 in response to the detection of the insertion of the commodity TP by the detection sensor 13, and control the operation of the imaging device 16 to capture an image of the inserted commodity TP. The control device 11 may control the imaging device 16 to capture an image of the commodity TP at any timing after the detection of the insertion of the commodity TP. For example, the control device 11 may control the imaging device 16 to capture an image of the commodity TP immediately after the detection of the insertion of the commodity TP. The control device 11 may appropriately acquire one or more captured images 165 generated by this imaging from the imaging device 16. After generating one or more captured images 165, the control device 11 may stop the operation of the imaging device 16 or switch it to a standby state. According to this example, since the period during which the imaging device 16 is activated can be narrowed down, it is expected that the processing load and power consumption in the smart grip 1 can be reduced. In another example, the imaging device 16 may be controlled to continuously generate captured images. The generated captured images may be stored in a storage area such as the storage unit 12 for at least a certain period of time. The captured images generated at or around the time when the detection sensor 13 detects the insertion of the commodity TP are highly likely to include the inserted commodity TP. Therefore, the control device 11 may acquire, as the captured image 165, one or more captured images generated at or around the time when the detection sensor 13 detects the insertion of the commodity TP from the captured images stored in the storage area.

[0051] After acquiring the captured image 165, the control device 11 may identify the commodity TP shown in the captured image 165 by performing image analysis on the acquired captured image 165. The identification method by image analysis may be appropriately determined according to the embodiment. In one example, when a code generated by encoding identification information of the commodity TP is assigned to the commodity TP, analyzing the captured image 165 may include decoding the code assigned to the commodity TP. The type of the code may be selected arbitrarily. The code may be, for example, a one-dimensional code, a two-dimensional code, or the like. In another example, the control device 11 may identify the commodity TP from the characteristics (appearance, etc.) of the commodity TP in the image analysis, without relying on the code.

[0052] The method of image analysis is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the control device 11 may analyze the captured image 165 by a general image analysis method such as edge detection, pattern matching, etc. In another example, the control device 11 may analyze the image. The captured image 165 may be analyzed using a trained machine learning model that has acquired the capability. The machine learning model is configured to include one or more calculation parameters that can be adjusted by machine learning. The one or more calculation parameters are used for the calculation of the desired inference (such as analyzing an image). The machine learning model may be configured, for example, by a neural network, a support vector machine, or other functional expressions (calculation models). The machine learning method may be appropriately selected according to the machine learning model to be adopted (for example, backpropagation method, etc.). Training the machine learning model means adjusting (optimizing) the values ​​of the calculation parameters using training samples. The machine learning model may be appropriately trained to derive the true value of the corresponding analysis result when given an image of the training sample. A large-scale model such as a large-scale visual language model (VLM) may be used as the trained machine learning model.

[0053] The control device 11 may output the identification result of the product TP as appropriate. The identification result of the product TP may be used for any purpose. In one example, the identification result of the product TP may be added to the product insertion record 20. That is, generating the product insertion record 20 including the event position 201 may be configured by generating the product insertion record 20 including the event position 201 and the identification result of the product TP. According to this example, by referring to the identification result of the product TP, the product TP inserted in the event indicated by the product insertion record 20 can be specified. When the output device 15 is provided, the control device 11 may output the identification result of the product TP from the output device 15. Also, when the communication module 17 is provided, the control device 11 may output the identification result of the product TP to an external computer (a store terminal, a user terminal, a server device, etc.).

[0054] According to one example of the present embodiment, the inserted commodity TP can be identified on the smart grip 1. The computer that performs the image analysis of the captured image 165 is not limited to the control device 11. In another example, the control device 11 may transmit a request to perform image analysis on the captured image 165 together with the captured image 165 to an external computer such as a store terminal, a user terminal, or a server device. Then, the control device 11 may obtain the identification result of the commodity TP by receiving a reply of the result of the image analysis including the identification result of the commodity TP from the external computer. The identification method of the commodity TP is not limited to the above-mentioned method by image analysis, and may be appropriately changed according to the embodiment. In another example, the smart grip 1 may further include a code reading device. The control device 11 may identify the commodity TP by reading the code attached to the commodity TP with the reading device. Also, the identification process on the smart grip 1 may be omitted. The commodity insertion record 20 may be generated without identifying the commodity TP. The process of identifying the commodity TP may be executed by an external computer (including the information processing device 2 described later) instead of the control device 11.

[0055] [Output related information] In one example of this embodiment, the smart grip 1 may further include an output device 15. The output device 15 may be configured to output information. The output device 15 may be configured, for example, with a display, a speaker, an indicator light, etc. The output format of the information may be appropriately selected from text, images, sounds, etc. When the output device 15 includes a display, it is preferable that the display is disposed in a location where the output information can be viewed while the user is holding the smart grip 1. This allows the user to check the information displayed on the display while using the smart grip 1.

[0056] The conditions for outputting information and the information to be output may be appropriately determined according to the embodiment. In one example, the control device 11 may be further configured to output related information 155 related to the introduction of the product TP from the output device 15 when the introduction of the product TP is detected by the detection sensor 13. According to one example of the present embodiment, by providing the related information 155 when the product TP is introduced, it is possible to expect activation of purchasing activities such as stimulating the desire to purchase further products.

[0057] The related information 155 may be output at any timing after the detection sensor 13 detects the insertion of the product TP. For example, the control device 11 may output the related information 155 from the output device 15 in real time after the detection sensor 13 detects the insertion of the product TP. The source of the related information 155 may be appropriately determined according to the embodiment. For example, when the smart grip 1 includes a storage unit 12, the related information 155 may be stored in advance in the storage unit 12. The control device 11 may acquire the related information 155 from the storage unit 12. Also, for example, the related information 155 may be stored in a storage area of ​​an external computer (a store terminal, a user terminal, a server device, etc.). The control device 11 may acquire the related information 155 from the external computer at any timing before output.

[0058] In addition, the contents of the related information 155 are not particularly limited as long as they relate to the event position 201, the product TP to be inserted, the act of inserting the product TP, and other purchasing activities, and may be appropriately selected according to the embodiment. When there are multiple candidates for the related information 155, the related information 155 to be output may be appropriately selected from the multiple candidates. For example, the related information 155 may be selected from the multiple candidates according to at least one of the event position 201, the product TP, and the user information. Also, for example, the related information 155 may be selected randomly. The selection of the related information 155 may be performed by either the control device 11 or an external computer (a store terminal, a user terminal, a server device, etc.). When selecting the related information 155 from the multiple candidates, any information may be referenced. The information used to select the related information 155 may be held in at least one of the control device 11 and the external computer, and may be appropriately referenced when selecting the related information 155.

[0059] In one example of this embodiment, the contents of the related information 155 may be configured according to at least one of the following first to seventh examples. In each of the following examples, for convenience of explanation, a form in which the control device 11 selects the related information 155 is assumed. However, as described above, the computer that executes the process of selecting the related information 155 does not have to be limited to the control device 11. At least a part of the process of the control device 11 described below may be executed by an external computer. Information that the control device 11 refers to when selecting the related information 155 may be held in either the control device 11 or the external computer.

[0060] (1) First Case In one example, the related information 155 may include at least one of advertisement information 1550 and coupon information 1555 of other products placed around the event location 201. The other products are products other than the inserted product TP (target product). The advertisement information 1550 includes advertisements for the other products. The advertisement content may be determined appropriately. In a simple example, the advertisement information 1550 may be composed of the names of the other products. The coupon information 1555 includes information indicating discounts on the other products.

[0061] The other products may be specified as appropriate. For example, candidates for the related information 155 may be predefined for each area in the store. In response to this, when the insertion of the product TP is detected, the control device 11 may specify the area to which the event position 201 belongs, and may select the related information 155 according to the specified area. When identifying the product TP, the control device 11 may exclude candidates corresponding to the product TP indicated by the identification result, and select the related information 155 from the remaining candidates. In addition, for example, the control device 11 may refer to information (such as map information 30 described later) indicating the correspondence between the display positions of products existing in the store and the identification information of the products. The control device 11 may determine the other products by selecting products displayed in the vicinity of the event position 201 and at a position different from the event position 201 from the products existing in the store indicated by the referenced information. The range of the "periphery" may be determined as appropriate. For example, whether or not the product belongs to the "periphery" may be determined by a method such as threshold value determination. The control device 11 may acquire the related information 155 by selecting a candidate linked to another product from among multiple candidates.

[0062] According to an example of the present embodiment, by providing at least one of the advertisement information 1550 and the coupon information 1555, it is expected that the purchase motivation of the other product will be stimulated. In addition, in an example of the present embodiment, it is preferable that the control device 11 outputs the related information 155 including at least one of the advertisement information 1550 and the coupon information 1555 of the other product in real time immediately after the detection sensor 13 detects the insertion of the product TP. In this way, while the target user is located in the vicinity of the other product, at least one of the advertisement information 1550 and the coupon information 1555 of the product can be provided to the target user. In other words, the related information 155 of the product located in the vicinity of the target user can be provided to the target user. As a result, since the effort of picking up the other product is reduced, it is expected that the probability that the target user will purchase the other product will be improved.

[0063] (2) Second Case In one example, the related information 155 may be related to the product TP. For example, it may include at least one of advertising information, coupon information, and attribute information of the product TP. The attribute information may include any kind of information related to the characteristics of the product TP. For example, the attribute information may be composed of the expiration date, release date, allergy indication, reviews, related products, recommended products, or a combination of these of the product TP. The allergy indication indicates allergens contained in the product TP. The related products may be other products related to the product TP. For example, the other products related to the product TP may be products used together with the product TP. The recommended products may be other products recommended to people who purchase the product TP. The related products and the recommended products may partially overlap, or may not overlap. The related information 155 related to the product TP may be appropriately specified. For example, when identifying the product TP, the control device 11 may acquire the related information 155 related to the product TP according to the identification result of the product TP. According to one example of the present embodiment, information related to the product TP can be provided to the user.

[0064] (3) Third Case In one example, the content of the related information 155 may be determined according to the past purchase history. The control device 11 may identify a product that has not been purchased for a certain period of time by referring to the purchase history. The related information 155 may be configured by a notification informing the user of the identified product. In addition, when identifying the product TP, the control device 11 may extract a history related to the product TP from the purchase history according to the identification result of the product TP. The related information 155 may be configured by information related to the purchase history of the extracted product TP. For example, in the purchase history of the product TP, when it is determined that the input product TP has been purchased multiple times in the past, the related information 155 may be configured by a notification indicating that a change to a subscription is recommended. The threshold value of the number of purchases for recommending a change to a subscription may be arbitrarily defined. In addition, for example, when the input product TP has been purchased within a predetermined period in the past, the related information 155 may be configured by a notification indicating that the product TP has been purchased in the past. The predetermined period may be arbitrarily defined. According to one example of the present embodiment, information related to the past purchase history can be provided to the user.

[0065] (4) Fourth Case In one example, the related information 155 may be configured with information related to the budget. For example, the related information 155 may be configured with a notification (warning) indicating that the budget upper limit is being approached. The control device 11 may appropriately acquire the amount (price) of the product TP. For example, the information indicating the amount may be linked to the product TP (identification information, etc.). When identifying the product TP, the control device 11 may acquire information indicating the amount of the product TP according to the identification result of the product TP. Each time the control device 11 detects the insertion of a product TP by the detection sensor 13, the control device 11 may calculate the total amount of the one or more products TP inserted into the basket B1 by adding up the acquired amounts. Then, the control device 11 may determine whether the calculated total amount is close to the budget upper limit. If it is determined that the total amount is close to the budget upper limit, the control device 11 may calculate the amount of the product TP that is close to the budget upper limit. If not, the control device 11 may omit output of the related information 155 configured by the notification indicating that the budget upper limit has been reached. The budget upper limit may be set arbitrarily. For example, the budget upper limit may be input by the operation of the control device 11 or via an external computer. Furthermore, whether the budget upper limit has been reached may be determined by any method such as threshold determination. The threshold may be specified arbitrarily. According to one example of the present embodiment, it is possible to provide the user with information related to the budget, such as a warning that the budget upper limit has been reached. Note that the information related to the budget may be changed as appropriate. For example, whether the budget upper limit has been reached may be replaced with whether the budget upper limit has been exceeded. The stage at which the warning is notified may not be limited to the budget upper limit. Furthermore, multiple stages at which the warning is notified may be set, and the notification content may be changed for each stage.

[0066] (5) Fifth Case In one example, the related information 155 may be composed of any information derived from the information of the product TP and the user information of the target user. The user information may include any information of the target user. The user information may include information about the family of the target user. For example, when at least one of the allergens of the target user himself or her and the family of the target user is contained in the product TP, the related information 155 may be composed of a warning indicating that the product TP contains an allergen. In this case, the user information may include allergy information indicating at least one of the allergens of the target user himself or her and the family of the target user. The allergy indication of the product TP may be acquired as appropriate. In one example, the attribute information including the allergy indication may be linked to the product TP (identification information, etc.). When identifying the product, the control device 11 may acquire the allergy indication of the product TP according to the identification result of the product. The control device 11 may refer to the user information as appropriate. The control device 11 may compare the allergy indication of the product TP with the allergy information included in the user information. The control device 11 may determine whether at least one of the allergens of the target user and the family of the target user is contained in the product TP according to the result of the comparison. When it is determined that an allergen is contained in the product TP, the control device 11 may output a warning indicating that an allergen is contained in the product TP as the related information 155. According to one example of this embodiment, it is possible to provide the user with information about the product TP relating to the target user, such as a warning indicating that an allergen is contained in the product TP. Note that the information derived from the product TP and the user information is not limited to the information about allergies described above, and may be changed as appropriate depending on the embodiment.

[0067] (6) Sixth Case In one example, the related information 155 may be information obtained by searching the Web for the product TP to be inserted. For example, the related information 155 may be information showing a comparison between the selling price of the product TP to be inserted in a store and the selling price on an EC (electronic commerce) site. The control device 11 may be configured as follows. The selling prices at the store and the EC site may be appropriately acquired. As an example, the information indicating the selling price at the store may be linked to the product TP (identification information, etc.). When identifying the product TP, the control device 11 may acquire information indicating the selling price of the product TP at the store according to the identification result of the product TP. In addition, the control device 11 may acquire information indicating the selling price at the EC site by performing a web search according to the identification result of the product TP. The EC site to search for the selling price may be selected arbitrarily. In an example, one or more EC sites may be predefined. The control device 11 may generate information indicating a comparison between the selling price at the store and the selling price at the EC site based on the acquired information. The configuration of the information indicating the comparison may be appropriately determined according to the embodiment. In a simple example, the control device 11 may output the selling price at the store and the selling price at the EC site as the related information 155 as they are. In another example, the control device 11 may determine whether the selling price at the store is lower than the selling price at the EC site. If the selling price at the store is lower than the selling price on the EC site, the control device 11 sets information indicating that the selling price at the store is lower than the selling price on the EC site (for example, a message "NNN yen cheaper than the AAA site") as related information 155. According to one example of this embodiment, information about the product TP available on the web, such as a comparison with the selling price on an EC site, can be provided to the user.

[0068] The information obtained by the web search does not have to be limited to a comparison with the selling price on the EC site, and may be appropriately selected depending on the embodiment. The information search may be performed by any method. In one example, a known search engine may be used to search for information. In another example, the information search may be performed by a general method such as extracting information containing a specific word on a specific site without using a known search engine. A trained machine learning model such as a large language model (LLM) may be used to search for information.

[0069] (7) Seventh Case In one example, the related information 155 may be composed of coupon information with set usage conditions. The coupon information may be related to the inserted product TP, as in the above first and second examples. Alternatively, the coupon information may be unrelated to the inserted product TP. The usage conditions are conditions for receiving the discount indicated in the coupon information. The usage conditions may be set as appropriate. The usage conditions may include, for example, designation of the number of products to be purchased, the purchased product, the expiration date, etc. According to one example of this embodiment, by providing coupon information with set usage conditions, it is expected that the target user's purchasing motivation will be stimulated.

[0070] [Use of product input records] The product input record 20 may be used for any purpose, such as identifying the product TP, analyzing purchasing behavior in the store, etc. Information processing related to the product input record 20 may be performed in real time in response to the generation of the product input record 20, or may be performed after the product input record 20 is generated. Information processing related to the product input record 20 may be performed on the control device 11, or may be performed on an external computer (a store terminal, a user terminal, a server device, etc.).

[0071] 2 is a schematic diagram showing an example of a usage scenario of the product input record 20 according to this embodiment. In one example, the smart grip 1 (control device 11) may directly or indirectly provide the generated product input record 20 to the information processing device 2 via a network. Providing it indirectly means providing it via another computer. The network standard is not particularly limited and may be appropriately selected depending on the embodiment.

[0072] The information processing device 2 is one or more computers configured to execute information processing on the product input record 20. The information processing device 2 is an example of the above-mentioned external computer. For example, the information processing device 2 may be a store terminal provided in a store, a user terminal carried by a target user, a server device, etc. The information processing on the product input record 20 may be appropriately selected depending on the embodiment. For example, the information processing on the product input record 20 may include identification of the product TP, analysis of purchasing behavior (including generation of movement history), provision of information, etc. In one example of this embodiment, the information processing executed by the information processing device 2 may include at least any one of the following first to third cases.

[0073] The smart grip 1 is an example of a shopping device. In each of the following cases, for the sake of convenience, it is assumed that the information processing device 2 executes information processing on the product insertion record 20 generated by the smart grip 1. However, the product insertion record processed by the information processing device 2 is not limited to the product insertion record 20 generated by the smart grip 1. The product insertion record may be generated by a shopping device other than the smart grip 1 as long as it includes the event location. The other shopping device may be appropriately configured to be used during shopping and to generate a product insertion record including the event location. The information processing device 2 acquires the product insertion record generated by the shopping device other than the smart grip 1, and executes information processing on the acquired product insertion record. In addition, a part of the product input record group processed by the information processing device 2 may be collected from the smart grips 1 of one or more target users, and another part of the product input record group may be collected from other shopping devices.

[0074] (A) Case 1 FIG. 3 shows a schematic example of a method of using the product insertion record 20 according to this embodiment (product identification). In one example, the information processing device 2 may acquire the product insertion record 20 of the target user generated by the shopping device (smart grip 1). The product insertion record 20 may include the event position 201 where the target product TP1 was inserted. The target product TP1 corresponds to the product TP. The information processing device 2 may identify the inserted target product TP1 according to the acquired product insertion record 20. Then, the information processing device 2 may output information related to the result of identifying the target product TP1.

[0075] According to one example of the present embodiment, a technique can be provided for identifying an inserted product (target product TP1) by utilizing the product insertion record 20. Note that the method for identifying the target product TP1 according to the product insertion record 20 (event position 201) may be appropriately defined according to the embodiment. In one example of the present embodiment, at least one of the following three methods may be adopted as a method for identifying the target product TP1 from the product insertion record 20.

[0076] (A-1) First Identification Method 4 is a schematic diagram showing an example of a first method for identifying a product using the product input record 20 according to the present embodiment. In one example, identifying the target product TP1 according to the product input record 20 may consist of identifying the target product TP1 input at the event location 201 by matching the product input record 20 with the map information 30.

[0077] The map information 30 may be configured to include identification information of products in association with the display positions of the products in the store. In other words, the map information 30 may be configured to indicate the correspondence between the display positions of products in the store and the identification information of the products. The data format of the map information 30 may be appropriately selected depending on the embodiment. For example, the map information 30 may be configured with a plurality of records (such as a table-format database) that hold various information such as display positions and identification information for each product.

[0078] Furthermore, the method of matching the product input record 20 and the map information 30 may be defined arbitrarily. For example, the information processing device 2 may compare the display position of each product shown in the map information 30 with the event position 201 included in the product input record 20. Depending on the result of the comparison, the information processing device 2 may extract products displayed at a position matching the event position 201 from among the products included in the map information 30. The position matching the event position 201 may be, for example, the same position as the event position 201, or the position closest to the event position 201. The information processing device 2 may identify the target product TP1 by recognizing that the extracted product is the target product TP1.

[0079] According to one example of this embodiment, the target product TP1 can be appropriately identified by using the map information 30. There are cases where it is difficult to uniquely identify the target product TP1 only from the event position 201, such as when multiple types of products are mixed and arranged at the same position. In this case, the information processing device 2 may further use other information other than the event position 201 to identify the target product TP1. For example, the other information may include the weight of the product. As one example, the smart grip 1 may be configured to be able to measure the weight of the inserted target product TP1 by including a load sensor (weight sensor) in the detection sensor 13, or by further providing a load sensor in addition to the detection sensor 13. The amount of change (increase) in weight when the insertion of the target product TP1 is detected corresponds to the weight of the target product TP1. Product insertion record 2 0 may include the measurement result of the weight of the target product TP1. Furthermore, the map information 30 may further include information indicating the weight per unit of each product. When multiple types of products are displayed at the same position, multiple products are extracted in the search using the event position 201. The information processing device 2 may extract a product having a weight that matches the weight of the target product TP1 included in the product input record 20 from multiple products displayed at a position that matches the event position 201. Matching the weight of the target product TP1 may mean, for example, that the weight (increase) included in the product input record 20 is an integer multiple (including 1) of the weight of the product shown in the map information 30, or that the weight included in the product input record 20 is approximate to an integer multiple of the weight of the product shown in the map information 30. The information processing device 2 may identify the target product TP1 by recognizing that the extracted product is the target product TP1. This makes it possible to appropriately identify the target product TP1 even in cases where it is difficult to identify the target product TP1 using only the event position 201, such as cases where multiple types of products are displayed at the same position.

[0080] (A-2) Second identification method FIG. 5 is a schematic diagram showing an example of a second identification method of a commodity using the commodity input record 20 according to the present embodiment. In one example, identifying the target commodity TP1 according to the commodity input record 20 may be configured by acquiring a captured image 400 of the imaging device 40 including the event position 201 of the commodity input record 20 in its imaging range, and identifying the target commodity TP1 appearing in the captured image 400 by performing image analysis on the acquired captured image 400. The imaging device 40 may directly or indirectly provide the captured image 400 to the information processing device 2. The information processing device 2 may actively or passively acquire the captured image 400 from the imaging device 40. The second identification method is an example of a method for identifying the commodity TP in cooperation with sensors in a store.

[0081] Similar to imaging device 16 described above, imaging device 40 may be any sensor that captures data in the form of an image or a representation of an image, such as, for example, an RGB camera, a depth sensor, an infrared sensor, radar, LiDAR, etc. The imaging device 40 may include a sensor having higher performance than the imaging device 16, since the imaging device 40 has fewer physical constraints than the imaging device 16. The method of image analysis of the captured image 400 may be the same as the method of image analysis of the captured image 165. The captured image 400 used to identify the target product TP1 may be appropriately acquired from captured images generated at the time when the insertion of the target product TP1 is detected and at a time around that time.

[0082] One or more imaging devices 40 may be appropriately installed in the store. In the example of Fig. 5, at least three imaging devices 40 are installed in the store. The number of imaging devices 40 to be installed is not limited to the example of Fig. 5, and may be appropriately determined depending on the embodiment of the store, etc. The imaging device 40 that captures the captured image 400 may be appropriately identified from one or more imaging devices 40 installed in the store depending on the event position 201.

[0083] In the example of FIG. 5, the event position 201 belongs to the imaging range of one imaging device 40. However, when two or more imaging devices 40 are installed in a store, the imaging devices 40 may be arranged so that the imaging range includes the event position 201. When the imaging devices 40 are arranged so that the imaging range includes the event position 201, the information processing device 2 may acquire the captured image 400 from each of the imaging devices 40, and may perform image analysis on each of the acquired captured images 400. Alternatively, the information processing device 2 may select one or more imaging devices 40 from the imaging devices 40, acquire the captured image 400 from each of the selected one or more imaging devices 40, and perform image analysis on each of the acquired captured images 400. The method of selecting the imaging device 40 is not particularly limited and may be appropriately specified according to the embodiment. For example, a priority may be set in advance for each imaging device 40, and the information processing device 2 may select one or more imaging devices 40 in order of the priority. Also, for example, the information processing device 2 may select one or more imaging devices 40 based on any criteria, such as whether the event position 201 is close to the center of the imaging range.

[0084] Selecting an imaging device 40 may be replaced with selecting a captured image 400. For example, the information processing device 2 may acquire captured images 400 from each of the multiple imaging devices 40, and select one or more captured images 400 that satisfy a predetermined condition from among the acquired captured images 400. The predetermined condition may be appropriately defined so as to select an image suitable for identifying the target product TP1. For example, the predetermined condition may include criteria such as high clarity of the image of the target product TP1, a large image of the target product TP1, and the entire target product TP1 being captured. The information processing device 2 may perform image analysis on the selected one or more captured images 400.

[0085] If image analysis is performed on all captured images obtained by all imaging devices installed in a store, the amount of processing may become enormous. In contrast, according to this embodiment, the range of captured images (captured images 400) to be subjected to image analysis can be narrowed down to the range of event detection by the product insertion record 20. This can improve the efficiency of the identification process of the product (target product TP1) by image analysis. In this example, since the product insertion record 20 includes the identification information 203, it is possible to identify the target user who inserted the identified target product TP1 into the basket B1. This makes it possible to identify the target user corresponding to the obtained identification result without tracking the target user with sensors in the store.

[0086] (A-3) Third identification method FIG. 6 is a schematic diagram showing an example of a third identification method of a commodity using the commodity input record 20 according to the present embodiment. In one example, a load sensor 45 (weight sensor) may be installed on each commodity shelf PS in the store. Identifying the target commodity TP1 according to the commodity input record 20 may be configured by acquiring measurement data 450 of the load sensor 45 installed in the vicinity of the event position 201 of the commodity input record 20, and identifying the commodity shelf PS whose weight has decreased according to the acquired measurement data 450, thereby identifying the target commodity TP1. The load sensor 45 may be appropriately arranged to measure the weight of the commodity displayed on the commodity shelf PS. The load sensor 45 may directly or indirectly provide the measurement data 450 to the information processing device 2. The information processing device 2 may actively or passively acquire the measurement data 450 from the load sensor 45. As with the second identification method, the third identification method is also an example of a method for identifying a commodity TP in cooperation with sensors in the store.

[0087] The layout of the product shelves PS may be determined as appropriate depending on the embodiment. The product shelves PS may be arranged side by side in any direction, such as the horizontal direction or the vertical direction. Whether or not the load sensor 45 is present near the event position 201 may be determined depending on whether or not the load sensor 45 (product shelves PS) is within a range 2010 near the event position 201. The range 2010 near the event position 201 and the determination method may be defined as appropriate depending on the embodiment.

[0088] For example, the information processing device 2 may set a range 2010 based on the event position 201, and determine whether the position of the load sensor 45 (product shelf PS) belongs to the set range 2010. The position of the load sensor 45 may be identified from the load sensor 45 itself. Or, the position of the product shelf PS may be used as the position of the load sensor 45. That is, the presence of the load sensor 45 in the vicinity of the event position 201 may be constituted by the product shelf PS on which the load sensor 45 is installed being present in the vicinity of the event position 201. The information processing device 2 may appropriately refer to information indicating the position of the load sensor 45 (product shelf PS). The information indicating the position of the load sensor 45 may be held in the information processing device 2, or may be held in another computer other than the information processing device 2. In addition, the attribute values ​​(shape, size, etc.) of the range 2010 may be arbitrarily defined. The event position 201 may be disposed at an arbitrary position relative to the range 2010. In one example, the event position 201 may be used as the center of the range 2010. However, the arrangement of the event position 201 is not limited to this example. The event position 201 may be arranged at a position other than the center of the range 2010. The information processing device 2 may determine whether or not the load sensor 45 exists near the event position 201, depending on the result of determining whether or not the position of the load sensor 45 belongs to the range 2010. Specifically, the information processing device 2 may determine that the load sensor 45 exists near the event position 201, depending on the determination that the load sensor 45 belongs to the set range 2010.

[0089] Also, for example, the information processing device 2 may calculate the distance between the position of the load sensor 45 (product shelf PS) and the event position 201. The information processing device 2 may determine whether the calculated distance is less than a threshold value. The information processing device 2 may specify whether the load sensor 45 is present near the event position 201 according to the result of determining whether the calculated distance is less than the threshold value. Specifically, the information processing device 2 may specify that the load sensor 45 is present near the event position 201 according to the determination that the calculated distance is less than the threshold value. On the other hand, the information processing device 2 may specify that the target load sensor 45 is not present near the event position 201 according to the determination that the calculated distance exceeds the threshold value. When the calculated distance is equal to the threshold value, the information processing device 2 may specify that the load sensor 45 is either present or not present near the event position 201. The threshold value may be arbitrarily defined. The range specified by the threshold value is an example of the range 2010.

[0090] Depending on the positional relationship of each product shelf PS, multiple load sensors 45 (product shelves PS) may be present near the event position 201 (i.e., may belong within a range 2010 near the event position 201). The information processing device 2 may acquire measurement data 450 from each of the multiple load sensors 45 present near the event position 201. In the example of FIG. 6, three load sensors 45 belong near the event position 201. In this case, the information processing device 2 may acquire measurement data 450 from each of the three load sensors 45. The information processing device 2 may identify the target product TP1 by specifying the product shelf PS whose weight has decreased from among the multiple product shelves PS in the acquired measurement data 450.

[0091] The product corresponding to the product shelf PS may be appropriately identified. For example, the information processing device 2 may identify the product (target product TP1) corresponding to the product shelf PS by referring to information (such as the map information 30) indicating the correspondence between the product shelf PS (load sensor 45) and the products. The information indicating the correspondence between the product shelf PS and the products may be held in the information processing device 2, or may be held in a computer other than the information processing device 2.

[0092] When multiple types of products are displayed on one product shelf PS, the information processing device 2 may acquire the amount of change (amount of reduction) in weight in the measurement data 450 of the load sensor 45 as the weight of the target product TP1. The information processing device 2 may extract a product having a weight that matches the acquired weight from among multiple types of products displayed on the product shelf PS whose weight has been reduced. Comparing to the acquired weight may mean, for example, that the acquired weight (amount of reduction) is an integer multiple (including 1) of the weight of the product, or that the acquired weight is approximate to an integer multiple of the weight of the product. The information processing device 2 may identify the target product TP1 by recognizing that the extracted product is the target product TP1.

[0093] Furthermore, the target product TP1 is usually picked up by the target user from the product shelf PS and then put into the basket B1 (basket body B10). Therefore, the start of the weight change caused by taking the target product TP1 from the product shelf PS occurs before the time (time 205) when the detection sensor 13 detects the putting of the product TP. Meanwhile, depending on the configuration of the product shelf PS, the putting speed of the target product TP1, etc., the end time of the weight change caused by taking the target product TP1 may be before or after the time when the detection sensor 13 detects the putting of the product TP. Therefore, the information processing device 2 may acquire partial data of a predetermined section of the measurement data 450 accumulated by the measurement of the load sensor 45, the start time of which is a time before the time 205 of the product putting record 20. The predetermined section may be defined as appropriate depending on the embodiment. The information processing device The device 2 may identify the product shelf PS where the weight has decreased, according to the acquired partial data.

[0094] If all the load sensors installed in the store are constantly monitored, the amount of processing may become enormous. In contrast, according to the present embodiment, the range of the load sensors 45 to be monitored can be narrowed down to the range of event detection by the product insertion record 20. This can improve the efficiency of the identification process of the product (target product TP1) based on the weight change. Note that, even in this example, the product insertion record 20 includes the identification information 203, so that the target user who inserted the identified target product TP1 into the basket B1 can be identified. This makes it possible to identify the target user corresponding to the obtained identification result without tracking the target user with sensors in the store.

[0095] (A-4)Other In one example of this embodiment, the information processing device 2 may identify the target product TP1 by employing at least one of the above-mentioned first to third identification methods. The information processing device 2 may identify the target product TP1 by a plurality of methods. For example, the information processing device 2 may employ all of the above-mentioned first to third identification methods. In this case, the information processing device 2 may determine the final identification result of the target product TP1 by ensembling (for example, by majority voting) the identification results of each method.

[0096] The process of identifying the target product TP1 may be executed at any timing. For example, the information processing device 2 may sequentially identify the target product TP1 corresponding to the generated product input record 20 in response to the generation of the product input record 20. The information processing device 2 may identify the target product TP1 in real time when detecting the input of the target product TP1. Also, for example, the information processing device 2 may execute the identification process for a plurality of product input records 20 at once. In this case, the identification process for at least a portion of the target products TP1 may be executed after the fact.

[0097] Furthermore, the identification result of the target product TP1 may be used for any purpose, such as calculating the purchase amount of the target user, managing the number of products in stock, etc. In one example, the information processing device 2 may use the identification result of the target product TP1 to calculate the total amount of the products purchased by the target user. As in the above-mentioned example of the budget upper limit, the information processing device 2 may acquire information indicating the amount of the target product TP1 according to the identification result of the target product TP1. The information processing device 2 may calculate the total amount of all the inserted target products TP1 by adding up the acquired amount for each target product TP1. The calculated total amount is an example of information on the result of identifying the target product TP1. The calculated total amount may be used for the payment of the target user. For example, the information processing device 2 may be a store terminal or a server device. The information processing device 2 may execute the payment process of the target user based on the calculated total amount. Also, for example, the information processing device 2 may be independent of a store system including a register. The information processing device 2 may notify the store system of the calculated total amount, thereby causing the store system to execute the payment process of the target user. In addition, whether the target user has completed the purchasing behavior (i.e., put all products to be purchased into the shopping cart) may be specified by any method. For example, whether the target user's purchasing behavior has been completed may be determined depending on whether any end condition is satisfied, such as the target user reaching the cash register in the store, the cash register being operated, or the target user performing an operation to end the purchasing behavior on the shopping device (smart grip 1).

[0098] In another example, the information processing device 2 may use the identification result of the target product TP1 to manage the stock quantity of the target product TP1. For example, the information processing device 2 may reduce the stock quantity of the identified target product TP1 in response to identifying the target product TP1. The number to be reduced may be determined appropriately. In one example, the information processing device 2 may obtain the amount of change (increase) in weight of the basket B1 due to the addition of the target product TP1 via the product insertion record 20. The information processing device 2 may divide the obtained amount of change (increase) in weight by the weight per unit of the target product TP1. The number to be reduced may be determined by determining whether or not the stock quantity of the target product TP1 is less than a threshold value. Information indicating the stock quantity of each product may be stored in the information processing device 2, or may be stored in another computer accessible to the information processing device 2. After reducing the stock quantity of the target product TP1, the information processing device 2 may determine whether or not the stock quantity of the target product TP1 is less than a threshold value. The threshold value may be arbitrarily defined. If the stock quantity of the target product TP1 is less than the threshold value, the information processing device 2 may output an instruction to replenish the target product TP1. The instruction to replenish the target product TP1 is an example of information on the result of identifying the target product TP1. As an example of an instruction to replenish the target product TP1, the information processing device 2 may transmit a notification (message, etc.) to a terminal held by a store manager to prompt the replenishment of the target product TP1. In addition, if a robot device capable of automatically transporting the target product TP1 is installed in the store, the information processing device 2 may give a command to the robot device to replenish the target product TP1. This makes it possible to automate the replenishment of products.

[0099] (B) Second case FIG. 7 shows a schematic example of a method of using the product input record 20 according to this embodiment (generation of movement history). In one example, the information processing device 2 may acquire the product input record 20 of the target user generated by the shopping device (smart grip 1). The product input record 20 may include the event position 201 where the target product TP1 was inserted. The information processing device 2 may generate a movement history 50 of the target user from the event position 201 of the acquired product input record 20. The information processing device 2 may output the generated movement history 50.

[0100] As long as the movement history 50 includes the event position 201, the configuration of the movement history 50 is not particularly limited and may be appropriately determined depending on the embodiment. In one example, the movement history 50 may be configured by the event position 201 of the product insertion record 20 generated in the shopping device (smart grip 1) of the target user from the time of entering the store through the entrance ET to the time of payment at the register RS. In another example, the movement history 50 may further include a trajectory 500 within the store. The trajectory 500 may be appropriately configured to show the progress of the movement of the target user within the store.

[0101] The trajectory 500 may be generated by any method. In one example, the information processing device 2 may generate the trajectory 500 by connecting characteristic points of movement, such as connecting the entrance / exit ET and the first event position 201, connecting the kth event position 201 and the k+1th event position 201 (k is an integer equal to or greater than 1), or connecting the last event position 201 and the register RS. In another example, the shopping device (smart grip 1) may measure the position by the positioning sensor (positioning sensor 14) while the target user is present in the store, even at a time other than the time when the insertion of the product is detected. The information processing device 2 may further use the measurement result of this position to generate the trajectory 500. For example, the shopping device (smart grip 1) may continue to measure the position by the positioning sensor (positioning sensor 14) for a predetermined period after measuring the event position 201. The predetermined period may be appropriately defined according to the embodiment. The information processing device 2 may acquire the measurement results, and generate the trajectory 500 after identifying the progress of movement (movement direction, etc.) from each event position 201 based on the acquired measurement results.

[0102] In the example of FIG. 7, the store has one entrance / exit ET, eight product shelves PS, and three registers RS. It is assumed that the target user puts products into the basket at six locations, locations L1 to L6 (i.e., six product input records 20 are generated, and the event positions 201 of each product input record 20 indicate the locations L1 to L6). In one example, the information processing device 2 may generate a trajectory 500 by sequentially connecting the entrance / exit ET, the locations L1 to L6 indicated by each event position 201, and the register RS. The layout of the store is not limited to the example of FIG. 7, and may be appropriately changed depending on the embodiment. The store may have multiple floors, such as a basement and two or more floors. The store may have multiple entrances / exits ET. The entrance and exit may be provided separately.

[0103] According to one example of the present embodiment, a technique is provided for monitoring a movement history 50 of a target user using the product input record 20. The movement history 50 may be used for any purpose, such as analyzing the movement line of customers in a store. The generated movement history 50 may be output as is. In addition, the information processing device 2 may generate an analysis result such as a heat map by analyzing the movement history 50 of multiple users. The heat map may be appropriately generated so as to indicate the frequency of stay of customers (visit rate for each area, transition probability, etc.). The information processing device 2 may output the generated analysis result. This analysis result makes it possible to identify places in the store where customers pass frequently and places where they pass rarely. In one example, this analysis result may be reflected in determining the layout in the store (the arrangement, size, etc. of the product shelves PS). For example, the layout in the store may be determined to optimize customer passage according to the analysis result, such as widening the width of aisles where there is a high frequency of passage and reducing places where there is a low frequency of passage. The optimization of the layout may be performed manually or by computer information processing using a trained machine learning model, for example. The information processing related to the optimization of the layout may be executed by either the information processing device 2 or another computer. The information processing device 2 may output the result of the optimization of the layout.

[0104] (C) Third case FIG. 8 shows a schematic example of a method of using the product input record 20 according to this embodiment (locating the location). In one example, the information processing device 2 may acquire the product input record 20 of the target user generated by the shopping device (smart grip 1). The product input record 20 may include the event location 201 where the product (target product TP1) was inserted. The information processing device 2 may acquire the purchase record 35 of the product purchased by the target user. The information processing device 2 may identify the location where the target user put the purchased product into the basket (basket B1) by comparing the acquired product input record 20 and purchase record 35 with the map information 30. The information processing device 2 may output information related to the identified location.

[0105] As long as it includes identification information of the purchased item, the configuration of purchase record 35 is not particularly limited and may be determined appropriately depending on the embodiment. The data format of purchase record 35 may be selected appropriately depending on the embodiment. For example, purchase record 35 may be configured to hold various information such as identification information for each item. Purchase record 35 may be generated appropriately when payment is made for the purchased item. For example, purchase record 35 may be generated by a cash register. A POS (Point of Sale) cash register may be used as the cash register in the store. Purchase record 35 may be generated by a POS cash register. That is, the purchase record 35 may be composed of POS information. The POS information may be composed in a known manner.

[0106] When the information processing device 2 is a store terminal, the POS register may be included in the information processing device 2 or may be connected to the information processing device 2. The information processing device 2 may appropriately acquire the purchase record 35 from the POS register. When the information processing device 2 includes a POS register, the purchase record 35 may be generated by the information processing device 2 at the time of payment by the target user. Also, when the information processing device 2 is independent of a store system (such as a POS system) including a POS register, the store system may generate the purchase record 35 at the time of payment by the target user. The store system may provide the generated purchase record 35 to the information processing device 2 directly or indirectly.

[0107] The purchased item has already been identified by the identification information in the purchase record 35. When comparing the item insertion record 20 and the purchase record 35 with the map information 30, the information processing device 2 may compare the location on the map information 30 of the purchased item identified by the purchase record 35 with the event location 201. Depending on the comparison result, the information processing device 2 may specify the location where the target user put the purchased item into the cart.

[0108] In some cases, the same product may be placed in multiple locations (e.g., permanent shelves, special shelves, etc.) in a store for reasons such as strengthening sales. In response to this, according to one example of the present embodiment, a technology can be provided that uses the product input record 20 to identify the location where the target user put the purchased product into the basket. That is, after identifying the purchased product using the purchase record 35, the product input record 20 (event location 201) and the map information 30 can be used to identify the location where the purchased product was put into the basket. As a result, even when the same product is placed in multiple locations in a store, it is possible to analyze from which location each target user obtained the purchased product. This analysis makes it possible to measure the effect of each location in the store in encouraging purchases, such as the effect of special shelves in encouraging purchases.

[0109] (D) Other At least one of the information processing of identifying the product (first example), generating the movement history (second example), and specifying the location (third example) may be executed by a different computer. For example, a different computer may be used as the information processing device 2 for each information processing. Furthermore, at least two of the information processing of the first to third examples may be executed on the same computer. The computer that executes the information processing of each example may be selected as appropriate.

[0110] §2 Configuration Example [Hardware configuration] (Smart Grip) 1, the smart grip 1 includes a computer to which a control device 11, a memory unit 12, a detection sensor 13, a positioning sensor 14, an output device 15, an imaging device 16, and a communication module 17 are electrically connected. In the smart grip 1, at least a part of the computer may be housed in a box.

[0111] The control device 11 includes a processor P and is configured to execute information processing based on a program and various data. The processor P is a hardware processor and is an example of a processor resource. The type of the processor P may be appropriately selected depending on the embodiment. For example, the processor P may be a CPU (Central Processing Unit), a microcomputer, or the like. The microprocessor may be composed of a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), etc.

[0112] The storage unit 12 may be configured with a hard disk drive, a solid state drive, a semiconductor memory, or the like. The storage unit 12 is an example of a memory resource. In this embodiment, the storage unit 12 stores various information such as a program 81 and identification information 125. The program 81 is a program for causing the control device 11 to execute information processing (see FIG. 14 below) related to the generation of the commodity input record 20. The program 81 includes a series of instructions for the information processing. In one example, when the control device 11 executes information processing for identifying the commodity TP by image analysis of the captured image 165, the program 81 may further include a series of instructions for information processing (see FIG. 15 below) related to the identification of the commodity TP. In one example, when the control device 11 executes information processing for outputting the related information 155, the program 81 may further include a series of instructions for information processing (see FIG. 16 below) related to the output of the related information 155. The identification information 125 may be omitted.

[0113] At least one of the program 81 and the identification information 125 may be stored in a storage medium SR1 instead of or together with the storage unit 12. The storage medium SR1 is configured to accumulate various information (stored programs, etc.) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The storage unit 12 and the storage medium SR1 are examples of non-transitory storage media. The smart grip 1 (computer) stores at least one of the program 81 and the identification information 125 in the storage medium. The information may be acquired from the storage medium SR1. The storage medium SR1 may be a disk-type storage medium (CD, DVD, etc.), or may be a non-disk type storage medium such as a semiconductor memory (flash memory, etc.). Any drive device may be used to read the information stored in the storage medium SR1. The type of drive device may be selected according to the storage medium SR1.

[0114] The detection sensor 13 is configured to detect the insertion of a product TP into the basket B1. As long as the detection sensor 13 can detect the insertion of the product TP, the type of the detection sensor 13 is not particularly limited and may be appropriately selected depending on the embodiment. For example, the detection sensor 13 may be configured with one or more load sensors (weight sensors). In this case, the detection sensor 13 can detect the insertion of the product TP by detecting a change (increase) in weight due to the insertion of the product TP. The detection sensor 13 may be placed at any location where a change in load occurs due to the insertion of the product TP (i.e., where a change in weight due to the insertion of the product TP can be observed).

[0115] Also, for example, the detection sensor 13 may be configured with one or more imaging devices. The imaging device may be a device that captures an image or an image representation, such as an RGB camera, a depth sensor, an infrared sensor, a radar, or a LiDAR. The detection sensor 13 may include any sensor that acquires data by a sensor. In this case, the detection sensor 13 may be the same as the imaging device 16, or may be provided separately from the imaging device 16. The detection sensor 13 can detect the insertion of the product TP by capturing the product TP in an image captured by the imaging device. The detection sensor 13 may be disposed at any location where the product TP is within the imaging range at least during the period from when the user picks up the product TP to when the product TP is placed in the basket B1, and / or after the product TP is placed in the basket B1. When reducing the processing load and power consumption by narrowing the period during which the imaging device 16 is activated as described above, it is preferable to use a sensor of a different type from the imaging device 16, which has a lower processing load and power consumption, for the detection sensor 13.

[0116] Furthermore, for example, the detection sensor 13 may be composed of one or more contact sensors. In this case, the detection sensor 13 can detect the insertion of the product TP by contacting the detection sensor 13 with the inserted product TP or with another object (such as a switch member) displaced by the insertion of the product TP. The detection sensor 13 may be disposed at any location where the inserted product TP comes into direct or indirect contact. Indirect contact may include contact with another object displaced by the insertion of the product TP. The detection sensor 13 may be composed of multiple types of sensors.

[0117] The positioning sensor 14 is configured to measure a position. As long as the position can be measured, the type of the positioning sensor 14 is not particularly limited and may be appropriately selected depending on the embodiment. For example, the positioning sensor 14 may be configured with a GPS (Global Positioning System) sensor, a GNSS (Global Navigation Satellite System) sensor, or the like. In addition, for example, The positioning sensor 14 may be configured with a beacon receiver. In this case, a plurality of beacon transmitters may be installed in the store. The position of the smart grip 1 may be appropriately measured (estimated) from the strength of the beacon signal received by the positioning sensor 14 (beacon receiver). For example, the positioning sensor 14 may estimate the distance to each beacon transmitter according to the strength of the signal received by the beacon receiver. The positioning sensor 14 may measure the position of the smart grip 1 from the estimated distance to each beacon transmitter by triangulation. At least a part of the process of measuring this position may be executed by the control device 11.

[0118] The communication module 17 is configured to execute wired or wireless communication via a network. The communication module 17 may be configured, for example, by a wired LAN (Local Area Network) module, a wireless LAN module, or the like. The communication module 17 may include a close-proximity wireless communication module. The network standard is not particularly limited and may be appropriately selected depending on the embodiment. For example, the type of network may be appropriately selected from the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, or the like. The smart grip 1 (control The control device 11) can exchange data with other computers (such as the information processing device 2) via the communication module 17.

[0119] In addition, regarding the specific hardware configuration of the smart grip 1 (computer), components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control device 11 may include multiple hardware processors (processor P). At least one of the output device 15, the imaging device 16, and the communication module 17 may be omitted. At least one of the program 81 and the identification information 125 may be stored in an external storage device such as a NAS. The external storage device is also an example of a non-transitory storage medium. The smart grip 1 may be configured to be operable by further including an input device. The input device may be configured to accept input of information. The input device may be configured, for example, by a keyboard, a touch panel, an operator, a microphone, etc. The input device and the output device 15 may be integrally configured by a touch panel display, etc. The output device 15 is an example of a feedback module. The feedback module may be configured to feed back information to the user by at least one of vision, hearing, and touch. The smart grip 1 may further include a feedback module other than the output device 15. For example, the output device 15 may further include a haptic feedback module configured to feed back information tactilely. The haptic feedback module may be configured of any device capable of transmitting information tactilely. For example, the haptic feedback module may be configured of a servo motor, a vibration motor, or a combination thereof. When the haptic feedback module is included, the smart grip 1 may output any information, such as a notification of the output of the related information 155, by haptic feedback. The smart grip 1 may include multiple computers. The hardware configurations of the computers may or may not be the same. Furthermore, the computer included in the smart grip 1 may be a dedicated computer, as well as a general-purpose PC (Personal Computer), a tablet PC, a terminal device ( For example, the computer included in the smart grip 1 may be a smartphone. The smart grip 1 may be constructed by mounting a smartphone on a housing (attachment).

[0120] (Smart Grip Structure) 9 and 10 are a side view and a plan view showing an example of the structure of the smart grip 1 according to this embodiment. In one example, the smart grip 1 may further include a housing 90 that houses the detection sensor 13, the positioning sensor 14, and the control device 11. The housing 90 may include one or more grooves (groove 915, groove 935, etc.) for hooking the handle B11 provided on the basket B1. According to this example of the present embodiment, it is possible to provide a smart grip 1 that can be used as a handle for a basket by replacing the handle (handle B11) of an existing basket.

[0121] The structure of the smart grip 1, such as the shape of the housing 90, the size of the housing 90, the arrangement of each component (such as the control device 11) in the housing 90, the number of grooves, the shape of the grooves, and the arrangement of the grooves, may be appropriately determined depending on the embodiment. In the example of Fig. 9 and Fig. 10, a situation is assumed in which the detection sensor 13 is composed of a pair of load sensors 130, the output device 15 includes a display 151, and the imaging device 16 is provided separately from the detection sensor 13. A situation is assumed in which the housing 90 has a pair of grooves (915, 935). In addition, a situation is assumed in which the smart grip 1 has a vibration motor 180 and a servo motor 185 as a haptic feedback module.

[0122] For ease of explanation, the left-right direction in Figs. 9 and 10 will be referred to as the "front-rear direction." The left direction in Figs. 9 and 10 will be referred to as the "front direction," and the right direction in Figs. 9 and 10 will be referred to as the "rear direction." The direction perpendicular to the paper surface of Fig. 9 and the up-down direction in Fig. 10 will be referred to as the "left-right direction." The up direction in Fig. 10 will be referred to as the "right direction," and the down direction in Fig. 10 will be referred to as the "left direction." The up-down direction in Fig. 9 10 and the direction perpendicular to the paper surface of Fig. 10 are referred to as the "upper-lower direction". The upper direction in Fig. 9 is referred to as the "upper direction", and the lower direction in Fig. 9 is referred to as the "lower direction".

[0123] In one example, the housing 90 may include a first portion 91, a second portion 92, and a third portion 93. In one example of each figure, the first portion 91, the second portion 92, and the third portion 93 are arranged in a line in the front-rear direction from the rear. Each portion (91, 92, 93) is formed in a rectangular parallelepiped shape. The first portion 91 and the third portion 93 may have substantially the same shape. The width (length in the left-right direction) of the second portion 92 may be formed shorter than the first portion 91 and the third portion 93. As a result, the second portion 92 may constitute a portion to be held by the user. In one example of each figure, the second portion 92 extends along the front-rear direction. Therefore, the user can hold the smart grip 1 by holding the second portion 92 with his / her hand along the front-rear direction. The total length of the circumference of the second portion 92 around the axis in the front-rear direction may be determined within a range of a length that can be held by hand.

[0124] Side walls may be provided at the ends of each part (91, 92, 93) in the horizontal direction (left-right direction and front-back direction). A bottom surface may be provided at the lower end of each part (91, 92, 93). As a result, an internal space may be formed in each part (91, 92, 93) that is surrounded by each side wall and that houses the computer part of the smart grip 1. The internal spaces of the first part 91 and the second part 92 may be at least partially continuous. The internal spaces of the second part 92 and the third part 93 may also be at least partially continuous. The upper end of each part (91, 92, 93) may be closed by a lid or the like, or may be open.

[0125] A groove 915 may be provided on the rear end side of the first portion 91. In one example of each figure, the groove 915 is formed by recesses provided on side walls opposing each other in the left-right direction of the first portion 91. Each recess is formed in a rectangular shape. The recesses are aligned in the front-rear direction to form the groove 915. Even when the upper end of the first portion 91 is closed, the portion of the groove 915 on the upper end surface of the first portion 91 may be open.

[0126] Similarly, a groove 935 may be provided on the front end side of the third portion 93. In one example of each figure, the groove 935 is formed by recesses provided on side walls opposing each other in the left-right direction of the third portion 93. Each recess is formed in a rectangular shape. The recesses are aligned in the front-rear direction to form the groove 935. Even when the upper end of the third portion 93 is closed, the portion of the groove 935 on the upper end surface of the third portion 93 may be open. The width (length in the front-rear direction) of each groove (915, 935) may be formed to be the same as or slightly larger than the width of the corresponding handle portion B11 of the basket B1. In this way, each groove (915, 935) may be configured to be able to receive each handle portion B11 of the basket B1.

[0127] The pair of load sensors 130 may be arranged in the internal space of each of the first and third parts 91 and 93 in accordance with the positions of the pair of grooves (915, 935). When the portion of the handle part B11 that is hooked onto the smart grip 1 is formed in a straight line, in one example, the recesses constituting each groove (915, 935) may be formed so that the lower end of the recess is located slightly lower than the measurement part (contact surface with the handle part B11) of the load sensor 130. In this way, when each handle part B11 is hooked onto each groove (915, 935), each handle part B11 comes into contact with the measurement part of each load sensor 130. As a result, each load sensor 130 can observe the change in weight when the product TP is put into the basket B1 (basket body B10). The location of the load sensor 130 is an example of a location where a change in load occurs when the product TP is put into the basket B1. However, the positional relationship between each groove (915, 935) and each load sensor 130 is not limited to this example, and may be changed as appropriate depending on the embodiment, such as the shape of the handle part B11. For example, each handle part B11 may be formed so that the part located in the internal space of each part (91, 93) when hung on the smart grip 1 is curved downward. In this case, the recesses constituting each groove (915, 935) are positioned so that the lower end of the recess is positioned above the measuring part of the load sensor 130. The sensor may be configured to be placed in a position such that the sensor is fixed to the sensor.

[0128] In one example of each figure, the imaging device 16 is disposed in the internal space of the first part 91. For example, an opening may be provided on the bottom surface of the first part 91 in accordance with the position of the imaging section 161 (light receiving section, etc.) of the imaging device 16. The imaging device 16 may be disposed so that the imaging section 161 is exposed from the bottom surface of the first part 91 by fitting the imaging section 161 into the opening on the bottom surface. This allows the imaging device 16 to capture an image of the area below the smart grip 1. Therefore, when the smart grip 1 is attached to the basket B1 by hooking each of the handle parts B11 of the basket B1 into each of the grooves (915, 935), the imaging device 16 can include the storage section of the basket body B10 in the imaging range. As a result, when the product TP is inserted into the basket body B10, the imaging device 16 can observe the inserted product TP. The location where the imaging device 16 is disposed is an example of a location where the inserted product TP can be observed.

[0129] In one example of each figure, the vibration motor 180 and the servo motor 185 are disposed in the internal space of the second part 92. In this example, the second part 92 constitutes the part that is gripped by the user. Therefore, by disposing the vibration motor 180 and the servo motor 185 in the second part 92, it is possible to provide tactile feedback to the user using the smart grip 1. The types of the vibration motor 180 and the servo motor 185 are not particularly limited and may be appropriately selected depending on the embodiment.

[0130] The vibration motor 180 is configured to provide tactile feedback to the user by vibrating. That is, the vibration motor 180 can provide feedback to the user by vibration. Feedback by vibration may be used arbitrarily. For example, when the smart grip 1 outputs the related information 155 to the display 151, the smart grip 1 may notify the user that the related information 155 has been output by vibrating the vibration motor 180.

[0131] The servo motor 185 may include a shaft 1851 and an arm 1852. The shaft 1851 constitutes a rotation axis. The arm 1852 is attached to the shaft 1851 and rotates around the axis of the shaft 1851 in response to the rotation of the shaft 1851. The arm 1852 may be configured so that a part (for example, a tip) of the arm 1852 comes into contact with the hand of the user holding the second part 92 by rotating clockwise or counterclockwise. Alternatively, the arm 1852 may be configured so that the tip of the arm 1852 hits the side wall of the second part 92 by rotating clockwise or counterclockwise, and the vibration generated by this is transmitted to the hand of the user holding the second part 92. By having such a configuration, the servo motor 185 can return feedback in a specific direction. In one example of each figure, the arm 1852 can rotate clockwise to provide feedback to the left side of the hand of the user holding the second part 92. On the other hand, by rotating the arm 1852 counterclockwise, feedback can be given to the right side of the user's hand gripping the second portion 92. This feedback indicating a specific direction may be used arbitrarily. For example, when the information output to the display 151 includes information regarding a direction (for example, the location of a product, etc.), the smart grip 1 may indicate the direction of the target to the user by outputting feedback indicating the target direction by the servo motor 185. In one example of each figure, the arm 1852 is formed in a flat plate shape. However, the shape of the arm 1852 is not limited to such an example. As long as it is possible to return tactile feedback in a specific direction by rotating, the shape of the arm 1852 is not particularly limited and may be appropriately selected according to the embodiment.

[0132] In addition, in one example of each figure, the control device 11, the storage unit 12, the positioning sensor 14, and the communication module 17 are housed in a box 1100. The box 1100 is formed in a rectangular parallelepiped shape, and is disposed in the internal space of the third portion 93. A display 151 is disposed on the upper surface of the box 1100. The third portion 93 is positioned in front of the second portion 92. When the top end of the third portion 93 is closed, the display 151 may be positioned such that the screen is exposed from the top end surface of the third portion 93. In this example, when the user holds the smart grip 1 properly, the third portion 93 is located in front of the second portion 92. Therefore, while using the smart grip 1, the screen of the display 151 is positioned slightly in front of the user's gripping hand. This allows the user to easily view the screen of the display 151 without being obstructed by body parts while using the smart grip 1.

[0133] The structure of the smart grip 1 is not limited to the examples shown in FIGS. 9 and 10, and may be modified as appropriate depending on the embodiment. For example, the shape of the housing 90 may be modified as appropriate. In another example, the housing of the smart grip 1 may be formed in a ring shape, and a part of the ring may be opened and closed by attaching and detaching the ends. As a result, the housing of the smart grip 1 may be configured to replace the handle by receiving the handle part B11 of the basket B1 in the ring formed by the housing. In yet another example, the housing of the smart grip 1 may be formed to receive a terminal device such as a smartphone. In this case, the smart grip 1 may be constructed by attaching the terminal device to the housing. In addition, the position of each groove (915, 935) may be changed as appropriate. The number of grooves provided in the housing (housing 90) is not limited to two, and may be one, or may be three or more. In addition, the arrangement of each component may be changed as appropriate. The arrangement of the load sensor 130 may be changed as appropriate depending on the structure of the housing (housing 90). When a sensor other than the load sensor 130 is used as the detection sensor 13, the detection sensor 13 may be appropriately placed in a location where it can detect the insertion of the product TP. The location of the imaging device 16 may be changed appropriately depending on the embodiment. The imaging device 16 may be placed in a location other than the internal space of the first part 91. The components housed in the box 1100 may be changed appropriately. The box 1100 may be placed in a location other than the internal space of the third part 93. The box 1100 may be omitted.

[0134] (Information processing device) 11 is a schematic diagram showing an example of a hardware configuration of the information processing device 2 according to the present embodiment. The information processing device 2 according to the present embodiment is a computer in which a control unit 21, a storage unit 22, a communication module 23, an input device 24, an output device 25, and a drive 26 are electrically connected.

[0135] The control unit 21 is configured to execute information processing based on programs and various data by including a CPU, which is a hardware processor, a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The control unit 21 (CPU) is an example of a processor resource.

[0136] The storage unit 22 may be configured with a hard disk drive, a solid state drive, a semiconductor memory, or the like. The storage unit 22, the RAM, and the ROM are examples of memory resources. In this embodiment, the storage unit 22 stores various information such as a program 82. The program 82 is a program for causing the information processing device 2 to execute information processing on the product input record 20. The program 82 includes a series of instructions for the information processing. In one example, when causing the information processing device 2 to execute a product identification process, the program 82 may include a series of instructions for information processing on product identification (FIG. 17 described below). In one example, when causing the information processing device 2 to execute a movement history generation process, the program 82 may include a series of instructions for information processing on the generation of the movement history (FIG. 18 described below). In one example, when causing the information processing device 2 to execute a location identification process, the program 82 may include a series of instructions for information processing on location identification (FIG. 19 described below).

[0137] The communication module 23 is configured to perform wired or wireless communication over a network. The communication module 23 may be configured, for example, by a wired LAN module, a wireless LAN module, etc. The information processing device 2 can exchange data with another computer (for example, the computer portion of the smart grip 1, etc.) via the communication module 23.

[0138] The input device 24 is configured to receive input of information. The input device 24 may be configured with, for example, a mouse, a keyboard, an operator, etc. The output device 25 is configured to output information. The output device 25 may be configured with, for example, a display, a speaker, etc. An operator can operate the information processing device 2 by using the input device 24 and the output device 25. The input device 24 and the output device 25 may be connected via an external interface. The external interface may be appropriately configured to connect to an external device by wire or wirelessly, for example, via a Universal Serial Bus (USB) port, a dedicated port, etc. The input device 24 and the output device 25 may be integrally configured with a touch panel display, etc.

[0139] The drive 26 is a device for reading various information such as programs stored in the storage medium SR2. The program 82 may be stored in the storage medium SR2 instead of or together with the storage unit 22. The information processing device 2 may acquire the program 82 from the storage medium SR2. The storage medium SR2 may be configured similarly to the storage medium SR1. The storage unit 22 and the storage medium SR2 are examples of non-transitory storage media. The type of the drive 26 may be selected according to the storage medium SR2. The drive 26 may be connected via an external interface.

[0140] Regarding the specific hardware configuration of the information processing device 2, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, DSP, GPU, ASIC, etc. At least one of the communication module 23, the input device 24, the output device 25, and the drive 26 may be omitted. The program 82 may be stored in an external storage device such as a NAS. The external storage device is also an example of a non-transitory storage medium. The information processing device 2 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. In addition, the information processing device 2 may be a general-purpose server device, a general-purpose PC, a tablet PC, a terminal device, etc., in addition to a computer designed for a special purpose.

[0141] [Software configuration] (Smart Grip) 12 shows an example of a software configuration of the smart grip 1 (control device 11) according to this embodiment. The control device 11 of the smart grip 1 executes instructions included in a program 81 stored in the storage unit 12 by the processor P. As a result, the smart grip 1 (control device 11) operates as a computer including a detection unit 111, a generation unit 112, and a storage processing unit 113 as software modules. That is, in this embodiment, each software module of the smart grip 1 is realized by the control device 11 (processor P).

[0142] The detection unit 111 is configured to detect the insertion of the product TP according to the measurement result of the detection sensor 13. The generation unit 112 is configured to generate a product insertion record 20 including the event position 201 where the product TP was inserted, which is the event position 201 measured by the positioning sensor 14, when the insertion of the product TP is detected by the detection sensor 13. The storage processing unit 113 is configured to store the generated product insertion record 20.

[0143] When the control device 11 is configured to execute output processing of the related information 155, the smart grip 1 (control device 11) may further include an output processing unit 114 as a software module. The output processing unit 114 may be configured to output related information 155 related to the insertion of the product TP from the output device 15 when the insertion of the product TP is detected by the detection sensor 13. When the output of the related information 155 is not executed, the output processing unit 114 may be omitted from the software configuration of the smart grip 1.

[0144] When the control device 11 is configured to execute the identification process of the commodity TP by image analysis, the smart grip 1 (control device 11) may further include an image acquisition unit 115, an identification unit 116, and a result output unit 117 as software modules. The image acquisition unit 115 is configured to acquire a captured image 165 of the imaging device 16 when the detection sensor 13 detects the insertion of the commodity TP. The identification unit 116 is configured to identify the commodity TP appearing in the captured image 165 by image analysis of the acquired captured image 165. The result output unit 117 is configured to output information related to the identification result of the commodity TP. When the identification process of the commodity TP is not executed on the smart grip 1, the image acquisition unit 115, the identification unit 116, and the result output unit 117 may be omitted from the software configuration of the smart grip 1.

[0145] (Information processing device) 13 shows an example of a software configuration of the information processing device 2 (controller 21) according to this embodiment. The controller 21 of the information processing device 2 executes instructions included in the program 82 stored in the storage unit 22 by means of the CPU. As a result, the information processing device 2 (controller 21) operates as a computer including an acquisition unit 211, an information processing unit 212, and an output processing unit 213 as software modules. That is, in this embodiment, each software module of the information processing device 2 is realized by the controller 21 (CPU).

[0146] The acquisition unit 211 is configured to acquire the product insertion record 20 of the target user generated by the shopping device (smart grip 1). The product insertion record 20 may include the event position 201 where the target product TP1 was inserted. The information processing unit 212 is configured to execute information processing on the product insertion record 20. The output processing unit 213 is configured to output information on the execution result of the information processing on the product insertion record 20.

[0147] When the control unit 21 is configured to execute a product identification process, the information processing unit 212 may include a recognition unit 2121. The recognition unit 2121 is configured to recognize the inserted target product TP1 according to the acquired product insertion record 20. Any of the above-mentioned first to third recognition methods may be adopted as the recognition method. In response to this, the output processing unit 213 may be configured to output information related to the result of identifying the target product TP1.

[0148] When the control unit 21 is configured to execute a process of generating a movement history, the information processing unit 212 may include a history generating unit 2122. The history generating unit 2122 is configured to generate a movement history 50 of the target user from the event position 201 of the acquired product input record 20. In response to this, the output processing unit 213 may be configured to output the generated movement history 50.

[0149] When the control unit 21 is configured to execute location characteristic processing, the acquisition unit 211 may be further configured to acquire a purchase record 35 of the purchased item of the target user. The information processing unit 212 may include a location identification unit 2123. The location identification unit 2123 is configured to identify the location where the target user put the purchased item into the basket (basket B1) by matching the acquired product input record 20 and purchase record 35 with the map information 30. In response to this, the output processing unit 213 may be configured to output information related to the identified location. In the information processing unit 212, at least the identification unit 2121, the history generation unit 2122, and the location identification unit 2123 Either one may be omitted.

[0150] (others) In this embodiment, an example is described in which each software module of the smart grip 1 and the information processing device 2 is realized by a general-purpose processor (processor P, CPU). However, some or all of the above software modules may be realized by one or more dedicated processors or chipsets. Each of the above modules may be realized as a hardware module. With regard to the software configuration of the smart grip 1 and the information processing device 2, modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0151] §3 Example of operation [Smart Grip] (Generating product input records) FIG. 14 is a flowchart showing an example of a processing procedure for generating the product input record 20 by the control device 11 of the smart grip 1 according to this embodiment. The processing procedure for generating the product input record 20 described below is an example of a generation method (information processing method) executed by a computer (control device 11). However, the processing procedure for generating the product input record 20 described below is merely an example, and each step may be modified as much as possible. In addition, steps may be omitted, replaced, or added to the processing procedure for generating the product input record 20 described below as appropriate depending on the embodiment.

[0152] In step S101, the control device 11 operates as the detection unit 111. The control device 11 acquires a measurement result from the detection sensor 13. The control device 11 detects the insertion of the product TP according to the measurement result of the detection sensor 13. When the insertion of the product TP is detected, the control device 11 proceeds to the next step S102. When the insertion of the product TP is not detected, the control device 11 repeatedly executes the process of step S101 and waits until the insertion of the product TP is detected.

[0153] In step S102, the control device 11 operates as the generation unit 112. The control device 11 acquires the event position 201 where the product TP was inserted by referring to the measurement result of the positioning sensor 14. The control device 11 generates a product insertion record 20 including the acquired event position 201. After generating the product insertion record 20, the control device 11 advances the process to the next step S103.

[0154] In step S103, the control device 11 operates as the storage processing unit 113. The control device 11 stores the generated product input record 20. In one example, the control device 11 may store the generated product input record 20 in the memory unit 12. In another example, the control device 11 may store the product input record 20 in an external memory area by transmitting the product input record 20 to an external computer such as the information processing device 2. After storing the product input record 20, the control device 11 proceeds to the next step S104.

[0155] In step S104, the control device 11 judges whether or not to end the process. The judgment criteria may be set arbitrarily. In one example, the control device 11 may judge not to end the process until an end instruction is given. On the other hand, when an end instruction is given, the control device 11 may judge to end the process. The end instruction may be given by any method. In another example, if the target user is present in the store, the control device 11 may judge not to end the process. On the other hand, if the target user is not present in the store (has left the store), the control device 11 may judge to end the process. Whether or not the target user is present in the store may be judged appropriately according to the measurement result of the positioning sensor 14, etc.

[0156] If it is determined not to end the process, the control device 11 returns the process to step S101 and executes the process again from step S101. If it is determined to end the process, the control device 11 ends the process procedure related to the generation of the product input record 20 according to this operation example. Note that the timing of ending the process is not limited to this example. The control device 11 may end the process procedure related to the generation of the product input record 20 at any timing.

[0157] In one example, the control device 11 may execute a series of processes from step S101 to step S104 in real time. The order of the processes in each step may be changed as appropriate. In another example, the control device 11 may execute the processes in step S101 and step S102 multiple times, and then execute the process in step S103 all at once.

[0158] According to this embodiment, the detection sensor 13 and the positioning sensor 14 are used to generate the product input record 20 by the process of step S102, so that the purchasing behavior of the user can be monitored without installing sensors in the store. This is expected to simplify the monitoring of purchasing behavior. The smart grip 1 according to this embodiment may be used to monitor the purchasing behavior of the user together with sensors installed in the store. In this case, as in the second identification method, the third identification method, and the like, the product input record 20 is used to improve the efficiency of processing the sensing data obtained by the sensors installed in the store. This is expected to simplify the monitoring of purchasing behavior.

[0159] (Product Identification) FIG. 15 is a flowchart showing an example of a processing procedure for identifying the product TP by the control device 11 of the smart grip 1 according to this embodiment. The processing procedure for identifying the product TP described below is an example of an identification method (information processing method) executed by a computer (control device 11). However, the processing procedure for identifying the product TP described below is merely an example, and each step may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added to the processing procedure for identifying the product TP described below as appropriate depending on the embodiment.

[0160] In step S201, the control device 11 operates as the detection unit 111. The process of step S201 may be the same as that of step S101. If the control device 11 detects the insertion of the product TP, the control device 11 proceeds to the next step S202. If the insertion of the product TP is not detected, the control device 11 repeatedly executes the process of step S201 and waits until the insertion of the product TP is detected.

[0161] In step S202, the control device 11 operates as the image acquisition unit 115. That is, the control device 11 acquires the captured image 165 from the imaging device 16. In step S203, the control device 11 operates as the identification unit 116. That is, the control device 11 identifies the product TP appearing in the captured image 165 by performing image analysis on the acquired captured image 165. In step S204, the control device 11 operates as the result output unit 117. That is, the control device 11 outputs information related to the identification result of the product TP. The output destination and content of the information may be appropriately selected according to the embodiment. After outputting the information related to the identification result, the control device 11 proceeds to the next step S205.

[0162] In step S205, the control device 11 judges whether or not to end the process. The process in step S205 may be the same as step S104 described above. If it is judged not to end the process, the control device 11 returns the process to step S201 and executes the process again from step S201. If it is judged to end the process, the control device 11 ends the processing procedure related to the identification of the product TP according to this operation example. Note that the timing of ending the process is not limited to this example. The control device 11 may execute the processing related to the identification of the product TP at any timing. The processing procedure may then be terminated.

[0163] According to one example of the present embodiment, the inserted product TP can be identified on the smart grip 1 by the processes of steps S202 and S203. In one example, the control device 11 may execute a series of processes from steps S201 to S205 in real time. The order of the processes of each step may be changed as appropriate. In another example, the control device 11 may execute the processes of steps S201 to S203 multiple times and then execute the process of step S204 all at once. In addition, when the control device 11 executes the processes of both steps S101 to S104 and steps S201 to S205, the process of step S201 may be executed in common with the above step S101. The process of step S205 may be executed in common with the above step S104. The processes of steps S202 to S204 may be executed at any timing after step S101 and before step S104.

[0164] (Related information output) FIG. 16 is a flowchart showing an example of a processing procedure for outputting the related information 155 by the control device 11 of the smart grip 1 according to this embodiment. The processing procedure for outputting the related information 155 described below is an example of an output method (information processing method) executed by a computer (control device 11). However, the processing procedure for outputting the related information 155 described below is merely an example, and each step may be changed as much as possible. Furthermore, steps may be omitted, replaced, or added to the processing procedure for outputting the related information 155 described below as appropriate depending on the embodiment.

[0165] In step S301, the control device 11 operates as the detection unit 111. The process of step S301 may be the same as that of step S101. If the control device 11 detects the insertion of the product TP, the control device 11 proceeds to the next step S302. If the insertion of the product TP is not detected, the control device 11 repeatedly executes the process of step S301 and waits until the insertion of the product TP is detected.

[0166] In step S302, the control device 11 operates as the output processing unit 114. That is, the control device 11 outputs related information 155 related to the insertion of the product TP from the output device 15. In one example, the related information 155 may be configured from at least one of the above first to seventh cases. For example, when the first case is adopted, the related information 155 may include at least one of advertisement information 1550 of other products arranged in the vicinity of the event position 201 and coupon information 1555. After outputting the related information 155, the control device 11 advances the process to the next step S303.

[0167] In step S303, the control device 11 judges whether or not to end the process. The process of step S303 may be the same as step S104. If it is judged not to end the process, the control device 11 returns the process to step S301 and executes the process again from step S301. If it is judged to end the process, the control device 11 ends the process procedure related to the output of the related information 155 according to this operation example. Note that the timing of ending the process is not limited to this example. The control device 11 may end the process procedure related to the output of the related information 155 at any timing.

[0168] According to one example of this embodiment, by providing related information 155 when the product TP is inserted by the process of step S302, it is expected that purchasing activity will be stimulated, such as by stimulating further purchasing desire for the product. In one example, the control device 11 may execute a series of processes from step S301 to step S303 in real time. The processing order of each step may be changed as appropriate. In another example, the control device 11 may execute a predetermined It may be determined whether the above condition is satisfied. The predetermined condition may be arbitrarily defined. For example, the predetermined condition may include that the output of the related information 155 is permitted. Whether or not the output of the related information 155 is permitted may be set appropriately. The control device 11 may execute the process of step S302 only when the predetermined condition is satisfied. In addition, when the control device 11 executes both the processes of steps S101 to S104 and steps S301 to S303, the process of step S301 may be executed in common with the above step S101. The process of step S303 may be executed in common with the above step S104. The process of step S302 may be executed at any timing after step S101 and before step S104.

[0169] [Information processing device] (Product Identification) FIG. 17 is a flowchart showing an example of a processing procedure for product identification using the product input record 20 by the information processing device 2 according to this embodiment. The following processing procedure for product identification is an example of an identification method (information processing method) executed by a computer. However, the following processing procedure for product identification is merely an example, and each step may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added to the following processing procedure for product identification as appropriate depending on the embodiment.

[0170] In step S501, the control unit 21 operates as the acquisition unit 211. That is, the control unit 21 acquires the product input record 20 of the target user generated by the shopping device. In one example, the control unit 21 may acquire the product input record 20 from the smart grip 1. In step S502, the control unit 21 operates as the identification unit 2121. That is, the control unit 21 identifies the input target product TP1 according to the acquired product input record 20. In one example, the control unit 21 may identify the target product TP1 by at least one of the above-mentioned first identification method to third identification method. For example, when the first identification method is adopted, the control unit 21 may identify the target product TP1 input at the event position 201 by collating the product input record 20 with the map information 30. Also, for example, when the second identification method is adopted, the control unit 21 may acquire the captured image 400 of the imaging device 40 including the event position 201 of the product input record 20 in the imaging range. The control unit 21 may identify the target product TP1 appearing in the captured image 400 by performing image analysis on the captured image 400. In addition, for example, when the third identification method is adopted, the control unit 21 may acquire measurement data 450 of a load sensor 45 installed near the event position 201 of the product input record 20. The control unit 21 may identify the target product TP1 by identifying the product shelf PS with a reduced weight according to the acquired measurement data 450. In one example, the control unit 21 may determine the final identification result of the target product TP1 by ensembling the identification results obtained by a plurality of methods. In step S503, the control unit 21 outputs information related to the result of identifying the target product TP1.

[0171] The output destination and contents of the information may be appropriately selected according to the embodiment. In one example, the control unit 21 may directly output the identification result of the target product TP1. In another example, the control unit 21 may execute any information processing (such as the calculation of the purchase amount and management of the number of stocks) according to the obtained identification result. The control unit 21 may output the result of executing the information processing as information related to the identification result. The output of the result of executing this information processing may include, for example, outputting specific information according to the identification result, controlling the operation of the controlled device according to the identification result, and the like. Outputting specific information may include notifying the completion of the payment process, notifying the store system of the calculated total amount, notifying a message encouraging the refilling of the target product TP1, and the like. Controlling the operation of the controlled device may include giving a command to the robot device to refill the target product TP1, and the like. The output destination may be, for example, a RAM, a memory unit 22, an output device 25, another computer (including the smart grip 1), a controlled device, and the like. When the output of the information is completed, the control unit 21 returns to the present operation example. The process for identifying the product is then completed.

[0172] According to one example of the present embodiment, the input product (target product TP1) can be identified by using the product input record 20 through the processes of steps S501 and S502. The control unit 21 may execute a series of processes from steps S501 to S503 at any timing. In one example, the control unit 21 may execute a series of processes from steps S501 to S503 in real time in response to the generation of the product input record 20. In another example, the control unit 21 may acquire a plurality of product input records 20 in the process of step S501. The control unit 21 may execute the process of step S502 for each acquired product input record 20. In this case, the identification process (process of step S502) of at least a part of the target products TP1 may be executed ex post.

[0173] (Generating movement history) FIG. 18 is a flowchart showing an example of a processing procedure for generating a movement history 50 using the product input record 20 by the information processing device 2 according to this embodiment. The processing procedure for generating the movement history 50 described below is an example of a generation method (information processing method) executed by a computer. However, the processing procedure for generating the movement history 50 described below is merely an example, and each step may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added to the processing procedure for generating the movement history 50 described below as appropriate depending on the embodiment.

[0174] In step S601, the control unit 21 operates as the acquisition unit 211. The process of step S601 may be the same as that of step S501. In step S602, the control unit 21 operates as the history generation unit 2122. That is, the control unit 21 generates the movement history 50 of the target user from the event position 201 of the acquired product input record 20. In step S603, the control unit 21 operates as the output processing unit 213. That is, the control unit 21 outputs the generated movement history 50. As in the above step S503, the output destination and contents of the information may be appropriately selected according to the embodiment. When the output of the information is completed, the control unit 21 ends the processing procedure related to the generation of the movement history 50 according to this operation example.

[0175] According to an example of the present embodiment, the movement history 50 of the target user can be generated using the product input record 20 by the processes of steps S601 and S602. The target user's purchasing behavior can be analyzed by the generated movement history 50. The control unit 21 may execute a series of processes from steps S601 to S603 at any timing. The control unit 21 may execute both the processes from steps S501 to S503 and steps S601 to S603. In this case, the process of step S601 may be executed in common with the above step S501. The order of the processes of steps S502, S503, S602, and S603 is not particularly limited and may be appropriately determined according to the embodiment.

[0176] (Location identification) FIG. 19 is a flowchart showing an example of a processing procedure for identifying the insertion location of a product (target product TP1) using the product insertion record 20 by the information processing device 2 according to this embodiment. The processing procedure for identifying the insertion location described below is an example of an identification method (information processing method) executed by a computer. However, the processing procedure for identifying the insertion location described below is merely an example, and each step may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added to the processing procedure for identifying the insertion location described below as appropriate depending on the embodiment.

[0177] In steps S701 and S702, the control unit 21 operates as the acquisition unit 211. The process in step S701 may be the same as that in step S501. In S702, the control unit 21 acquires the purchase record 35 of the purchased item of the target user. In step S703, the control unit 21 operates as the location identification unit 2123. That is, the control unit 21 collates the acquired product input record 20 and purchase record 35 with the map information 30 to identify the location where the target user put the purchased item into the basket (basket B1). In step S704, the control unit 21 operates as the output processing unit 213. That is, the control unit 21 outputs information related to the identified location. As in the above step S503, the output destination and contents of the information may be appropriately selected according to the embodiment. When the output of the information is completed, the control unit 21 ends the processing procedure related to the identification of the input location in this operation example.

[0178] According to an example of the present embodiment, the process of steps S701 to S703 can specify the location where the purchased product was placed in the shopping cart. As a result, even if the same product is placed in multiple locations in a store, it is possible to analyze from which location each target user obtained the purchased product. The control unit 21 may execute a series of processes of steps S701 to S704 at any timing. The processing order of each step may be changed as appropriate. In another example, the processing of step S702 may be executed before step S701. The processing of step S702 may be executed at least partially in parallel with the processing of step S701. The control unit 21 may also execute the processing of both steps S501 to S503 and steps S701 to S704. In this case, the processing of step S701 may be executed in common with the above-mentioned step S501. The processing order of steps S502, S503, S702, S703, and S704 is not particularly limited and may be determined as appropriate according to the embodiment. The control unit 21 may execute both the processes of steps S601 to S603 and steps S701 to S704. In this case, the process of step S701 may be executed in common with the above-mentioned step S601. The order of the processes of steps S602, S603, S702, S703, and S704 is not particularly limited and may be appropriately determined depending on the embodiment. When all the processes of identifying the product, generating the movement history, and specifying the insertion location are executed, the processes of steps S501, S601, and S701 may be executed in common.

[0179] § 4 Variations Although the embodiment of the present disclosure has been described in detail above, the above description is merely an example of the present disclosure in every respect. The processes and means described in the present disclosure can be freely combined and implemented as long as no technical contradiction occurs. In addition, various improvements or modifications may be made to the above embodiment as appropriate. For example, the following modifications are possible. In the following, the same reference numerals are used for components similar to those in the above embodiment, and the description of the same points as those in the above embodiment is omitted as appropriate.

[0180] <4.1> The above embodiment is configured as a smart grip 1. However, the embodiment of the present disclosure may not be limited to the above smart grip 1. The embodiment of the present disclosure may be a component of a basket used for shopping, or may be a basket (including a cart) itself. The type of basket is not particularly limited, and may be appropriately selected depending on the embodiment.

[0181] (First Modification) FIG. 20 is a schematic diagram showing another example of a situation to which the present disclosure is applied. In the example shown in FIG. 20, the embodiment of the present disclosure is a shopping basket 1A. The shopping basket 1A includes a basket body 102A, a detection sensor 13, a positioning sensor 14, and a control device 11. The basket body 102A may be configured similarly to the basket body B10 of the basket B1 described above. Except for replacing the basket body B10 with the basket body 102A, the detection sensor 13, the positioning sensor 14, and the control device 11 may be configured similarly to the above embodiment. That is, the detection sensor 13 detects a position of the basket body 102A. The control device 11 may be configured to detect the insertion of a product TP (target product). When the detection sensor 13 detects the insertion of the product TP into the basket body 102A, the control device 11 may generate a product insertion record 20 including the event position 201 where the product TP was inserted, which is the event position 201 measured by the positioning sensor 14.

[0182] Other configurations of the shopping basket 1A are not particularly limited and may be appropriately determined depending on the embodiment. In one example, the shopping basket 1A may include a pair of handles 101A attached to the basket body 102A, and a grip 100A spanning the pair of handles 101A. Each handle 101A may be configured similarly to each handle B11 of the basket B1. When adopting these configurations, in one example, the shopping basket 1A may be constructed by using the above-mentioned smart grip 1 for the grip 100A. However, the method of constructing the shopping basket 1A is not limited to such an example.

[0183] Like the smart grip 1, the shopping basket 1A may further include a memory unit 12. The shopping basket 1A may further include at least one of an output device 15, an imaging device 16, and a communication module 17. When the shopping basket 1A further includes the output device 15, the control device 11 may output related information 155 from the output device 15 when the detection sensor 13 detects the insertion of the product TP. When the shopping basket 1A further includes the imaging device 16, the control device 11 may acquire a captured image 165 of the imaging device 16 when the detection sensor 13 detects the insertion of the product TP, and identify the product TP by performing image analysis on the acquired captured image 165.

[0184] The location where the computer portion of the shopping basket 1A (control device 11 to communication module 17) is arranged does not have to be limited to the grip 100A. At least a part of the computer portion of the shopping basket 1A may be arranged at any location other than the grip 100A. In another example, when a load sensor is used for the detection sensor 13, the detection sensor 13 may be arranged on the bottom surface of the storage section of the basket body 102A. The imaging device 16 may be arranged on the edge of the opening of the basket body 102A with the imaging unit 161 facing toward the storage section of the basket body 102A.

[0185] The shopping basket 1A is an example of a shopping device. According to this modification, in a situation where the basket form is adopted, it is expected that the monitoring of purchasing behavior by a user can be simplified. The configuration of the shopping basket 1A may be appropriately changed depending on the embodiment. In another example, the grip 100A may be omitted. In this case, the user may hold the shopping basket 1A by grasping the pair of handle portions 101A.

[0186] (Second Modification) FIG. 21 is a schematic diagram showing another example of a scene to which the present disclosure is applied. In the example of FIG. 21, the embodiment of the present disclosure is a shopping basket 1B. The shopping basket 1B corresponds to an embodiment in which the shopping basket 1A is further provided with one or more wheels 103B, thereby forming a cart. That is, the shopping basket 1B includes a basket body 102B, a detection sensor 13, a positioning sensor 14, a control device 11, and one or more wheels 103B. The basket body 102B may be the same as the basket body 102A of the first modified example. The detection sensor 13 may be configured to detect the insertion of a product TP (target product) into the basket body 102B. When the control device 11 detects the insertion of the product TP into the basket body 102B by the detection sensor 13, the control device 11 may generate a product insertion record 20 including the event position 201 where the product TP was inserted, which is the event position 201 measured by the positioning sensor 14. The number of wheels 103B may be appropriately selected according to the embodiment. In a typical example, the shopping basket 1B may include four wheels 103B. The shopping basket 1B may be referred to as a shopping cart.

[0187] The arrangement of the wheels 103B and the structure for supporting the basket body 102B may be appropriately determined depending on the embodiment. It may be determined appropriately. In one example, the shopping basket 1B may include a grip 100B and a base 101B. The base 101B may include a leg 1011B and a support 1012B. The leg 1011B may be disposed below the basket body 102B. Each wheel 103B may be attached to the leg 1011B. The support 1012B may be configured to extend vertically upward from one end of the leg 1011B. The basket body 102B may be fixed to the support 1012B. The grip 100B may be attached to the upper end of the support 1012B.

[0188] As a result, the shopping basket 1B may have a configuration similar to that of a general shopping cart. However, the configuration of the shopping basket 1B need not be limited to this example. The shopping basket 1B may have a configuration similar to that of other types of carts, such as a silver cart. Furthermore, when these configurations are adopted, in one example, the shopping basket 1B may be constructed by using the above-mentioned smart grip 1 for the grip 100B. However, the method of constructing the shopping basket 1B need not be limited to this example.

[0189] Like the smart grip 1, the shopping basket 1B may further include a memory unit 12. The shopping basket 1B may further include at least one of an output device 15, an imaging device 16, and a communication module 17. Like the first modified example, when the shopping basket 1B further includes the output device 15, the control device 11 may execute output processing of related information 155. When the shopping basket 1B further includes the imaging device 16, the control device 11 may execute identification processing of the product TP by image analysis.

[0190] The location where the computer part of the shopping basket 1B (control device 11 to communication module 17) is arranged does not have to be limited to the grip 100B. At least a part of the computer part of the shopping basket 1B may be arranged at any location other than the grip 100B. In another example, when a load sensor is used for the detection sensor 13, the detection sensor 13 may be arranged on the bottom surface of the storage part of the basket body 102B. The imaging device 16 may be arranged on the edge of the opening of the basket body 102B or in the area of ​​the support part 1012B from the upper end of the basket body 102B to the grip 100B, with the imaging unit 161 facing toward the storage part of the basket body 102B.

[0191] The shopping basket 1B is an example of a shopping device. According to this modification, in a situation where the form of a cart is adopted, it is expected that the monitoring of the purchasing behavior of the user can be simplified. The basket body 102B may be used as a basket receiver. In this case, other baskets may be placed in the basket body 102B, and the product TP may be put into the other basket. The detection sensor 13 may be appropriately arranged so as to be able to detect the putting of the product TP into the other basket. Detecting the putting of the product TP into the basket body 102B may include detecting the putting of the product TP into the other basket placed in the basket body 102B. In addition, the leg 1011B may be configured to receive luggage. In this case, the detection sensor 13 may be further arranged in the leg 1011B. When the identification of the product TP by image analysis is also performed in the leg 1011B, the imaging device 16 may be further arranged in the leg 1011B. [Explanation of symbols]

[0192] 1. Smart Grip, 1A·1B…Shopping basket, 102A·102B…Basket body, 11: control device, 13: detection sensor, 14: positioning sensor, 15...output device, 16...imaging device, 90…Housing, 915·935…Groove, 20...Product input record, 201...Event location, 2...information processing device, 30...Map information, 35...Purchase record, 40: imaging device; 400: captured image; 45...load sensor, 450...measurement data, 50...Travel history, TP...product, TP1...target product, PS…Product shelf

Claims

1. A smart grip that can be used as a handle for a basket, Housing, Detection sensor, Positioning sensor, and Control device Equipped with The detection sensor is configured to detect the insertion of a commodity into the basket, The positioning sensor is configured to measure a position; The control device includes: generating a product insertion record including an event location measured by the positioning sensor when the insertion of the product is detected by the detection sensor, the event location including the event location where the product was inserted; and storing the generated product input record; configured to run The housing accommodates the detection sensor, the positioning sensor, and the control device, and has one or more grooves for hanging a handle provided on the basket. Smart grip.

2. The detection sensor is configured to detect the insertion of a commodity into the basket by one or more load sensors, Each of the one or more load sensors is arranged in accordance with the position of each of the one or more grooves so that a handle portion provided on the basket contacts a measuring portion of each of the one or more load sensors. The smart grip according to claim 1.

3. Further comprising an output device, The control device is further configured to, when the detection sensor detects the insertion of the product, output information related to the insertion of the product from the output device. The smart grip according to claim 1 or 2.

4. The related information includes at least one of advertisement information and coupon information for other products arranged in the vicinity of the event location. The smart grip according to claim 3.

5. Further comprising an imaging device, The control device includes: When the detection sensor detects the insertion of the commodity, an image captured by the imaging device is acquired; and Identifying the product shown in the captured image by performing image analysis on the captured image; and further configured to perform The smart grip according to claim 1 or 2.

Images