Information processing device

Abstract

To provide a technology for easily monitoring user purchasing behavior. [Solution] An information processing device according to one aspect of the present disclosure acquires a product input record of a target user, which is a product input record generated by a shopping device and includes the event location where the target product, which is a purchased product, was input, acquires a purchase record of the purchased product of the target user, and compares the acquired product input record and purchase record with map information which includes identification information of the product in association with the display location of the product in the store, thereby identifying the location where the target user put the purchased product into their basket and outputting information related to the identified location.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device. [Background technology]

[0002] There is a system that monitors the purchasing behavior of users in a store by installing multiple sensors such as an imaging device and a load sensor in the store and measuring the user's behavior with the installed multiple sensors. For example, Patent Document 1 proposes a system that receives an indicator that indicates the user's location in the facility, receives a product order via an input device, and determines whether the order is associated with the user based on sensing data generated by one or more sensors in the facility. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2023-509844 Summary of the Invention [Problem to be solved by the invention]

[0004] When monitoring user purchasing behavior using only sensors installed in stores, the monitoring process may become complicated due to the difference in sensor placement in each store, the large number of sensors, the large amount of sensing data to be processed, etc. In other words, it may become difficult to simply monitor user purchasing behavior.

[0005] In one aspect, the present disclosure has been made in consideration of the above circumstances, and one of its objectives is to provide a technology for easily monitoring user purchasing behavior, and another objective is to provide a technology for utilizing product input records generated by monitoring purchasing behavior. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present disclosure employs the following configurations. Note that the following configurations can be combined as appropriate.

[0007] A smart grip according to one aspect of the present disclosure is configured to be usable as a handle for a basket. The smart grip includes a detection sensor, a positioning sensor, and a control device. The detection sensor is configured to detect the insertion of a product (target product) into the basket. The positioning sensor is configured to measure a position. The control device is configured, when the detection sensor detects the insertion of the product, to generate a product insertion record including the event position measured by the positioning sensor, where the product was inserted, and to store the generated product insertion record.

[0008] According to this configuration, by generating a product insertion record using the detection sensor and positioning sensor provided in the smart grip, it is possible to monitor user purchasing behavior without installing sensors in the store. This is expected to simplify monitoring. Note that the smart grip according to this configuration may also be used to monitor user purchasing behavior together with sensors installed in the store. In this case, by using the product insertion record to improve the efficiency of processing sensing data obtained by sensors installed in the store, it is expected to simplify monitoring.

[0009] The smart grip according to the above aspect may further include a housing that houses the detection sensor, the positioning sensor, and the control device, and that has one or more grooves for hooking onto a handle provided on the basket. With this configuration, it is possible to provide a smart grip that can be used as a handle for the basket by replacing the handle of an existing basket.

[0010] The smart grip according to the above aspect may further include an output device. The control device may be further configured to output related information relating to the insertion of a product from the output device when the detection sensor detects the insertion of a product. With this configuration, providing related information when a product is inserted can be expected to stimulate purchasing activity, such as by stimulating further product purchases.

[0011] In the smart grip according to the above aspect, the related information may include at least one of advertisement information and coupon information for other products placed around the event location. With this configuration, providing at least one of the advertisement information and coupon information can be expected to stimulate purchasing desire for other products.

[0012] The smart grip according to the above aspect may further include an imaging device. The control device may be configured to acquire an image captured by the imaging device when the detection sensor detects the insertion of a commodity, and to identify the commodity shown in the image by analyzing the acquired image. With this configuration, the inserted commodity can be identified on the smart grip.

[0013] The present disclosure is not limited to the smart grip. An aspect of the present disclosure may be a component of a shopping basket. An aspect of the present disclosure may be the basket (including a cart) itself. Furthermore, an aspect of the present disclosure may be an information processing method for implementing all or part of the control device (computer), a program, or a machine-readable storage medium storing such a program. Here, the machine-readable storage medium may be a non-transitory medium that stores information such as a program through electrical, magnetic, optical, mechanical, or chemical action. The non-transitory storage medium may include a storage medium (e.g., a CD, a DVD, a semiconductor memory), an auxiliary storage device of a computer, an external storage device connected to a computer, etc.

[0014] For example, a shopping basket according to one aspect of the present disclosure may include a basket body, a detection sensor, a positioning sensor, and a control device. The detection sensor may be configured to detect the insertion of an item into the basket body. The positioning sensor may be configured to measure a position. The control device may be configured, when the detection sensor detects the insertion of the item, to generate an item insertion record including an event position measured by the positioning sensor, where the item was inserted, and to store the generated item insertion record.

[0015] The shopping basket according to the above aspect may be configured as a cart by further including one or more wheels. With this configuration, the cart configuration can be expected to simplify monitoring of purchasing behavior by users.

[0016] According to another aspect of the present disclosure, an information processing device includes a control unit. The control unit is configured to acquire a product insertion record of a target user generated by a shopping device, the product insertion record including an event location where the target product was inserted, identify the inserted target product based on the acquired product insertion record, and output information related to the result of identifying the target product. This configuration provides a technology for identifying inserted products using the product insertion record.

[0017] In the information processing device according to the above aspect, identifying the target product in accordance with the product input record may be configured to identify the target product input at the event location by comparing the product input record with map information including identification information of the product in association with the display position of the product in the store. With this configuration, the target product can be appropriately identified by using the map information.

[0018] In the information processing device according to the above aspect, identifying the target product in accordance with the product insertion record may be configured by acquiring an image captured by an imaging device whose imaging range includes the event location of the product insertion record, and performing image analysis on the acquired image to identify the target product appearing in the image. Performing image analysis on all images acquired by all imaging devices installed in the store may result in a huge amount of processing. In contrast, this configuration allows the range of captured images to be subjected to image analysis to be narrowed to the range of event detection based on the product insertion record. This allows for more efficient product identification processing (image analysis).

[0019] In the information processing device according to the above aspect, a load sensor may be installed on each product shelf in the store. Identifying the target product based on the product insertion record may be configured to acquire measurement data from a load sensor installed near the event location in the product insertion record, and identify the product shelf where the weight has decreased based on the acquired measurement data, thereby identifying the target product. Constantly monitoring all load sensors installed in the store may result in a huge amount of processing. In contrast, with this configuration, the range of load sensors to be monitored can be narrowed to the range of event detection based on the product insertion record. This can improve the efficiency of the product identification process.

[0020] According to another aspect of the present disclosure, there is provided an information processing device including a control unit configured to: acquire a product insertion record of a target user generated by a shopping device, the product insertion record including an event location where a target product was inserted; generate a movement history of the target user from the event location in the acquired product insertion record; and output the generated movement history. This configuration provides a technology for monitoring the movement history of a target user using the product insertion record.

[0021] According to another aspect of the present disclosure, an information processing device includes a control unit. The control unit is configured to: acquire a product input record of a target user, the product input record being generated by a shopping device and including an event location where the product was input; acquire a purchase record of the target user's purchased products; identify the location where the target user put the purchased products into a cart by comparing the acquired product input record and purchase record with map information including product identification information associated with the display location of the products in the store; and output information related to the identified location. The same product may be placed in multiple locations within a store (e.g., permanent shelves, special shelves, etc.). This configuration provides a technology for identifying the location where the target user put the purchased products into a cart using the product input record.

[0022] In the information processing device according to each of the above aspects, the shopping device may be at least one of the smart grip and shopping basket according to any of the above aspects. That is, the smart grip and shopping basket according to the above aspect are examples of a shopping device. However, the shopping device is not limited to the smart grip and shopping basket according to the above aspect. As long as it is possible to generate a product insertion record including the event location, the configuration and type of the shopping device are not particularly limited and may be selected appropriately depending on the embodiment.

[0023] The embodiment of the present disclosure is not limited to the information processing device (computer). One aspect of the present invention may be an information processing method, an information processing system, a program, or a storage medium that stores such a program and is readable by a machine such as a computer. The information processing system may be configured by a plurality of computers, such as a server device and a store terminal. [Effects of the Invention]

[0024] According to one aspect of the present disclosure, it is possible to provide a technology for easily monitoring user purchasing behavior. Also, according to another aspect of the present disclosure, it is possible to provide a technology for utilizing product input records generated by monitoring purchasing behavior. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 schematically illustrates an example of a situation to which the present disclosure is applied. [Figure 2] FIG. 2 shows a schematic example of a usage scenario of the product input record. [Figure 3] FIG. 3 shows a schematic example of how the product input record is used. [Figure 4] FIG. 4 is a diagram illustrating an example of a method for identifying a product using a product input record. [Figure 5] FIG. 5 is a diagram illustrating an example of a method for identifying a product using a product input record. [Figure 6] FIG. 6 is a diagram showing an example of a method for identifying a product using a product input record. [Figure 7] FIG. 7 is a diagram showing another example of a method for using the product input record. [Figure 8] FIG. 8 is a diagram showing another example of a method for using the product input record. [Figure 9] FIG. 9 is a side view schematically showing an example of the structure of a smart grip. [Figure 10] FIG. 10 is a plan view schematically showing an example of the structure of a smart grip. [Figure 11] FIG. 11 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 12] FIG. 12 is a diagram illustrating an example of the software configuration of the smart grip (control device). [Figure 13] FIG. 13 is a diagram illustrating an example of the software configuration of the information processing device. [Figure 14] FIG. 14 is a flowchart illustrating an example of a processing procedure for generating a product input record by the control device. [Figure 15] FIG. 15 is a flowchart illustrating an example of a processing procedure for product identification by the control device. [Figure 16] FIG. 16 is a flowchart illustrating an example of a processing procedure regarding output of related information by the control device. [Figure 17] FIG. 17 is a flowchart illustrating an example of a processing procedure for product identification using a product insertion record by an information processing device. [Figure 18] FIG. 18 is a flowchart illustrating an example of a processing procedure for generating a movement history using a product insertion record by an information processing device. [Figure 19] FIG. 19 is a flowchart illustrating an example of a processing procedure for location identification using a product insertion record by an information processing device. [Figure 20] FIG. 20 schematically illustrates another example of a situation to which the present disclosure is applied. [Figure 21] FIG. 21 schematically illustrates another example of a situation to which the present disclosure is applied. DETAILED DESCRIPTION OF THE INVENTION

[0026] An embodiment according to one aspect of the present disclosure will be described below with reference to the drawings. However, the embodiment described below is merely an example of the present disclosure in all respects. Various improvements or modifications may be made without departing from the scope of the present disclosure. In implementing the present disclosure, specific configurations according to the embodiment may be appropriately adopted. Note that while data appearing in the present embodiment is described in natural language, more specifically, it may be specified using pseudo-language, commands, parameters, machine language, electrical signals, etc. that can be recognized by a machine such as a computer.

[0027] §1 Application Examples FIG. 1 schematically illustrates an example of a situation in which the present disclosure is applied. A smart grip 1 according to this embodiment is configured to be usable as a handle for a basket B1. The smart grip 1 includes a control device 11, a detection sensor 13, and a positioning sensor 14. The detection sensor 13 is configured to detect the insertion of a product TP into the basket B1. The positioning sensor 14 is configured to measure a position. When the detection sensor 13 detects the insertion of the product TP, the control device 11 is configured to generate a product insertion record 20 including the event position 201 where the product TP was inserted, which is measured by the positioning sensor 14, and to store the generated product insertion record 20.

[0028] According to this embodiment, the product insertion record 20 is generated using the detection sensor 13 and the positioning sensor 14 provided in the smart grip 1, thereby making it possible to monitor user purchasing behavior without installing sensors in the store. This is expected to simplify the monitoring of purchasing behavior. The smart grip 1 according to this embodiment may also be used to monitor user purchasing behavior together with sensors installed in the store. In this case, the product insertion record 20 can be used to improve the efficiency of processing sensing data obtained by sensors installed in the store, thereby making it possible to simplify the monitoring of purchasing behavior.

[0029] [Basket] 1, in a typical example, the basket B1 may be a shopping basket including a basket body B10 and a pair of handles B11. When the basket B1 includes the basket body B10, putting a product TP into the basket body B10 may correspond to putting the product TP into the basket B1.

[0030] The configuration of the basket body B10 and each handle portion B11 may be determined appropriately depending on the embodiment. For example, the basket body B10 may be configured so that the upper end is open and the lower end is closed at the bottom, allowing products TP to be received from the upper end side. The basket body B10 may have any shape, such as a substantially rectangular parallelepiped or a substantially truncated quadrangular pyramid. The pair of handle portions B11 may be arranged separately in the longitudinal direction of the basket body B10. Each end of the handle portion B11 may be connected to each side surface in the lateral direction of the basket body B10 so that the handle portion B11 can be rotated around an axis in the lateral direction. As a result, each handle portion B11 may be configured to be rotated at the connection portion with the basket body B10 to change its angle and be gripped along the lateral direction.

[0031] The configuration of the basket B1 is not limited to the example shown in FIG. 1 and may be modified as appropriate depending on the embodiment. For example, the number of handle portions B11 is not limited to two. The number of handle portions B11 may be one, or three or more. Furthermore, each end of one handle portion B11 may be separated and connected to one side surface in the short direction of the basket body B10 in the longitudinal direction, and each end of the other handle portion B11 may be separated and connected to the other side surface in the short direction of the basket body B10 in the longitudinal direction. In this way, each handle portion B11 may be configured to be gripped along the longitudinal direction.

[0032] The type of basket B1 is not limited to the example shown in FIG. 1 and may be changed as appropriate depending on the embodiment. As long as it can be used to carry the product TP, the type of basket B1 is not particularly limited and may be selected as appropriate depending on the embodiment. In another example, the basket B1 may include a bag that can be used for shopping, such as a plastic bag or a paper bag. The basket B1 may also include a device that can hold products by hanging them, such as a hanger.

[0033] [Can be used as a basket handle] The fact that it can be used as a handle for a basket (such as the basket B1) means that the smart grip 1 is disposed as at least a part of the handle that is held by a user when carrying a product (product TP). The smart grip 1 may be configured in any form that can be placed on a basket. In one example, the smart grip 1 can be used as a handle for a basket by hanging the handle of the basket (handle B11, etc.) on the smart grip 1 to replace an existing handle provided on the basket. The handle of the basket may include any component that can be hung on the smart grip 1, such as a bag handle or a hanger hook. In another example, the smart grip 1 can be used as a handle for a basket by being attached to the handle of the basket. In yet another example, the smart grip 1 can be used as a handle for a basket by being attached to the basket body (basket body B10, etc.) as a handle. In other words, the smart grip 1 can be configured as the handle of the basket itself.

[0034] [Product / Store] The product TP may include any type of carried item. The product TP may also be referred to as a target product. The smart grip 1 may be used in any store. The configuration of the store is not particularly limited and may be determined appropriately depending on the embodiment. The store's sales format may be either unmanned or manned. The store may or may not be equipped with sensors such as an imaging device and a load sensor. If sensors are installed in the store, the purchasing behavior of the target user may be monitored by linking the sensors of the smart grip 1 (detection sensor 13, positioning sensor 14, etc.) with the sensors installed in the store. Alternatively, the smart grip 1 may monitor the purchasing behavior of the target user independently of the sensors installed in the store (i.e., without linking with the sensors of the store).

[0035] [Product insertion] Putting a product (product TP) into a basket (basket B1) may include any form of carrying a product from the sales floor. As illustrated in FIG. 1 , in a typical example, putting a product into a basket may be putting a new product TP into the basket body B10 of the basket B1, which includes the smart grip 1. Putting a product into a basket may also include any action similar to putting a product into a basket, such as hanging the handle of a basket containing a product on the smart grip 1. Hanging the handle of a basket containing a product may include hanging the handle of a plastic bag containing the product, hanging the hook of a hanger that is hanging the product, etc.

[0036] [Product input record] The product insertion record 20 is configured to include an event location 201. The event location 201 is derived from measurements of the positioning sensor 14. For example, the measurements of the positioning sensor 14 may be used as the event location 201. Alternatively, the event location 201 may be calculated from measurements of the positioning sensor 14 by applying any arithmetic processing to one or more measurements of the positioning sensor 14. The event location 201 may be derived from values measured in real time by the positioning sensor 14 in response to the detection of insertion of the product TP by the detection sensor 13. The event location 201 may be appropriately configured to indicate the location where the product TP was inserted (insertion location). In a typical example, the event location 201 may be configured to indicate the insertion location using two-dimensional or three-dimensional coordinate values. However, the configuration of the event location 201 is not limited to this example and may be appropriately changed depending on the embodiment. As long as the product insertion record 20 includes the event location 201, the configuration of the product insertion record 20 is not particularly limited and may be appropriately defined depending on the embodiment. The product insertion record 20 may be composed of only the event location 201, or may further include other information other than the event location 201.

[0037] (Identification information) As an example of other information, the product insertion record 20 may further include identification information 203 for identifying a user (target user) who uses the smart grip 1. The source of the identification information 203 may be determined appropriately depending on the embodiment. In one example, the smart grip 1 stores The device may further include a memory unit 12. Identification information 125 may be stored in advance in the memory unit 12. The identification information 125 may be configured arbitrarily as long as it can identify the target user. For example, the identification information 125 may be configured to directly indicate the target user, such as the target user's account or user identifier. The identification information 125 may also be configured to indirectly indicate the target user, such as an identifier of the smart grip 1 or an identifier of the control device 11. The identification information 203 of the product insertion record 20 may be derived as appropriate from the identification information 125 in the memory unit 12. In one example, the identification information 125 in the memory unit 12 may be used directly as the identification information 203 of each product insertion record 20. In another example, the identification information 203 may be calculated from the identification information 125 by applying any arithmetic operation (deletion of a value, addition of a value, arithmetic operation, conversion, etc.) to the identification information 125.

[0038] According to one example of this embodiment, since the product input record 20 includes the identification information 203, it is possible to easily identify the user corresponding to the product input record 20 (i.e., the user who input the product TP at the event position 201 indicated in the product input record 20). When the product input records 20 of multiple users are collected, it is possible to easily extract the product input record 20 of a specific user from the collected product input records 20 by searching using the identification information 203.

[0039] Furthermore, when identifying a user corresponding to a product insertion event using equipment installed in a store, for example, imaging devices are installed in multiple locations in the store and image analysis (such as facial recognition) is periodically performed on images captured by the imaging devices, which may increase the burden of installing sensors and the processing load of tracking people. In contrast, according to one example of the present embodiment, the identification information 125 of the smart grip 1 is linked to the product insertion record 20 as the identification information 203, making it possible to identify a user corresponding to a product insertion event. This reduces at least one of the burden of installing sensors in the store and the processing load of tracking people.

[0040] The identification information 125 stored in the smart grip 1 may be used for purposes other than deriving the identification information 203 of the product insertion record 20. As an example, the identification information 125 of the smart grip 1 may be used to authenticate a user at least when entering or leaving a store. For example, the smart grip 1 may exchange data with a store system (such as a store terminal) when entering or leaving the store. The method of data exchange may be selected arbitrarily.

[0041] As an example of a data exchange method, the smart grip 1 may further include an output device 15. The output device 15 may include a display. The smart grip 1 (control device 11) may convert the identification information 125 into a code and output the obtained code from the output device 15. The code may further include any information other than the identification information 125. For example, the code may be generated as a one-time code by further including temporary information such as a time or a random number. The type of code may be selected arbitrarily. For example, the code may be composed of a one-dimensional code or a two-dimensional code. Code readers may be installed at the entrances and exits of the store. The store's system may exchange data with the smart grip 1 by reading the information (including the code) of the smart grip 1 using the reader.

[0042] As another example of a data exchange method, the smart grip 1 may further include a communication module 17. The communication module 17 may include a near-field communication module. The near-field communication standard is not particularly limited and may be selected appropriately depending on the embodiment. Accordingly, modules (readers, etc.) compatible with the near-field communication module of the smart grip 1 may be provided at the entrance and exit of the store. Accordingly, the smart grip 1 and the store system may exchange data via the near-field communication module.

[0043] The store system may acquire the identification information 125 from the smart grip 1 in the data exchange. The store system may authenticate the user according to the acquired identification information 125. The store system may be configured to control the locking and unlocking of the entrance and exit doors in response to the user authentication. For example, the store system may keep the entrance and exit doors locked unless the user authentication is performed. The store system may unlock the corresponding entrance or exit door in response to successful user authentication. After the user passes through the entrance or exit (e.g., opens the door once and then closes it), the store system may relock the corresponding entrance or exit door. According to this example of the present embodiment, it is possible to easily identify a user present in the store. Furthermore, after successful authentication at the entrance, the control device 11 of the smart grip 1 may start executing a process to monitor the user's purchasing behavior, such as generating a product insertion record 20. After successful authentication at the exit, the control device 11 may stop executing the monitoring process. This allows the operation of the control device 11 to be appropriately controlled so that the monitoring process is performed within the store.

[0044] (time) As another example of information, the product insertion record 20 may further include a time 205 when the product insertion record 20 was generated (i.e., when the product TP was inserted). The source of the time 205 may be determined appropriately depending on the embodiment. In one example, the smart grip 1 may further include a timer. The control device 11 may refer to the timer when the detection sensor 13 detects the insertion of the product TP (when the product insertion record 20 is generated) and obtain the time indicated by the timer as the time 205. According to one example of the present embodiment, by using the time 205, it is possible to extract a product insertion record 20 (insertion event) corresponding to a specific time period from multiple product insertion records 20. Furthermore, by using the time 205, it is possible to classify the product insertion records 20 by time period and analyze the characteristics of insertion events in each time period (e.g., which location of the product TP is most likely to be purchased).

[0045] (others) In the example of FIG. 1 , the product input record 20 includes an event location 201, identification information 203, and time 205. However, the configuration of the product input record 20 is not limited to this example and may be modified as appropriate depending on the embodiment. For example, at least one of the identification information 203 and the time 205 may be omitted. Furthermore, the product input record 20 may further include information other than the identification information 203 and the time 205. In one example, the product input record 20 may further include an identifier for identifying the product input record 20.

[0046] [Maintaining product input records] The product insertion record 20 may be stored in any memory area. In one example, the smart grip 1 may further include a memory unit 12. The generated product insertion record 20 may be stored in the memory unit 12. In another example, the smart grip 1 may further include a communication module 17. In this case, the smart grip 1 may transmit the product insertion record 20 to an external computer such as a terminal or a server device (including the information processing device 2 described below) via the communication module 17. As a result, the product insertion record 20 may be stored in the memory area of the external computer. The terminal may include a user terminal, a store terminal (a terminal installed in a store), etc. The server device may include an external storage device such as a NAS (Network Attached Storage). In either case of storing the product insertion record 20 in the memory unit 12 in the smart grip 1 or in an external computer, storing the product insertion record 20 may be performed using a cache memory, a RAM (Random Access Memory), etc. This may include temporarily storing the item input record 20 in a memory such as a USB flash drive (USB Read Only Memory).

[0047] Furthermore, the process of saving the product input record 20 may be executed at any timing after the process of generating the product input record 20 has been executed one or more times. In one example, the smart grip 1 may sequentially save the generated product input record 20 in response to the generation of the product input record 20. In another example, the smart grip 1 may repeatedly execute the generation of the product input record 20 to form a product input record group, and save at least a part of the formed product input record group. (Multiple product input records 20) may be saved together.

[0048] [Product identification using smart grips] In one example of this embodiment, the control device 11 may be configured to identify the product TP according to the observation result of the product TP by a sensor provided in the smart grip 1. The sensor used for identification may be selected appropriately depending on the embodiment. The detection sensor 13 may be used for identification, or a sensor other than the detection sensor 13 may be used for identification. The identification method may be determined appropriately depending on the sensor used.

[0049] In one example, the smart grip 1 may further include an imaging device 16. The imaging device 16 may be appropriately positioned so as to capture the product TP when the product TP is placed in the basket B1 (basket body B10). For example, the imaging device 16 may be positioned so that the basket B1 (basket body B10, etc.) is within its imaging range. As long as it is possible to generate an image in which the product TP can be identified, the type of imaging device 16 is not particularly limited and may be appropriately selected depending on the embodiment. The imaging device 16 may include any sensor that acquires data in the form of an image or an image representation, such as an RGB camera, a depth sensor, an infrared sensor, radar, or LiDAR (light detection and ranging).

[0050] The control device 11 may acquire a captured image 165 from the imaging device 16 when the detection sensor 13 detects the insertion of the product TP. The timing of imaging by the imaging device 16 (generation of the captured image 165) may be determined as appropriate depending on the embodiment. In one example, when the imaging device 16 is disposed in a position where it can capture an image of the product TP after the insertion is detected by the detection sensor 13, the control device 11 may activate the imaging device 16 and control the operation of the imaging device 16 to capture an image of the inserted product TP in response to the detection of the insertion of the product TP by the detection sensor 13. The control device 11 may control the imaging device 16 to capture an image of the product TP at any timing after detecting the insertion of the product TP. For example, the control device 11 may control the imaging device 16 to capture an image of the product TP immediately after detecting the insertion of the product TP. The control device 11 may appropriately acquire one or more captured images 165 generated by this imaging from the imaging device 16. After generating one or more captured images 165, the control device 11 may stop the operation of the imaging device 16 or switch it to a standby state. According to this example, the period during which the imaging device 16 is activated can be narrowed, which is expected to reduce the processing load and power consumption of the smart grip 1. In another example, the imaging device 16 may be controlled to continuously generate captured images. The generated captured images may be accumulated in a storage area, such as the storage unit 12, for at least a certain period of time. Captured images generated at or around the time when the detection sensor 13 detects the insertion of the product TP are likely to include the inserted product TP. Therefore, the control device 11 may acquire, from the captured images accumulated in the storage area, one or more captured images generated at or around the time when the detection sensor 13 detects the insertion of the product TP as captured images 165.

[0051] After acquiring the captured image 165, the control device 11 may identify the product TP shown in the captured image 165 by performing image analysis on the acquired captured image 165. The method of identification by image analysis may be determined appropriately depending on the embodiment. In one example, if a code generated by encoding identification information of the product TP is assigned to the product TP, analyzing the captured image 165 may include decoding the code assigned to the product TP. The type of code may be selected arbitrarily. The code may be, for example, a one-dimensional code, a two-dimensional code, etc. In another example, the control device 11 may identify the product TP from the characteristics (appearance, etc.) of the product TP without using the code in the image analysis.

[0052] The image analysis method is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the control device 11 may analyze the captured image 165 using a general image analysis method such as edge detection, pattern matching, etc. In another example, the control device 11 may analyze the image by The captured image 165 may be analyzed using a trained machine learning model that has acquired capabilities. The machine learning model is configured to include one or more calculation parameters that can be adjusted by machine learning. The one or more calculation parameters are used to calculate the desired inference (such as analyzing an image). The machine learning model may be configured, for example, using a neural network, a support vector machine, or other functional formulas (calculation models). The machine learning method may be appropriately selected depending on the machine learning model to be adopted (for example, backpropagation). Training the machine learning model means adjusting (optimizing) the values of the calculation parameters using training samples. The machine learning model may be appropriately trained so that, when given images of the training samples, it derives the true value of the corresponding analysis result. The trained machine learning model may be a large-scale model such as a large-scale visual language model (VLM).

[0053] The control device 11 may output the identification result of the product TP as appropriate. The identification result of the product TP may be used for any purpose. In one example, the identification result of the product TP may be added to the product insertion record 20. That is, generating the product insertion record 20 including the event location 201 may be configured by generating the product insertion record 20 including the event location 201 and the identification result of the product TP. According to this example, by referring to the identification result of the product TP, it is possible to identify the product TP inserted in the event indicated by the product insertion record 20. If the output device 15 is provided, the control device 11 may output the identification result of the product TP from the output device 15. Furthermore, if the communication module 17 is provided, the control device 11 may output the identification result of the product TP to an external computer (such as a store terminal, a user terminal, or a server device).

[0054] According to one example of the present embodiment, the inserted product TP can be identified on the smart grip 1. Note that the computer that performs image analysis of the captured image 165 does not have to be limited to the control device 11. In another example, the control device 11 may transmit a request to perform image analysis on the captured image 165, along with the captured image 165, to an external computer such as a store terminal, a user terminal, or a server device. The control device 11 may then obtain the identification result of the product TP by receiving a reply containing the image analysis result, including the identification result of the product TP, from the external computer. The method of identifying the product TP is not limited to the image analysis method described above and may be changed as appropriate depending on the embodiment. In another example, the smart grip 1 may further include a code reader. The control device 11 may identify the product TP by reading a code attached to the product TP with the reader. Furthermore, the identification process on the smart grip 1 may be omitted. The product insertion record 20 may be generated without identifying the product TP. The process of identifying the product TP may be performed by an external computer (including the information processing device 2, described later) instead of the control device 11.

[0055] [Related information output] In one example of this embodiment, the smart grip 1 may further include an output device 15. The output device 15 may be configured to output information. The output device 15 may be configured, for example, with a display, a speaker, an indicator light, etc. The information output format may be appropriately selected from text, images, sound, etc. When the output device 15 includes a display, it is preferable that the display be placed in a location where the output information can be viewed while the user is holding the smart grip 1. This allows the user to check the information displayed on the display while using the smart grip 1.

[0056] The conditions for outputting information and the information to be output may be determined as appropriate depending on the embodiment. In one example, the control device 11 may be further configured to output related information 155 related to the insertion of the product TP from the output device 15 when the detection sensor 13 detects the insertion of the product TP. According to one example of the present embodiment, by providing the related information 155 when the product TP is inserted, it is possible to expect activation of purchasing activity, such as stimulating the desire to purchase further products.

[0057] The related information 155 may be output at any timing after the detection sensor 13 detects the insertion of the product TP. For example, the control device 11 may output the related information 155 from the output device 15 in real time after the detection sensor 13 detects the insertion of the product TP. The source of the related information 155 may be determined appropriately depending on the embodiment. For example, if the smart grip 1 includes a memory unit 12, the related information 155 may be stored in advance in the memory unit 12. The control device 11 may acquire the related information 155 from the memory unit 12. Also, for example, the related information 155 may be stored in a memory area of an external computer (such as a store terminal, a user terminal, or a server device). The control device 11 may acquire the related information 155 from the external computer at any timing before outputting the information.

[0058] Furthermore, the content of the related information 155 is not particularly limited as long as it relates to purchasing activities such as the event location 201, the product TP to be inserted, and the act of inserting the product TP, and may be selected appropriately depending on the embodiment. When multiple candidates for the related information 155 exist, the related information 155 to be output may be selected from the multiple candidates as appropriate. For example, the related information 155 may be selected from the multiple candidates depending on at least one of the event location 201, the product TP, and the user information. Furthermore, for example, the related information 155 may be selected randomly. The selection of the related information 155 may be performed by either the control device 11 or an external computer (such as a store terminal, a user terminal, or a server device). When selecting the related information 155 from the multiple candidates, any information may be referenced. The information used to select the related information 155 may be stored in at least one of the control device 11 and the external computer, and may be referenced appropriately when selecting the related information 155.

[0059] In one example of this embodiment, the content of the related information 155 may be configured according to at least one of the following first to seventh examples. In each of the following examples, for convenience of explanation, it is assumed that the control device 11 selects the related information 155. However, as described above, the computer that executes the process of selecting the related information 155 does not have to be limited to the control device 11. At least a part of the process of the control device 11 described below may be executed by an external computer. Information that the control device 11 refers to when selecting the related information 155 may be held in either the control device 11 or the external computer.

[0060] (1) First Case In one example, the related information 155 may include at least one of advertisement information 1550 and coupon information 1555 for other products placed around the event location 201. The other products are products other than the inserted product TP (target product). The advertisement information 1550 includes advertisements for the other products. The advertisement content may be determined as appropriate. In a simple example, the advertisement information 1550 may be composed of the names of the other products. The coupon information 1555 includes information indicating discounts on the other products.

[0061] The other products may be identified as appropriate. For example, candidates for the related information 155 may be predefined for each area in the store. Accordingly, when the insertion of the product TP is detected, the control device 11 may identify the area to which the event location 201 belongs and select the related information 155 according to the identified area. When identifying the product TP, the control device 11 may exclude candidates corresponding to the product TP indicated by the identification result and select the related information 155 from the remaining candidates. Also, for example, the control device 11 may refer to information (such as map information 30 described below) indicating the correspondence between the display positions of products present in the store and the identification information of the products. The control device 11 may determine the other products by selecting products displayed in the vicinity of the event location 201 but at a position different from the event location 201 from the products present in the store indicated by the referenced information. The range of the "periphery" may be determined as appropriate. For example, whether or not a product belongs to the "periphery" may be determined using a method such as threshold value determination. The control device 11 may acquire the related information 155 by selecting a candidate linked to another product from among multiple candidates.

[0062] According to one example of the present embodiment, providing at least one of the advertising information 1550 and the coupon information 1555 can be expected to stimulate the purchase desire of other products. Note that, in one example of the present embodiment, the control device 11 preferably outputs related information 155 including at least one of the advertising information 1550 and the coupon information 1555 of other products in real time immediately after the detection sensor 13 detects the insertion of the product TP. This makes it possible to provide the target user with at least one of the advertising information 1550 and the coupon information 1555 of the other product while the target user is located near the other product. In other words, it is possible to provide the target user with related information 155 of products located close to the target user. As a result, the target user does not need to make an effort to pick up the other product, and it is expected that the probability that the target user will purchase the other product will increase.

[0063] (2) Second Case In one example, the related information 155 may be related to the product TP. For example, it may include at least one of advertising information, coupon information, and attribute information of the product TP. The attribute information may include any type of information related to the characteristics of the product TP. For example, the attribute information may be composed of the product TP's expiration date, release date, allergy labeling, reviews, related products, recommended products, or a combination thereof. The allergy labeling indicates allergens contained in the product TP. The related products may be other products related to the product TP. For example, the other products related to the product TP may be products used together with the product TP. The recommended products may be other products recommended to people who purchase the product TP. The related products and recommended products may or may not partially overlap. The related information 155 related to the product TP may be identified as appropriate. For example, when identifying the product TP, the control device 11 may acquire the related information 155 related to the product TP according to the identification result of the product TP. According to an example of this embodiment, information about the product TP can be provided to the user.

[0064] (3) Third Case In one example, the content of the related information 155 may be determined according to past purchase history. The control device 11 may identify products that have not been purchased for a certain period of time by referring to the purchase history. The related information 155 may be configured as a notification informing the user of the identified product. Furthermore, when identifying the product TP, the control device 11 may extract history related to the product TP from the purchase history according to the identification result of the product TP. The related information 155 may be configured as information related to the purchase history of the extracted product TP. For example, if it is determined in the purchase history of the product TP that the input product TP has been purchased multiple times in the past, the related information 155 may be configured as a notification recommending a change to a subscription. The threshold number of purchases recommended for a change to a subscription may be defined arbitrarily. Furthermore, for example, if the input product TP has been purchased within a predetermined period of time in the past, the related information 155 may be configured as a notification indicating that the product TP has been purchased in the past. The predetermined period may be defined arbitrarily. According to one example of the present embodiment, information related to past purchase history can be provided to the user.

[0065] (4) Fourth Case In one example, the related information 155 may be configured with information related to the budget. For example, the related information 155 may be configured with a notification (warning) indicating that the budget upper limit is being approached. The control device 11 may acquire the amount (price) of the product TP as appropriate. For example, information indicating the amount may be linked to the product TP (identification information, etc.). When identifying the product TP, the control device 11 may acquire information indicating the amount of the product TP according to the identification result of the product TP. Each time the detection sensor 13 detects the insertion of a product TP, the control device 11 may calculate the total amount of the one or more products TP inserted into the basket B1 by adding up the acquired amounts. Then, the control device 11 may determine whether the calculated total amount is close to the budget upper limit. If it is determined that the total amount is close to the budget upper limit, the control device 11 may calculate the total amount of the products TP that are close to the budget upper limit. The control device 11 may output the related information 155 configured with a notification indicating that the budget limit has been reached. If this is not the case, the control device 11 may omit outputting the related information 155 configured with a notification indicating that the budget limit has been reached. The budget limit may be set arbitrarily. For example, the budget limit may be input by operating the control device 11 or via an external computer. Whether the budget limit has been reached may be determined by any method, such as threshold determination. The threshold may be defined arbitrarily. According to one example of the present embodiment, it is possible to provide the user with information related to the budget, such as a warning that the budget limit has been reached. The information related to the budget may be changed as appropriate. For example, whether the budget limit has been reached may be replaced with whether the budget limit has been exceeded. The stage at which the warning is issued does not have to be limited to the budget limit. Furthermore, multiple stages at which the warning is issued may be set, and the content of the notification may be changed for each stage.

[0066] (5) Fifth Case In one example, the related information 155 may be composed of any information derived from information on the product TP and user information on the target user. The user information may include any information on the target user. The user information may include information on the target user's family. For example, if the product TP contains an allergen of at least one of the target user's own allergens and the target user's family, the related information 155 may be composed of a warning indicating that the product TP contains an allergen. In this case, the user information may include allergy information indicating at least one of the allergens of the target user's own allergens and the target user's family. The allergy indication of the product TP may be acquired as appropriate. In one example, attribute information including the allergy indication may be linked to the product TP (identification information, etc.). When identifying the product, the control device 11 may acquire the allergy indication of the product TP according to the product identification result. The control device 11 may refer to the user information as appropriate. The control device 11 may compare the allergy indication of the product TP with the allergy information included in the user information. The control device 11 may determine whether the product TP contains at least one of the allergens of the target user and the target user's family according to the comparison result. If it is determined that an allergen is contained in the product TP, the control device 11 may output a warning indicating that the product TP contains an allergen as the related information 155. According to one example of this embodiment, information about the product TP related to the target user, such as a warning indicating that the product TP contains an allergen, can be provided to the user. Note that the information derived from the product TP and user information is not limited to the information about allergies described above, and may be changed as appropriate depending on the embodiment.

[0067] (6) Sixth Case In one example, the related information 155 may be configured by information obtained by searching the input product TP on the web. For example, the related information 155 may be configured by information showing a comparison of the sales price of the input product TP at a store and the sales price at an EC (electronic commerce) site. The control device 11 may be configured as follows. The selling prices at the store and the EC site may be acquired as appropriate. As an example, information indicating the selling price at the store may be linked to the product TP (identification information, etc.). When identifying the product TP, the control device 11 may acquire information indicating the selling price of the product TP at the store according to the identification result of the product TP. Furthermore, the control device 11 may acquire information indicating the selling price on the EC site by performing a web search according to the identification result of the product TP. The EC site on which the selling price is searched may be selected arbitrarily. As an example, one or more EC sites may be specified in advance. The control device 11 may generate information indicating a comparison between the selling price at the store and the selling price on the EC site based on the acquired information. The configuration of the information indicating the comparison may be determined as appropriate depending on the embodiment. In a simple example, the control device 11 may output the selling price at the store and the selling price on the EC site as the related information 155. In another example, the control device 11 may determine whether the selling price at the store is lower than the selling price on the EC site. If the selling price at the store is lower than the selling price on the EC site, the control device 11 sets information indicating that the selling price at the store is lower than the selling price on the EC site (for example, a message such as "NNN yen cheaper than the AAA site") as related information 155. According to one example of this embodiment, it is possible to provide the user with information about the product TP obtained on the web, such as a comparison result with the selling price on an EC site.

[0068] The information obtained by a web search does not have to be limited to a comparison with the selling price on an e-commerce site, and may be selected appropriately depending on the embodiment. The information search may be performed by any method. In one example, a known search engine may be used to search for information. In another example, the information search may be performed by a general method, such as extracting information containing specific words from a specific site, without using a known search engine. A trained machine learning model, such as a large language model (LLM), may be used to search for information.

[0069] (7) Seventh Case In one example, the related information 155 may be configured with coupon information for which usage conditions are set. The coupon information may be related to the inserted product TP, as in the first and second examples above. Alternatively, the coupon information may be unrelated to the inserted product TP. The usage conditions are conditions for receiving the discount indicated in the coupon information. The usage conditions may be set as appropriate. The usage conditions may include, for example, specifications such as the number of products to be purchased, the purchased product, and an expiration date. According to one example of this embodiment, providing coupon information for which usage conditions are set can be expected to stimulate purchasing motivation among target users.

[0070] [Use of product input records] The product input record 20 may be used for any purpose, such as identifying the product TP, analyzing purchasing behavior in the store, etc. Information processing related to the product input record 20 may be performed in real time as the product input record 20 is generated, or may be performed after the product input record 20 is generated. Information processing related to the product input record 20 may be performed on the control device 11, or may be performed on an external computer (a store terminal, a user terminal, a server device, etc.).

[0071] FIG. 2 is a schematic diagram showing an example of a usage scenario of the product input record 20 according to this embodiment. In one example, the smart grip 1 (control device 11) may directly or indirectly provide the generated product input record 20 to the information processing device 2 via a network. Providing it indirectly means providing it via another computer. The network standard is not particularly limited and may be selected appropriately depending on the embodiment.

[0072] The information processing device 2 is one or more computers configured to execute information processing related to the product input record 20. The information processing device 2 is an example of the external computer. For example, the information processing device 2 may be a store terminal provided in a store, a user terminal carried by a target user, a server device, etc. The information processing related to the product input record 20 may be selected appropriately depending on the embodiment. For example, the information processing related to the product input record 20 may include identifying the product TP, analyzing purchasing behavior (including generating a movement history), providing information, etc. In one example of this embodiment, the information processing executed by the information processing device 2 may include at least one of the following first to third cases.

[0073] The smart grip 1 is an example of a shopping device. In the following examples, for the sake of convenience, it is assumed that the information processing device 2 performs information processing on the product insertion record 20 generated by the smart grip 1. However, the product insertion record processed by the information processing device 2 is not limited to the product insertion record 20 generated by the smart grip 1. The product insertion record may be generated by a shopping device other than the smart grip 1 as long as it includes the event location. The other shopping device may be appropriately configured to be used during shopping and to generate a product insertion record including the event location. The information processing device 2 acquires the product insertion record generated by a shopping device other than the smart grip 1 and performs information processing on the acquired product insertion record. In addition, a part of the product insertion record group processed by the information processing device 2 may be collected from the smart grips 1 of one or more target users, and another part of the product insertion record group may be collected from other shopping devices.

[0074] (A) Case 1 FIG. 3 schematically shows an example of a method of using the product insertion record 20 according to this embodiment (product identification). In one example, the information processing device 2 may acquire the product insertion record 20 of the target user generated by the shopping device (smart grip 1). The product insertion record 20 may include the event position 201 where the target product TP1 was inserted. The target product TP1 corresponds to the product TP. The information processing device 2 may identify the inserted target product TP1 according to the acquired product insertion record 20. Then, the information processing device 2 may output information related to the result of identifying the target product TP1.

[0075] According to one example of the present embodiment, a technology can be provided for identifying an inserted product (target product TP1) by utilizing the product insertion record 20. Note that the method for identifying the target product TP1 according to the product insertion record 20 (event position 201) may be defined as appropriate depending on the embodiment. In one example of the present embodiment, at least one of the following three methods may be adopted as a method for identifying the target product TP1 from the product insertion record 20.

[0076] (A-1) First Identification Method 4 is a schematic diagram illustrating an example of a first method for identifying a product using the product input record 20 according to the present embodiment. In one example, identifying the target product TP1 according to the product input record 20 may consist of identifying the target product TP1 input at the event location 201 by matching the product input record 20 with the map information 30.

[0077] The map information 30 may be configured to include product identification information associated with the display position of the product within the store. In other words, the map information 30 may be configured to indicate the correspondence between the display position of the product within the store and the product identification information. The data format of the map information 30 may be selected appropriately depending on the embodiment. For example, the map information 30 may be configured with multiple records (e.g., a table-format database) that hold various information such as the display position and identification information for each product.

[0078] Furthermore, the method of comparing the product input record 20 and the map information 30 may be defined arbitrarily. For example, the information processing device 2 may compare the display position of each product indicated in the map information 30 with the event position 201 included in the product input record 20. Based on the comparison result, the information processing device 2 may extract products displayed at a position that matches the event position 201 from among the products included in the map information 30. The position that matches the event position 201 may be, for example, the same position as the event position 201 or the position closest to the event position 201. The information processing device 2 may identify the target product TP1 by recognizing that the extracted product is the target product TP1.

[0079] According to one example of this embodiment, the target product TP1 can be appropriately identified by using the map information 30. However, there are cases where it is difficult to uniquely identify the target product TP1 using only the event location 201, such as when multiple types of products are mixed and placed in the same location. In this case, the information processing device 2 may further use information other than the event location 201 to identify the target product TP1. For example, the other information may include the weight of the product. As one example, the smart grip 1 may be configured to be able to measure the weight of the inserted target product TP1 by including a load sensor (weight sensor) in the detection sensor 13 or by further providing a load sensor in addition to the detection sensor 13. The amount of change (increase) in weight when the insertion of the target product TP1 is detected corresponds to the weight of the target product TP1. Product Insertion Record 2 0 may include the measurement result of the weight of the target product TP1. Furthermore, the map information 30 may further include information indicating the weight per unit of each product. When multiple types of products are displayed at the same location, the search using the event location 201 extracts multiple products. The information processing device 2 may extract a product having a weight that matches the weight of the target product TP1 included in the product insertion record 20 from the multiple products displayed at a location that matches the event location 201. Matching the weight of the target product TP1 may mean, for example, that the weight (increase) included in the product insertion record 20 is an integer multiple (including 1) of the weight of the product shown in the map information 30, or that the weight included in the product insertion record 20 is approximately an integer multiple of the weight of the product shown in the map information 30. The information processing device 2 may identify the target product TP1 by recognizing that the extracted product is the target product TP1. This allows the target product TP1 to be appropriately identified even in cases where it is difficult to identify the target product TP1 using only the event location 201, such as when multiple types of products are displayed at the same location.

[0080] (A-2) Second Identification Method FIG. 5 schematically illustrates an example of a second identification method for a commodity using the commodity input record 20 according to the present embodiment. In one example, identifying the target commodity TP1 in accordance with the commodity input record 20 may be configured by acquiring a captured image 400 by an imaging device 40 whose imaging range includes the event position 201 of the commodity input record 20, and performing image analysis on the acquired captured image 400 to identify the target commodity TP1 appearing in the captured image 400. The imaging device 40 may directly or indirectly provide the captured image 400 to the information processing device 2. The information processing device 2 may actively or passively acquire the captured image 400 from the imaging device 40. The second identification method is an example of a method for identifying the commodity TP in cooperation with sensors in a store.

[0081] Similar to image capture device 16 described above, image capture device 40 may be any sensor that captures data in the form of an image or a representation of an image, such as, for example, an RGB camera, a depth sensor, an infrared sensor, radar, LiDAR, etc. The imaging device 40 may include a sensor having higher performance than the imaging device 16, since the imaging device 40 has fewer physical constraints than the imaging device 16. The method of image analysis of the captured image 400 may be the same as the method of image analysis of the captured image 165. The captured image 400 used to identify the target product TP1 may be appropriately acquired from captured images generated at the time when the insertion of the target product TP1 is detected or at a time around that time.

[0082] One or more imaging devices 40 may be installed in a store as appropriate. In the example of FIG. 5, at least three imaging devices 40 are installed in the store. The number of imaging devices 40 to be installed is not limited to the example of FIG. 5 and may be determined as appropriate depending on the embodiment of the store, etc. The imaging device 40 that acquires the captured image 400 may be identified as appropriate from among one or more imaging devices 40 installed in the store according to the event location 201.

[0083] In the example of FIG. 5 , the event location 201 belongs to the imaging range of one imaging device 40. However, when two or more imaging devices 40 are installed in a store, the imaging ranges of the multiple imaging devices 40 may be arranged so that the event location 201 is included in the imaging range. When the multiple imaging devices 40 are arranged so that the event location 201 is included in the imaging range, the information processing device 2 may acquire captured images 400 from each of the multiple imaging devices 40 and perform image analysis on each of the acquired captured images 400. Alternatively, the information processing device 2 may select one or more imaging devices 40 from the multiple imaging devices 40, acquire captured images 400 from each of the selected one or more imaging devices 40, and perform image analysis on each of the acquired captured images 400. The method of selecting the imaging device 40 is not particularly limited and may be appropriately determined depending on the embodiment. For example, a priority may be set in advance for each imaging device 40, and the information processing device 2 may select one or more imaging devices 40 in descending order of priority. Furthermore, for example, the information processing device 2 may select one or more imaging devices 40 based on any criteria, such as whether the event position 201 is close to the center of the imaging range.

[0084] Selecting an imaging device 40 may be replaced with selecting a captured image 400. For example, the information processing device 2 may acquire a captured image 400 from each of the multiple imaging devices 40, and select one or more captured images 400 that satisfy predetermined conditions from the acquired multiple captured images 400. The predetermined conditions may be defined as appropriate to select an image suitable for identifying the target product TP1. For example, the predetermined conditions may include criteria such as high clarity of the image of the target product TP1, a large image of the target product TP1, and the entire target product TP1 being captured. The information processing device 2 may perform image analysis on the selected one or more captured images 400.

[0085] Performing image analysis on all captured images obtained by all imaging devices installed in a store may result in a huge amount of processing. In contrast, according to this embodiment, the range of captured images (captured images 400) to be subjected to image analysis can be narrowed to the range of event detection using the product insertion record 20. This can improve the efficiency of the product (target product TP1) identification process using image analysis. In this example, since the product insertion record 20 includes the identification information 203, it is possible to identify the target user who inserted the identified target product TP1 into the cart B1. This makes it possible to identify the target user corresponding to the obtained identification result without tracking the target user using store sensors.

[0086] (A-3) Third Identification Method FIG. 6 schematically illustrates an example of a third identification method for products using the product input record 20 according to this embodiment. In one example, a load sensor 45 (weight sensor) may be installed on each product shelf PS in a store. Identifying the target product TP1 in accordance with the product input record 20 may include acquiring measurement data 450 from the load sensor 45 installed near the event position 201 in the product input record 20, and identifying the product shelf PS with a weight loss based on the acquired measurement data 450, thereby identifying the target product TP1. The load sensor 45 may be appropriately positioned to measure the weight of the products displayed on the product shelf PS. The load sensor 45 may directly or indirectly provide the measurement data 450 to the information processing device 2. The information processing device 2 may actively or passively acquire the measurement data 450 from the load sensor 45. Similar to the second identification method, the third identification method is also an example of a method for identifying a product TP in cooperation with store sensors.

[0087] The layout of the product shelves PS may be determined as appropriate depending on the embodiment. The product shelves PS may be arranged in any direction, such as horizontally or vertically. Whether the load sensor 45 is present near the event position 201 may be determined based on whether the load sensor 45 (product shelf PS) is within a range 2010 near the event position 201. The range 2010 near the event position 201 and the determination method may be defined as appropriate depending on the embodiment.

[0088] For example, the information processing device 2 may set a range 2010 based on the event position 201, and determine whether the position of the load sensor 45 (product shelf PS) falls within the set range 2010. The position of the load sensor 45 may be identified from the load sensor 45 itself. Alternatively, the position of the product shelf PS may be used as the position of the load sensor 45. In other words, the presence of the load sensor 45 near the event position 201 may be determined by the product shelf PS on which the load sensor 45 is installed being near the event position 201. The information processing device 2 may refer to information indicating the position of the load sensor 45 (product shelf PS) as appropriate. The information indicating the position of the load sensor 45 may be held in the information processing device 2, or may be held in a computer other than the information processing device 2. Furthermore, the attribute values (shape, size, etc.) of the range 2010 may be defined arbitrarily. The event position 201 may be located at an arbitrary position relative to the range 2010. In one example, the event position 201 may be used as the center of the range 2010. However, the location of the event position 201 is not limited to this example. The event position 201 may be located at a position other than the center of the range 2010. The information processing device 2 may determine whether the load sensor 45 is present near the event position 201 based on the result of determining whether the position of the load sensor 45 is within the range 2010. Specifically, the information processing device 2 may determine that the load sensor 45 is present near the event position 201 based on the determination that the load sensor 45 is present within the set range 2010.

[0089] Furthermore, for example, the information processing device 2 may calculate the distance between the position of the load sensor 45 (product shelf PS) and the event position 201. The information processing device 2 may determine whether the calculated distance is less than a threshold value. The information processing device 2 may determine whether the load sensor 45 is present near the event position 201 based on the result of determining whether the calculated distance is less than the threshold value. Specifically, the information processing device 2 may determine that the load sensor 45 is present near the event position 201 based on the determination that the calculated distance is less than the threshold value. On the other hand, the information processing device 2 may determine that the target load sensor 45 is not present near the event position 201 based on the determination that the calculated distance exceeds the threshold value. If the calculated distance is equal to the threshold value, the information processing device 2 may determine that the load sensor 45 is either present or not present near the event position 201. The threshold value may be defined arbitrarily. The range specified by the threshold value is an example of the range 2010.

[0090] Depending on the positional relationship of each product shelf PS, multiple load sensors 45 (product shelves PS) may be present near the event position 201 (i.e., they may belong to a range 2010 near the event position 201). The information processing device 2 may acquire measurement data 450 from each of the multiple load sensors 45 present near the event position 201. In the example of FIG. 6, three load sensors 45 belong to the vicinity of the event position 201. In this case, the information processing device 2 may acquire measurement data 450 from each of the three load sensors 45. The information processing device 2 may identify the target product TP1 by specifying the product shelf PS whose weight has decreased from among the multiple product shelves PS in the acquired measurement data 450.

[0091] The product corresponding to the product shelf PS may be identified as appropriate. For example, the information processing device 2 may identify the product (target product TP1) corresponding to the product shelf PS by referring to information (such as the map information 30) indicating the correspondence between the product shelf PS (load sensor 45) and the product. The information indicating the correspondence between the product shelf PS and the product may be held by the information processing device 2, or may be held by a computer other than the information processing device 2.

[0092] When multiple types of products are displayed on one product shelf PS, the information processing device 2 may acquire the amount of change (amount of decrease) in weight in the measurement data 450 of the load sensor 45 as the weight of the target product TP1. The information processing device 2 may extract a product having a weight that matches the acquired weight from the multiple types of products displayed on the product shelf PS whose weight has decreased. Matching the acquired weight may mean, for example, that the acquired weight (amount of decrease) is an integer multiple (including 1) of the product weight, or that the acquired weight is approximate to an integer multiple of the product weight. The information processing device 2 may identify the target product TP1 by recognizing that the extracted product is the target product TP1.

[0093] Furthermore, the target product TP1 is usually picked up by the target user from the product shelf PS and then put into the basket B1 (basket body B10). Therefore, the weight change caused by taking the target product TP1 from the product shelf PS starts before the time (time 205) when the detection sensor 13 detects the putting of the product TP. On the other hand, depending on the configuration of the product shelf PS, the putting speed of the target product TP1, etc., the end time of the weight change caused by taking the target product TP1 may be before or after the time when the detection sensor 13 detects the putting of the product TP. Therefore, the information processing device 2 may acquire partial data of a predetermined section of the measurement data 450 accumulated by measurement of the load sensor 45, the start time of which is before the time 205 of the product putting record 20. The predetermined section may be defined as appropriate depending on the embodiment. The device 2 may identify the product shelf PS where the weight has decreased, according to the acquired partial data.

[0094] Constant monitoring of all load sensors installed in a store can potentially result in a huge amount of processing. In contrast, according to this embodiment, the range of load sensors 45 to be monitored can be narrowed down to the range of event detection using the product insertion record 20. This can improve the efficiency of the process of identifying a product (target product TP1) based on weight changes. Note that, in this example, the product insertion record 20 also includes the identification information 203, making it possible to identify the target user who inserted the identified target product TP1 into the basket B1. This makes it possible to identify the target user corresponding to the obtained identification result without tracking the target user using sensors in the store.

[0095] (A-4) Other In one example of this embodiment, the information processing device 2 may identify the target product TP1 by employing at least one of the above-described first to third identification methods. The information processing device 2 may also identify the target product TP1 using a plurality of methods. For example, the information processing device 2 may employ all of the above-described first to third identification methods. In this case, the information processing device 2 may determine the final identification result of the target product TP1 by ensembling (for example, by majority voting) the identification results of each method.

[0096] The process of identifying the target product TP1 may be executed at any timing. For example, the information processing device 2 may sequentially identify the target product TP1 corresponding to the generated product insertion record 20 in response to the generation of the product insertion record 20. The information processing device 2 may identify the target product TP1 in real time when detecting the insertion of the target product TP1. Furthermore, for example, the information processing device 2 may execute the identification process for a plurality of product insertion records 20 in a batch. In this case, the identification process for at least some of the target products TP1 may be executed after the fact.

[0097] Furthermore, the identification result of the target product TP1 may be used for any purpose, such as calculating the purchase amount of the target user or managing the inventory of products. In one example, the information processing device 2 may use the identification result of the target product TP1 to calculate the total amount of the products purchased by the target user. Similar to the above-described budget upper limit example, the information processing device 2 may acquire information indicating the price of the target product TP1 according to the identification result of the target product TP1. The information processing device 2 may calculate the total price of all the inserted target products TP1 by adding up the acquired prices for each target product TP1. The calculated total price is an example of information related to the identification result of the target product TP1. The calculated total price may be used for the payment of the target user. For example, the information processing device 2 may be a store terminal or a server device. The information processing device 2 may execute a payment process for the target user using the calculated total price. Furthermore, for example, the information processing device 2 may be independent of a store system including a cash register. The information processing device 2 may notify the store system of the calculated total price, thereby causing the store system to execute a payment process for the target user. Whether the target user has completed the purchasing behavior (i.e., whether all products to be purchased have been placed in the cart) may be determined by any method. For example, whether the target user has completed the purchasing behavior may be determined depending on whether any termination condition is met, such as whether the target user has reached the cash register in the store, the cash register has been operated, or the target user has performed an operation to terminate the purchasing behavior on the shopping device (smart grip 1).

[0098] In another example, the information processing device 2 may use the identification result of the target product TP1 to manage the inventory quantity of the target product TP1. For example, the information processing device 2 may reduce the inventory quantity of the identified target product TP1 in response to identifying the target product TP1. The number to reduce may be determined appropriately. In one example, the information processing device 2 may obtain the change (increase) in weight of the basket B1 due to the addition of the target product TP1 via the product insertion record 20. The information processing device 2 divides the obtained change (increase) in weight by the weight per unit of the target product TP1. The number to be reduced may be determined by determining the number of items to be reduced. Information indicating the inventory quantity of each item may be stored in the information processing device 2 or in another computer accessible to the information processing device 2. After reducing the inventory quantity of the target item TP1, the information processing device 2 may determine whether the inventory quantity of the target item TP1 is less than a threshold value. The threshold value may be arbitrarily defined. If the inventory quantity of the target item TP1 is less than the threshold value, the information processing device 2 may output an instruction to replenish the target item TP1. The instruction to replenish the target item TP1 is an example of information related to the result of identifying the target item TP1. As an example of an instruction to replenish the target item TP1, the information processing device 2 may send a notification (e.g., a message) to a terminal carried by a store manager to prompt the replenishment of the target item TP1. Furthermore, if a robot device capable of automatically transporting the target item TP1 is installed in the store, the information processing device 2 may issue a command to the robot device to replenish the target item TP1. This makes it possible to automate the replenishment of items.

[0099] (B) Second case FIG. 7 schematically shows an example of a method of using the product insertion record 20 according to this embodiment (generation of a movement history). In one example, the information processing device 2 may acquire the product insertion record 20 of the target user generated by the shopping device (smart grip 1). The product insertion record 20 may include the event position 201 where the target product TP1 was inserted. The information processing device 2 may generate a movement history 50 of the target user from the event position 201 of the acquired product insertion record 20. The information processing device 2 may output the generated movement history 50.

[0100] The configuration of the movement history 50 is not particularly limited as long as it includes the event location 201, and may be determined appropriately depending on the embodiment. In one example, the movement history 50 may be configured from the event location 201 of the product insertion record 20 generated in the target user's shopping device (smart grip 1) from the time the target user enters the store through the entrance / exit ET until the time the target user makes a payment at the register RS. In another example, the movement history 50 may further include a trajectory 500 within the store. The trajectory 500 may be configured appropriately to show the progress of the target user's movement within the store.

[0101] The trajectory 500 may be generated by any method. In one example, the information processing device 2 may generate the trajectory 500 by connecting characteristic points of movement, such as connecting the entrance / exit ET and the first event position 201, connecting the kth event position 201 and the k+1th event position 201 (k is an integer equal to or greater than 1), or connecting the last event position 201 and the register RS. In another example, the shopping device (smart grip 1) may measure its position using a positioning sensor (positioning sensor 14) while the target user is in the store, even at times other than when the insertion of a product is detected. The information processing device 2 may further use the result of this position measurement to generate the trajectory 500. For example, the shopping device (smart grip 1) may continue to measure its position using the positioning sensor (positioning sensor 14) for a predetermined period after measuring the event position 201. The predetermined period may be defined as appropriate depending on the embodiment. The information processing device 2 may acquire the measurement results, and generate the trajectory 500 after identifying the progress of movement (movement direction, etc.) from each event position 201 based on the acquired measurement results.

[0102] In the example of FIG. 7, the store has one entrance / exit ET, eight product shelves PS, and three registers RS. A scenario is assumed in which the target user puts products into the cart at six locations, locations L1 to L6 (i.e., six product insertion records 20 are generated, and the event locations 201 of each product insertion record 20 indicate the locations L1 to L6). In one example, the information processing device 2 may generate a trajectory 500 by sequentially connecting the entrance / exit ET, the locations L1 to L6 indicated by each event location 201, and the register RS. Note that the layout of the store is not limited to the example of FIG. 7 and may be changed as appropriate depending on the embodiment. The store may have multiple floors, such as a basement and two or more floors. The store may have multiple entrances / exits ET. The entrance and exit may be provided separately.

[0103] According to an example of the present embodiment, a technology is provided for monitoring a movement history 50 of a target user using a product input record 20. The movement history 50 may be used for any purpose, such as analyzing customer traffic patterns within a store. The generated movement history 50 may be output as is. Furthermore, the information processing device 2 may generate analysis results, such as a heat map, by analyzing the movement histories 50 of multiple users. The heat map may be generated as appropriate to indicate the frequency of customer stays (e.g., visit rate per area, transition probability, etc.). The information processing device 2 may output the generated analysis results. Based on the analysis results, areas in the store where customers frequently pass and areas where customers rarely pass can be identified. In one example, the analysis results may be reflected in determining the store layout (e.g., the placement and size of product shelves PS). For example, the store layout may be determined to optimize customer traffic based on the analysis results, such as by widening aisles with high traffic and reducing aisles with low traffic. The layout optimization may be performed manually or by computerized information processing, such as by using a trained machine learning model. The information processing related to the layout optimization may be executed by either the information processing device 2 or another computer. The information processing device 2 may output the results of the layout optimization.

[0104] (C) Third case FIG. 8 schematically shows an example of a method of using the product insertion record 20 according to this embodiment (locating a location). In one example, the information processing device 2 may acquire the product insertion record 20 of the target user generated by the shopping device (smart grip 1). The product insertion record 20 may include the event location 201 where the product (target product TP1) was inserted. The information processing device 2 may acquire a purchase record 35 of the product purchased by the target user. The information processing device 2 may identify the location where the target user placed the purchased product in the basket (basket B1) by comparing the acquired product insertion record 20 and purchase record 35 with the map information 30. The information processing device 2 may output information related to the identified location.

[0105] The configuration of the purchase record 35 is not particularly limited as long as it includes identification information of the purchased item, and may be determined appropriately depending on the embodiment. The data format of the purchase record 35 may be selected appropriately depending on the embodiment. For example, the purchase record 35 may be configured to hold various information such as identification information for each item. The purchase record 35 may be generated appropriately when paying for the purchased item. For example, the purchase record 35 may be generated by a cash register. A POS (Point of Sale) cash register may be used as the cash register in the store. The purchase record 35 may be generated by a POS cash register. That is, the purchase record 35 may be composed of POS information. The POS information may be composed in a known manner.

[0106] If the information processing device 2 is a store terminal, the POS register may be included in the information processing device 2 or may be connected to the information processing device 2. The information processing device 2 may appropriately acquire the purchase record 35 from the POS register. If the information processing device 2 includes a POS register, the purchase record 35 may be generated by the information processing device 2 when the target user makes a payment. Furthermore, if the information processing device 2 is independent of a store system (such as a POS system) that includes a POS register, the store system may generate the purchase record 35 when the target user makes a payment. The store system may provide the generated purchase record 35 to the information processing device 2 directly or indirectly.

[0107] The purchased items have already been identified by the identification information in the purchase record 35. When comparing the item insertion record 20 and the purchase record 35 with the map information 30, the information processing device 2 may compare the location on the map information 30 of the purchased items identified by the purchase record 35 with the event location 201. Based on the comparison result, the information processing device 2 may identify the location where the target user put the purchased items into the cart.

[0108] In some cases, the same product may be placed in multiple locations within a store (e.g., permanent shelves, special shelves, etc.) for reasons such as strengthening sales. In response to this, according to one example of the present embodiment, a technology can be provided that uses the product input record 20 to identify the location where a target user placed a purchased product in a cart. That is, after identifying the purchased product using the purchase record 35, the location where the purchased product was placed in the cart can be identified using the product input record 20 (event location 201) and map information 30. As a result, even when the same product is placed in multiple locations within a store, it is possible to analyze from which location each target user obtained the purchased product. This analysis can measure the effect of each location within the store in encouraging purchases, such as the effect of special shelves in encouraging purchases.

[0109] (D) Other At least one of the information processing of identifying a product (first example), generating a movement history (second example), and specifying a location (third example) may be executed by a different computer. For example, a different computer may be used as the information processing device 2 for each information processing. Furthermore, at least two of the information processing of the first to third examples may be executed on the same computer. The computer that executes the information processing of each example may be selected as appropriate.

[0110] §2 Configuration example [Hardware configuration] (Smart Grip) 1, the smart grip 1 includes a computer to which a control device 11, a memory unit 12, a detection sensor 13, a positioning sensor 14, an output device 15, an imaging device 16, and a communication module 17 are electrically connected. In the smart grip 1, at least a part of the computer may be housed in a box.

[0111] The control device 11 includes a processor P and is configured to execute information processing based on programs and various data. The processor P is a hardware processor and is an example of a processor resource. The type of processor P may be selected appropriately depending on the embodiment. For example, the processor P may be a CPU (Central Processing Unit), a MIMO (Micom), or a ROM (Multi-Processing Unit). The device may be configured using a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or the like.

[0112] The storage unit 12 may be configured with a hard disk drive, a solid state drive, a semiconductor memory, or the like. The storage unit 12 is an example of a memory resource. In this embodiment, the storage unit 12 stores various information such as a program 81 and identification information 125. The program 81 is a program for causing the control device 11 to execute information processing related to the generation of the product insertion record 20 (see FIG. 14, described below). The program 81 includes a series of instructions for the information processing. For example, when the control device 11 executes information processing for identifying the product TP through image analysis of the captured image 165, the program 81 may further include a series of instructions for information processing related to the identification of the product TP (see FIG. 15, described below). For example, when the control device 11 executes information processing for outputting the related information 155, the program 81 may further include a series of instructions for information processing related to the output of the related information 155 (see FIG. 16, described below). The identification information 125 may be omitted.

[0113] At least one of the program 81 and the identification information 125 may be stored in a storage medium SR1 instead of or together with the storage unit 12. The storage medium SR1 is configured to store various information (stored programs, etc.) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The storage unit 12 and the storage medium SR1 are examples of non-transitory storage media. The smart grip 1 (computer) stores at least one of the program 81 and the identification information 125 in the storage medium. The information may be acquired from the storage medium SR1. The storage medium SR1 may be a disk-type storage medium (CD, DVD, etc.), or may be a non-disk-type storage medium such as a semiconductor memory (flash memory, etc.). Any drive device may be used to read the information stored in the storage medium SR1. The type of drive device may be selected depending on the storage medium SR1.

[0114] The detection sensor 13 is configured to detect the insertion of a product TP into the basket B1. As long as the detection sensor 13 can detect the insertion of the product TP, the type of the detection sensor 13 is not particularly limited and may be selected appropriately depending on the embodiment. For example, the detection sensor 13 may be configured with one or more load sensors (weight sensors). In this case, the detection sensor 13 can detect the insertion of the product TP by detecting a change (increase) in weight due to the insertion of the product TP. The detection sensor 13 may be placed at any location where a change in load occurs due to the insertion of the product TP (i.e., where a change in weight due to the insertion of the product TP can be observed).

[0115] Furthermore, for example, the detection sensor 13 may be configured by one or more imaging devices. The imaging device may be a device that captures an image or an image representation, such as an RGB camera, a depth sensor, an infrared sensor, a radar, or a LiDAR. The detection sensor 13 may include any sensor that acquires data by detecting a product TP. In this case, the detection sensor 13 may be the same as the imaging device 16, or may be provided separately from the imaging device 16. The detection sensor 13 can detect the insertion of the product TP by capturing the product TP in an image captured by the imaging device. The detection sensor 13 may be placed in any location where the product TP is within its imaging range at least during the period from when the user picks up the product TP to when the product TP is placed in the basket B1, and / or after the product TP is placed in the basket B1. Note that when reducing the processing load and power consumption by limiting the period during which the imaging device 16 is activated as described above, it is preferable to use, as the detection sensor 13, a sensor of a different type from the imaging device 16 that has a lower processing load and power consumption than the imaging device 16.

[0116] Furthermore, for example, the detection sensor 13 may be composed of one or more contact sensors. In this case, the detection sensor 13 can detect the insertion of the product TP by contact with the inserted product TP or another object (such as a switch member) displaced by the insertion of the product TP. The detection sensor 13 may be placed at any location where the inserted product TP comes into direct or indirect contact. Indirect contact may include contact with another object displaced by the insertion of the product TP. The detection sensor 13 may be composed of multiple types of sensors.

[0117] The positioning sensor 14 is configured to measure a position. As long as the position can be measured, the type of the positioning sensor 14 is not particularly limited and may be appropriately selected depending on the embodiment. For example, the positioning sensor 14 may be configured by a GPS (Global Positioning System) sensor, a GNSS (Global Navigation Satellite System) sensor, or the like. Also, for example, The positioning sensor 14 may be configured with a beacon receiver. In this case, a store may be equipped with multiple beacon transmitters. The position of the smart grip 1 may be appropriately measured (estimated) from the strength of the beacon signal received by the positioning sensor 14 (beacon receiver). For example, the positioning sensor 14 may estimate the distance to each beacon transmitter according to the strength of the signal received by the beacon receiver. The positioning sensor 14 may measure the position of the smart grip 1 from the estimated distance to each beacon transmitter by triangulation. At least a part of the process of measuring this position may be executed by the control device 11.

[0118] The communication module 17 is configured to perform wired or wireless communication via a network. The communication module 17 may be configured, for example, by a wired LAN (Local Area Network) module, a wireless LAN module, or the like. The communication module 17 may also include a close-proximity wireless communication module. The network standard is not particularly limited and may be selected appropriately depending on the embodiment. For example, the type of network may be selected appropriately from the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, or the like. The smart grip 1 (controller The control device 11 can exchange data with other computers (for example, the information processing device 2, etc.) via the communication module 17.

[0119] Note that, with regard to the specific hardware configuration of the smart grip 1 (computer), components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control device 11 may include multiple hardware processors (processor P). At least one of the output device 15, the imaging device 16, and the communication module 17 may be omitted. At least one of the program 81 and the identification information 125 may be stored in an external storage device such as a NAS. The external storage device is also an example of a non-transitory storage medium. The smart grip 1 may be configured to be operable by further including an input device. The input device may be configured to accept information input. The input device may be configured, for example, by a keyboard, a touch panel, a control, a microphone, etc. The input device and the output device 15 may be integrated into a touch panel display, etc. The output device 15 is an example of a feedback module. The feedback module may be configured to provide information to the user visually, aurally, and / or tactilely. The smart grip 1 may further include a feedback module in addition to the output device 15. For example, the output device 15 may further include a haptic feedback module configured to provide haptic feedback of information. The haptic feedback module may be configured with any device capable of transmitting information haptically. For example, the haptic feedback module may be configured with a servo motor, a vibration motor, or a combination thereof. When the haptic feedback module is provided, the smart grip 1 may output any information, such as a notification of the output of the related information 155, by haptic feedback. The smart grip 1 may include multiple computers. The hardware configurations of the computers may or may not be the same. Furthermore, the computer included in the smart grip 1 may be a dedicated computer, as well as a general-purpose PC (Personal Computer), a tablet PC, a terminal device ( For example, the computer included in the smart grip 1 may be configured as a smartphone. The smart grip 1 may be constructed by mounting a smartphone on a housing (attachment).

[0120] (Smart Grip structure) 9 and 10 are a side view and a plan view schematically illustrating an example of the structure of the smart grip 1 according to this embodiment. In one example, the smart grip 1 may further include a housing 90 that houses the detection sensor 13, the positioning sensor 14, and the control device 11. The housing 90 may include one or more grooves (groove 915, groove 935, etc.) for hooking the handle B11 provided on the basket B1. According to this example of the present embodiment, it is possible to provide a smart grip 1 that can be used as a basket handle by replacing the handle (handle B11) of an existing basket.

[0121] The structure of the smart grip 1, such as the shape of the housing 90, the size of the housing 90, the arrangement of each component (such as the control device 11) within the housing 90, the number of grooves, the shape of the grooves, and the arrangement of the grooves, may be determined appropriately depending on the embodiment. In the example of Figures 9 and 10, a situation is assumed in which the detection sensor 13 is configured with a pair of load sensors 130, the output device 15 includes a display 151, and the imaging device 16 is provided separately from the detection sensor 13. A situation is assumed in which the housing 90 has a pair of grooves (915, 935). It is also assumed in which the smart grip 1 has a vibration motor 180 and a servo motor 185 as a haptic feedback module.

[0122] For ease of explanation, the left-right direction in Figs. 9 and 10 will be referred to as the "front-rear direction." The left direction in Figs. 9 and 10 will be referred to as the "front direction," and the right direction in Figs. 9 and 10 will be referred to as the "rear direction." The direction perpendicular to the paper surface of Fig. 9 and the up-down direction in Fig. 10 will be referred to as the "left-right direction." The up direction in Fig. 10 will be referred to as the "right direction," and the down direction in Fig. 10 will be referred to as the "left direction." Up-down in Fig. 9 10 and the direction perpendicular to the paper surface of Fig. 10 will be referred to as the "upper direction." The upper direction in Fig. 9 will be referred to as the "upper direction," and the lower direction in Fig. 9 will be referred to as the "lower direction."

[0123] In one example, the housing 90 may include a first portion 91, a second portion 92, and a third portion 93. In one example of each figure, the first portion 91, the second portion 92, and the third portion 93 are aligned in a line in the front-to-rear direction from the rear. Each portion (91, 92, 93) is formed in a rectangular parallelepiped shape. The first portion 91 and the third portion 93 may have substantially the same shape. The width (length in the left-to-right direction) of the second portion 92 may be formed shorter than the first portion 91 and the third portion 93. As a result, the second portion 92 may constitute a portion to be held by the user. In one example of each figure, the second portion 92 extends along the front-to-rear direction. Therefore, the user can hold the smart grip 1 by grasping the second portion 92 with their hand along the front-to-rear direction. The total length of the circumference of the second portion 92 around the axis in the front-to-rear direction may be determined within a range of length that can be grasped by hand.

[0124] Side walls may be provided at the horizontal (left-right and front-rear) ends of each section (91, 92, 93). A bottom surface may be provided at the lower end of each section (91, 92, 93). This may form an internal space in each section (91, 92, 93) surrounded by the side walls and accommodating the computer section of the smart grip 1. The internal spaces of the first section 91 and the second section 92 may be at least partially continuous. The internal spaces of the second section 92 and the third section 93 may also be at least partially continuous. The upper end of each section (91, 92, 93) may be closed by a lid or the like, or may be open.

[0125] A groove 915 may be provided on the rear end side of the first portion 91. In one example shown in each figure, the groove 915 is formed by recesses provided on side walls opposing each other in the left-right direction of the first portion 91. Each recess is formed in a rectangular shape. The recesses are aligned in the front-rear direction to form the groove 915. Even when the top end of the first portion 91 is closed, the portion of the groove 915 on the top end surface of the first portion 91 may be open.

[0126] Similarly, a groove 935 may be provided on the front end side of the third portion 93. In one example shown in each figure, the groove 935 is formed by a recess provided on each of the side walls of the third portion 93 facing each other in the left-right direction. Each recess is formed in a rectangular shape. The recesses are aligned in the front-to-rear direction to form the groove 935. Even when the top end of the third portion 93 is closed, the portion of the groove 935 on the top end surface of the third portion 93 may be open. The width (length in the front-to-rear direction) of each groove (915, 935) may be formed to be the same as or slightly larger than the width of the corresponding handle portion B11 of the basket B1. In this way, each groove (915, 935) may be configured to be able to receive the handle portion B11 of the basket B1.

[0127] The pair of load sensors 130 may be disposed in the internal spaces of the first portion 91 and the third portion 93, respectively, in accordance with the positions of the pair of grooves (915, 935). When the portion of the handle portion B11 that is hooked onto the smart grip 1 is formed linearly, for example, the recesses constituting each groove (915, 935) may be formed so that the lower end of the recess is positioned slightly lower than the measuring portion of the load sensor 130 (the contact surface with the handle portion B11). This allows each handle portion B11 to come into contact with the measuring portion of the load sensor 130 when hooked onto each groove (915, 935). As a result, each load sensor 130 can observe a change in weight when a product TP is placed into the basket B1 (basket body B10). The location of this load sensor 130 is an example of a location where a change in load occurs when a product TP is placed. However, the positional relationship between each groove (915, 935) and each load sensor 130 is not limited to this example, and may be changed as appropriate depending on the embodiment, such as the shape of the handle portion B11. For example, each handle portion B11 may be formed so that the portion located in the internal space of each portion (91, 93) curves downward when hung on the smart grip 1. In this case, the recesses constituting each groove (915, 935) are positioned so that the lower end of the recess is positioned above the measuring portion of the load sensor 130. The sensor may be configured to be placed in a position where the sensor is located.

[0128] In one example of each figure, the imaging device 16 is disposed in the internal space of the first portion 91. For example, an opening may be provided in the bottom surface of the first portion 91 to align with the position of the imaging unit 161 (e.g., light receiving unit) of the imaging device 16. The imaging device 16 may be disposed so that the imaging unit 161 is exposed from the bottom surface of the first portion 91 by fitting the imaging unit 161 into the opening in the bottom surface. This allows the imaging device 16 to capture an image of the area below the smart grip 1. Therefore, when the smart grip 1 is attached to the basket B1 by hooking each handle portion B11 of the basket B1 into each groove (915, 935), the imaging device 16 can include the storage section of the basket body B10 in its imaging range. As a result, when a product TP is inserted into the basket body B10, the imaging device 16 can observe the inserted product TP. The location of the imaging device 16 is an example of a location where the inserted product TP can be observed.

[0129] In one example of each figure, the vibration motor 180 and the servo motor 185 are disposed in the internal space of the second portion 92. In this example, the second portion 92 constitutes the portion that is gripped by the user. Therefore, by disposing the vibration motor 180 and the servo motor 185 in the second portion 92, it is possible to provide tactile feedback to the user using the smart grip 1. The types of the vibration motor 180 and the servo motor 185 are not particularly limited and may be selected appropriately depending on the embodiment.

[0130] The vibration motor 180 is configured to vibrate to provide tactile feedback to the user. That is, the vibration motor 180 can provide feedback to the user through vibration. Feedback through vibration may be used as desired. For example, when the smart grip 1 outputs the related information 155 to the display 151, the smart grip 1 may notify the user that the related information 155 has been output by vibrating the vibration motor 180.

[0131] The servo motor 185 may include a shaft 1851 and an arm 1852. The shaft 1851 forms a rotation axis. The arm 1852 is attached to the shaft 1851 and rotates around the axis of the shaft 1851 in response to the rotation of the shaft 1851. The arm 1852 may be configured so that a portion (e.g., a tip) of the arm 1852 comes into contact with the hand of a user holding the second portion 92 when the arm 1852 rotates clockwise or counterclockwise. Alternatively, the arm 1852 may be configured so that the tip of the arm 1852 hits the side wall of the second portion 92 when the arm 1852 rotates clockwise or counterclockwise, and the resulting vibration is transmitted to the user's hand holding the second portion 92. This configuration allows the servo motor 185 to provide feedback in a specific direction. In one example shown in each figure, the arm 1852 rotates clockwise to provide feedback to the left side of the user's hand holding the second portion 92. On the other hand, by rotating the arm 1852 counterclockwise, feedback can be provided to the right side of the user's hand gripping the second portion 92. This feedback indicating a specific direction may be used arbitrarily. For example, if the information output to the display 151 includes directional information (e.g., the location of a product), the smart grip 1 may indicate the direction of the target to the user by outputting feedback indicating the target direction using the servo motor 185. In the example shown in each figure, the arm 1852 is formed in a flat plate shape. However, the shape of the arm 1852 is not limited to this example. As long as tactile feedback can be provided in a specific direction by rotating the arm 1852, the shape of the arm 1852 is not particularly limited and may be selected appropriately depending on the embodiment.

[0132] In the example shown in each figure, the control device 11, the storage unit 12, the positioning sensor 14, and the communication module 17 are housed in a box 1100. The box 1100 is formed in a rectangular parallelepiped shape and is disposed in the internal space of the third section 93. A display 151 is disposed on the top surface of the box 1100. When the top end of the third portion 93 is closed, the display 151 may be positioned so that the screen is exposed from the top end surface of the third portion 93. In this example, when the user holds the smart grip 1 properly, the third portion 93 is located in front of the second portion 92. Therefore, while using the smart grip 1, the screen of the display 151 is located slightly in front of the user's gripping hand. This allows the user to easily view the screen of the display 151 without it being obstructed by other body parts while using the smart grip 1.

[0133] The structure of the smart grip 1 is not limited to the examples shown in FIGS. 9 and 10 and may be modified as appropriate depending on the embodiment. For example, the shape of the housing 90 may be modified as appropriate. In another example, the housing of the smart grip 1 may be formed in a ring shape, and a portion of the ring may be opened and closed by attaching and detaching the ends. As a result, the housing of the smart grip 1 may be configured to replace a handle by receiving the handle portion B11 of the basket B1 inside the ring formed by the housing. In yet another example, the housing of the smart grip 1 may be formed to receive a terminal device such as a smartphone. In this case, the smart grip 1 may be constructed by attaching the terminal device to the housing. Furthermore, the positions of the grooves (915, 935) may be modified as appropriate. The number of grooves provided in the housing (housing 90) is not limited to two and may be one, or three or more. Furthermore, the arrangement of each component may be modified as appropriate. The arrangement of the load sensor 130 may be modified as appropriate depending on the structure of the housing (housing 90). When a sensor other than the load sensor 130 is used as the detection sensor 13, the detection sensor 13 may be appropriately placed in a location where it can detect the insertion of the product TP. The location of the imaging device 16 may be changed appropriately depending on the embodiment. The imaging device 16 may be placed in a location other than the internal space of the first part 91. The components housed in the box 1100 may be changed appropriately. The box 1100 may be placed in a location other than the internal space of the third part 93. The box 1100 may be omitted.

[0134] (Information processing device) 11 schematically shows an example of the hardware configuration of an information processing device 2 according to this embodiment. The information processing device 2 according to this embodiment is a computer in which a control unit 21, a storage unit 22, a communication module 23, an input device 24, an output device 25, and a drive 26 are electrically connected.

[0135] The control unit 21 is configured to execute information processing based on programs and various data by including a CPU, which is a hardware processor, a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The control unit 21 (CPU) is an example of a processor resource.

[0136] The memory unit 22 may be configured with a hard disk drive, a solid state drive, a semiconductor memory, etc. The memory unit 22, RAM, and ROM are examples of memory resources. In this embodiment, the memory unit 22 stores various information such as a program 82. The program 82 is a program for causing the information processing device 2 to execute information processing related to the product insertion record 20. The program 82 includes a series of instructions for the information processing. For example, when causing the information processing device 2 to execute a product identification process, the program 82 may include a series of instructions for information processing related to product identification (see FIG. 17, which will be described later). For example, when causing the information processing device 2 to execute a movement history generation process, the program 82 may include a series of instructions for information processing related to movement history generation (see FIG. 18, which will be described later). For example, when causing the information processing device 2 to execute a location identification process, the program 82 may include a series of instructions for information processing related to location identification (see FIG. 19, which will be described later).

[0137] The communication module 23 is configured to perform wired or wireless communication over a network. The communication module 23 may be configured, for example, by a wired LAN module, a wireless LAN module, etc. The information processing device 2 can exchange data with other computers (for example, the computer portion of the smart grip 1, etc.) via the communication module 23.

[0138] The input device 24 is configured to accept input of information. The input device 24 may be configured, for example, by a mouse, a keyboard, an operator, etc. The output device 25 is configured to output information. The output device 25 may be configured, for example, by a display, a speaker, etc. An operator can operate the information processing device 2 by using the input device 24 and the output device 25. The input device 24 and the output device 25 may be connected via an external interface. The external interface may be appropriately configured to connect to an external device by wire or wirelessly, for example, by a USB (Universal Serial Bus) port, a dedicated port, etc. The input device 24 and the output device 25 may be integrated into one device such as a touch panel display.

[0139] The drive 26 is a device for reading various information such as programs stored in the storage medium SR2. The program 82 may be stored in the storage medium SR2 instead of or together with the storage unit 22. The information processing device 2 may acquire the program 82 from the storage medium SR2. The storage medium SR2 may be configured similarly to the storage medium SR1. The storage unit 22 and the storage medium SR2 are examples of non-transitory storage media. The type of the drive 26 may be selected according to the storage medium SR2. The drive 26 may be connected via an external interface.

[0140] Note that, with regard to the specific hardware configuration of the information processing device 2, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, DSP, GPU, ASIC, etc. At least one of the communication module 23, the input device 24, the output device 25, and the drive 26 may be omitted. The program 82 may be stored in an external storage device such as a NAS. An external storage device is also an example of a non-transitory storage medium. The information processing device 2 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. Furthermore, the information processing device 2 may be a dedicated computer, a general-purpose server device, a general-purpose PC, a tablet PC, a terminal device, etc.

[0141] [Software configuration] (Smart Grip) 12 schematically shows an example of the software configuration of the smart grip 1 (control device 11) according to this embodiment. The control device 11 of the smart grip 1 executes instructions contained in a program 81 stored in the storage unit 12 using the processor P. As a result, the smart grip 1 (control device 11) operates as a computer that includes a detection unit 111, a generation unit 112, and a storage processing unit 113 as software modules. That is, in this embodiment, each software module of the smart grip 1 is realized by the control device 11 (processor P).

[0142] The detection unit 111 is configured to detect the insertion of the product TP according to the measurement result of the detection sensor 13. The generation unit 112 is configured to generate a product insertion record 20 including the event position 201 where the product TP was inserted, which is the event position 201 measured by the positioning sensor 14, when the insertion of the product TP is detected by the detection sensor 13. The storage processing unit 113 is configured to store the generated product insertion record 20.

[0143] When the control device 11 is configured to execute the output process of the related information 155, the smart grip 1 (control device 11) may further include an output processing unit 114 as a software module. The output processing unit 114 may be configured to output related information 155 related to the insertion of the product TP from the output device 15 when the insertion of the product TP is detected by the detection sensor 13. When the output of the related information 155 is not executed, the output processing unit 114 may be omitted from the software configuration of the smart grip 1.

[0144] When the control device 11 is configured to perform the identification process of the product TP by image analysis, the smart grip 1 (control device 11) may further include an image acquisition unit 115, an identification unit 116, and a result output unit 117 as software modules. The image acquisition unit 115 is configured to acquire a captured image 165 from the imaging device 16 when the detection sensor 13 detects the insertion of the product TP. The identification unit 116 is configured to identify the product TP appearing in the captured image 165 by performing image analysis on the acquired captured image 165. The result output unit 117 is configured to output information related to the identification result of the product TP. When the identification process of the product TP is not performed on the smart grip 1, the image acquisition unit 115, the identification unit 116, and the result output unit 117 may be omitted from the software configuration of the smart grip 1.

[0145] (Information processing device) 13 schematically shows an example of the software configuration of the information processing device 2 (controller 21) according to this embodiment. The controller 21 of the information processing device 2 executes instructions included in the program 82 stored in the storage unit 22 using the CPU. As a result, the information processing device 2 (controller 21) operates as a computer including an acquisition unit 211, an information processing unit 212, and an output processing unit 213 as software modules. That is, in this embodiment, each software module of the information processing device 2 is realized by the controller 21 (CPU).

[0146] The acquisition unit 211 is configured to acquire the product insertion record 20 of the target user generated by the shopping device (smart grip 1). The product insertion record 20 may include the event position 201 where the target product TP1 was inserted. The information processing unit 212 is configured to execute information processing on the product insertion record 20. The output processing unit 213 is configured to output information on the execution result of the information processing on the product insertion record 20.

[0147] When the control unit 21 is configured to execute a product identification process, the information processing unit 212 may include an identification unit 2121. The identification unit 2121 is configured to identify the inserted target product TP1 according to the acquired product insertion record 20. Any of the above-described first to third identification methods may be adopted as the identification method. In response to this, the output processing unit 213 may be configured to output information related to the result of identifying the target product TP1.

[0148] When the control unit 21 is configured to execute a movement history generation process, the information processing unit 212 may include a history generation unit 2122. The history generation unit 2122 is configured to generate a movement history 50 of the target user from the event position 201 of the acquired product insertion record 20. In response to this, the output processing unit 213 may be configured to output the generated movement history 50.

[0149] When the control unit 21 is configured to execute location characteristic processing, the acquisition unit 211 may be further configured to acquire a purchase record 35 of the purchased items of the target user. The information processing unit 212 may include a location identification unit 2123. The location identification unit 2123 is configured to identify the location where the target user put the purchased items into the basket (basket B1) by comparing the acquired product input record 20 and purchase record 35 with the map information 30. In response to this, the output processing unit 213 may be configured to output information related to the identified location. In the information processing unit 212, at least the identification unit 2121, the history generation unit 2122, and the location identification unit 2123 Either one may be omitted.

[0150] (others) In this embodiment, an example is described in which each software module of the smart grip 1 and the information processing device 2 is implemented by a general-purpose processor (processor P, CPU). However, some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may be implemented as a hardware module. With regard to the software configuration of the smart grip 1 and the information processing device 2, modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0151] §3 Example of operation [Smart Grip] (Generating product input records) FIG. 14 is a flowchart showing an example of a processing procedure for generating the product insertion record 20 by the control device 11 of the smart grip 1 according to this embodiment. The processing procedure for generating the product insertion record 20 described below is an example of a generation method (information processing method) executed by a computer (control device 11). However, the processing procedure for generating the product insertion record 20 described below is merely an example, and each step may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added to the processing procedure for generating the product insertion record 20 described below as appropriate depending on the embodiment.

[0152] In step S101, the control device 11 operates as the detection unit 111. The control device 11 acquires a measurement result from the detection sensor 13. The control device 11 detects the insertion of the product TP according to the measurement result of the detection sensor 13. If the insertion of the product TP is detected, the control device 11 proceeds to the next step S102. If the insertion of the product TP is not detected, the control device 11 repeatedly executes the process of step S101 and waits until the insertion of the product TP is detected.

[0153] In step S102, the control device 11 operates as the generation unit 112. The control device 11 acquires the event position 201 where the product TP was inserted by referring to the measurement results of the positioning sensor 14. The control device 11 generates a product insertion record 20 including the acquired event position 201. After generating the product insertion record 20, the control device 11 proceeds to the next step S103.

[0154] In step S103, the control device 11 operates as the storage processing unit 113. The control device 11 stores the generated product insertion record 20. In one example, the control device 11 may store the generated product insertion record 20 in the memory unit 12. In another example, the control device 11 may store the product insertion record 20 in an external memory area by transmitting the product insertion record 20 to an external computer such as the information processing device 2. After storing the product insertion record 20, the control device 11 proceeds to the next step S104.

[0155] In step S104, the control device 11 determines whether or not to terminate the processing. The criteria for the determination may be set arbitrarily. In one example, the control device 11 may determine not to terminate the processing until an instruction to terminate is given. On the other hand, when an instruction to terminate is given, the control device 11 may determine to terminate the processing. The instruction to terminate may be given in any manner. In another example, if the target user is present in the store, the control device 11 may determine not to terminate the processing. On the other hand, if the target user is not present in the store (has left the store), the control device 11 may determine to terminate the processing. Whether or not the target user is present in the store may be determined appropriately based on the measurement results of the positioning sensor 14, etc.

[0156] If it is determined not to end the processing, the control device 11 returns the processing to step S101 and executes the processing again from step S101. If it is determined to end the processing, the control device 11 ends the processing procedure related to the generation of the product input record 20 according to this operation example. Note that the timing for ending the processing is not limited to this example. The control device 11 may end the processing procedure related to the generation of the product input record 20 at any timing.

[0157] In one example, the control device 11 may execute a series of processes from step S101 to step S104 in real time. The order of the processes in each step may be changed as appropriate. In another example, the control device 11 may execute the processes in step S101 and step S102 multiple times, and then execute the process in step S103 all at once.

[0158] According to this embodiment, by generating the product insertion record 20 using the detection sensor 13 and the positioning sensor 14 through the process of step S102, it is possible to monitor the purchasing behavior of users without installing sensors in the store. This is expected to simplify the monitoring of purchasing behavior. Furthermore, the smart grip 1 according to this embodiment may be used together with sensors installed in the store to monitor the purchasing behavior of users. In this case, as in the second identification method, the third identification method, etc., by using the product insertion record 20, it is possible to improve the efficiency of processing the sensing data obtained by the sensors installed in the store. This is expected to simplify the monitoring of purchasing behavior.

[0159] (Product Identification) FIG. 15 is a flowchart showing an example of a processing procedure for identifying a product TP by the control device 11 of the smart grip 1 according to this embodiment. The following processing procedure for identifying a product TP is an example of an identification method (information processing method) executed by a computer (control device 11). However, the following processing procedure for identifying a product TP is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure for identifying a product TP can be omitted, replaced, or added as appropriate depending on the embodiment.

[0160] In step S201, the control device 11 operates as the detection unit 111. The processing in step S201 may be the same as that in step S101 above. If the insertion of the product TP is detected, the control device 11 proceeds to the next step S202. If the insertion of the product TP is not detected, the control device 11 repeatedly executes the processing in step S201 and waits until the insertion of the product TP is detected.

[0161] In step S202, the control device 11 operates as the image acquisition unit 115. That is, the control device 11 acquires the captured image 165 from the imaging device 16. In step S203, the control device 11 operates as the identification unit 116. That is, the control device 11 identifies the product TP appearing in the captured image 165 by performing image analysis on the acquired captured image 165. In step S204, the control device 11 operates as the result output unit 117. That is, the control device 11 outputs information related to the identification result of the product TP. The output destination and content of the information may each be selected appropriately depending on the embodiment. After outputting the information related to the identification result, the control device 11 proceeds to the next step S205.

[0162] In step S205, the control device 11 determines whether or not to end the process. The process in step S205 may be the same as step S104 above. If it is determined not to end the process, the control device 11 returns the process to step S201 and executes the process again from step S201. If it is determined to end the process, the control device 11 ends the processing procedure related to the identification of the product TP according to this operation example. Note that the timing for ending the process is not limited to this example. The control device 11 may, at any timing, execute the processing related to the identification of the product TP. The procedure may then be terminated.

[0163] According to one example of the present embodiment, the inserted product TP can be identified on the smart grip 1 by the processes of steps S202 and S203. In one example, the control device 11 may execute a series of processes from steps S201 to S205 in real time. The order of the steps may be changed as appropriate. In another example, the control device 11 may execute the processes of steps S201 to S203 multiple times, and then execute the process of step S204 all at once. Furthermore, when the control device 11 executes the processes of both steps S101 to S104 and steps S201 to S205, the process of step S201 may be executed in common with step S101. The process of step S205 may be executed in common with step S104. The processes of steps S202 to S204 may be executed at any timing after step S101 and before step S104.

[0164] (Related information output) FIG. 16 is a flowchart showing an example of a processing procedure for outputting the related information 155 by the control device 11 of the smart grip 1 according to this embodiment. The processing procedure for outputting the related information 155 below is an example of an output method (information processing method) executed by a computer (control device 11). However, the processing procedure for outputting the related information 155 below is merely an example, and each step may be changed as much as possible. Furthermore, steps may be omitted, replaced, or added to the processing procedure for outputting the related information 155 below as appropriate depending on the embodiment.

[0165] In step S301, the control device 11 operates as the detection unit 111. The processing of step S301 may be the same as that of step S101 described above. If the insertion of the product TP is detected, the control device 11 proceeds to the next step S302. If the insertion of the product TP is not detected, the control device 11 repeatedly executes the processing of step S301 and waits until the insertion of the product TP is detected.

[0166] In step S302, the control device 11 operates as the output processing unit 114. That is, the control device 11 outputs related information 155 related to the insertion of the product TP from the output device 15. In one example, the related information 155 may be configured from at least one of the first to seventh cases described above. For example, when the first case is adopted, the related information 155 may include at least one of advertisement information 1550 and coupon information 1555 of other products arranged in the vicinity of the event position 201. After outputting the related information 155, the control device 11 proceeds to the next step S303.

[0167] In step S303, the control device 11 determines whether or not to end the processing. The processing of step S303 may be the same as step S104 described above. If it is determined not to end the processing, the control device 11 returns the processing to step S301 and executes the processing again from step S301. If it is determined to end the processing, the control device 11 ends the processing procedure related to the output of the related information 155 according to this operation example. Note that the timing for ending the processing is not limited to this example. The control device 11 may end the processing procedure related to the output of the related information 155 at any timing.

[0168] According to one example of this embodiment, by providing the related information 155 when the product TP is inserted by the process of step S302, it is possible to expect activation of purchasing activities such as stimulating further purchasing desire of the product. Note that, in one example, the control device 11 may execute a series of processes from step S301 to step S303 in real time. The processing order of each step may be changed as appropriate. In another example, the control device 11 may execute a predetermined The control device 11 may determine whether the predetermined condition is satisfied. The predetermined condition may be defined arbitrarily. For example, the predetermined condition may include that the output of the related information 155 is permitted. Whether or not the output of the related information 155 is permitted may be set appropriately. The control device 11 may execute the process of step S302 only if the predetermined condition is satisfied. Furthermore, when the control device 11 executes the processes of both steps S101 to S104 and steps S301 to S303, the process of step S301 may be executed in common with the above-mentioned step S101. The process of step S303 may be executed in common with the above-mentioned step S104. The process of step S302 may be executed at any timing after step S101 and before step S104.

[0169] [Information processing device] (Product Identification) FIG. 17 is a flowchart showing an example of a processing procedure for product identification using the product input record 20 by the information processing device 2 according to this embodiment. The following processing procedure for product identification is an example of an identification method (information processing method) executed by a computer. However, the following processing procedure for product identification is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure for product identification can be omitted, replaced, or added as appropriate depending on the embodiment.

[0170] In step S501, the control unit 21 operates as the acquisition unit 211. That is, the control unit 21 acquires the product insertion record 20 of the target user generated by the shopping device. For example, the control unit 21 may acquire the product insertion record 20 from the smart grip 1. In step S502, the control unit 21 operates as the identification unit 2121. That is, the control unit 21 identifies the inserted target product TP1 according to the acquired product insertion record 20. For example, the control unit 21 may identify the target product TP1 by at least one of the first to third identification methods. For example, when the first identification method is adopted, the control unit 21 may identify the target product TP1 inserted at the event position 201 by comparing the product insertion record 20 with the map information 30. Furthermore, for example, when the second identification method is adopted, the control unit 21 may acquire a captured image 400 of the imaging device 40 whose imaging range includes the event position 201 of the product insertion record 20. The control unit 21 may identify the target product TP1 appearing in the captured image 400 by performing image analysis on the acquired captured image 400. Furthermore, for example, when the third identification method is adopted, the control unit 21 may acquire measurement data 450 from a load sensor 45 installed near the event position 201 of the product input record 20. The control unit 21 may identify the target product TP1 by identifying the product shelf PS where the weight has decreased according to the acquired measurement data 450. In one example, the control unit 21 may determine the final identification result for the target product TP1 by ensembling the identification results obtained by multiple methods. In step S503, the control unit 21 outputs information related to the result of identifying the target product TP1.

[0171] The output destination and content of the information may be selected appropriately depending on the embodiment. In one example, the control unit 21 may output the identification result of the target product TP1 as is. In another example, the control unit 21 may execute any information processing (such as the calculation of the purchase amount and inventory management) depending on the obtained identification result. The control unit 21 may output the result of the information processing as information related to the identification result. The output of the result of the information processing may include, for example, outputting specific information depending on the identification result, or controlling the operation of the controlled device depending on the identification result. Outputting specific information may include notifying the store system that the payment process has been completed, notifying the store system of the calculated total amount, or notifying a message prompting the store system to replenish the target product TP1. Controlling the operation of the controlled device may include issuing a command to the robot device to replenish the target product TP1. The output destination may be, for example, RAM, the memory unit 22, the output device 25, another computer (including the smart grip 1), the controlled device, etc. When the output of the information is completed, the control unit 21 returns to the operation example. The processing procedure for product identification is then completed.

[0172] According to one example of the present embodiment, the inserted product (target product TP1) can be identified by using the product insertion record 20 through the processes of steps S501 and S502. The control unit 21 may execute the series of processes of steps S501 to S503 at any timing. In one example, the control unit 21 may execute the series of processes of steps S501 to S503 in real time in response to the generation of the product insertion record 20. In another example, the control unit 21 may acquire multiple product insertion records 20 in the process of step S501. The control unit 21 may execute the process of step S502 for each acquired product insertion record 20. In this case, the identification process of at least some of the target products TP1 (the process of step S502) may be executed ex post.

[0173] (Generating movement history) FIG. 18 is a flowchart showing an example of a processing procedure for generating a movement history 50 using the product input record 20 by the information processing device 2 according to this embodiment. The processing procedure for generating the movement history 50 described below is an example of a generation method (information processing method) executed by a computer. However, the processing procedure for generating the movement history 50 described below is merely an example, and each step may be changed as much as possible. Furthermore, steps may be omitted, replaced, or added to the processing procedure for generating the movement history 50 described below as appropriate depending on the embodiment.

[0174] In step S601, the control unit 21 operates as the acquisition unit 211. The processing of step S601 may be the same as that of step S501 above. In step S602, the control unit 21 operates as the history generation unit 2122. That is, the control unit 21 generates a movement history 50 of the target user from the event position 201 of the acquired product insertion record 20. In step S603, the control unit 21 operates as the output processing unit 213. That is, the control unit 21 outputs the generated movement history 50. As with step S503 above, the output destination and content of the information may each be selected appropriately depending on the embodiment. When the output of the information is completed, the control unit 21 ends the processing procedure related to the generation of the movement history 50 according to this operation example.

[0175] According to one example of the present embodiment, the movement history 50 of the target user can be generated using the product input record 20 by the processes of steps S601 and S602. The generated movement history 50 can be used to analyze the purchasing behavior of the target user. The control unit 21 may execute the series of processes of steps S601 to S603 at any timing. The control unit 21 may also execute the processes of both steps S501 to S503 and steps S601 to S603. In this case, the process of step S601 may be executed in common with the above-mentioned step S501. The order of the processes of steps S502, S503, S602, and S603 is not particularly limited and may be determined appropriately depending on the embodiment.

[0176] (Location identification) FIG. 19 is a flowchart showing an example of a processing procedure for identifying a location where a product (target product TP1) is inserted using the product insertion record 20 by the information processing device 2 according to this embodiment. The following processing procedure for identifying a location where an item is inserted is an example of an identification method (information processing method) executed by a computer. However, the following processing procedure for identifying a location where an item is inserted is merely an example, and each step may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added as appropriate to the processing procedure for identifying a location where an item is inserted, depending on the embodiment.

[0177] In steps S701 and S702, the control unit 21 operates as the acquisition unit 211. The process in step S701 may be the same as that in step S501. In S702, the control unit 21 acquires the purchase record 35 of the purchased item of the target user. In step S703, the control unit 21 operates as the location identification unit 2123. That is, the control unit 21 compares the acquired item insertion record 20 and purchase record 35 with the map information 30 to identify the location where the target user placed the purchased item in the basket (basket B1). In step S704, the control unit 21 operates as the output processing unit 213. That is, the control unit 21 outputs information related to the identified location. As with step S503 above, the output destination and content of the information may each be selected appropriately depending on the embodiment. When the output of the information is completed, the control unit 21 terminates the processing procedure related to identifying the insertion location related to this operation example.

[0178] According to an example of the present embodiment, the processing of steps S701 to S703 can identify the location where the purchased item was placed in the shopping cart. As a result, even if the same item is placed in multiple locations within a store, it is possible to analyze from which location each target user obtained the purchased item. The control unit 21 may execute the series of processes of steps S701 to S704 at any timing. The order of the processes of each step may be changed as appropriate. In another example, the process of step S702 may be executed before step S701. The process of step S702 may be executed at least partially in parallel with the process of step S701. Furthermore, the control unit 21 may execute the processes of both steps S501 to S503 and steps S701 to S704. In this case, the process of step S701 may be executed in common with step S501. The order of the processes of steps S502, S503, S702, S703, and S704 is not particularly limited and may be determined as appropriate depending on the embodiment. The control unit 21 may execute the processes of both steps S601 to S603 and steps S701 to S704. In this case, the process of step S701 may be executed in common with the above-mentioned step S601. The order of the processes of steps S602, S603, S702, S703, and S704 is not particularly limited and may be determined appropriately depending on the embodiment. When all the processes of identifying products, generating movement histories, and specifying insertion locations are executed, the processes of steps S501, S601, and S701 may be executed in common.

[0179] §4 Variations Although the embodiments of the present disclosure have been described in detail above, the above description is merely an example of the present disclosure in every respect. The processes and means described in the present disclosure can be freely combined and implemented as long as no technical contradiction occurs. Furthermore, various improvements or modifications may be made to the above embodiments as appropriate. For example, the following modifications are possible. Note that, in the following, the same reference numerals are used for components similar to those in the above embodiments, and descriptions of the same points as those in the above embodiments are omitted as appropriate.

[0180] <4.1> The above embodiment is configured as a smart grip 1. However, the embodiment of the present disclosure is not limited to the above smart grip 1. The embodiment of the present disclosure may be a component of a basket used for shopping, or may be the basket (including a cart) itself. The type of basket is not particularly limited, and may be selected appropriately depending on the embodiment.

[0181] (First Modification) FIG. 20 schematically shows another example of a situation to which the present disclosure is applied. In the example of FIG. 20, the embodiment of the present disclosure is a shopping basket 1A. The shopping basket 1A includes a basket body 102A, a detection sensor 13, a positioning sensor 14, and a control device 11. The basket body 102A may be configured in the same manner as the basket body B10 of the basket B1. Except for the basket body B10 being replaced with the basket body 102A, the detection sensor 13, the positioning sensor 14, and the control device 11 may be configured in the same manner as in the above embodiment. That is, the detection sensor 13 detects a position of the basket body 102A. The control device 11 may be configured to detect the insertion of a product TP (target product). When the detection sensor 13 detects the insertion of the product TP into the basket body 102A, the control device 11 may generate a product insertion record 20 including the event position 201 where the product TP was inserted, which is measured by the positioning sensor 14.

[0182] Other configurations of the shopping basket 1A are not particularly limited and may be determined appropriately depending on the embodiment. In one example, the shopping basket 1A may include a pair of handles 101A attached to the basket body 102A, and a grip 100A spanning the pair of handles 101A. Each handle 101A may be configured similarly to each handle B11 of the basket B1. When adopting these configurations, in one example, the shopping basket 1A may be constructed by using the above-mentioned smart grip 1 for the grip 100A. However, the method of constructing the shopping basket 1A is not limited to this example.

[0183] Like the smart grip 1, the shopping basket 1A may further include a memory unit 12. The shopping basket 1A may further include at least one of an output device 15, an imaging device 16, and a communication module 17. If the shopping basket 1A further includes the output device 15, the control device 11 may output related information 155 from the output device 15 when the detection sensor 13 detects that the product TP has been inserted. If the shopping basket 1A further includes the imaging device 16, the control device 11 may acquire a captured image 165 by the imaging device 16 when the detection sensor 13 detects that the product TP has been inserted, and identify the product TP by performing image analysis on the acquired captured image 165.

[0184] The location where the computer portion of the shopping basket 1A (control device 11 to communication module 17) is arranged does not have to be limited to the grip 100A. At least a part of the computer portion of the shopping basket 1A may be arranged anywhere other than the grip 100A. In another example, when a load sensor is used as the detection sensor 13, the detection sensor 13 may be arranged on the bottom surface of the storage section of the basket body 102A. The imaging device 16 may be arranged on the edge of the opening of the basket body 102A, with the imaging unit 161 facing toward the storage section of the basket body 102A.

[0185] The shopping basket 1A is an example of a shopping device. According to this modification, in situations where a basket configuration is adopted, it is expected that monitoring of purchasing behavior by users will be simplified. The configuration of the shopping basket 1A may be modified as appropriate depending on the embodiment. In another example, the grip 100A may be omitted. In this case, the user may hold the shopping basket 1A by grasping the pair of handles 101A.

[0186] (Second Modification) FIG. 21 schematically illustrates another example of a situation to which the present disclosure is applied. In the example illustrated in FIG. 21, the embodiment of the present disclosure is a shopping basket 1B. The shopping basket 1B corresponds to the above-described shopping basket 1A further equipped with one or more wheels 103B, thereby configuring it as a cart. That is, the shopping basket 1B includes a basket body 102B, a detection sensor 13, a positioning sensor 14, a control device 11, and one or more wheels 103B. The basket body 102B may be similar to the basket body 102A of the first modified example. The detection sensor 13 may be configured to detect the insertion of a product TP (target product) into the basket body 102B. When the detection sensor 13 detects the insertion of the product TP into the basket body 102B, the control device 11 may generate a product insertion record 20 including the event position 201 where the product TP was inserted, the event position 201 being measured by the positioning sensor 14. The number of wheels 103B may be selected appropriately depending on the embodiment. In a typical example, the shopping basket 1B may include four wheels 103B. The shopping basket 1B may also be referred to as a shopping cart.

[0187] The arrangement of the wheels 103B and the structure for supporting the basket body 102B may be appropriately determined depending on the embodiment. The dimensions may be determined as appropriate. In one example, the shopping basket 1B may include a grip 100B and a base 101B. The base 101B may include legs 1011B and a support pole 1012B. The legs 1011B may be disposed below the basket body 102B. Each wheel 103B may be attached to the leg 1011B. The support pole 1012B may be configured to extend vertically upward from one end of the leg 1011B. The basket body 102B may be fixed to the support pole 1012B. The grip 100B may be attached to the upper end of the support pole 1012B.

[0188] As a result, the shopping basket 1B may have a configuration similar to that of a general shopping cart. However, the configuration of the shopping basket 1B is not limited to this example. The shopping basket 1B may have a configuration similar to that of other types of carts, such as a silver cart. Furthermore, when adopting these configurations, in one example, the shopping basket 1B may be constructed by using the above-mentioned smart grip 1 for the grip 100B. However, the method of constructing the shopping basket 1B is not limited to this example.

[0189] Similar to the smart grip 1, the shopping basket 1B may further include a memory unit 12. The shopping basket 1B may further include at least one of an output device 15, an imaging device 16, and a communication module 17. Similar to the first modified example, when the shopping basket 1B further includes the output device 15, the control device 11 may execute an output process of the related information 155. When the shopping basket 1B further includes the imaging device 16, the control device 11 may execute an identification process of the product TP by image analysis.

[0190] The location where the computer portion of the shopping basket 1B (control device 11 to communication module 17) is arranged does not have to be limited to the grip 100B. At least a part of the computer portion of the shopping basket 1B may be arranged anywhere other than the grip 100B. In another example, when a load sensor is used as the detection sensor 13, the detection sensor 13 may be arranged on the bottom surface of the storage section of the basket body 102B. The imaging device 16 may be arranged on the edge of the opening of the basket body 102B or in the area of the support section 1012B from the upper end of the basket body 102B to the grip 100B, with the imaging unit 161 facing toward the storage section of the basket body 102B.

[0191] The shopping basket 1B is an example of a shopping device. This modification is expected to simplify monitoring of purchasing behavior by users in situations where a cart configuration is adopted. The basket body 102B may be used as a basket holder. In this case, other baskets may be placed in the basket body 102B, and the product TP may be placed in the other basket. The detection sensor 13 may be appropriately arranged to detect the placement of the product TP in the other basket. Detecting the placement of the product TP in the basket body 102B may include detecting the placement of the product TP in the other basket placed in the basket body 102B. The leg 1011B may also be configured to receive luggage. In this case, the detection sensor 13 may be further arranged in the leg 1011B. If the identification of the product TP through image analysis is also performed in the leg 1011B, the imaging device 16 may also be arranged in the leg 1011B. [Explanation of symbols]

[0192] 1...Smart Grip, 1A·1B…Shopping cart, 102A·102B…Basket body, 11...control device, 13...detection sensor, 14...positioning sensor, 15...output device, 16...imaging device, 90...Housing, 915·935...Groove, 20...Product input record, 201...Event location, 2...information processing device, 30...Map information, 35...Purchase history, 40...imaging device, 400...captured image, 45...load sensor, 450...measurement data, 50...Travel history, TP...product, TP1...target product, PS…Product shelf

Claims

1. acquiring a product insertion record of the target user generated by a shopping device, the product insertion record including an event location where the target product, which is a purchased product, was inserted; Obtaining a purchase record of the target user's purchased products; Identifying the location where the target user placed the purchased item in the cart by comparing the acquired product input record and purchase record with map information including identification information of the item in association with the display location of the item in the store; and outputting information about the identified location; a controller configured to execute Identifying the location where the purchased item was placed in the cart by comparing the map information, the item insertion record, and the purchase record includes, when the purchased item is placed in multiple locations within a store, identifying from which of the multiple locations the purchased item was placed in the cart by comparing the location on the map information of the purchased item identified by the purchase record and the event location. Information processing device.

2. The control unit is further configured to identify the inserted target product in accordance with the acquired product insertion record, and output information related to the result of identifying the target product. The information processing device according to claim 1 .

3. and identifying the target product according to the product input record by comparing the product input record with the map information to identify the target product input at the event location. The information processing device according to claim 2 .

4. The control unit is further configured to generate a movement history of the target user from the event location of the acquired product insertion record, and output the generated movement history. The information processing device according to claim 1 .

5. Obtaining a product insertion record of a target user generated by a shopping device, the product insertion record including an event location where a target product, which is a purchased product, was inserted; Obtaining a purchase record of the target user's purchased products; Identifying the location where the target user put the purchased item into the cart by correlating the acquired product input record and purchase record with map information including identification information of the item in association with the display location of the item in the store; outputting information about the identified location; Identifying the inserted target product according to the acquired product insertion record; and outputting information regarding the result of identifying the target product; a control unit configured to perform Identifying the target product according to the product input record includes: Identifying an imaging device that includes the event position of the product insertion record in an imaging range from among a plurality of imaging devices installed in the store according to the event position of the product insertion record; Acquiring an image captured by the identified imaging device; and Identifying the target product shown in the captured image by performing image analysis on the captured image; It consists of Information processing device.

6. Obtaining a product insertion record of a target user generated by a shopping device, the product insertion record including an event location where a target product, which is a purchased product, was inserted; Obtaining a purchase record of the target user's purchased products; Identifying the location where the target user put the purchased item into the cart by correlating the acquired product input record and purchase record with map information including identification information of the item in association with the display location of the item in the store; outputting information about the identified location; Identifying the inserted target product according to the acquired product insertion record; and outputting information regarding the result of identifying the target product; a control unit configured to perform Load sensors are installed on each shelf in the store. Identifying the target product according to the product input record includes: Identifying, according to the event position in the product input record, a load sensor installed near the event position in the product input record from among the load sensors installed on each product shelf in the store; Obtaining measurement data of the identified load sensor; and Identifying the target product by identifying a product shelf where the weight has decreased according to the acquired measurement data; It consists of Information processing device.

7. obtaining a product deposit record for the target user generated by a shopping device, the product deposit record including an event location where the target product was deposited; Identifying the inserted target product according to the acquired product insertion record; and outputting information regarding the result of identifying the target product; a control unit configured to perform Identifying the target product according to the product input record includes: Identifying an imaging device that includes the event position of the product insertion record in an imaging range from among a plurality of imaging devices installed in the store according to the event position of the product insertion record; Acquiring an image captured by the identified imaging device; and Identifying the target product shown in the captured image by performing image analysis on the captured image; It consists of Information processing device.

8. Obtaining a product insertion record for a target user, the product insertion record being generated by a shopping device and including an event location where the target product was inserted; Identifying the inserted target product according to the acquired product insertion record; and outputting information regarding the result of identifying the target product; a control unit configured to perform Load sensors are installed on each shelf in the store. Identifying the target product according to the product input record includes: Identifying, according to the event position in the product input record, a load sensor installed near the event position in the product input record from among the load sensors installed on each product shelf in the store; Obtaining measurement data of the identified load sensor; and Identifying the target product by identifying a product shelf where the weight has decreased according to the acquired measurement data; It consists of Information processing device.

Images