User behavior detection system, user behavior detection device, and user behavior detection method

Abstract

This invention provides a user behavior detection system, a user behavior detection device, and a user behavior detection method capable of detecting a user's actions before check-in. [Solution] The user behavior detection system 10 includes a camera device that acquires the user's actions before check-in as image data, a device communication device that transmits product identification information to identify the product as the user moves the product, a product detection device that detects the product identification information, a user terminal device that acquires the time and place when the user checks in, and a user behavior detection device that detects the user's movement path within the store based on the image data acquired from the camera device, and detects the user's actions toward the product before check-in based on the movement path, the product identification information acquired from the product detection device, and the place and time acquired from the user terminal device.

Description

Technical Field

[0001] The present invention relates to a user behavior detection system, a user behavior detection device, and a user behavior detection method.

Background Art

[0002] Conventionally, there has been a user behavior detection system for detecting a user's behavior in a store. Examples of the user behavior detection system include the following. That is, an information processing device that obtains the degree of association between the movement line information and the sales information of products based on the product type included in the product sales information (POS (Point of Sale) data) and the product type specified from the movement line information of the purchaser, and associates the movement line information with the sales information based on the degree of association. According to this information processing device, it is said that the behavior of the purchaser in the store can be associated with the actual purchase result by the purchaser.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The above-described information processing device assumes that a wireless tag is attached to a cart that can move in the store, and a receiver provided in the store acquires a signal from the wireless tag.

[0005] However, for example, there may be a case where the wireless tag cannot transmit a signal to the receiver due to a failure. Also, for example, there may be a case where the purchaser does not permit the detection of the behavior in the store. That is, in the information processing device, it may not be possible to detect the behavior of the purchaser in the store, and it may not be possible to acquire the movement line information of the purchaser. The above-described information processing device does not disclose how to cope with the case where the movement line information cannot be detected.

[0006] On the other hand, a user may hold a user terminal device and perform a check-in operation through the user terminal device. Checking in means that the user authorizes the system to detect the user's actions within the store. When a user performs a check-in operation, the user behavior detection system can detect the user's actions. However, when a user does not perform a check-in operation, the user behavior detection system cannot detect the user's actions.

[0007] Therefore, the purpose of this disclosure is to provide a user behavior detection system, a user behavior detection device, and a user behavior detection method that are capable of detecting a user's behavior before check-in. [Means for solving the problem]

[0008] The first embodiment of the user behavior detection system is a user behavior detection system that detects the behavior of a user who has entered a store. The user behavior detection system has a camera device that acquires the user's behavior before check-in as image data. The user behavior detection system also has a device communication device that transmits product identification information that identifies a product as the user moves the product. Furthermore, the user behavior detection system has a product detection device that detects the product identification information. Furthermore, the user behavior detection system has a user terminal device that acquires the time and place when the user checks in. Furthermore, the user behavior detection system has a user behavior detection device that detects the user's movement path within the store based on the image data acquired from the camera device, and detects the user's behavior towards a product before check-in based on the movement path, the product identification information acquired from the product detection device, and the place and time acquired from the user terminal device. Here, check-in indicates that the user has given permission for the user behavior detection system to detect the user's behavior within the store.

[0009] The user behavior detection device according to the second embodiment is a user behavior detection device that detects the behavior of a user who has entered a store. The user behavior detection device has a receiving unit that receives the user's behavior before check-in as image data from a camera device, receives product identification information that identifies products from a product detection device, and receives the time and place when the user checks in from a user terminal device. The user behavior detection device also has a control unit that detects the user's movement path within the store based on the image data acquired from the camera device, and detects the user's behavior towards products before check-in based on the movement path, the product identification information, and the place and time when the user checks in. Here, check-in indicates that the user has given permission for the user behavior detection system to detect the user's behavior within the store.

[0010] The third aspect of the user behavior detection method is a user behavior detection method in a user behavior detection system that detects the behavior of a user who has entered a store. The user behavior detection method includes a step in which a camera device acquires the user's behavior before check-in as image data. The user behavior detection method also includes a step in which a device communication device transmits product identification information that identifies a product as the user moves the product. Furthermore, the user behavior detection method includes a step in which a product detection device detects the product identification information. Furthermore, the user behavior detection method includes a step in which a user terminal device acquires the time and place when the user checks in. Furthermore, the user behavior detection method includes a step in which the user behavior detection device detects the user's movement path within the store based on the image data acquired from the camera device, and detects the user's behavior towards a product before check-in based on the movement path, the product identification information acquired from the product detection device, and the place and time acquired from the user terminal device. Here, check-in indicates that the user has given permission for the user behavior detection system to detect the user's behavior within the store. [Effects of the Invention]

[0011] This disclosure provides a user behavior detection system, a user behavior detection device, and a user behavior detection method that are capable of detecting a user's behavior before check-in. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 is a diagram showing an example configuration of a user behavior detection system according to the first embodiment. [Figure 2] Figure 2 is a diagram illustrating an example of an application scenario for the user behavior detection system according to the first embodiment. [Figure 3] Figure 3(A) shows an example configuration of a device communication device according to the first embodiment, and Figure 3(B) shows an example configuration of a beacon transmitting device according to the first embodiment. [Figure 4] Figure 4(A) shows an example of the configuration of the product detection device according to the first embodiment, and Figure 4(B) shows an example of the configuration of the camera device according to the first embodiment. [Figure 5] Figure 5(A) shows an example configuration of a user terminal device according to the first embodiment, and Figure 5(B) shows an example configuration of a user behavior detection device according to the first embodiment. [Figure 6] Figures 6(A) to 6(C) illustrate an example of a user behavior detection method in the first example of operation. [Figure 7] Figures 7(A) to 7(C) are diagrams illustrating an overview of the first operation example according to the first embodiment. [Figure 8] Figure 8 is a diagram showing a first example of operation according to the first embodiment. [Figure 9] Figures 9(A) and 9(B) are diagrams illustrating an overview of a second operation example according to the first embodiment. [Figure 10] Figures 10(A) and 10(B) are diagrams illustrating an overview of a second operation example according to the first embodiment. [Figure 11] Figure 11 is a diagram showing a second example of operation according to the first embodiment. [Figure 12] Figures 12(A) and 12(B) are diagrams illustrating the problems according to the first embodiment.

Mode for Carrying Out the Invention

[0013] [First Embodiment] Hereinafter, embodiments will be specifically described with reference to the drawings. In the description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0014] (1) Configuration Example of User Behavior Detection System First, a configuration example of the user behavior detection system according to the first embodiment will be described. FIG. 1 is a diagram showing a configuration example of the user behavior detection system 10.

[0015] The user behavior detection system includes a plurality of appliance communication devices 100 (100-1, 100-2, 100-3,...), a plurality of product tags 200 (200-1, 200-2, 200-3,...), a plurality of beacon transmission devices 300 (300-1, 300-2, 300-3,...), a plurality of product detection devices 400, a camera device 500, a user terminal device 600, and a user behavior detection device 700.

[0016] In one embodiment, the appliance communication device 100, the beacon transmission device 300, the product detection device 400, and the camera device 500 are provided in the store S. Hereinafter, an example in which the store S is a clothing store will be described, but the store S may be any store that sells the product P, and may be, for example, a supermarket, a greengrocer, a fishmonger, etc.

[0017] Also, in one embodiment, the appliance communication device 100 is provided in the display zone Z. Hereinafter, as the display zone Z, a hanger rack and a display shelf will be described as an example, but the display zone Z may be any zone as long as it is a zone for displaying the product P, and may be, for example, a showcase, etc. Also, hereinafter, an example in which there is one display zone Z in the store S will be described, but there may be a plurality of display zones Z in the store S.

[0018] The device communication device 100 (100-1 and 100-2) is attached to the holding devices H (H1 and H2) that hold the products P (P1 and P2). In the following description, a hanger is used as an example of the holding device H, but the holding device H can be any device capable of holding the products P, such as a tray or a product holder. The device communication device 100 may be incorporated into the holding device H or attached to the surface of the holding device H. The device communication device 100 is a communication device attached to the holding device H. In the following description, an example is given in which multiple sets of holding devices H and products P are provided as one set in the display zone Z, but there may be only one such set.

[0019] Furthermore, the device communication device 100 (100-3) may be attached to product P (P3). Specifically, the device communication device 100 may be attached to the price tag of product P. Alternatively, the device communication device 100 may be attached to a product tag 200 provided on product P. Alternatively, the device communication device 100 may be directly attached to the front (or back) surface of product P. When the device communication device 100 is attached to product P, product P may be placed on a product shelf. Even when the device communication device 100 is attached to product P, the device communication device 100 and product P may be provided as a single set, with one set provided in the display zone Z, or multiple sets may be provided in the display zone Z. In the first embodiment, an example in which the device communication device 100 is attached to product P will be described.

[0020] Product tag 200 is a tag attached to product P. Below, we will describe product tag 200 using a barcode as an example, but product tag 200 may also be a wireless tag or a QR code (registered trademark; hereinafter the same). Product tag 200 contains product identification information (or product ID) that distinguishes product P from other products. If product tag 200 is a barcode, the barcode contains product identification information. Similarly, if product tag 200 is a wireless tag or a QR code, the wireless tag or QR code contains product identification information. If product tag 200 is a wireless tag, it may be called an IC tag, RFID, or RF tag. The wireless tag may be a passive or active wireless tag. Product tag 200 may be attached to the surface of product P, or it may be attached to a label attached to product P.

[0021] In one embodiment, the beacon transmitter 300 is installed in a specific location within the store S. Specifically, the beacon transmitter 300 may be installed for each piece of equipment E, or it may be installed at the entrance of the store S. The following describes an example in which there are multiple pieces of equipment E within the store S, but there may be only one piece of equipment E within the store S. Pieces of equipment E may be, for example, a full-length mirror (or standing mirror) or a fitting room. The beacon transmitter 300 transmits beacons continuously (or periodically). A beacon is a signal that contains identification information (or an ID). A beacon transmitter 300 installed in a fitting room (for example, 300-1) transmits a beacon (sometimes called a "fitting room beacon") that contains identification information (or a fitting room ID) representing the fitting room. Similarly, a beacon transmitter 300 installed in a full-length mirror (for example, 300-2) transmits a beacon (sometimes called a "full-length mirror beacon") that contains identification information (or a full-length mirror ID) representing the full-length mirror. Furthermore, a beacon transmitter (e.g., 300-3) installed at the entrance transmits a beacon (sometimes called an "entry beacon") containing identification information (or entrance ID) representing the entrance. The beacon transmitted by the beacon transmitter 300 may also contain identification information representing a specific location.

[0022] The product detection device 400 detects product identification information (or product ID) that identifies product P. The product detection device 400 may be installed within the store S, in the display zone Z, or in the equipment E. Alternatively, the product detection device 400 may be installed in a passageway within the store S. The product detection device 400 can connect to a network NW via a wireless line. The product detection device 400 transmits the detected product ID to the user behavior detection device 700 via the network NW.

[0023] The camera device 500 is installed inside the store S in a location that provides a view of the entire store S. The camera device 500 can be connected to the network NW via a wired or wireless line. The camera device 500 transmits the captured images as image data to the user behavior detection device 700 via the network NW.

[0024] The user terminal device 600 is a terminal device held or used by a user. Here, a user is a customer who can purchase product P within the store S. In the following, the terms user and customer may not be distinguished. The user terminal device 600 can connect to the network NW via a wireless line. The user terminal device 600 can communicate with the user behavior detection device 700 via the network NW. The user terminal device 600 is a portable communication terminal device. Specifically, the user terminal device 600 is a smartphone, tablet, or wearable device. The user terminal device 600 may have a program installed that can be used within the store S. The user can receive various services within the store S via the user terminal device 600 on which the program is installed.

[0025] The user behavior detection device 700 detects the actions of user U who enters store S. The user behavior detection device 700 can connect to a network NW via a wired or wireless line. The user behavior detection device 700 can communicate with the product detection device 400, the camera device 500, and the user terminal device 600 via the network NW. The following description will explain an example in which the user behavior detection device 700 is located outside store S, but the user behavior detection device 700 may also be located inside store S.

[0026] (2) An example of an application scenario for a user behavior detection system Figure 2 is a diagram illustrating an example of an application scenario for the user behavior detection system 10 according to the first embodiment.

[0027] As shown in Figure 2, one hanger rack L is provided within the display zone Z. A product detection device 400 is installed on the hanger rack L. Although Figure 2 shows an example where the product detection device 400 is attached to the hanger rack L, the product detection device 400 may be installed anywhere within the store S. In one embodiment, the product detection device 400 may be placed in each mesh of the store S, which is virtually divided into a mesh. Therefore, the product detection device 400 may be installed in equipment E (E1 and E2) or on the ceiling of the store S.

[0028] Furthermore, as shown in Figure 2, beacon transmitters 300 (300-1 and 300-2) are provided in the equipment E (E1 and E2). Beacon transmitter 300-1, installed in fitting room E1, transmits a beacon containing the fitting room ID. Beacon transmitter 300-2, installed in the full-length mirror E2, transmits a beacon containing the full-length mirror ID. The beacon transmitters 300 are shown as being attached to the equipment E, but they only need to be installed near the equipment E, for example, on a wall or ceiling near the equipment E.

[0029] Furthermore, as shown in Figure 2, the camera device 500 is installed inside the store S. In the example shown in Figure 2, the camera device 500 is installed on the ceiling of the store S, but it is sufficient to install it in a location that can oversee the entire store S, for example, it may be installed on a wall inside the store S.

[0030] In Figure 2, user U, who possesses a user terminal device 600, enters store S. By executing an application program installed on the user terminal device 600, user U can receive various services within store S. After executing the application, user U performs a check-in operation via the user terminal device 600. Check-in means that user U authorizes the user behavior detection system 10 to detect user U's actions within store S. After the check-in operation, the user behavior detection system 10 becomes able to detect user U's actions.

[0031] After check-in, the user behavior detection system 10 can detect the following actions of user U.

[0032] • "Entering the store" The user behavior detection system 10 can detect when user U "enters" store S. Specifically, the detection is performed as follows:

[0033] In other words, when user U enters store S, user terminal device 600 receives a beacon ("entry beacon") transmitted from beacon transmitter 300-3 installed at the entrance. Upon receiving the beacon, user terminal device 600 obtains an entrance ID from it. Then, user terminal device 600 transmits user identification information (customer ID or user ID; hereinafter sometimes referred to as "user ID") that identifies user U, along with the entrance ID, to user behavior detection device 700. The user behavior detection device 700 can detect user U's "entry" in response to receiving the user ID and entrance ID.

[0034] Furthermore, "entry into the store" can also be detected from images captured by the camera device 500. That is, the camera device 500 can also capture images of user U entering the store. The camera device 500 transmits the captured images as image data to the user behavior detection device 700. The user behavior detection device 700 may use machine learning to detect "entry into the store". Specifically, the user behavior detection device 700 stores images showing a person entering the store as training data. The user behavior detection device 700 may then use the images acquired from the camera device 500 and the training data to infer whether or not "entry into the store" has occurred, thereby detecting "entry into the store".

[0035] • "Pull the hanger" If user U sees an item P of interest leaning against a holding device H, they may pull the holding device H from the hanger rack L in order to take the item P from the holding device H. The user behavior detection system 10 can detect the "pulling the hanger" behavior. Specifically, this is as follows:

[0036] In other words, when user U pulls the holding device H from the hanger rack L, a beacon is transmitted from the device communication device 100 (100-1). This beacon contains the device ID. This beacon is sometimes called a "pull beacon". When user terminal device 600 receives this beacon, it extracts the device ID from it. Then, in response to receiving the beacon, user terminal device 600 transmits the device ID along with the user ID to user behavior detection device 700. In response to receiving the user ID and device ID, user behavior detection device 700 can detect user U's action of "pulling the hanger".

[0037] Furthermore, the action of "pulling the hanger" can also be detected from images captured by the camera device 500. That is, the camera device 500 transmits image data to the user behavior detection device 700. The user behavior detection device 700 may detect the action of "pulling the hanger" using machine learning functions. Specifically, the user behavior detection device 700 holds image data showing the action of "pulling the hanger" as training data. Then, the user behavior detection device 700 may detect the action of "pulling the hanger" by inferring whether or not the action of "pulling the hanger" has occurred using the image data acquired from the camera device 500 and the training data.

[0038] • "Holding a product" If user U finds product P of interest, they may pick up product P to examine it. The user behavior detection system 10 can detect the "holding the product" behavior. Specifically, this is as follows:

[0039] In other words, when user U pulls the holding device H, removes product P from the holding device H, and holds product P in their hand, a beacon is transmitted from the device communication device 100 attached to product P. Alternatively, when user U picks up product P from a display shelf, a beacon is transmitted from the device communication device 100 attached to product P. This beacon contains the product ID. This beacon is sometimes called a "holding beacon". When user terminal device 600 receives this beacon, it extracts the product ID from it. In response to receiving the beacon, user terminal device 600 transmits the user ID and product ID to user behavior detection device 700. In response to receiving the user ID and product ID, user behavior detection device 700 can detect user U's "holding product" action.

[0040] Furthermore, the action of "holding a product" can also be detected from images captured by the camera device 500. That is, the camera device 500 transmits image data to the user behavior detection device 700. The user behavior detection device 700 may detect the action of "holding a product" using machine learning functions. Specifically, the user behavior detection device 700 stores image data showing the action of "holding a product" as training data. Then, the user behavior detection device 700 may detect the action of "holding a product" by inferring whether or not the action of "holding a product" has occurred using the image data acquired from the camera device 500 and the training data.

[0041] Thus, the actions of "entering the store," "pulling the hanger," and "holding the product" may be detected via the user terminal device 600 or via the camera device 500.

[0042] "Take it to the full-length mirror." User U may take an item P that interests them to a full-length mirror E2 to check whether the item P suits them. In such cases, the user behavior detection system 10 can detect the action of "taking the item to the mirror." Specifically, this is as follows:

[0043] In other words, when user U takes product P to the mirror E2, user terminal device 600 can receive a beacon ("mirror beacon") transmitted from beacon transmitter 300-2. This beacon contains a mirror ID. Upon receiving the beacon, user terminal device 600 extracts the mirror ID from it. User terminal device 600 transmits the mirror ID along with user U's user ID to user behavior detection device 700. The user behavior detection device 700 can detect user U's action of "taking the product to the mirror" upon receiving the user ID and mirror ID.

[0044] "Take it to the fitting room." User U may take product P, which they are interested in, to fitting room E1 and try on product P. The user behavior detection system 10 can detect the action of "taking it to the fitting room." Specifically, this is as follows:

[0045] In other words, when user U takes product P to fitting room E12, user terminal device 600 can receive a beacon ("fitting room beacon") transmitted from beacon transmitter 300-1. This beacon contains a fitting room ID. Upon receiving the beacon, user terminal device 600 extracts the fitting room ID from it. In response to receiving the beacon, user terminal device 600 transmits the user ID and fitting room ID to user behavior detection device 700. In response to receiving the user ID and fitting room ID, user behavior detection device 700 can detect user U's action of "taking the product to the fitting room".

[0046] Thus, the actions of "taking it to the full-length mirror" and "taking it to the fitting room" can be detected via the user terminal device 600.

[0047] • "Check the fabric and the price tag." User U may check the fabric of product P that they are interested in, or check the price tag attached to product P. The user behavior detection system 10 can detect these two actions, "checking the fabric" and "checking the price tag," as a single action. Specifically, this is as follows:

[0048] In other words, the camera device 500 captures images of user U checking the fabric of product P and checking the price tag of product P. The camera device 500 transmits these images as image data to the user behavior detection device 700. The user behavior detection device 700 may use machine learning to detect the behavior of "checking the fabric and checking the price tag". Specifically, the user behavior detection device 700 holds image data that shows the behavior of "checking the fabric and checking the price tag" as training data. The user behavior detection device 700 may then use the image data acquired from the camera device 500 and the training data to infer whether or not the behavior of "checking the fabric and checking the price tag" has occurred, thereby detecting the behavior of "checking the fabric and checking the price tag".

[0049] • "Try it on right there" User U may try on product P in display zone Z where product P is displayed, without taking it to fitting room E1. The user behavior detection system 10 can detect this behavior as "trying on in place." Specifically, it is as follows:

[0050] In other words, the camera device 500 captures an image of the user trying on clothes on the spot and transmits the image as image data to the user behavior detection device 700. The user behavior detection device 700 may use machine learning to detect the behavior of "trying on clothes on the spot". Specifically, the user behavior detection device 700 stores images that show the behavior of "trying on clothes on the spot" as training data. The user behavior detection device 700 may then detect the behavior of "trying on clothes on the spot" by inferring whether or not the behavior of "trying on clothes on the spot" occurred using the image acquired from the camera device 500 and the training data.

[0051] Thus, the actions of "checking the fabric and price tag" and "trying on the clothes on the spot" can be detected via the camera device 500.

[0052] The following table summarizes each behavior and its detection method.

[0053] [Table 1] (3) Example of equipment communication device configuration Next, an example of the configuration of the device communication device 100 according to the first embodiment will be described.

[0054] Figure 3(A) is a diagram showing an example configuration of the equipment communication device 100.

[0055] As shown in Figure 3(A), the device communication device 100 includes a beacon transmission unit 110, a control unit 130, a detection unit 140, and a power supply unit 150.

[0056] The beacon transmitting unit 110 transmits a beacon under the control of the control unit 130. When the beacon transmitting unit 110 is instructed by the control unit 130 to transmit a beacon, it generates a beacon containing the information received from the control unit 130 and transmits the beacon. In one embodiment, the beacon transmitting unit 110 transmits a beacon containing the device ID.

[0057] In the following explanation, beacons will be described using BT beacons based on the Bluetooth (registered trademark; hereinafter the same) standard as an example. However, beacons conforming to standards other than Bluetooth may also be used.

[0058] The control unit 130 controls various functions of the device communication device 100. The control unit 130 has at least one memory 132 and at least one processor 131 electrically connected to the memory 132. The memory 132 includes volatile memory and non-volatile memory and stores information used for processing by the processor 131 and programs executed by the processor 131. The processor 131 may perform various processes by executing programs stored in the memory 132. In one embodiment, the memory 132 stores a device ID or product ID. The information stored in the memory 132 is read out as appropriate under the control of the processor 131. In the following example of operation, the operations or processes performed in the device communication device 100 may be mainly performed by the control unit 130.

[0059] The detection unit 140 detects the movement of the holding device H. Alternatively, the detection unit 140 detects the movement of the product P. For example, the detection unit 140 detects that the holding device H has been removed from the hanger rack L. Alternatively, the detection unit 140 detects that the product P has been moved in the user's hand. The detection unit 140 may include a switch that turns on or off in response to the movement of the holding device H or the product P. Alternatively, the detection unit 140 may include a sensor (e.g., an acceleration sensor) that detects a change in a physical quantity in response to the movement of the holding device H or the product P. The detection unit 140 outputs a signal indicating the detection result to the control unit 130.

[0060] In the device communication device 100 configured as described above, the control unit 130 reads the device ID from the memory 132 in response to the detection unit 140 detecting the movement of the holding device H, and controls the beacon transmission unit 110 to continuously (or periodically) transmit a beacon containing the device ID. As a result, the device communication device 100 can transmit a beacon containing the device ID ("pulling beacon") when user U moves the holding device H. In addition, the control unit 130 reads the product ID from the memory 132 in response to the detection unit 140 detecting the movement of product P, and controls the beacon transmission unit 110 to continuously (or periodically) transmit a beacon containing the product ID. As a result, the device communication device 100 can transmit a beacon containing the product ID ("holding beacon") when user U picks up and moves product P.

[0061] The power supply unit 150 supplies power to each part of the device communication device 100 under the control of the control unit 130. The power supply unit 150 may be a battery (or rechargeable battery) that can be repeatedly charged and discharged.

[0062] (4) Example of a beacon transmitter configuration Next, we will describe an example configuration of the beacon transmitter 300.

[0063] Figure 3(B) shows an example of the configuration of the beacon transmitter 300.

[0064] As shown in Figure 3(B), the beacon transmitting device 300 includes a beacon transmitting unit 310, a control unit 330, and a power supply unit 340.

[0065] The beacon transmitting unit 310 transmits beacons under the control of the control unit 330. When the beacon transmitting unit 310 is instructed by the control unit 330 to transmit a beacon, it generates a beacon containing the information received from the control unit 330 and transmits the beacon. In one embodiment, the beacon transmitting unit 110 transmits a beacon containing a fitting room ID. In another embodiment, the beacon transmitting unit 110 transmits a beacon containing a full-length mirror ID. Furthermore, in another embodiment, the beacon transmitting unit 110 transmits a beacon containing an entrance ID.

[0066] The control unit 330 controls various functions of the beacon transmitter 300. The control unit 330 has at least one memory 332 and at least one processor 331 electrically connected to the memory 332. The memory 332 includes volatile memory and non-volatile memory and stores information used for processing in the processor 331 and programs executed by the processor 331. The processor 331 may perform various processes by executing programs stored in the memory 332. In one embodiment, the memory 332 stores a fitting room ID, a full-length mirror ID, or an entrance ID. The information stored in the memory 332 is read out as appropriate under the control of the processor 331. In the following example of operation, the operations or processes performed in the beacon transmitter 300 may be mainly performed in the control unit 330.

[0067] In the beacon transmitter 300 configured as described above, the control unit 330 reads the fitting room ID, mirror ID, and entrance ID from the memory 332 and controls the beacon transmitter 310 to continuously (or periodically) transmit a beacon containing one of these IDs. As a result, the beacon transmitter 300-1 installed in fitting room E1 can transmit a beacon containing the fitting room ID. Similarly, the beacon transmitter 300-2 installed in mirror E2 can transmit a beacon containing the mirror ID. Furthermore, the beacon transmitter 300 installed at the entrance can transmit a beacon containing the entrance ID.

[0068] The power supply unit 340 supplies power to each part of the device communication device 100 under the control of the control unit 330. The power supply unit 340 may be a battery (or secondary battery) that can be repeatedly charged and discharged.

[0069] (5) Example of product detection device configuration Next, an example of the configuration of the product detection device 400 according to the first embodiment will be described.

[0070] Figure 4(A) is a diagram showing an example configuration of the product detection device 400.

[0071] As shown in Figure 4(A), the product detection device 400 includes a beacon receiver 410, a wireless communication unit 420, a control unit 430, and a power supply unit 440.

[0072] The beacon receiver 410 receives beacons transmitted from the device communication device 100 under the control of the control unit 430. In one embodiment, the beacon receiver 410 receives a beacon ("possessed beacon") transmitted from the device communication device 100 attached to product P. The beacon contains a product ID. The beacon receiver 410 extracts the product ID contained in the beacon and outputs the product ID to the control unit 430. The beacon receiver 410 may also output the time the beacon was received, along with the product ID, to the control unit 430 as the reception time of the product ID.

[0073] The wireless communication unit 420 can connect to a network NW via a wireless line under the control of the control unit 430. The wireless communication unit 420 can communicate with the user behavior detection device 700 via the network NW. The wireless communication unit 420 can also communicate with the user terminal device 600. The wireless communication unit 420 may communicate using a wireless LAN standard wireless communication method or a cellular wireless communication method.

[0074] The control unit 430 controls various functions of the product detection device 400. The control unit 430 has at least one memory 432 and at least one processor 431 electrically connected to the memory 432. The memory 432 includes volatile memory and non-volatile memory and stores information used for processing by the processor 431 and programs executed by the processor 431. The processor 431 may perform various processes by executing programs stored in the memory 432. In one embodiment, the memory 432 stores product detection device identification information (or product detection device ID) that identifies the product detection device 400. The information stored in the memory 432 is read out as appropriate under the control of the processor 431. In the following example of operation, the operations or processes performed by the product detection device 400 may be mainly performed by the control unit 430.

[0075] In the product detection device 400 configured as described above, when the control unit 430 receives the product ID and reception time from the beacon receiver 410, it reads the product detection device ID from the memory 432 and controls the wireless communication unit 420 to transmit the product ID, reception time, and product detection device ID to the user behavior detection device 700. As a result, the product detection device 400 transmits the product ID, reception time, and product detection device ID to the user behavior detection device 700.

[0076] The power supply unit 440 supplies power to each part of the product detection device 400 under the control of the control unit 430. The power supply unit 440 may be a battery (or rechargeable battery) that can be repeatedly charged and discharged.

[0077] (6) Example of camera device configuration Next, an example of the configuration of the camera device 500 according to the first embodiment will be described.

[0078] Figure 4(B) shows an example of the configuration of a camera device.

[0079] As shown in Figure 4(B), the camera device 500 includes an imaging unit 510, a communication unit 520, a control unit 530, and a power supply unit 540.

[0080] The imaging unit 510 captures images of the store S under the control of the control unit 530. The imaging unit 510 includes multiple image sensors and outputs image data from the image by converting detected light into electrical signals. The image sensors may also be called image sensors. The imaging unit 510 outputs the image data to the control unit 530.

[0081] The communication unit 520 can connect to the network NW via a wireless or wired line under the control of the control unit 530. The communication unit 520 can communicate with the user behavior detection device 700 via the network NW. In the case of a wireless line, the communication unit 520 may communicate using a wireless communication method based on the wireless LAN standard, or it may communicate using a wireless communication method based on the cellular standard.

[0082] The control unit 530 controls various functions of the camera device 500. The control unit 530 has at least one memory 532 and at least one processor 531 electrically connected to the memory 532. The memory 532 includes volatile memory and non-volatile memory and stores information used for processing by the processor 531 and programs executed by the processor 531. The processor 531 may perform various processes by executing programs stored in the memory 532. The information stored in the memory 532 is read out as appropriate under the control of the processor 531. In the following example of operation, the operations or processes performed in the camera device 500 may be mainly performed by the control unit 530.

[0083] In the camera device 500 configured as described above, when the control unit 530 receives image data from the imaging unit 510, it acquires time information representing the time of imaging and controls the communication unit 520 to transmit the time information along with the image data to the user behavior detection device 700. As a result, the camera device 500 transmits image data and time information to the user behavior detection device 700. The image data includes at least image data showing the user U's actions before check-in. The image data may also include image data showing the user U's actions after check-in. The camera device 500 may transmit image data to the user behavior detection device 700 continuously (or periodically).

[0084] (7) Example of user terminal device configuration Next, an example configuration of the user terminal device 600 according to the first embodiment will be described.

[0085] Figure 5(A) is a diagram showing an example configuration of the user terminal device 600.

[0086] As shown in Figure 5(A), the user terminal device 600 includes a beacon receiving unit 610, a wireless communication unit 620, a control unit 430, and a power supply unit 640.

[0087] The beacon receiver 610 receives beacons ("pulling beacons" or "holding beacons") transmitted from the device communication device 100 under the control of the control unit 630. The beacon receiver 610 extracts the device ID or product ID contained in the beacon and outputs the device ID or product ID to the control unit 630.

[0088] Furthermore, the beacon receiver 610 receives beacons ("store entrance beacon", "fitting room beacon", or "mirror beacon") transmitted from the beacon transmitter 300 under the control of the control unit 630. The beacon receiver 610 extracts the entrance ID, fitting room ID, or mirror ID contained in the beacon and outputs the entrance ID, fitting room ID, or mirror ID to the control unit 630.

[0089] The beacon receiver 610 may also output the reception time of each beacon received to the control unit 630.

[0090] The wireless communication unit 620 can connect to a network NW via a wireless line under the control of the control unit 630. The wireless communication unit 620 can communicate with the user behavior detection device 700 via the network NW. The wireless communication unit 620 can also communicate with the product detection device 400. The wireless communication unit 420 may communicate using a wireless LAN standard wireless communication method or a cellular wireless communication method.

[0091] The control unit 630 controls various functions in the user terminal device 600. The control unit 630 has at least one memory 632 and at least one processor 631 electrically connected to the memory 632. The memory 632 includes volatile memory and non-volatile memory and stores information used for processing by the processor 631 and programs executed by the processor 631. The processor 631 may perform various processes by executing programs stored in the memory 632. In one embodiment, the memory 432 stores the user ID. The information stored in the memory 632 is read out as appropriate under the control of the processor 631. In the following example of operation, the operations or processes performed in the user terminal device 600 may be mainly performed by the control unit 630.

[0092] In the user terminal device 600 configured as described above, when the control unit 630 receives an equipment ID or product ID from the beacon receiver 610, it reads the user ID from the memory 632 and controls the wireless communication unit 620 to transmit the equipment ID and user ID, or the product ID and user ID, to the user behavior detection device 700. As a result, the user terminal device 600 can transmit the equipment ID and user ID, or the product ID and user ID, to the user behavior detection device 700.

[0093] Furthermore, when the control unit 630 receives an entrance ID, fitting room ID, or full-length mirror ID from the beacon receiver unit 610, it reads the user ID from the memory 632 and controls the wireless communication unit 620 to transmit the entrance ID and user ID, fitting room ID and user ID, or full-length mirror ID and user ID to the user behavior detection device 700. As a result, the user terminal device 600 can transmit the entrance ID and user ID, fitting room ID and user ID, or full-length mirror ID and user ID to the user behavior detection device 700.

[0094] Furthermore, the control unit 630 acquires the time and location when user U checks in. For example, the control unit 630 may acquire this information as follows: When the control unit 630 detects that user U has performed a check-in operation on the screen displayed on the display unit of the user terminal device 600, it acquires the time the check-in operation was performed using a timer. Also, when the control unit 630 detects that a check-in operation has been performed on the screen, it controls the wireless communication unit 620 to transmit a signal indicating that a check-in operation has been performed. The wireless communication unit 620 transmits this signal, and when a product detection device 400 installed at a specific location within the store S (for example, each mesh if the store S is virtually divided into a mesh, or each rack on which product P is displayed) detects it, the product detection device 400 transmits a product detection device ID. The wireless communication unit 620 receives the product detection device ID. The control unit 630 can determine the relative position within the store S (for example, the mesh position within the store S, or the rack position within the store S) using the product detection device ID, and thereby identify the check-in location. Alternatively, when the control unit 630 detects that a check-in operation has been performed on the screen, it may activate the GNSS (Global Navigation Satellite System) receiving function as an application program and obtain location information representing the location where the check-in operation was performed using the GNSS receiving function. The control unit 630 obtains the time and location, respectively, when the check-in was performed, based on the time and location information. The control unit 630 controls the wireless communication unit 620 to transmit the user ID along with the acquired time and location to the user activity detection device 700. As a result, the user terminal device 600 can transmit the time and location when user U checked in (i.e., the check-in time, the check-in location, and the user ID of user U who checked in) to the user activity detection device 700.

[0095] The power supply unit 640 supplies power to each part of the user terminal device 600 under the control of the control unit 630. The power supply unit 640 may be a battery (or rechargeable battery) that can be repeatedly charged and discharged.

[0096] (8) Example of User Behavior Detection Device Configuration Next, an example configuration of the user behavior detection device 700 according to the first embodiment will be described.

[0097] Figure 5(B) shows an example of the configuration of the user behavior detection device 700.

[0098] As shown in Figure 5(B), the user behavior detection device 700 includes a communication unit 710, a control unit 730, and a power supply unit 740.

[0099] The communication unit 710 can connect to a network NW via a wireless or wired line under the control of the control unit 730. The communication unit 710 can communicate with the product detection device 400, the camera device 500, and the user terminal device 600 via the network NW. In the case of a wireless line, the communication unit 710 may communicate using a wireless communication method based on the wireless LAN standard, or it may communicate using a wireless communication method based on the cellular standard.

[0100] The communication unit 710 receives image data and time information from the camera device 500. The communication unit 710 outputs the received image data and time information to the control unit 730.

[0101] Furthermore, the communication unit 710 receives the product ID, reception time, and product detection device ID from the product detection device 400. The communication unit 710 outputs the received product ID, reception time, and product detection device ID to the control unit 730.

[0102] Furthermore, the communication unit 710 receives the time and place (i.e., check-in time, check-in location, and user ID of user U who checked in) from the user terminal device 600. The communication unit 710 outputs the time, place, and user ID to the control unit 730.

[0103] The control unit 730 controls various functions of the user behavior detection device 700. The control unit 730 has at least one memory 732 and at least one processor 731 electrically connected to the memory 732. The memory 732 includes volatile memory and non-volatile memory and stores information used for processing in the processor 731 and programs executed by the processor 731. The processor 731 may perform various processes by executing programs stored in the memory 732. The information stored in the memory 732 is read out as appropriate under the control of the processor 731. In the following example of operation, the operations or processes performed in the user behavior detection device 700 may be mainly performed by the control unit 730.

[0104] In the user behavior detection device 700 configured as described above, when the control unit 730 receives image data and time information from the communication unit 710, it performs the following processing.

[0105] Firstly, the control unit 730 uses machine learning functionality to detect each action of user U based on image data. Each action may be one of the actions shown in Table 1. Specifically, as described above, the control unit 730 may detect each action by reading the training data for each action from the memory 732 and inferring the presence or absence of each action using the image data received from the camera device 500 and the training data.

[0106] Secondly, the control unit 730 detects a user's movement path (route) within the store S based on image data. The movement path may represent the user U's position (location) within the store S over time. Specifically, the control unit 730 acquires the movement path from the camera device 500 based on image data and time information. Then, based on the time information, the control unit 730 associates each detected action with the movement path. The movement path will include location, time, and action.

[0107] Furthermore, when the control unit 730 receives the product ID, reception time, and product detection device ID from the communication unit 710, it obtains the time when the product ID was detected and the location where the product ID was detected, based on the product ID, reception time, and product detection device ID. In one embodiment, the store S is virtually divided into a mesh, and a product detection device 400 is placed in each divided mesh. Therefore, by obtaining the product detection device ID, the control unit 730 can determine the location of the product detection device 400 in which the product ID was detected.

[0108] Furthermore, when the control unit 730 receives the check-in time, check-in location, and user ID from the communication unit 710, it confirms the check-in time and location of user U, indicated by the user ID.

[0109] The control unit 730 can then detect user U's actions prior to check-in based on the travel route, product ID, and the time and location of user U's check-in. Further details will be explained in the operation example.

[0110] The power supply unit 740 supplies power to each part of the user behavior detection device 700 under the control of the control unit 730. The power supply unit 740 may be a battery (or rechargeable battery) that can be repeatedly charged and discharged.

[0111] (9) Problems according to the first embodiment Next, we will describe the problems associated with the first embodiment.

[0112] Figures 12(A) and 12(B) are diagrams illustrating the problems according to the first embodiment.

[0113] As described above, by performing a check-in operation, user U can receive various services within store S. User U's check-in operation allows the user behavior detection system 10 to detect user U's actions within store S. As described above (Table 1), these actions can be detected via the user terminal device 600 or the camera device 500.

[0114] However, there are cases where user U's check-in operation is not performed upon entering store S, but rather after entering. For example, as shown in Figure 12(A), the check-in operation may be performed when purchasing product P. Alternatively, as shown in Figure 12(B), the user may perform the check-in operation while moving around store S with product P in hand, prompted by a store employee.

[0115] When the check-in process is performed with a delay from the time of entry, the user behavior detection system 10 may not be able to detect the actions of user U from the time of entry until the check-in process.

[0116] Therefore, the objective of the first embodiment is to enable the detection of user U's actions within the store S before the check-in operation.

[0117] Therefore, in the first embodiment, the user behavior detection system 10 associates the product ID with the user ID based on the user U's actions before check-in detected by the camera device 500, thereby detecting the user U's actions toward product P before check-in.

[0118] Specifically, firstly, the camera device 500 acquires image data of user U's actions before check-in. Secondly, the equipment communication device 100 transmits a product ID that identifies product P as user U moves product P. Thirdly, the product detection device 400 detects the product ID. Fourthly, the user terminal device 600 acquires the time and place when user U checks in. Fifthly, the user action detection device detects user U's movement path within the store S based on the image data acquired from the camera device 500, and detects user U's actions toward product P before check-in based on the movement path, the product ID acquired from the product detection device 400, and the place and time acquired from the user terminal device 600. Further details will be explained in the operation example.

[0119] (10) User Behavior Detection Method According to the First Embodiment Next, the user behavior detection method according to the first embodiment will be described.

[0120] FIGS. 6(A) to 6(C) are diagrams showing an example of the user behavior detection method.

[0121] FIG. 6(A) shows an example of the behavior of user U detected via the user terminal device 600. For example, at a certain time T1, user U performs the action of "pulling the hanger", and then, at time T2 (T2>T1), user U performs the action of "holding the product" for product P. As described above, in the user behavior detection device 700, by receiving the user ID and the appliance ID from the user terminal device 600, the user behavior detection device 700 can detect that user U having the user ID performs the action of "pulling the hanger". At that time, the user behavior detection device 700 can receive the reception time of the appliance ID from the user terminal device 600. Therefore, the user behavior detection device 700 can obtain the time when the action was performed based on the reception time. The detection of the action of "holding the product" is basically the same. The user behavior detection device 700 can detect that user U having the user ID performs the action of "taking it to the fitting room" at time T2 based on the user ID, the reception time, and the product ID.

[0122] On the other hand, FIG. 6(B) shows an example of the behavior of user U detected via the camera device 500. In FIG. 6(B), in the user behavior detection device 700, at a certain time t1 (t1<T1), the action of "checking the fabric and checking the price tag" is detected, and then, at time T1, the action of "pulling the hanger" is detected, and then, at time t2 (T1<t2), the action of "checking the fabric and checking the price tag" is detected, and then, at time T2 (t2<T2), the action of "holding the product" is detected.

[0123] However, even though the camera device 500 can detect each of the above actions, it cannot identify user U individually.

[0124] Therefore, the user behavior detection device 700 checks whether the same behavior was detected at the same time for both the behavior detected via the user terminal device 600 and the behavior detected via the camera device 500. In the examples of Figures 6(A) and 6(B), the user behavior detection device 700 detects the behavior of "pulling the hanger" at time T1 in both cases, and the behavior of "holding the product" at time T2 in both cases. Therefore, the user behavior detection device 700 can determine that each behavior shown in Figure 6(B) was performed by user U as detected in Figure 6(A). In other words, if the user behavior detection device 700 detects the same behavior at the same time for both the behavior acquired via the user terminal device 600 (Figure 6(A)) and the behavior acquired via the camera device 500 (Figure 6(B)), it can determine that user U performed each of the behaviors detected via the camera device 500 (Figure 6(C)).

[0125] The user behavior detection device 700 according to the first embodiment associates each action detected via the camera device 500 with a user ID, as described above. Basically, the actions of user U are detected using this user behavior detection method.

[0126] (11) First operation example according to the first embodiment

[0127] (11-1) Overview of the first operation example Figures 7(A) to 7(C) are diagrams illustrating an overview of the first operation example according to the first embodiment.

[0128] Figure 7(A) shows an example of a user's movement path (route) detected via the camera device 500. The control unit 730 of the user behavior detection device 700 generates the movement path based on image data transmitted from the camera device 500. Specifically, the control unit 730 may generate the movement path by plotting each point representing the user's location within the store S on the memory 732 over time, based on the image data. Each point on the movement path contains information about time and location.

[0129] Furthermore, as described above, the control unit 730 of the user behavior detection device 700 can detect the actions of a particular user (Table 1) based on the image data received from the camera device 500. The control unit 730 can also obtain time information from the camera device 500 along with the image data. Therefore, when the control unit 730 of the user behavior detection device 700 detects the actions of a particular user at a certain time, it can associate the detected actions with each point along the movement path based on the time information. As a result, each point along the movement path will contain information on time, location, and actions.

[0130] Furthermore, the control unit 730 of the user behavior detection device 700 assigns a route ID (or movement path ID) to the generated movement path.

[0131] At this point, when the travel path is generated, it is not possible to determine which product P the path is for. This is because the image data acquired from the camera device 500 alone cannot individually identify product P.

[0132] Therefore, the control unit 730 of the user behavior detection device 700 associates the movement path with the product ID. Specifically, the control unit 730 acquires (generates) time-series information of product P based on the movement path and the product ID. Figure 7(B) is a diagram showing an example of time-series information of product P. As shown in Figure 7(B), the time-series information of product P is information that associates the product ID with the movement path. The control unit 730 of the user behavior detection device 700 generates time-series information of product P in the following way, for example.

[0133] Specifically, the control unit 730 obtains the product ID from the product detection device 400, and at the same time, receives the reception time when the product detection device 400 obtained the product ID. The product detection device 400 is located in each mesh that is divided into a mesh-like structure within the store S. The product detection device 400 transmits the product detection device ID along with the detected product ID. By obtaining the product detection device ID, the control unit 730 can obtain the location (position) within the store S where the product ID was detected. Thus, the control unit 730 can obtain the time when the product ID was detected and the location (position) within the store S where the product ID was obtained. The control unit 730 then checks whether there is a time and location on the travel path that matches the time and location when the product ID was detected, and if there is, it generates time-series information for product P by associating the product ID with the travel path. Figure 7(B) shows an example where each product P—product A with product ID=“A”, product B with product ID=“B”, and product C with product ID=“C”—is linked to a movement path, and time-series information for product P is generated.

[0134] The control unit 730 of the user behavior detection device 700 then detects user U's actions toward product P before check-in, based on the time-series information of product P and the time and place where user U checked in. Specifically, if the control unit 730 has time-series information that matches the time and place where user U checked in, it associates the time and place where user U checked in with that time-series information. Figure 7(C) shows an example where the time and place where user U (user ID=“X”) checked in is associated with the time-series information of product P.

[0135] As a result, the movement path (Figure 7(A)) can be linked to product P (Figure 7(B)) and user U (Figure 7(C)). This allows the user behavior detection device 700 to detect that the movement path shown in Figure 7(A) represents the movement path of user U (user ID=“X”) to product P (product A, product B, and product C). Furthermore, this movement path includes actions detected by the user behavior detection device 700. In other words, this movement path also includes the movement path of user U before check-in. Therefore, the user behavior detection device 700 can detect what actions user U took with respect to product A before check-in.

[0136] (11-2) First operation example according to the first embodiment Next, a first example of operation according to the first embodiment will be described.

[0137] Figure 8 is a diagram showing a first example of operation according to the first embodiment. Figure 8 shows an example of operation in the user behavior detection device 700.

[0138] As shown in Figure 8, in step S10, the control unit 730 of the user behavior detection device 700 starts processing.

[0139] In step S11, the control unit 730 of the user behavior detection device 700 detects a movement path (movement route) based on image data received from the camera device 500. For example, the control unit 730 detects the movement path shown in Figure 7(A).

[0140] In step S12, the control unit 730 of the user behavior detection device 700 receives the product ID detected by the product detection device 400 from the product detection device 400. The control unit 730 receives the time information of the product ID along with the product ID from the product detection device 400, and obtains the location where the product ID was detected based on the product detection device ID received from the product detection device 400.

[0141] In step S13, the control unit 730 of the user behavior detection device 700 acquires time-series information of product P detected on the movement path (path). For example, based on the movement path detected in step S11 and the product ID (and time information and location) received in step S12, the control unit 730 acquires time-series information of product P as shown in Figure 7(B).

[0142] In step S14, the control unit 730 of the user behavior detection device 700 receives the check-in time and location of user U from the user terminal device 600. Specifically, the control unit 730 receives the check-in time, check-in location, and the user ID of user U who checked in from the user terminal device 600.

[0143] In step S15, the control unit 730 of the user behavior detection device 700 compares the time-series information of product P along the movement path (movement route) acquired in step S13 with the check-in time, check-in location, and user ID received in step S14.

[0144] In step S16, the control unit 730 of the user behavior detection device 700 detects user U's actions toward product P before check-in, based on the time-series information of product P on the movement path (route), the check-in time, the check-in location, and the user ID. Specifically, if the control unit 730 finds a check-in time and location that matches the time and location included in the time-series information of product P on the movement path (route), it associates user U's check-in time and location with that time and location. As a result, the control unit 730 detects that the time-series information of product P is user U's time-series information, that is, that the movement path detected in step S11 is user U's movement path toward product P. Then, by acquiring the actions before check-in included in the movement path, the control unit 730 can detect user U's actions toward product P before check-in.

[0145] Then, in step S17, the control unit 730 of the user behavior detection device 700 terminates the processing related to user behavior detection.

[0146] (12) Second operation example according to the first embodiment Next, a second operational example according to the first embodiment will be described. In the first operational example, an example was described in which the product detection device 400 is arranged within a mesh in the store S, but the arrangement of the product detection device 400 is not limited to this. For example, the product detection device 400 may be arranged for each rack on which the products P are displayed. In the second operational example, an example will be described in which the product detection device 400 is arranged for each rack.

[0147] Regarding the rack, the following explanation will use hanger rack L as an example, but the rack can be anything other than hanger rack L that can display product P, such as a mannequin, a showcase, or a display shelf.

[0148] The second operational example according to the first embodiment will be explained focusing on the differences from the first operational example.

[0149] (12-1) Overview of the second example of operation Figures 9(A) through 10(B) are diagrams illustrating the outline of the second operation example.

[0150] Figure 9(A) is a diagram showing an example of a movement path related to the second operation example. In the second operation example, as in the first operation example, the user behavior detection device 700 generates a movement path based on image data received from the camera device 500. However, in the example shown in Figure 9(A), the movement path includes racks placed inside the store S. Since the installation locations of the racks are fixed, it is possible to include the installation locations of the racks on the movement path, as shown in Figure 9(A). Figure 9(A) shows an example in which three racks (rack L1, rack L2, and rack L3) are installed inside the store S.

[0151] In the second operational example, the user behavior detection device 700 converts the movement path into a rack transition. A rack transition is a path that passes through racks. As shown in Figure 9(B), a rack transition is a conversion of a user's movement path into a path that passes through racks. The control unit 730 of the user behavior detection device 700 generates rack transitions based on the movement path. Specifically, the control unit 730 may determine which racks the movement path passes through based on the location of each point on the movement path and the location of the racks, and generate rack transitions by connecting the racks that were passed through with straight lines. In this case, if there is a user action associated with the movement path, the control unit 730 associates that action with the rack closest to the location where the action took place. In other words, each rack can be associated with information about the actions that took place in each rack. The rack transition will include time information and information about the user's actions for each rack, along with the rack itself. The control unit 730 assigns a rack transition ID to the generated rack transitions.

[0152] In the rack transition shown in Figure 9(B), it is not possible to determine which product P is being transitioned to. Therefore, the user behavior detection device 700 associates a product ID with each rack transition. Figure 10(A) shows the time-series information for each rack of product P on the rack transition, where the product ID is associated with each rack transition.

[0153] In the user behavior detection device 700, similar to the first operation example, the product ID is obtained from the product detection device 400. However, since the product detection device 400 is installed for each rack, it is possible to detect which products have been detected for each rack. Specifically, the product detection device 400 transmits the product ID, the time at which the product ID was obtained, and the rack ID of the rack in which the product detection device 400 is installed to the user behavior detection device 700. The control unit 730 of the user behavior detection device 700 detects rack transitions that have a time and rack that matches the time information and rack ID received from the product detection device 400. The control unit 730 then associates the product ID with the rack on the detected rack transition. As a result, the control unit 730 can obtain time-series information of product P (and its product ID) for each rack detected on the path through the racks (i.e., rack transitions), as shown in Figure 10(A). Figure 10(A) shows an example where product A is detected first in rack L1, then product B is detected in rack L2, and finally product C is detected in rack L3.

[0154] Next, as shown in Figure 10(B), the user activity detection device 700 receives the check-in time, check-in location, and user ID from the user terminal device 600, similar to the first operation example. In the second operation example, the user activity detection device 700 detects which rack the check-in was performed on. Specifically, the control unit 730 of the user activity detection device 700 detects the rack closest to the check-in location as the rack where the check-in was performed, based on the check-in location and the location of the rack. Figure 10(B) shows an example where user U with user ID = "X" checked in on rack L2.

[0155] The user behavior detection device 700 then detects user U's actions toward product P before check-in, based on the time-series information of product P for each rack on the rack transition (Figure 10(A)) and the time and place when user U checked in. Specifically, the control unit 730 of the user behavior detection device 700 matches the time-series information of product P for each rack on the rack transition (Figure 10(A)) with the rack where check-in occurred (Figure 10(B)) to detect whether there is a rack and check-in time that matches the rack and time when product P was detected. If there is a rack and check-in time that matches the rack and time when product P was detected, the control unit 730 associates the time and place when user U checked in with the time-series information of product P on that rack transition. As a result, the control unit 730 can associate user U with the rack transition to which product P is associated, that is, it can detect that the rack transition was performed by user U toward product P. Furthermore, since the actions taken by a particular user are linked to each rack transition, the control unit 730 can detect the actions taken by user U in rack L1 before check-in (Figure 10(B)).

[0156] (12-2) Second operation example according to the first embodiment Next, a second example of operation according to the first embodiment will be described.

[0157] Figure 11 is a diagram showing a second example of operation according to the first embodiment.

[0158] As shown in Figure 11, in step S20, the control unit 730 of the user behavior detection device 700 starts processing.

[0159] In step S21, the control unit 730 of the user behavior detection device 700 detects a movement path (route) based on image data received from the camera device 500. Similar to the first operation example, the control unit 730 detects the actions of a user (Table 1) based on image data received from the camera device 500 and associates the detected actions with the movement path.

[0160] In step S22, the control unit 730 of the user behavior detection device 700 converts the movement path into rack transitions. For example, the control unit 730 may obtain the rack transitions shown in Figure 10(B) based on the location of each point on the movement path and the location of the racks.

[0161] In step S23, when the product detection device 400 receives a beacon transmitted from the equipment communication device 100, it transmits the product ID contained in the beacon, along with time information and rack ID, to the user behavior detection device 700, and the control unit 730 of the user behavior detection device 700 receives the product ID, time information, and rack ID.

[0162] In step S24, the control unit 730 of the user behavior detection device 700 acquires time-series information of product P for each rack detected on the rack transition. Specifically, the control unit 730 acquires time-series information of product P for each rack on the rack transition by associating the rack transition with product P based on the rack transition acquired in step S22 and the product ID, time information, and rack ID received in step S23 (Figure 10(A)).

[0163] In step S25, the control unit 730 of the user behavior detection device 700 receives from the user terminal device 600 the time and place where user U checked in (i.e., check-in time, check-in location, and user ID).

[0164] In step S26, the control unit 730 of the user behavior detection device 700 compares the time-series information of product P for each rack on the rack transition acquired in step S24 with the time and place of check-in of user U received in step S25.

[0165] In step S27, the control unit 730 of the user behavior detection device 700 detects user U's actions toward product P before check-in, based on the time-series information of product P for each rack on the rack transition and the time and place of user U's check-in. Specifically, the control unit 730 associates the time and place of user U's check-in with the time-series information of product P for each rack on the rack transition and detects it as user U's rack transition. Then, by acquiring the actions before check-in included in the rack transition, the control unit 730 can detect user U's actions toward product P for each rack before check-in.

[0166] Then, in step S28, the control unit 730 of the user behavior detection device 700 terminates a series of processes related to user behavior detection.

[0167] In the second operational example, since the product detection devices 400 are installed on a rack-by-rack basis, the number of product detection devices 400 can be reduced compared to the case where product detection devices 400 are installed on a mesh-by-mesh basis as described in the first operational example. Furthermore, since processing is performed on a rack-by-rack basis during user behavior detection (steps S22, S24, and S26 in Figure 11, etc.), it is possible to reduce processing load compared to the first operational example where processing is performed on a point along the movement path.

[0168] (13) Other examples of operation according to the first embodiment Next, other examples of operation according to the first embodiment will be described.

[0169] (13-1) Other examples of operation 1 In the first embodiment, an example was described in which the product detection device 400 acquires a product ID transmitted from a device communication device 100 attached to product P. The product detection device 400 may also receive a device ID transmitted from a device communication device 100 attached to a holding device H. In this case, the product detection device 400 transmits the device ID to the user behavior detection device 700. In the user behavior detection device 700, the device ID and product ID are pre-associated (paired), and the product ID can be obtained from the device ID. Thereafter, the user behavior detection device 700 can perform user behavior detection operations in the same manner as in the first embodiment.

[0170] (13-2) Other examples of operation 2 In the first embodiment, an example was described in which the user behavior detection device 700 detects the behavior of a certain user. For example, the camera device 500 may detect the behavior of a certain user. In this case, the control unit 530 of the camera device 500 may use machine learning to detect the behavior. Specifically, the memory 632 stores training data indicating each behavior, and the control unit 530 may use the training data and the captured image data to infer the presence or absence of each behavior. The camera device 500 transmits the inference result to the user behavior detection device 700. The user behavior detection device 700 no longer detects each behavior and performs user behavior detection operations in the same manner as in the first embodiment using the inference result.

[0171] (13-3) Other examples of operation 3 In the first embodiment, an example was described in which the actions "take to the full-length mirror" and "take to the fitting room" are detected via the user terminal device 600, but they may also be detected via the camera device 500. In this case, the user action detection device 700 may detect each action based on image data acquired from the camera device 500. The user action detection device 700 may use machine learning functions to store training data for each action and use the training data and image data to infer whether or not each action is performed.

[0172] (13-4) Other examples of operation 4 In the first embodiment, an example using beacons was described, but signals other than beacons may be used. For example, signals using short-range wireless communication methods such as UWB (Ultra Wide Band) or NFC (Near Field Communication) may be used. These signals include identification information (each ID), similar to the first example of operation.

[0173] [Other embodiments] A program (e.g., a customer service support program) that causes a computer to execute each process (or operation) described in each of the embodiments described above may be provided. This program can be installed on a computer using a computer-readable medium. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, but may be a CD-ROM or DVD-ROM, for example. Such a recording medium may be included in the control unit 130 of the device communication device 100, the control unit 330 of the beacon transmission device 300, the control unit 430 of the product detection device 400, the control unit 530 of the camera device 500, the control unit 630 of the user terminal device 600, and the control unit 730 of the user behavior detection device 700. Each control unit 130, 330, 430, 530, 630, and 730 may read the program from the recording medium and execute the program to realize the functions (or processes, or operations) described in the embodiments described above.

[0174] Although the embodiments have been described in detail above with reference to the drawings, the specific configuration is not limited to those described above, and various design changes can be made without departing from the gist of the work. Furthermore, the embodiments, operations, or processes described above can be combined as appropriate, as long as they do not contradict each other.

[0175] (Note) The following notes are provided, but they do not limit the embodiments.

[0176] (Note 1) In a user behavior detection system that detects the actions of users who enter a store, A camera device that acquires the user's actions before check-in as image data, A device communication device that transmits product identification information to identify a product when the user moves the product, A product detection device for detecting the aforementioned product identification information, A user terminal device that acquires the time and place when the user checks in, The system includes a user behavior detection device that detects the user's movement path within the store based on the image data acquired from the camera device, and detects the user's actions toward the product before check-in based on the movement path, the product identification information acquired from the product detection device, and the location and time acquired from the user terminal device. The aforementioned check-in indicates that the user authorizes the user behavior detection system to detect the user's actions within the store. User behavior detection system.

[0177] (Note 2) The user behavior detection device acquires time-series information of the products detected along the travel path based on the travel path and the product identification information, and detects the user's actions toward the products before check-in based on the location and time acquired from the user terminal device and the time-series information. The user behavior detection system described in Appendix 1.

[0178] (Note 3) The product detection device is positioned within each mesh in the store, which is virtually divided into a mesh. The user behavior detection system described in Appendix 1 or Appendix 2.

[0179] (Note 4) The product detection device detects the user's actions toward the product before check-in, based on the time-series information including the location and time that match the location and time obtained from the user terminal device. A user behavior detection system as described in any of Appendix 1 to Appendix 3.

[0180] (Note 5) The aforementioned travel path includes information about the user's actions prior to the check-in. A user behavior detection system as described in any of Appendix 1 to Appendix 4.

[0181] (Note 6) The product detection device is positioned for each rack on which the products are displayed. A user behavior detection system as described in any of Appendix 1 to Appendix 5.

[0182] (Note 7) The product detection device converts the movement path into a path via the racks based on the movement path, acquires the products for each rack detected in chronological order along the path via the racks based on the product identification information, and detects the user's actions toward the products before check-in based on the products for each rack detected in chronological order along the path and the time and place when the user checked in. A user behavior detection system as described in any of Appendix 1 to Appendix 6.

[0183] (Note 8) In a user behavior detection device that detects the actions of users who enter a store, A receiving unit that receives the user's actions before check-in as image data from a camera device, receives product identification information to identify products from a product detection device, and receives the time and place when the user checks in from a user terminal device, The system includes a control unit that detects the user's movement path within the store based on the image data acquired from the camera device, and detects the user's actions toward the product before check-in based on the movement path, the product identification information, and the location and time when the user checked in. The aforementioned check-in indicates that the user authorizes the user behavior detection device to detect the user's actions within the store. User behavior detection device.

[0184] (Note 9) A user behavior detection method in a user behavior detection system that detects the actions of users who enter a store, The camera device acquires the user's actions before check-in as image data, The steps include: the device communication device transmits product identification information that identifies the product as the user moves the product; The product detection device includes the step of detecting the product identification information, The user terminal device obtains the time and place when the user checked in, The user behavior detection device includes the steps of detecting the user's movement path within the store based on the image data acquired from the camera device, and detecting the user's actions toward the product before check-in based on the movement path, the product identification information acquired from the product detection device, and the location and time acquired from the user terminal device. The aforementioned check-in indicates that the user authorizes the user behavior detection system to detect the user's actions within the store. User behavior detection method. [Explanation of Symbols]

[0185] 10: User behavior detection system 100: Device communication device 110: Beacon transmission unit 130: Control unit 140: Detection unit 200: Product tag 300: Beacon transmitter 310: Beacon transmitter unit 330: Control unit 332: Memory 400: Product detection device 410: Beacon receiver 420: Wireless communication unit 430: Control unit 432: Memory 500: Camera device 510: Imaging unit 520: Communication unit 530: Control unit 600: User terminal device 610: Beacon receiver unit 620: Wireless communication unit 630: Control unit 632: Memory 700: User behavior detection device 710: Communication unit 730: Control unit H: Holding device L: Hanger rack P: Product S: Store U: User Z: Display Zone

Claims

1. In a user behavior detection system that detects the actions of users who enter a store, A camera device that acquires the user's actions before check-in as image data, A device communication device that transmits product identification information to identify a product when the user moves the product, A product detection device for detecting the aforementioned product identification information, A user terminal device that acquires the time and place when the user checks in, The system includes a user behavior detection device that detects the user's movement path within the store based on the image data acquired from the camera device, and detects the user's actions toward the product before check-in based on the movement path, the product identification information acquired from the product detection device, and the location and time acquired from the user terminal device. The aforementioned check-in indicates that the user authorizes the user behavior detection system to detect the user's actions within the store. User behavior detection system.

2. The user behavior detection device acquires time-series information of the products detected along the travel path based on the travel path and the product identification information, and detects the user's actions toward the products before check-in based on the location and time acquired from the user terminal device and the time-series information. The user behavior detection system according to claim 1.

3. The product detection device is positioned within each mesh in the store, which is virtually divided into a mesh. The user behavior detection system according to claim 1.

4. The product detection device detects the user's actions toward the product before check-in, based on the time-series information including the location and time that match the location and time obtained from the user terminal device. The user behavior detection system according to claim 2.

5. The aforementioned travel path includes information about the user's actions prior to the check-in. The user behavior detection system according to claim 1.

6. The product detection device is positioned for each rack on which the products are displayed. The user behavior detection system according to claim 1.

7. The product detection device converts the movement path into a path via the racks based on the movement path, acquires the products for each rack detected in chronological order along the path via the racks based on the product identification information, and detects the user's actions toward the products before check-in based on the products for each rack detected in chronological order along the path and the time and place when the user checked in. The user behavior detection system according to claim 6.

8. In a user behavior detection device that detects the actions of users who enter a store, A receiving unit that receives the user's actions before check-in as image data from a camera device, receives product identification information to identify products from a product detection device, and receives the time and place when the user checks in from a user terminal device, The system includes a control unit that detects the user's movement path within the store based on the image data acquired from the camera device, and detects the user's actions toward the product before check-in based on the movement path, the product identification information, and the location and time when the user checked in. The aforementioned check-in indicates that the user authorizes the user behavior detection device to detect the user's actions within the store. User behavior detection device.

9. A user behavior detection method in a user behavior detection system that detects the actions of users who enter a store, The camera device acquires the user's actions before check-in as image data, The steps include: the device communication device transmits product identification information to identify the product as the user moves the product; The product detection device includes the step of detecting the product identification information, The user terminal device obtains the time and place when the user checked in, The user behavior detection device includes the steps of detecting the user's movement path within the store based on the image data acquired from the camera device, and detecting the user's actions toward the product before check-in based on the movement path, the product identification information acquired from the product detection device, and the location and time acquired from the user terminal device. The aforementioned check-in indicates that the user authorizes the user behavior detection system to detect the user's actions within the store. User behavior detection method.

Images