Information processing devices, support methods, programs, and store systems

Abstract

The present invention provides an information processing device comprising a customer information generation unit and an information output unit. The customer information generation unit generates customer information, which is information pertaining to a customer visiting a store, on the basis of a captured image of the store interior. The customer information includes at least one of the following: time spent in the store, distance traveled in the store, expected value of products purchased, and expected number of products purchased. When the customer information for a certain customer meets a predetermined first criterion, the information output unit outputs predetermined information for assisting said customer.

Description

Information processing device, support method, non-transitory computer-readable recording medium, and store system

[0001] The present invention relates to an information processing device, a support method, a non-transitory computer-readable recording medium, and a store system.

[0002] Patent Document 1 listed below discloses a system that supports providing appropriate customer service and a shopping-friendly environment. The system in Patent Document 1 uses sensors installed in the store to detect the movement of customers entering the store, determines whether the customer requires assistance based on the degree to which the detected movement matches predetermined conditions, and if the customer is identified as requiring assistance, sets a priority and notifies the store staff.

[0003] Japanese Patent Application Laid-Open No. 2016-153935

[0004] It is difficult to accurately identify customers who need assistance among those staying in a store. Attempting to provide assistance to customers who do not need assistance may end up making them feel inconvenienced.

[0005] One example of the object of the present invention is to provide an information processing device, a support method, a non-transitory computer-readable recording medium, and a store system that solves the problem of accurately identifying customers who need support in a store and providing support services to those customers.

[0006] The information processing device of the present disclosure includes: a customer information generation means for generating customer information about a customer visiting the store based on an image taken inside the store, the customer information including at least one of the length of time spent in the store, the distance traveled in the store, the planned purchase amount of the product, and the planned number of products to be purchased; and an information output means for outputting predetermined information when the customer information satisfies a first criterion.

[0007] The support method in the present disclosure includes at least one computer generating customer information about customers visiting the store based on images taken inside the store, the customer information including at least one of the following information: length of stay in the store, distance traveled in the store, purchase amount of products, and number of products purchased; and outputting specified information when the customer information satisfies a first criterion.

[0008] The non-transitory computer-readable recording medium of the present disclosure records a program that, when executed by at least one computer, causes the at least one computer to function as: customer information generation means that generates customer information about customers visiting the store based on images taken inside the store, the customer information including at least one of the length of time spent in the store, the distance traveled in the store, the planned purchase amount of products, and the planned number of products to be purchased; and information output means that outputs specified information when the customer information satisfies a first criterion.

[0009] A store system according to the present disclosure includes one or more image capture devices installed in a store, and an information processing device communicably connected to the one or more image capture devices. The information processing device also includes: customer information generation means for generating, based on images captured inside the store using the one or more image capture devices, customer information about customers visiting the store, the customer information including at least one of a duration of stay in the store, a distance traveled in the store, a planned purchase amount of a product, and a planned number of products to be purchased; and information output means for outputting predetermined information when the customer information satisfies a first criterion.

[0010] According to the present disclosure, it is possible to accurately identify customers who need assistance in a store and provide assistance services to those customers.

[0011] 1 is a diagram illustrating an example of a functional configuration of an information processing device according to the present disclosure. FIG. 1 is a diagram illustrating an example of a hardware configuration of a computer that realizes an information processing device according to the present disclosure. FIG. 2 is a flowchart illustrating an overview of operation of an information processing device according to the present disclosure. FIG. 3 is a diagram for explaining an overview of condition information. FIG. 4 is a diagram illustrating an example of a configuration of a store system configured including an information processing device according to the present disclosure. FIG. 5 is a sequence diagram illustrating an example operation of a store system including an information processing device according to the present disclosure. FIG. 6 is a diagram for explaining a first specific example of a flow in which a support device operates in response to the output of predetermined information in a store system according to the present disclosure. FIG. 7 is a diagram illustrating a first example of a screen output on a store clerk display or store clerk terminal. FIG. 8 is a diagram illustrating a second example of a screen output on a store clerk display or store clerk terminal. FIG. 9 is a diagram for explaining a second specific example of a flow in which a support device operates in response to the output of predetermined information in a store system according to the present disclosure. FIG. 10 is a diagram illustrating a first example of a screen output on a customer operation terminal or customer terminal. FIG. 11 is a diagram illustrating a second example of a screen output on a customer operation terminal or customer terminal. FIG. 12 is a diagram illustrating a third specific example of a flow in which a support device operates in response to the output of predetermined information in a store system according to the present disclosure. FIG. 13 is a flowchart illustrating an example of a process for determining a support target, executed by an information output unit according to the present disclosure. FIG. 14 is a diagram illustrating an example of information for registering a predetermined action. Fig. 1 is a flowchart showing an example of a process in which a customer information generation unit according to the present disclosure generates customer information; Fig. 2 is a diagram showing an example of information defining a specific area; Fig. 3 is a diagram showing another example of the functional configuration of an information processing device according to the present disclosure; Fig. 4 is a flowchart showing an example of a process executed by a condition information setting unit according to the present disclosure;

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. Note that in all drawings, similar components are denoted by similar reference numerals, and descriptions thereof will be omitted as appropriate. Furthermore, unless otherwise specified, in each block diagram, each block represents a functional configuration, not a hardware configuration. Furthermore, when elements such as blocks in a diagram are linked by arrows, the direction of the arrows is merely intended to make the flow of information easier to understand. In this case, the direction of the arrows does not limit the direction of communication (one-way communication / two-way communication) unless otherwise specified.

[0013] In the following description, "acquisition" includes at least one of the following: a device going to retrieve data or information stored in another device or storage medium (active acquisition), and a device inputting data or information output from another device (passive acquisition). Examples of active acquisition include making a request or inquiry to another device and receiving a reply, and accessing and reading information from another device or storage medium. Examples of passive acquisition include receiving information that is distributed (or transmitted, push notification, etc.). Furthermore, "acquisition" may also mean selecting and acquiring data or information from received data or information, or selecting and receiving distributed data or information.

[0014] First Embodiment <Functional Configuration Example> Fig. 1 is a diagram illustrating an example of the functional configuration of an information processing device according to the present disclosure. The information processing device 10 illustrated in this figure includes a customer information generation unit 110 and an information output unit 120.

[0015] The customer information generation unit 110 generates information about customers visiting a store (hereinafter also referred to as "customer information"). The customer information generation unit 110 acquires images of the store from one or more image capture devices (not shown) installed in the store. Then, based on the acquired images, the customer information generation unit 110 generates customer information including at least one of the following: (1) length of stay in the store (2) distance traveled in the store (3) planned purchase amount of products (4) planned number of products to purchase

[0016] The information output unit 120 outputs predetermined information when the customer information generated by the customer information generation unit 110 satisfies a predetermined criterion (hereinafter also referred to as "first criterion"). Here, "predetermined information" refers to any information used to assist the customer. The information output unit 120 also outputs the predetermined information to a specific device used to assist the customer (hereinafter also referred to as "assistance device"). Specific examples of the "predetermined information" and specific examples of the operation of the information output unit 120 will be described later.

[0017] <Hardware Configuration Example> FIG. 2 is a diagram illustrating an example of the hardware configuration of a computer that realizes an information processing device according to the present disclosure.

[0018] The computer 1000 illustrated in this figure includes a bus 1010 , a processor 1020 , a memory 1030 , a storage device 1040 , an input / output interface 1050 , and a network interface 1060 .

[0019] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0020] The processor 1020 is a central processing unit (CPU) or a graphics processing unit (GPU).

[0021] The memory 1030 is a main storage device realized by a random access memory (RAM) or the like.

[0022] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), etc. The storage device 1040 stores at least a program module that realizes the functions of the information processing device 10 described above (the customer information generation unit 110 and the information output unit 120).

[0023] The processor 1020 loads a program module read from the storage device 1040 onto the memory 1030 and executes the program module, thereby realizing the function corresponding to the program module. For example, the processor 1020 reads a program module corresponding to the customer information generation unit 110 onto the memory 1030 and executes the program module, thereby realizing the function of the customer information generation unit 110 described in the present disclosure. Similarly, the processor 1020 reads a program module corresponding to the information output unit 120 onto the memory 1030 and executes the program module, thereby realizing the function of the information output unit 120 described in the present disclosure. The operation of the processor 1020 is common to each embodiment included in the present disclosure.

[0024] The above-described program modules may be recorded on a recording medium other than the storage device 1040. The recording medium on which the program modules are recorded includes any non-transitory, tangible medium usable by the computer 1000. The recording medium on which the program modules are recorded may have embedded therein program code that is readable by the computer 1000 (processor 1020).

[0025] The input / output interface 1050 is an interface for connecting the computer 1000 (information processing device 10) to various input / output devices.

[0026] The network interface 1060 is an interface for connecting the computer 1000 (information processing device 10) to a network. This network includes, for example, a local area network (LAN) and a wide area network (WAN). The network interface 1060 connects the computer 1000 (information processing device 10) to various networks via a wireless or wired connection.

[0027] The computer 1000 (information processing device 10) can also be connected to various devices (not shown) via the input / output interface 1050 or the network interface 1060. The "various devices" referred to here are not particularly limited, but include, for example, input devices such as operation buttons, a keyboard, a touch panel, a mouse, and a microphone, and output devices such as a display, a speaker, and a printer.

[0028] <Outline of Operation> An outline of the operation of the information processing device 10 according to the present disclosure will now be described with reference to the drawings. Fig. 3 is a flowchart showing an outline of the operation of the information processing device 10 according to the present disclosure.

[0029] First, the customer information generation unit 110 acquires images generated by one or more imaging devices installed in the store (step S102). For example, the customer information generation unit 110 acquires video images (multiple frame images) from each imaging device in approximately real time. The customer information generation unit 110 then analyzes the acquired frame images and generates customer information for the customer appearing in the frame images (step S104). First, the customer information generation unit 110 sequentially processes images acquired from the imaging device 20 using a known tracking algorithm to identify multiple frame images containing the same customer. The customer information generation unit 110 then calculates the length of time the customer spent in the store based on the length of time the image was captured corresponding to the identified multiple frame images. The customer information generation unit 110 may also calculate the stay time for each area within the store (e.g., for each product area classified by the type of product sold). For example, the customer information generation unit 110 may also calculate the amount of change in the customer's position based on the identified multiple frame images. In this case, the information output unit 120 can calculate the customer's movement distance by summing the amount of change in the customer's position between frame images. The customer's movement distance may be based on an image coordinate system or a real-world coordinate system. The customer information generation unit 110 can acquire information about the products the customer plans to purchase by processing the images acquired from the imaging device 20 using a known object recognition algorithm. For example, the customer information generation unit 110 can determine whether the customer has added any products to their shopping cart by processing the images. The customer information generation unit 110 then refers to a product master (not shown) based on the results of identifying the products added to the shopping cart using their feature information. As a result, the customer information generation unit 110 can acquire price information for the products identified as "products the customer plans to purchase." The customer information generation unit 110 can calculate the "planned purchase amount" by summing the product prices acquired in this way. The customer information generation unit 110 can generate information indicating the "planned purchase quantity" by counting the number of products added to the shopping cart.

[0030] The information output unit 120 determines whether the customer information generated by the customer information generation unit 110 satisfies a predetermined criterion (first criterion) (step S106). If the customer information satisfies the first criterion (step S106: YES), the information output unit 120 determines that the customer corresponding to the customer information is a support target and outputs predetermined information (step S108). On the other hand, if the customer information does not satisfy the first criterion (step S106: NO), the information output unit 120 determines that the customer corresponding to the customer information is not a support target and does not output the predetermined information.

[0031] The illustrated information processing device 10 generates customer information for a customer based on an image of the customer visiting a store. Here, the customer information includes at least one of the following: time spent in the store, distance traveled in the store, planned purchase amount, and planned purchase quantity. As will become clear from the description below, these pieces of information can be used individually or in combination as indicators for determining the need for assistance. The illustrated information processing device 10 determines whether the customer needs assistance based on such customer information. The operation of this information processing device 10 is expected to improve the accuracy of detecting customers who need assistance from among customers visiting a store. If the illustrated information processing device 10 determines that the customer needs assistance, it outputs predetermined information to be used to provide assistance to the customer. The predetermined information is output to a specific device used in customer assistance services, as described in detail below. In other words, the present disclosure provides an information processing device, assistance method, and non-transitory computer-readable recording medium that solve the problem of "accurately identifying customers who need assistance in a store and providing assistance services to those customers." The present disclosure also provides a store system for customer assistance using the information processing device 10. Such a store system will be described in another embodiment.

[0032] A detailed example of the information processing device 10 according to the present disclosure will be described below.

[0033] As an example, the customer information generation unit 110 analyzes images acquired from the imaging device 20 to generate customer information indicating the length of time a customer has spent in a store or an area of ​​the store. In this case, the information output unit 120 can determine whether the customer needs assistance based on the customer's length of stay. For example, if a customer has been staying in a store or an area within the store for a long time, the information output unit 120 determines that the customer is having trouble finding the product they are looking for. For example, the information output unit 120 compares the length of stay in the store indicated by the customer information with a predetermined reference length of stay as a determination criterion to determine whether the customer should be supported. If the length of stay in the customer information generated for a customer is longer than the reference length of stay (i.e., satisfies the first criterion), the information output unit 120 detects (determines) the customer as a support recipient. If a customer is detected as a support recipient, the information output unit 120 outputs predetermined information to, for example, an assistance device described in another embodiment below. As a result, assistance is provided to the customer determined to be a support recipient.

[0034] As another example, the customer information generation unit 110 analyzes images acquired from the imaging device 20 to generate customer information indicating the distance a customer travels in the store. In this case, the information output unit 120 can determine whether the customer needs assistance based on the distance traveled in the store. For example, if a customer travels a long distance (i.e., walks around the store for a long time), the information output unit 120 determines that the customer is having trouble finding the product they are looking for. For example, the information output unit 120 compares the distance traveled in the store indicated by the customer information with a predetermined reference distance as a judgment criterion to determine whether the customer should be supported. If the distance traveled in the customer information generated for a customer is longer than the reference distance (i.e., satisfies the first criterion), the information output unit 120 detects (determines) the customer as a support target. When a customer is detected as a support target, the information output unit 120 outputs predetermined information to, for example, an assistance device described in another embodiment below. As a result, assistance is provided to the customer determined as a support target.

[0035] The following example illustrates a case in which the customer information generation unit 110 analyzes an image acquired from the imaging device 20 and includes multiple pieces of the above four pieces of information in the customer information. For example, the customer information generation unit 110 generates customer information indicating at least one of the customer's stay time in a store or an area of ​​the store and the distance traveled within the store, and at least one of the total price or total number of items the customer plans to purchase. In this case, the information output unit 120 determines whether at least one of the customer's stay time and the distance traveled, and at least one of the planned price and the number of items the customer plans to purchase, respectively, satisfy preset conditions. In this case, the information output unit 120 references information indicating conditions based on at least one of the customer's stay time and the distance traveled, and at least one of the planned price and the number of items the customer plans to purchase (hereinafter also referred to as "condition information"). The condition information is pre-stored in a storage area accessible to the information output unit 120, such as the memory 1030 or the storage device 1040. The information output unit 120 determines whether or not to make a customer a support target based on the result of comparing pre-stored condition information with the customer information generated by the customer information generation unit 110. In other words, the information output unit 120 determines whether or not to output predetermined information based on the result of comparing pre-stored condition information with the customer information generated by the customer information generation unit 110. A more detailed flow will be described below.

[0036] As an example, suppose the condition information defines the length of stay in the store (reference stay time) and the planned purchase amount of a product (reference planned purchase amount) as "criteria for determining whether or not assistance is needed." In this case, the customer information generation unit 110 generates customer information about customers visiting the store, including at least the length of stay in the store and the planned purchase amount of the product. The information output unit 120 then operates to output predetermined information if the stay time indicated by the customer information is equal to or longer than the reference stay time and the planned purchase amount indicated by the customer information is equal to or less than the reference planned purchase amount. This operation allows the information output unit 120 to determine (detect) "customers who stay in the store for a long time but have a small planned purchase amount of the product." In other words, the information output unit 120 can determine (detect) "customers who are having trouble finding the product they are looking for," i.e., "customers who are likely to need assistance."

[0037] As another example, suppose the condition information defines the distance traveled within the store (reference distance traveled) and the planned purchase amount of a product (reference planned purchase amount) as "criteria for determining whether assistance is needed." In this case, the customer information generation unit 110 generates customer information for customers visiting the store, including at least the distance traveled within the store and the planned purchase amount of the product. The information output unit 120 then operates to output predetermined information if the distance traveled indicated by the customer information is equal to or greater than the reference distance traveled and the planned purchase amount indicated by the customer information is equal to or less than the reference planned purchase amount. Here, the longer the customer travels within the store, the more likely the customer is to stay in the store for a long time. This operation allows the information output unit 120 to determine (detect) "customers who stay in the store for a long time but have a small planned purchase amount of the product." In other words, the information output unit 120 can determine (detect) "customers who are having trouble finding the product they are looking for," i.e., "customers who are likely to need assistance."

[0038] As another example, suppose the condition information defines the length of time spent in the store (reference length of time spent) and the number of items planned to be purchased (reference number of items planned to be purchased) as "criteria for determining whether assistance is needed." In this case, the customer information generation unit 110 generates customer information about customers visiting the store, including at least the length of time spent in the store and the number of items planned to be purchased. The information output unit 120 then operates to output predetermined information if the length of time spent in the store indicated by the customer information is equal to or greater than the reference length of time spent, and the number of items planned to be purchased indicated by the customer information is equal to or less than the reference number of items planned to be purchased. This operation allows the information output unit 120 to determine (detect) "customers who have spent a long time in the store but have few items planned to purchase." In other words, the information output unit 120 can determine (detect) "customers who are having difficulty finding the products they are looking for," i.e., "customers who are likely to need assistance." Here, the price of each product varies. Therefore, using the "number of items planned to be purchased" as an indicator rather than the "amount planned to be purchased" may more accurately identify customers who need assistance.

[0039] The condition information may define a single criterion by associating the "reference stay time," "reference travel distance," "reference planned purchase amount," and "reference planned purchase quantity." This point will be explained using FIG. 4. FIG. 4 is a diagram for explaining the overview of the condition information. As an example of condition information, this diagram shows information in a two-dimensional space with the "stay time" and "planned purchase amount" on the horizontal and vertical axes, respectively. In the example shown in this diagram, multiple (nine) subspaces are defined in the two-dimensional space. Using the condition information (information in the two-dimensional space) shown in this diagram, the information output unit 120 can determine whether a customer needs assistance based on the spatial location determined by the stay time and planned purchase amount indicated by the customer information. For example, the information output unit 120 can determine that the longer the customer stays in the store, the lower right (the shaded subspace) the location of the space containing the intersection of the stay time and planned purchase amount indicated by the customer information, the lower the position of the space. If customer information is located in this space, it can be inferred that the customer corresponding to that customer information is hesitating to purchase a product for some reason. The same applies to the relationship between the "standard travel distance" and the "standard planned purchase amount," and the relationship between the "standard stay time" and the "standard planned purchase number of items." Furthermore, three or more types of information may be used among the "standard stay time," "standard travel distance," "standard planned purchase amount," and "standard planned purchase number of items." In this case, the information output unit 120 reads condition information that defines a three-dimensional or higher space with each piece of information as its axis. Even when using such condition information, the information output unit 120 can identify a spatial location specified by the customer information in the same way as when using the condition information in FIG. 4. Then, the information output unit 120 can determine whether or not it is necessary to output predetermined information (whether or not the customer is a target customer for support) based on the identified spatial location.

[0040] The values ​​of the "standard stay time," "standard travel distance," "standard planned purchase amount," and "standard planned purchase item count" indicated by the condition information may not be fixed but may vary depending on the situation. For example, the information output unit 120 may be configured to correct the values ​​of the "standard stay time," "standard travel distance," "standard planned purchase amount," and "standard planned purchase item count" using customer attribute information generated by the customer information generation unit 110 and attribute information of products in the area where the customer is staying. As an example, the information output unit 120 may correct the value used as the judgment standard according to the customer's attributes (e.g., age, gender, etc.). For example, if the customer attribute information indicates an elderly demographic (e.g., 65 years or older), the information output unit 120 may correct the standard value defined by the condition information to a smaller value. In other words, the information output unit 120 may relax the standard for determining whether assistance is required. As another example, if a product (e.g., a prepared meal) that tends to take customers a relatively long time to decide which product to purchase is identified by image analysis, the information output unit 120 may correct the standard value defined by the definition information to a larger value. That is, the information output unit 120 may tighten the criteria for determining whether assistance is needed. As another example, the information output unit 120 may correct the reference value defined by the condition information based on the degree of match between the customer's attribute information (age and gender) and the product's attribute information (target age group and gender). For example, assume that the customer's attribute information, "male, in his 40s," and the product's attribute information, "female, in his 20s," are obtained through image analysis by the customer information generation unit 110. In this case, the degree of match between the customer's attribute information and the product's attribute information is low. In such a case, the information output unit 120 may correct the reference value defined by the definition information to a smaller value. That is, the information output unit 120 may relax the criteria for determining whether assistance is needed. This is expected to improve the accuracy of determining (detecting) customers who need assistance.

[0041] Second Embodiment <System Configuration Example> Fig. 5 is a diagram illustrating the configuration of a store system configured to include an information processing device according to the present disclosure. The store system 1 illustrated in this diagram is configured to include an information processing device 10 according to the present disclosure, one or more imaging devices 20, and one or more support devices 30. In the example of this diagram, the information processing device 10, the one or more imaging devices 20, and the one or more support devices 30 are each connected via an arbitrary network 40.

[0042] <About the Imaging Device> The imaging device 20 is installed in an appropriate position in the store to capture images of customers visiting the store. The imaging device 20 is installed in a position determined based on the shape and layout of the store (such as the ceiling or wall of the store, or shelves displaying products). By analyzing the images generated by the imaging device, the following information is generated for each customer, for example: - Length of stay in the entire store or length of stay in each specific area (product sales floor) within the store - Distance traveled or route traveled within the store - Attribute information such as the customer's gender, age, and whether or not they are accompanied by someone

[0043] <About the Assistance Device> The assistance device 30 is a variety of devices used to assist customers. For example, the following devices are used as assistance devices 30: - Displays installed in the store to display information for store staff - Staff terminals such as mobile terminals carried by store staff and management terminals installed in the back room - Customer operation terminals placed in areas where products are displayed - Customer terminals such as smartphones carried by each customer - Autonomous mobile robots that patrol the store to assist customers

[0044] <Operation Example of Store System> An operation example of a store system according to the present disclosure will be described with reference to Fig. 6. Fig. 6 is a sequence diagram showing an operation example of a store system including an information processing device according to the present disclosure.

[0045] First, one or more image capturing devices 20 installed in the store generate images of the store locations corresponding to their respective fields of view (step S202).

[0046] The information processing device 10 acquires images generated by one or more imaging devices 20 installed in the store, for example, in substantially real time (step S204). The customer information generation unit 110 of the information processing device 10 analyzes the acquired images to generate customer information including at least one of the four types of information exemplified in the first embodiment (step S206). The information output unit 120 of the information processing device 10 determines whether the generated customer information satisfies a predetermined criterion (first criterion) (step S208). If it is determined that the customer information generated for a certain customer satisfies the first criterion (step S208: YES), the information output unit 120 outputs predetermined information to the support device 30 to support the customer. On the other hand, if it is determined that the customer information generated for a certain customer does not satisfy the first criterion, no processing is performed by the information output unit 120. Note that the process of generating customer information based on the images generated by the imaging device 20 and the process of determining whether the generated customer information satisfies the first criterion (determining whether to output predetermined information) are repeatedly performed.

[0047] In response to acquiring the predetermined information from the information processing device 10, the support device 30 executes an operation for supporting the customer (step S210).

[0048] The following describes, with reference to the drawings, the flow of operations of the support device 30 in response to the output of predetermined information in the store system 1 according to the present disclosure.

[0049] 7 is a diagram illustrating a first specific example of the flow of operations of the support device 30 in response to the output of predetermined information in the store system 1 according to the present disclosure. In the example of this diagram, the support device 30 is a display 31 provided for the store clerk or a store clerk terminal 32 used by the store clerk. Here, the store clerk terminal 32 may be, for example, a mobile terminal carried by each store clerk or a management terminal installed in the back room.

[0050] In a first specific example, the information output unit 120 outputs to the display 31 or the clerk terminal 32 at least information indicating that the customer needs assistance. Here, if the customer information generation unit 110 has acquired information indicating the customer's location, such as the area where the customer is staying, the information output unit 120 may output to the display 31 or the clerk terminal 32 information notifying the presence and location of the customer determined to need assistance. In response to acquiring information from the information output unit 120, the display 31 or the clerk terminal 32 outputs a screen such as that shown in FIG. 8 . FIG. 8 is a diagram showing a first example of a screen output on the clerk display 31 or the clerk terminal 32. The screen shown in this figure can notify the clerk that the customer needs assistance. Furthermore, if the screen includes information indicating the customer's location, such as the screen shown in this figure, the clerk can quickly move to the customer in need of assistance by checking the customer's location on the screen.

[0051] The information output unit 120 may also output the analysis results of the image captured by the imaging device 20 to the display 31 or the clerk terminal 32. In this case, for example, the customer information generation unit 110 extracts an image area of ​​the customer from the image to be processed to generate an image of the customer. For example, the customer information generation unit 110 may analyze the image area of ​​the customer to generate attribute information of the customer. The information output unit 120 may further output the generated image of the customer and attribute information of the customer to the display 31 or the clerk terminal 32. In this case, the display 31 or the clerk terminal 32 may output a screen such as that shown in FIG. 9 . FIG. 9 is a diagram showing a second example of a screen output on the clerk display 31 or the clerk terminal 32. The screen illustrated in this figure further includes an image of the customer (an image including at least the customer's face). This allows the clerk to easily identify the target customer. The screen illustrated in this figure further includes information about the customer's attributes (e.g., gender, age, whether the customer is accompanied, etc.). Information about customer attributes can help store staff create a favorable customer service plan.

[0052] The display 31 or the clerk terminal 32 may output sound information (a predetermined alarm sound or a predetermined voice message such as "There is a customer who needs assistance") along with the screens exemplified in Figures 8 and 9. By using sound information, the clerk can easily notice that a customer needs assistance even when the clerk is not looking at the display 31 or the clerk terminal 32. Furthermore, if the clerk terminal 32 has a vibration function, the clerk terminal 32 may also use the vibration function. In this way, the clerk can more easily notice that a customer needs assistance.

[0053] 10 is a diagram illustrating a second specific example of the flow of operations of the support device 30 in response to the output of predetermined information in the store system 1 according to the present disclosure. In the example of this figure, the support device 30 is a customer operation terminal 33 provided in an area where products are displayed or a customer terminal 34 carried by a customer.

[0054] In the second specific example, the information output unit 120 outputs information for displaying a product search screen to the operation terminal 33 or the customer terminal 34. The "information for displaying a product search screen" may be the drawing data of the product search screen itself, or a trigger command for displaying the product search screen on an application installed on the operation terminal 33 or the customer terminal 34. In response to acquiring information from the information output unit 120, the operation terminal 33 or the customer terminal 34 outputs a screen such as that shown in FIG. 11 . FIG. 11 is a diagram showing a first example of a screen output on the customer operation terminal 33 or the customer terminal 34. A customer can search for a desired product by operating the product search screen displayed on the operation terminal 33 or the customer terminal 34.

[0055] The information output unit 120 may also output the analysis results of the image captured by the imaging device 20 to the operation terminal 33 or the customer terminal 34. In this case, for example, the customer information generation unit 110 analyzes the image to be processed to generate information indicating the area (merchandise section) where the customer is staying and the identification results of the product the customer is holding. The information output unit 120 may further output the generated information to the operation terminal 33 or the customer terminal 34. In this case, the operation terminal 33 or the customer terminal 34 can output a screen such as that shown in FIG. 12 . FIG. 12 is a diagram showing a second example of a screen output on the customer operation terminal 33 or the customer terminal 34. The screen illustrated in this figure differs from the screen shown in the first specific example in that the initial state of the screen is determined based on the identification results of the area where the customer is staying and the product the customer is holding. For example, assume that the information obtained from the information output unit 120 indicates that (1) the customer is staying in the daily necessities area and (2) the customer is holding laundry detergent. In this case, the operation terminal 33 and the customer terminal 34 can output a screen, as shown in the figure, in which (1) the search target is the daily necessities area category, and (2) the search results for related products (e.g., products from the same manufacturer) of the product picked up are already displayed. By outputting such a screen, it is expected that usability will be improved.

[0056] The operation terminal 33 and the customer terminal 34 may output sound information (a predetermined alarm sound or a predetermined voice message such as "Please search for the product on this screen") along with the screens exemplified in Figures 11 and 12. By using sound information, even when the customer is not looking at the operation terminal 33 or the customer terminal 34, the customer can easily notice that assistance is being provided from the store system 1 according to the present disclosure. Furthermore, if the customer terminal 34 has a vibration function, the customer terminal 34 may also use the vibration function. In this way, the customer can more easily notice that assistance is being provided from the store system 1 according to the present disclosure.

[0057] FIG. 13 illustrates a third specific example of the flow of operations of the support device 30 in response to the output of predetermined information in the store system 1 according to the present disclosure. In this example, the support device 30 is an autonomous mobile robot 35. The autonomous mobile robot 35 patrols the store, one of its purposes being to provide support to customers. The autonomous mobile robot 35 may be programmed to automatically determine its patrol route based on past performance. For example, the autonomous mobile robot 35 patrols a preset initial route during initial operation. Thereafter, when a support task, such as that described below, occurs, the autonomous mobile robot 35 performs a process to optimize its route based on the location where the support task was performed. For example, the autonomous mobile robot 35 reconstructs its travel route so that it patrols around locations where many customers needing support are detected. This improves the efficiency of customer support tasks using the autonomous mobile robot 35. The optimization of the autonomous mobile robot 35's travel route is automatically performed, for example, every predetermined period (e.g., every other day). Alternatively, a store manager may appropriately optimize the autonomous mobile robot 35's travel route using a store management terminal (not shown).

[0058] In the third specific example, the information output unit 120 outputs information indicating the location of the customer to the autonomous mobile robot 35. The customer information generation unit 110 can generate, as information indicating the location of the customer to be assisted, for example, location information (information indicating the image capture area) associated with the imaging device 20 that captured the image to be processed, or information indicating the location coordinates within the store identified by analyzing the image. The information output unit 120 outputs such customer location information to the autonomous mobile robot 35.

[0059] The autonomous mobile robot 35 moves to the location indicated by the customer's location information obtained from the information output unit 120. The autonomous mobile robot 35 has, for example, a tablet terminal on which a product search application such as that described in the second specific example is installed. When the autonomous mobile robot 35 reaches a customer, it outputs a predetermined voice message such as "Please search for the product on this terminal." The customer can search for the product they are looking for using the tablet terminal possessed by the autonomous mobile robot 35. In this way, the store system 1 can support customers using the autonomous mobile robot 35.

[0060] Third Embodiment The information processing device 10 preferably further includes a mechanism for more accurately determining customers who need assistance. Such a mechanism will be described below as an example.

[0061] As an example, the customer information generation unit 110 is configured to analyze customer behavior based on an image generated by the imaging device 20. The customer information generation unit 110 analyzes the image to identify customer behavior in the image and generate information indicating the identified customer behavior. The customer information generation unit 110 then includes the generated information indicating the customer behavior in the customer information. The information output unit 120 determines whether the customer needs assistance based on the information regarding the customer behavior included in the customer information generated by the customer information generation unit 110. For example, the information output unit 120 determines whether the customer behavior indicated by the information included in the customer information corresponds to a behavior (predetermined behavior) pre-registered as a behavior indicating a low likelihood of needing assistance. If it is determined that the customer behavior indicated by the information included in the customer information corresponds to the predetermined behavior, the information output unit 120 determines that the customer corresponding to the customer information is not a customer eligible for assistance. In this case, the information output unit 120 does not output the predetermined information as described above.

[0062] FIG. 14 is a flowchart showing an example of a process for determining a support target, which is executed by the information output unit 120 according to the present disclosure.

[0063] First, the customer information generating unit 110 acquires images generated by one or more image capturing devices installed in the store (step S302). This process is similar to the process described above.

[0064] Next, the customer information generation unit 110 generates customer information by analyzing the acquired image (step S304). Here, the customer information generation unit 110 further executes a process of generating information indicating the behavior of customers present in the acquired image. The customer information generation unit 110 can identify the behavior of each person present in the image using known image analysis technology. For example, the customer information generation unit 110 can identify the movement of specific body parts (e.g., eyes, mouth, hands, feet, etc.) of each person or the object each person is holding using known image analysis technology. The customer information generation unit 110 can also identify a store clerk from among the people appearing in the image using, for example, an image database in which characteristic information of each store clerk is pre-registered. As a result, for example, the customer information generation unit 110 can identify behaviors such as "a customer operating his / her smartphone" or "a customer talking to a store clerk." The customer information generation unit 110 includes information indicating the customer's behavior identified through image analysis (e.g., identification information assigned to each behavior) in the customer information generated for that customer.

[0065] Next, the information output unit 120 determines whether the customer corresponding to the customer information generated by the customer information generation unit 110 is a support target based on the customer information generated by the customer information generation unit 110. This determination process includes a process of determining whether the customer information satisfies the first criterion as described above (step S306) and a process of determining whether the customer information satisfies the criterion for determining that the customer is not a support target (second criterion) (step S308). Note that the order in which these determinations are performed is not limited to the order shown in the figure. The process of step S304 may be performed after the process of step S306, or these processes may be performed in parallel.

[0066] The determination process in step S306 is the same as the determination process described above.

[0067] In the determination process of step S308, the information output unit 120 determines whether the customer behavior indicated by the information included in the customer information corresponds to a predetermined behavior. FIG. 15 is a diagram showing an example of information for registering a predetermined behavior. The information illustrated in FIG. 15 is pre-stored in a storage area accessible to the information output unit 120, such as the memory 1030 or the storage device 1040. The "predetermined behavior" illustrated in FIG. 15 includes at least the behavior of "operating a smartphone" and the behavior of "conversing with a store clerk." Note that these are merely examples of "predetermined behavior." Any behavior can be set as the "predetermined behavior." Assume now that the behavior indicated by the information included in the customer information corresponds to the predetermined behavior, i.e., satisfies the second criterion. In this case (step S308: YES), the information output unit 120 determines that the customer corresponding to the customer information is not eligible for support, regardless of the result of the determination process of step S306, and does not output the predetermined information. On the other hand, assume that the behavior indicated by the information included in the customer information does not correspond to the predetermined behavior, i.e., does not satisfy the second criterion. In this case (step S308: NO), the information output unit 120 determines that the customer corresponding to the customer information is a support target, and outputs predetermined information (S310). Note that if the result of the determination process in step S308 is "NO," the process is executed on the condition that the result of the determination process in step S306 is also "YES," as shown in the figure.

[0068] According to the example described here, whether a customer is eligible for assistance is determined based on whether the customer's behavior obtained as a result of analyzing the image corresponds to a predetermined behavior. Here, "predetermined behavior" means "behavior that is pre-registered as indicating a low likelihood of needing assistance." By taking "customer behavior" into account as shown in this example, it is possible to more accurately identify customers who truly need assistance. As a result, assistance work at the store can be carried out more smoothly.

[0069] As another example, characteristics of the area in which a customer is located may be used to accurately determine whether a customer needs assistance. For example, if a customer is located in an area near a checkout waiting area, the waiting time may be excluded from the waiting time used by the information output unit 120 when determining whether the customer is eligible for assistance. The presence of a customer in an area waiting for payment indicates that the customer has the desired product in hand and is likely not in need of further assistance. Therefore, when the customer information generation unit 110 identifies from an image that a customer is located in a specific area, such as an area waiting for payment, it may be configured to calculate the customer's stay time excluding the time spent in that specific area.

[0070] 16 is a flowchart showing an example of a process for generating customer information by the customer information generation unit 110 according to the present disclosure. The flowchart in this figure is executed in the process for generating customer information (e.g., step S104 in the flowchart in FIG. 3, step S206 in the sequence diagram in FIG. 6, and step S304 in the flowchart in FIG. 14).

[0071] First, the customer information generation unit 110 analyzes the acquired image and identifies the position of the customer in the image (step S402). For example, the customer information generation unit 110 can identify the customer's position within the store based on the characteristics of the store's interior that appear in the image (such as the position of pillars, the layout of product shelves, and character strings or marks printed on the floor or wall). The customer information generation unit 110 may calculate the customer's position as position coordinates in a specific coordinate system, or may acquire information corresponding to the position identified by the image analysis results (such as an identifier assigned to each area) as the customer's position.

[0072] Next, the customer information generation unit 110 determines whether the location of the customer identified in step S402 is within a predetermined "specific area" (step S404). The customer information generation unit 110 can determine whether the customer is located in a "specific area" by, for example, referring to information such as that shown in FIG. 17. FIG. 17 is a diagram showing an example of information defining a specific area. The example in FIG. 17 includes information indicating the respective ranges of a "queue area for waiting for checkout" and a "service counter area." The customer information generation unit 110 can determine whether the location of the customer identified in step S402 is within a "specific area" using the information such as that shown in FIG. 17. If it is determined that the customer's location is within a "specific area" (step S404: YES), the customer information generation unit 110 maintains the previously calculated information on the customer's stay time without changing it. On the other hand, if it is determined that the customer's location is not included in the "specific area" (step S404: NO), the customer information generation unit 110 updates the information on the customer's stay time that has already been calculated (step S406).

[0073] 16 , the information output unit 120 determines whether or not the customer corresponding to the customer information is a support target, based on the customer information generated by the customer information generation unit 110. This determination process is similar to the determination process described above.

[0074] According to another example described above, the time a customer spends in a specific area is excluded from the store's stay time. As a result, the stay time of a customer who is not considered to need assistance is counted unnecessarily, reducing the possibility of the customer being determined to be a customer requiring assistance. In other words, it is possible to prevent customers who are already unlikely to need assistance from being detected as customers needing assistance. As a result, assistance work at the store can be carried out more smoothly.

[0075] Fourth Embodiment The characteristics of customers visiting each store may differ. In such cases, the customers who need assistance may also differ from store to store. Therefore, the information processing device 10 may further have a configuration for setting, for each store, information (condition information) that defines conditions for determining whether a customer needs assistance.

[0076] FIG. 18 is a diagram illustrating another example of the functional configuration of an information processing device 10 according to the present disclosure. The information processing device 10 illustrated in this diagram includes a condition information setting unit 130 in addition to the customer information generation unit 110 and information output unit 120 described above. The condition information setting unit 130 sets condition information for each store based on performance information collected for each store. The performance information for each store is generated based on customer information actually measured or acquired at each store. For example, the performance information is generated by linking at least one of the measured time spent at the store and the distance traveled at the store for each customer with at least one of the total price and total number of items purchased by the customer. The performance information generated for each store is transmitted to and stored in the information processing device 10 or a database (not shown), for example, together with information identifying the store (e.g., an identifier assigned to each store).

[0077] FIG. 19 is a flowchart showing an example of processing executed by the condition information setting unit 130 according to the present disclosure.

[0078] First, the condition information setting unit 130 reads the performance information for each store stored in a predetermined storage area (step S502). Then, based on the read performance information for each store, the condition information setting unit 130 sets, for each store, condition information indicating the conditions for determining whether a customer is eligible for support (step S504). For example, based on the performance information for each store, the condition information setting unit 130 calculates a distribution based on the product sales performance (total price or total number of items purchased by the customer) and the customer's behavior at the store (time spent at the store or distance traveled at the store). Based on the calculated distribution, the condition information setting unit 130 then sets the position of the bottom X% of product sales performance (total price or total number of items purchased by the customer) and the top Y% of customer behavior performance (time spent at the store or distance traveled at the store) as the standard for that store. Note that "X" and "Y" can be set to any value within an appropriate range.

[0079] According to the example described above, the criteria for determining whether a customer is eligible for support are set for each store based on the performance information of each store. By changing the criteria according to the characteristics of each store, each store can more appropriately determine whether a customer needs support.

[0080] Although the present disclosure has been described with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0081] In addition, although the flowcharts and sequence diagrams used in the above description show multiple steps (processes) in a sequential order, the execution order of the steps in each embodiment is not limited to the order shown. In each embodiment, the order of the steps shown in the diagrams can be changed as long as it does not cause any problems in terms of the content. Furthermore, the above-described embodiments can be combined as long as the content is not contradictory.

[0082] Some or all of the above embodiments may be described as in the following supplementary notes, but are not limited to them. 1. An information processing device comprising: customer information generation means for generating customer information based on images taken inside the store, as information about a customer visiting the store, the customer information including at least one of the duration of stay in the store, the distance traveled in the store, the planned purchase amount of products, and the planned number of products to be purchased; and information output means for outputting predetermined information when the customer information satisfies a first criterion. 2. The information processing device described in 1., wherein the predetermined information includes information indicating that the customer needs assistance. 3. The information processing device described in 1., wherein the predetermined information includes information for displaying a product search screen. 4. The information processing device described in 1., wherein the predetermined information includes information indicating the location of the customer. 5. The information processing device described in any one of 1. to 4., wherein the customer information generation means includes a result of analyzing the customer's behavior based on the images in the customer information, and the information output means does not output the predetermined information when the customer's behavior included in the customer information corresponds to the predetermined behavior. 6. The information processing device of any one of 1. to 5., wherein the customer information generation means calculates the stay time excluding time the customer is located in a specific area. 7. The customer information generation means includes at least one of the stay time and the travel distance, and at least one of the planned purchase amount of the product and the planned number of products to be purchased in the customer information, and the information output means decides whether to output the predetermined information based on a result of comparing the customer information with condition information indicating conditions based on at least one of the stay time in a store and the travel distance in the store, and at least one of the planned purchase amount of the product and the planned number of products to be purchased. 8. The information processing device of any one of 1. to 6., wherein the condition information is information defining a standard stay time in a store and a standard planned purchase amount of the product, and the information output means outputs the predetermined information when the stay time indicated by the customer information is equal to or longer than the standard stay time and the planned purchase amount of the product indicated by the customer information is equal to or less than the standard planned purchase amount.9. The information processing device described in 7., wherein the condition information is information defining a standard travel distance in the store and a standard planned purchase amount of the product, and the information output means outputs the predetermined information when the travel distance indicated by the customer information is equal to or greater than the standard travel distance and the planned purchase amount of the product indicated by the customer information is equal to or less than the standard planned purchase amount. 10. The information processing device described in 7., wherein the condition information is information defining a standard stay time in the store and a standard planned purchase number of the product, and the information output means outputs the predetermined information when the stay time indicated by the customer information is equal to or greater than the standard stay time and the planned purchase number of the product indicated by the customer information is equal to or less than the standard planned purchase number. 11. The information processing device described in any one of 1. to 10., wherein the information output means outputs the predetermined information to a display provided for a store clerk or a store clerk terminal used by a store clerk. 12. The information output means outputs the predetermined information to a customer operation terminal arranged in an area where products are displayed or to a customer terminal carried by the customer. 13. The information processing device according to any one of 7. to 11., further comprising condition information setting means for setting the condition information for each store based on performance information collected for each store. 14. The information processing device according to 13., wherein the performance information is generated by linking at least one of the length of time spent in the store and the distance traveled in the store measured for each customer with at least one of the total price and total number of items of products purchased by the customer. 15. A support method comprising: at least one computer generating customer information, based on images taken inside the store, as information about customers visiting the store, the customer information including at least one of the length of time spent in the store, the distance traveled in the store, the purchase price of products, and the number of items purchased of products; and outputting predetermined information when the customer information satisfies a first criterion.16. A non-transitory computer-readable recording medium having recorded thereon a program that, when executed by at least one computer, causes the at least one computer to function as: customer information generation means that generates customer information, based on images taken inside the store, as information about customers visiting the store, the information including at least one of the duration of stay in the store, the distance traveled in the store, the planned amount of product purchases, and the number of products planned to purchase; and information output means that outputs predetermined information when the customer information satisfies a first criterion. 17. A store system comprising: one or more imaging devices provided in a store; and an information processing device communicatively connected to the one or more imaging devices, wherein the information processing device comprises: customer information generation means that generates customer information, based on images taken inside the store using the one or more imaging devices, as information about customers visiting the store, the information including at least one of the duration of stay in the store, the distance traveled in the store, the planned amount of product purchases, and the number of products planned to purchase; and information output means that outputs predetermined information when the customer information satisfies a first criterion.

[0083] Furthermore, some or all of the configurations described in Supplementary Notes 2 to 14 that are subordinate to Supplementary Note 1 (information processing device) may also be subordinate to Supplementary Note 15 (support method), Supplementary Note 16 (non-transitory computer-readable recording medium), and Supplementary Note 17 (store system) in the same subordinate relationship as Supplementary Note 2 to 14. Furthermore, not limited to Supplementary Note 1, Supplementary Note 15, Supplementary Note 16, and Supplementary Note 17, some or all of the configurations described as Supplements may be subordinate to various hardware, software, various recording means for recording software, or systems, within the scope of each of the above-described embodiments.

[0084] REFERENCE SIGNS LIST 1 Store system 10 Information processing device 1010 Bus 1020 Processor 1030 Memory 1040 Storage device 1050 Input / output interface 1060 Network interface 110 Customer information generation unit 120 Information output unit 130 Condition information setting unit 1000 Computer 20 Imaging device 30 Support device 31 Display 32 Store clerk terminal 33 Operation terminal 34 Customer terminal 35 Autonomous mobile robot 40 Network

Claims

1. A customer information generation means generates customer information based on images taken inside the store, including at least one of the following as information about a customer visiting the store: the time spent in the store, the distance traveled within the store, the planned purchase amount, and the planned number of items to be purchased. Information output means that outputs predetermined information when the customer information meets the first criterion, An information processing device equipped with the following features.

2. The aforementioned predetermined information includes information indicating that the customer requires assistance, The information processing apparatus according to claim 1.

3. The aforementioned predetermined information includes information for displaying a product search screen. The information processing apparatus according to claim 1.

4. The aforementioned predetermined information includes information indicating the customer's location. The information processing apparatus according to claim 1.

5. The customer information generation means includes in the customer information the results of analyzing the customer's behavior based on the image. The information output means does not output the predetermined information if the customer's actions included in the customer information correspond to a predetermined action. The information processing apparatus according to any one of claims 1 to 4.

6. The customer information generation means calculates the stay time excluding the time the customer is located in a specific area. The information processing apparatus according to any one of claims 1 to 4.

7. The customer information generation means includes in the customer information at least one of the stay time and the distance traveled, and at least one of the planned purchase amount and the planned number of items to be purchased. The information output means determines whether or not to output the predetermined information based on the result of comparing condition information, which indicates conditions based on at least one of the time spent in the store and the distance traveled in the store, and at least one of the planned purchase amount and the planned number of items to be purchased, with the customer information. The information processing apparatus according to any one of claims 1 to 4.

8. At least one computer, Based on images taken inside the store, customer information is generated that includes at least one of the following as information about the customer visiting the store: the time spent in the store, the distance traveled within the store, the planned purchase amount, and the planned number of items to be purchased. When the customer information meets the first criterion, predetermined information is output. Support methods that include this.

9. By being executed by at least one computer, the at least one computer, Customer information generation means that generates customer information based on images taken inside a store, including at least one of the following as information about a customer visiting the store: the time spent in the store, the distance traveled within the store, the planned purchase amount, and the planned number of items to be purchased. Information output means that outputs predetermined information when the customer information meets the first criterion, A program that makes it function as such.

10. One or more imaging devices installed inside the store, The system comprises one or more imaging devices and an information processing device that is communicatively connected to them, The aforementioned information processing device is A customer information generation means generates customer information based on images taken inside the store using one or more of the aforementioned imaging devices, which includes at least one of the following as information about a customer visiting the store: the time spent in the store, the distance traveled within the store, the planned purchase amount, and the planned number of items to be purchased. Information output means that outputs predetermined information when the customer information meets the first criterion, A store system equipped with these features.