Store operation support system and store operation support method

Abstract

To provide a store operation support system capable of allowing a user to comprehend objectively the relationship between a product that the user is interested in and other products.SOLUTION: In a store operation support system 5, a processor 41 is configured to perform the following steps: acquiring in-store footage obtained by picking up inside a store 2; performing a series of recognition processing of one or more persons and multiple products around each person in the in-store footage; accumulating the results of the recognition process including purchasing behavior of each person in a storage unit 44; generating a product display map 173 representing the relationship between the featured product selected by the user and multiple other related products in multiple products, in which the product display map 173 includes information about the results of the recognition processing; and displaying the product display map 173 as store operation support information on a display.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] The present disclosure relates to a store operations support device and a store operations support method that provide store operations support information for supporting store operations to a user. [Background technology]

[0002] By understanding the purchasing behavior of people who use the store (i.e., customer behavior regarding the purchase of products and services), store managers can develop measures to promote sales of products and improve customer satisfaction.

[0003] Conventionally, a customer purchasing behavior analysis system has been known that includes a camera that captures images of the inside of a store and an image analysis device that captures and analyzes images from the camera in order to understand customer purchasing behavior (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-106628 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the conventional analysis system disclosed in the above Patent Document 1, a video analysis device analyzes the behavior of a customer captured in a camera image. Furthermore, when the analysis system determines that the customer's behavior is a specific, preset behavior, it determines whether or not the specific customer will purchase a product that has caught their eye. In this conventional technology, the images included in the analysis results of customer purchasing behavior are limited to the customer's specific behavior and the product that the customer holds in his or her hand.

[0006] Incidentally, after careful consideration, the inventors of the present application have discovered that it would be extremely useful for store management if users (such as store managers) were able to objectively grasp the relationship between a product that catches their attention and other products among multiple products sold in the store.

[0007] However, in the above-mentioned conventional techniques, no particular consideration is given to allowing the user to objectively grasp the relevance of the product that the user is interested in to other products.

[0008] Therefore, a main objective of the present disclosure is to provide a store operation support device and a store operation support method that enable a user to objectively grasp the relationship between a product that the user is interested in and other products. [Means for solving the problem]

[0009] The store operations support device disclosed herein is a store operations support device equipped with a processor that executes processing for providing a user with store operations support information for supporting the operation of a store, wherein the processor acquires in-store video obtained by filming the inside of the store, performs a recognition process for one or more people in the in-store video and a number of products present around each of the people, stores the results of the recognition process including the purchasing behavior of each of the people in a storage device, and generates a product display map showing the relationship between a featured product selected by the user from among the multiple products and a number of other related products, wherein the product display map includes information related to the results of the recognition process, and is configured to display the product display map on a display as the store operations support information.

[0010] The store operations support method disclosed herein is a store operations support method using a store operations support device that provides a user with store operations support information to support the operation of the store, wherein the store operations support device acquires in-store video obtained by filming the inside of the store, performs a recognition process on one or more people in the in-store video and a plurality of products present around each of the people, stores the results of the recognition process including the purchasing behavior of each of the people in a storage device, generates a product display map showing the relationship between a featured product selected by the user from among the plurality of products and a plurality of other related products, and the product display map includes information regarding the results of the recognition process, and is configured to display the product display map on a display as the store operations support information. Effect of the Invention

[0011] According to the present disclosure, a user can objectively grasp the relationship between a product that the user is interested in and other products. [Brief description of the drawings]

[0012] [Figure 1] Overall configuration diagram of the store operation support system according to the first embodiment [Diagram 2] Functional block diagram of the store operation support system [Diagram 3] FIG. 1 is a block diagram showing a hardware configuration of a store operation support device. [Figure 4] Flow diagram showing the flow of the process of generating purchasing behavior information [Diagram 5] Flow diagram showing the output process of behavioral observation video [Figure 6] FIG. 13 is an explanatory diagram illustrating an example of a behavior confirmation screen. [Figure 7] FIG. 13 is an explanatory diagram showing an overview of a human behavior display image on a behavior confirmation screen; [Figure 8] FIG. 11 is an explanatory diagram showing an example of a change in an image of a line segment showing a relationship between a person and a product on a behavior confirmation screen. [Figure 9] FIG. 11 is an explanatory diagram of a first display mode setting field on the behavior confirmation screen. [Figure 10]FIG. 11 is an explanatory diagram showing an example of a display of an image of a line segment showing a relationship between a person and a carried object on a behavior confirmation screen; [Figure 11] FIG. 13 is an explanatory diagram showing a display example of a person behavior display image showing a person's stay state on a behavior confirmation screen; [Figure 12] FIG. 7 is an explanatory diagram showing a modified example of a human behavior display image on the behavior confirmation screen shown in FIG. 6; [Figure 13] Functional block diagram of a store operation support system according to a second embodiment [Figure 14] Flow diagram showing the output process of the product confirmation screen [Figure 15] FIG. 13 is an explanatory diagram showing an example of a sales management screen. [Figure 16] FIG. 13 is an explanatory diagram showing an example of a display of a purchase trend confirmation screen based on the relevance of products [Figure 17] FIG. 17 is an explanatory diagram of the configuration of the purchase tendency display map shown in FIG. 16. [Figure 18] FIG. 18 is an explanatory diagram of a node image in the purchase tendency display map shown in FIG. 17. [Figure 19] FIG. 13 is an explanatory diagram showing a display example of a purchase tendency confirmation screen based on the similarity of products [Figure 20] FIG. 20 is an explanatory diagram showing another display example of the purchase tendency confirmation screen shown in FIG. 19; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] A first invention made to solve the above problem is a store operations support device having a processor that executes processing for providing a user with store operations support information for supporting the operation of a store, wherein the processor acquires in-store video obtained by filming the inside of the store, performs a recognition process of one or more people in the in-store video and a plurality of products present around each of the people, stores the results of the recognition process including the purchasing behavior of each of the people in a storage device, and generates a product display map showing the relationship between a featured product selected by the user from among the plurality of products and a plurality of other related products, wherein the product display map includes information related to the results of the recognition process, and is configured to display the product display map on a display as the store operations support information.

[0014] This enables users to objectively grasp the relationship between the product they are interested in and other products (i.e., related products) based on the purchasing behavior of customers in the store (i.e., people recognized in the in-store video).

[0015] In addition, in a second aspect of the present invention, the result of the recognition process includes a degree of interest calculated based on a purchasing behavior of each person with respect to the corresponding product.

[0016] This allows the user to easily understand the relationship between the product that catches his / her attention and other products based on the customer's interest level.

[0017] In a third aspect of the present invention, the processor is configured to accumulate product management information regarding the plurality of products in a storage device, and to generate the product display map based on the product management information.

[0018] This allows the user to easily understand the relationship between the product that catches his / her attention and other products based on product management information (for example, information on sales figures, etc.).

[0019] In addition, a fourth invention is configured such that the product management information includes at least one of information regarding sales figures for each of the products, information regarding whether each of the products is on sale, information regarding whether each of the products is a new product, and information regarding the inventory of each of the products.

[0020] This allows users to easily understand the relationship between a product that interests them and other products based on appropriate information.

[0021] In addition, a fifth invention is configured such that the processor generates a sales management screen on which information about the plurality of products is displayed in a selectable manner based on the product management information, and the product of interest is selected based on the user's operation on the sales management screen.

[0022] This allows the user to appropriately select products that interest him or her while referring to the information of each product.

[0023] In addition, a sixth invention is configured such that the processor stores in a storage device, for each of the products, information on one or more linked products that may be purchased simultaneously, and information on one or more alternative products that may be purchased instead of the product, and the product display map shows the relationship between the featured product and the linked products or the alternative products as the related products.

[0024] This allows the user to objectively grasp the relationship between a product that catches his / her attention and other products that are closely related to that product (that is, linked products or alternative products).

[0025] In addition, in a seventh invention, the product display map includes an image showing nodes corresponding to each of the target product and the related products, and an image showing edges connecting the nodes corresponding to mutually related products.

[0026] This allows the user to easily grasp the relationship between the product that interests him or her and other products based on the images showing nodes and images showing edges.

[0027] In addition, the eighth invention is such that at least a portion of the image showing the node includes a first circle whose size is set according to a degree of interest calculated based on the behavior of each of the persons for each of the products, and a second circle whose size is set according to the percentage of each of the persons that ended up purchasing each of the products, and the first and second circles are concentric but have different diameters.

[0028] This allows the user to easily grasp the degree of customer interest and the product purchase ratio for at least one of the product that catches the user's attention and its related products.

[0029] In addition, a ninth invention is configured such that the processor changes the color of at least a part of the image showing the plurality of nodes according to a degree of interest calculated based on a purchasing behavior of each of the persons in the corresponding product.

[0030] This allows the user to easily grasp the degree of interest of customers in at least one of the product that catches the user's attention and its related products.

[0031] In addition, a tenth invention is configured such that the processor calculates the interest level for each of the results of the recognition processing accumulated over two different periods, and changes the color depending on the magnitude of the difference between the two interest levels.

[0032] This allows the user to easily grasp the change over time in the degree of interest of customers in at least one of the product that catches his / her attention and its related products.

[0033] In addition, an eleventh invention is configured such that the image showing each edge includes an image of a line segment, and the processor changes the thickness of the line segment depending on the degree to which the target product and the related products are purchased simultaneously.

[0034] This allows the user to easily grasp the degree to which the product he or she is interested in and its related products are purchased simultaneously.

[0035] In addition, a twelfth invention is configured such that the image showing each edge includes an image of a line segment, and the processor changes a thickness of the line segment depending on the similarity between the attention product and the related products.

[0036] This allows the user to easily grasp the degree of similarity between the product in which he or she is interested and related products.

[0037] In addition, the 13th invention is configured such that the processor displays a pop-up screen including information about the product corresponding to the node that is the target of the cursor operation, based on the user's cursor operation on an image showing each node in the product display map.

[0038] This allows the user to easily check detailed information about each product displayed in a simplified manner on the product display map by manipulating the cursor.

[0039] Further, a fourteenth invention is a store operations support method using a store operations support device that provides a user with store operations support information for supporting the operation of a store, wherein the store operations support device acquires an in-store image obtained by photographing the inside of the store, performs a recognition process on one or more people in the in-store image and a plurality of products present around each of the people, stores the results of the recognition process including the purchasing behavior of each of the people in a storage device, generates a product display map showing the relationship between a featured product selected by the user from among the plurality of products and a plurality of other related products, the product display map includes information related to the results of the recognition process, and displays the product display map on a display as the store operations support information.

[0040] This enables users to objectively grasp the relationship between the product they are interested in and other products (i.e., related products) based on the purchasing behavior of customers in the store (i.e., people recognized in the in-store video).

[0041] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0042] (First embodiment) FIG. 1 is an overall configuration diagram of a store operations support system 1 according to the first embodiment. The store operations support system 1 provides users with information to support the operations of a store. Users include store managers and store staff. Stores 2 that are targets of operation support include, for example, retail chain stores such as convenience stores and supermarkets.

[0043] As shown in Fig. 1, the store operations support system 1 includes multiple cameras 3, a recorder 4, a store operations support device 5, and an administrator terminal 6. The cameras 3 and recorders 4 installed in a store 2 are connected to a LAN (Local Area Network) 7. The LAN 7 is connected to a WAN (Wide Area Network) 9 such as the Internet via connection devices 8 including a gateway, a router, etc. The store operations support device 5 and the administrator terminal 6 are connected to the WAN 9. The cameras 3, the recorder 4, the store operations support device 5, and the administrator terminal 6 are capable of communicating with each other.

[0044] The number and arrangement of each component of the store operations support system 1 (here, the camera 3, the recorder 4, the store operations support device 5, and the manager terminal 6) can be changed as appropriate. The store operations support system 1 may include multiple stores similar to the store 2.

[0045] Each camera 3 is composed of a video camera equipped with known hardware. Each camera 3 is placed at an appropriate location in store 2 so as to be able to capture the purchasing behavior of multiple people (i.e., customers) using store 2.

[0046] The purchasing behavior may include any behavior of a person shopping in store 2. For example, the purchasing behavior may include an action of a person picking up a product and considering purchasing it, an action of a person placing a product that the person has decided to purchase in a shopping cart, and an action of a person moving around the store in search of a desired product.

[0047] Each camera 3 captures the purchasing behavior of each person, thereby generating a captured image (that is, an image of the inside of the store) including each person, the products that they purchase, and their belongings (such as a shopping basket).

[0048] Each camera 3 can sequentially transmit data of the in-store video generated by shooting and its associated information to the recorder 4. The associated information can include a camera ID (an identifier for each camera), location information (including the coordinates in the image of each camera and the installation position of each camera), shooting conditions (including the angle of view), and the like.

[0049] The recorder 4 is composed of a video recording device equipped with known hardware. The recorder 4 acquires data of the in-store video from each camera 3 and stores the video data in storage. The video data of the in-store video may include audio data. The recorder 4 may be installed outside the store 2. In this case, the recorder 4 is capable of communicating with other electronic devices via the WAN 9.

[0050] The store operations support device 5 provides users with store operations support information for supporting the operation of the store 2. The store operations support device 5 is composed of one or more computers capable of executing processes for generating store operations support information and outputting it to users. In this embodiment, the store operations support device 5 includes one or more servers that execute the necessary processes.

[0051] The manager terminal 6 is used by a user who uses the store operation support information. The manager terminal 6 includes one or more computers equipped with a display, such as a PC equipped with a display, a tablet terminal, and a smartphone. The manager terminal 6 may be installed within the store 2. In that case, the manager terminal 6 is capable of communicating with other electronic devices via the LAN 7.

[0052] Fig. 2 is a functional block diagram of the store operations support system 1. Fig. 3 is a block diagram showing the hardware configuration of the store operations support device 5. Fig. 4 is a flow diagram showing the flow of the process of generating purchasing behavior information by the store operations support device 5. Fig. 5 is a flow diagram showing the flow of the process of outputting behavior observation video by the store operations support device 5.

[0053] As shown in FIG. 2, in the store operations support system 1, data of in-store video generated by the camera 3 is sequentially stored in the recorder 4. The store operations support device 5 acquires the video data stored in the recorder 4 and generates store operations support information. In this embodiment, the store operations support device 5 transmits (outputs) the generated store operations support information to the manager terminal 6. The store operations support information includes a screen for the user to confirm the purchasing behavior of customers in the store (hereinafter referred to as a "behavior confirmation screen"; see the behavior confirmation screen 51 in FIG. 6). The behavior confirmation screen 51 includes video for observing a person's purchasing behavior (hereinafter referred to as a "behavior observation video") (see the behavior observation video 52 in FIG. 6). The behavior observation video 52 includes in-store video captured by the camera 3 for a predetermined period of time.

[0054] The user can set conditions (hereinafter, referred to as “user-set conditions”) for displaying a desired behavior observation video on the behavior confirmation screen 51 on the manager terminal 6, and transmit the conditions to the store operations support device 5.

[0055] The store operation support device 5 includes a control unit 11 and a storage unit 12 (an example of a storage device).

[0056] In the control unit 11, the video acquisition unit 21 executes a process for acquiring data of in-store video (hereinafter referred to as “video acquisition process”). In the video acquisition process, the in-store video data accumulated in the recorder 4 is appropriately transmitted and stored in the storage unit 12 as video data 31.

[0057] The object / behavior recognition unit 22 performs a recognition process (hereinafter referred to as "purchase behavior recognition process") of one or more people and multiple objects in the in-store video for a predetermined period of time on the video data 31. The multiple objects include one or more products related to each person (i.e., products that each person can purchase). The multiple objects may also include portable items (e.g., shopping baskets) used by each person in the store 2.

[0058] The purchasing behavior recognition process is executed using a purchasing behavior recognition model 32 pre-stored in the storage unit 12. The purchasing behavior recognition model 32 includes a learning model generated in advance by machine learning such as deep learning. The purchasing behavior recognition process includes extraction of people and objects in in-store video (including acquisition of position information of people and objects) and tracking of moving people and objects.

[0059] The purchasing behavior recognition process also includes recognition of the association between a person and an object (i.e., what action the person is taking with respect to the object). In the purchasing behavior recognition process, when the object to be recognized is a product, the degree of interest in the product, the time spent on the product (i.e., the time it takes the person to decide whether or not to purchase the product), and the degree of suspiciousness are obtained as the recognition result of the association between the person and the object.

[0060] The degree of interest in a product can be calculated based on, for example, the number of times a person picks up the target product within a given time period, the length of time the person visually looks at the target product while holding it in their hand, etc. The dwell time for the product can be calculated based on, for example, the time the person holds the target product in their hand, or the time the person holds the target product after returning it to the shelf. The degree of suspiciousness can be calculated based on suspicious behavior of a person that has been learned in advance (for example, the behavior of a person being cautious of the surroundings while holding the target product in their hand, or the behavior of a person hiding the target product that they have picked up in their clothes, etc.). In this embodiment, the degree of interest in a product, the dwell time for the product, and the degree of suspiciousness are each expressed as a numerical value in the range of 0-100.

[0061] In the purchasing behavior recognition process, an image showing the purchasing behavior of each person (hereinafter referred to as a "person behavior display image") is generated based on the recognition result of the association between the person and the corresponding object (e.g., the product to be purchased, or a portable object such as a shopping basket). The person behavior display image is superimposed on the in-store video to form a part of the behavior observation video. At this time, the type of the extracted object (e.g., the type of product) may be recognized.

[0062] Furthermore, in the purchasing behavior recognition process, a representative position of each person is determined based on the recognition result for each person. The person behavior display image may include an image showing such a representative position of each person. For example, the center of gravity of the image of each person or a predetermined part of each person (e.g., the head) may be used as the representative position of each person.

[0063] The store operations support device 5 stores the result of the purchasing behavior recognition process described above (including the object relevance display image) as purchasing behavior information 33 in the storage unit 12.

[0064] The person attribute estimation unit 23 performs a process (hereinafter, referred to as a "person attribute estimation process") of estimating the attributes of each person extracted from the video data 31 by the purchase behavior recognition process. The person attributes include at least one of the gender and age of each person.

[0065] The person attribute estimation process is executed using a person attribute estimation model 34 pre-stored in the storage unit 12. The person attribute estimation model 34 includes a learning model pre-generated by machine learning such as deep learning. Information on the gender and age of each person obtained by the person attribute estimation process is stored in the storage unit 12 as a part of the purchasing behavior information 33.

[0066] The product identification unit 24 performs a process of identifying products (hereinafter referred to as "product identification process") for objects (including products) extracted from the video data 31 by the purchasing behavior recognition process. In the product identification process, an image of the object extracted from the video data 31 is compared with a product image included in product information 35 (i.e., a product database) stored in the storage unit 12. The product information 35 may include a product ID, product name, product type, and the like in addition to a product image for each product sold in the store 2. Information regarding the identification result of each product obtained by the product identification process is stored in the storage unit 12 as a part of the purchasing behavior information 33. The product identification process by the product identification unit 24 may be performed using a learning model prepared in advance.

[0067] The screen generation / output unit 25 performs a process of outputting a behavior confirmation screen to the administrator terminal 6. The screen generation / output unit 25 also performs a process of generating a behavior observation video to be displayed on the behavior confirmation screen using the purchasing behavior information 33. The screen generation / output unit 25 appropriately changes the display mode of the behavior observation video on the display of the administrator terminal 6 based on the user setting conditions received from the administrator terminal 6.

[0068] The store operations support device 5 may be operated directly by a user without going through the manager terminal 6. In this case, the screen generation / output unit 25 can display (i.e., output) the behavior confirmation screen and the behavior observation video on the display 46 of the store operations support device 5 (see FIG. 3).

[0069] 3, the store operations support device 5 may include one or more processors 41 (CPU, MPU, etc.), a random access memory (RAM) 42, a read only memory (ROM) 43, storage 44 including a solid state drive (SSD) or a hard disk drive (HDD), and a communication interface (I / F) 45. Furthermore, the store operations support device 5 may include a display 46.

[0070] At least a part of the functions of each of the units 21-25 in the control unit 11 described above can be realized by the processor 41 executing a predetermined control program.

[0071] 4, the store operations support device 5 executes the above-mentioned video acquisition process (ST101), purchasing behavior recognition process (ST102), person attribute estimation process (ST103), and product identification process (ST104) as processes for generating purchasing behavior information 33. Furthermore, the store operations support device 5 stores the results of these processes in the storage unit 12 as purchasing behavior information 33 (ST105).

[0072] Furthermore, as shown in FIG. 5, the store operations support device 5 can execute a process for outputting a behavior confirmation screen.

[0073] More specifically, the store operations support device 5 first displays a behavior confirmation screen on the display of the manager terminal 6 (ST201). Next, the store operations support device 5 acquires user-defined conditions based on the user's operation on the manager terminal 6 (ST202). Furthermore, the store operations support device 5 acquires purchasing behavior information 33 that satisfies the user-defined conditions (e.g., corresponds to the set display period) from the storage unit 12 (ST203).

[0074] Therefore, the store operations support device 5 extracts necessary information from the purchasing behavior information 33 based on the conditions set by the user (ST203), and changes the display mode of the behavior observation video on the behavior confirmation screen (ST204).

[0075] The series of processes in steps ST202 to ST204 are repeatedly executed every time the user setting conditions are updated in the manager terminal 6.

[0076] Next, the details of the behavior confirmation screen displayed on the administrator terminal 6 will be described.

[0077] FIG. 6 is an explanatory diagram showing an example of the behavior confirmation screen 51. FIG. 7 is an explanatory diagram showing an outline of a person behavior display image on the behavior confirmation screen 51. FIG. 8 is an explanatory diagram showing an example of a change in an image of a line segment showing the relationship between a person and a product on the behavior confirmation screen 51. FIG. 9 is an explanatory diagram related to a first display mode setting field 75 on the behavior confirmation screen 51. FIG. 10 is an explanatory diagram showing an example of a display of an image of a line segment showing the relationship between a person and a portable item on the behavior confirmation screen 51. FIG. 11 is an explanatory diagram showing an example of a display of a person behavior display image showing a person's staying state on the behavior confirmation screen 51. FIG. 12 is an explanatory diagram showing a modified example of the person behavior display image on the behavior confirmation screen 51 shown in FIG. 6.

[0078] As shown in FIG. 6, the behavior confirmation screen 51 includes a video display area that displays the behavior observation video 52. The behavior confirmation screen 51 also includes a period setting field 53 that allows the user to set the period during which the behavior observation video 52 was shot (here, 12:00 to 13:00 on March 7, 2023). Furthermore, the behavior confirmation screen 51 includes a playback controller 54 that allows the user to set the playback of the behavior observation video 52. The playback controller 54 includes a playback button for starting or ending the playback of the behavior observation video 52, a time slider for setting the playback position (time), and a speed setting section for setting the playback speed.

[0079] The store image serving as the basis for the behavior observation image 52 includes a plurality of people 61A-61C. The store image also includes a plurality of product shelves 60 and a plurality of products 62 arranged on each of the product shelves 60 as a plurality of objects present around the plurality of people 61A-61C.

[0080] 7, a product 62A and a shopping basket 63A corresponding to the person 61A are included around the person 61A in the behavior observation video 52. The product 62A is recognized as one of the possessions of the person 61A (i.e., a product) because the person 61A picks it up and looks at it. The shopping basket 63A is also recognized as one of the possessions of the person 61A (i.e., a portable item) because the person 61A moves around holding it in his / her hand.

[0081] The behavior observation video 52 includes a human behavior display image superimposed on the in-store video. The human behavior display image for the person 61A includes a line segment image 65A arranged so as to overlap the person 61A and the corresponding product 62A. The line segment image 65A is arranged so as to connect the person 61A and the corresponding product 62A.

[0082] The human behavior display image for the person 61A also includes an image 66A of a line segment that is arranged so as to overlap the person 61A and a shopping basket 63A, which is a portable item corresponding to the person 61A.

[0083] An image showing the representative position superimposed on the in-store image is included in the behavior observation video 52. The image showing the representative position of the person 61A is a circular image 67A placed at the representative position of the person 61A.

[0084] Since there is a product 62A and a shopping basket 63A corresponding to the person 61A, the circle image 67A includes two concentric circles 67Aa, 67Ab having different diameters (see FIG. 7). One end of a line segment image 65A shown by a solid line is connected to a large diameter (i.e., outer) circle 67Aa also shown by a solid line. The other end of the line segment image 65A is connected to the product 62A. Also, one end of a line segment image 66A shown by a dashed line is connected to a small diameter (i.e., inner) circle 67Ab also shown by a dashed line.

[0085] 7 shows a rectangular image area 71 from which a person 61A is extracted, a rectangular image area 72 from which a product 62A is extracted, and a rectangular image area 73 from which a shopping cart 63A is extracted. The user can cause such image areas 71-73 to be displayed in the behavior observation video 52 in FIG. 6 as necessary.

[0086] Furthermore, the behavior observation video 52 (see FIG. 6) includes circular images 67B and 67C arranged at representative positions of the persons 61B and 61C, respectively, for the persons 61B and 61C. Note that, instead of the circular images 67A-67C, images of other shapes (e.g., polygonal) may be used as images showing the representative positions.

[0087] The behavior observation video 52 also includes a line segment image 66B that is arranged so as to overlap the person 61B and his / her carry-on bag 63B. One end of the line segment image 66B is connected to a circle image 67B. The other end of the line segment image 69B is connected to the reusable bag 63B.

[0088] The behavior confirmation screen 51 also includes first to third display mode setting fields 75-77. The user can change the display mode of the human behavior display image in the behavior observation video 52 by performing operations on the first to third display mode setting fields 75-77.

[0089] In the first display mode setting field 75, the user can change the display mode of the human behavior display image (here, the line thickness of the line segment image) based on the interest level for the product, the dwell time for the product, or the suspiciousness level. In FIG. 6, an example in which the user selects the interest level is shown.

[0090] For example, as shown in Fig. 8(A)-(C), the thickness of the line in the line segment image 65A gradually increases according to the change in the interest level (here, (A) interest level 30, (B) interest level 60, (C) interest level 100). If the interest level is set to increase according to the length of time that a person holds the target product in their hands and looks at it, the thickness of the line gradually increases according to the length of time (i.e., the passage of time). Similarly, if the user selects the dwell time or suspiciousness level in the first display mode setting field 75, the thickness of the line in the line segment image 65A changes according to those values.

[0091] As shown in FIG. 9, the user can set an invalid area in the first display mode setting field 75 in which the line thickness is not changed even if the interest level changes. FIG. 9 shows an example in which the user operates the slider 78 to set a range in which the interest level is less than 30 as the invalid area. In this case, in the range in which the interest level is from 0 to less than 30, the line thickness remains the initial thickness and does not change. Then, when the interest level becomes 30 or more (i.e., in the valid area in which the interest level is in the range of 30-100), the line thickness increases with the increase in the interest level. In this way, the user can easily grasp the change in the line thickness (i.e., the interest level) in the desired range of interest levels by appropriately setting an invalid area in which the line thickness is not changed.

[0092] In the second display mode setting field 76, the user can select whether to check the belongings of the persons 61A-61C or to check the stay status of the persons 61A-61C. FIG. 6 shows an example in which the user selects to check the belongings. Furthermore, here, the user has selected the product to be purchased, the shopping basket, the shopping cart, and the personal bag as belongings to be displayed. However, by checking only some of the belongings (e.g., the shopping basket) in the second display mode setting field 76, the user can easily check the belongings in the behavior observation video 52.

[0093] In the behavior observation video 52, a human behavior display image (e.g., line segment images 65A, 66A) related to the belongings selected by the user is displayed in a desired line shape (i.e., line type). Here, an example is shown in which the line shape of the human behavior display image is set to a solid line (see line segment image 65A) for a product, a dashed line (see line segment image 66A) for a shopping cart, and a double line (see line segment image 66B) for a reusable bag.

[0094] According to the user's settings in the second display mode setting field 76, for example, as shown in Fig. 10(A), a line segment image 66D is displayed in dashed lines so as to overlap person 61D and his / her carrying item, shopping cart 63D. More specifically, line segment image 66D is positioned so as to connect circle image 67D placed at the representative position of person 61D to shopping cart 63D.

[0095] 10(B), a line segment image 66B is shown by a double line so as to overlap person 61B and his / her carrying item, reusable bag 63B. More specifically, line segment image 66B is disposed so as to connect a circle image 67B disposed at a representative position of person 61B and reusable bag 63B.

[0096] In the third display mode setting field 77, the user can set the color of the line that displays the attribute of the person 61A-61C for the person behavior display image (for example, the line segment image 65A). In Fig. 6, an example is shown in which the user selects gender as an attribute of the person 61A-61C and sets the line colors corresponding to males and females, respectively.

[0097] This allows, for example, line segment image 65A to indicate the attribute of the corresponding person 61A (here, female) by the color applied to it.

[0098] Similarly to the line segment image 65A, the circular image 67A indicating the representative position of the person 61A can indicate the attribute of the corresponding person 61A (female in this case) by the color applied to the inner area.

[0099] On the other hand, when the user selects the stay state in the second display form setting field 76, for example, a behavior observation video 52 as shown in Fig. 11 is displayed. Here, both "moving" and "stationary" are selected as the stay states of the person to be displayed in the behavior observation video 52. As a result, a moving person 61E and a stationary person 61F are displayed in the behavior observation video 52.

[0100] Here, the shape of the lines in the person behavior display image (more precisely, the image showing the representative positions of the persons 61E and 61F) is set to a solid line when moving and a dashed line when stationary. As a result, in Fig. 11, the circular image 67E showing the representative position of the moving person 61E is shown in a solid line, and the circular image 67F showing the representative position of the stationary person 61F is shown in a dashed line.

[0101] As described above, the conditions set by the user in the period setting field 53 and the first to third display format setting fields 75-77 of the behavior confirmation screen 51 are appropriately transmitted to the store operations support device 5 as user-set conditions.

[0102] As the human behavior display image, an image of another form may be used instead of the above-mentioned line segment image (for example, see line segment image 65A in FIG. 6). For example, as shown in FIG. 12, as the human behavior display image, a frame image 165A surrounding a person 61A and a corresponding product 62A (i.e., a product to be purchased) or a frame image 166A surrounding a person 61A and a corresponding shopping basket 63A (i.e., a portable item) may be used. The frame image 165A may be in a form that surrounds at least a part of the person 61A and the product 62A. Similarly, the frame image 166A may be in a form that surrounds at least a part of the person 61A and the shopping basket 63A.

[0103] Furthermore, the store operations support device 5 may display a masked image or an avatar image by changing the image area of ​​each person included in the behavior observation video 52. This makes it possible to easily remove information that can identify each person (i.e., personal information) from the behavior observation video 52, and to check the behavior observation video 52 while protecting the privacy of the individuals.

[0104] Second embodiment Next, a store operations support system 1 according to a second embodiment will be described. In the store operations support system 1 according to the second embodiment, components similar to those in the first embodiment will be given the same reference numerals and detailed descriptions will be omitted. In addition, in the store operations support system 1 according to the second embodiment, matters that are not specifically mentioned below will be the same as those in the first embodiment.

[0105] 13 is a functional block diagram of a store operations support system 1 according to the second embodiment. In the store operations support device 5, the control unit 11 includes a product management information acquisition unit 126 that acquires management information 136 related to products sold in the store 2 (hereinafter referred to as "product management information"). The product management information 136 is associated with each product included in the product information 35 and sequentially stored in the storage unit 12. The product management information 136 may include sales information (including sales volume in a predetermined period, etc.) for each product, product price, sale information (discount information), sales start time, inventory information, etc.

[0106] The screen generation / output unit 25 performs processing to output a product confirmation screen for confirming products sold in the store 2 to the manager terminal 6. The product confirmation screen includes a product display map showing the relationships between the products. Furthermore, the screen generation / output unit 25 can add information such as interest level to the product display map using the purchasing behavior information 33. The screen generation / output unit 25 changes the display mode of the product display map based on the map setting conditions received from the manager terminal 6.

[0107] FIG. 14 is a flowchart showing the flow of the process of outputting a product confirmation screen by the store operation support device 5.

[0108] 14, first, the store operations support device 5 displays a product confirmation screen on the display of the manager terminal 6 (ST301). The product confirmation screen includes a sales management screen as its initial screen, and the sales information and the like related to the product (or product type) selected by the user are displayed on the sales management screen.

[0109] Next, the store operation support device 5 acquires map setting conditions based on a user's operation on the manager terminal 6 (ST302).

[0110] Furthermore, the store operations support device 5 acquires product information 35 and product management information 136 that satisfy the map setting conditions (for example, corresponding to the set display period) from the storage unit 12 (ST303, ST304). Furthermore, the store operations support device 5 acquires purchasing behavior information 33 that satisfies the map setting conditions from the storage unit 12 (ST305).

[0111] Then, the store operation support device 5 extracts necessary information from the purchasing behavior information 33 based on the map setting conditions (ST306), and changes the display format of the already displayed product display map, or displays a new product display map on the product confirmation screen (ST307).

[0112] The series of processes in the above steps ST302 to ST307 is repeatedly executed every time the map setting conditions are updated in the manager terminal 6.

[0113] FIG. 15 is an explanatory diagram showing an example of a sales management screen 151 in the store operation support system 1. As shown in FIG.

[0114] 15, the sales management screen 151 includes a product sales display area 152. The product sales display area 152 displays a product image, a product name, a sales volume for a predetermined period, and the like for each product selected by the user or each product belonging to a product type selected by the user.

[0115] In addition, the sales management screen 151 includes a category setting field 161 , a sort setting field 162 , and a product status setting field 163 .

[0116] In the category setting field 161, the user can search for individual products that the user wants to display in the product sales display area 152, and can also select the product type that the user wants to display. In this example, "soft drinks" is set as the product type.

[0117] In the sort setting field 162, the user can sort the products displayed in the product sales display area 152 in ascending or descending order based on the product sales volume, the sales increase rate for a given period, the interest level of the product (here, the average interest level of multiple customers), and the arrival date of the product. Here, an example is shown in which "sales volume" is selected as the sorting criterion.

[0118] In the product status setting field 163, the user can set the status of each product to be displayed in the product sales display area 152. Here, the user can select as the product status: products on sale, new products (i.e. products within a certain period of time since their release), and out-of-stock products (i.e. products that are out of stock). Here, an example is shown in which "on sale" has been selected as the product status.

[0119] Fig. 16 is an explanatory diagram showing a display example of a purchasing tendency confirmation screen 171 based on the relevance of products. Fig. 17 is an explanatory diagram relating to the configuration of the product display map shown in Fig. 16. Fig. 18 is an explanatory diagram relating to node images in the product display map shown in Fig. 17. Fig. 19 is an explanatory diagram showing a display example of a purchasing tendency confirmation screen 171 based on the similarity of products. Fig. 20 is an explanatory diagram showing another display example of the purchasing tendency confirmation screen 171 shown in Fig. 19.

[0120] When the user selects any product (e.g., clicks on the desired product image) on the sales management screen 151 shown in FIG. 15 above, a purchasing tendency confirmation screen 171 for the selected product is displayed on the display of the administrator terminal 6.

[0121] 16, the purchasing tendency confirmation screen 171 includes a product display map 173. The purchasing tendency confirmation screen 171 also includes a time setting field 174 for the user to set the time condition of the display target of the product display map 173. Here, an example is shown in which "present" is selected in the time setting field 174. As a result, the product display map 173 is generated based on information (e.g., product information 35, product management information 136, and purchasing behavior information 33) acquired within a predetermined period (e.g., one week) from the present.

[0122] The purchasing tendency confirmation screen 171 also includes a featured product display area 175 in which a product selected by the user on the sales management screen 151 (here, potatoes from Prefecture A, hereinafter referred to as the "featured product") is displayed. The featured product display area 175 includes, for example, an image of the featured product, the product name, and an interest level value. For example, when interest level data relating to multiple persons exists for the target product, the interest level value is determined based on the average interest level values ​​for those persons.

[0123] The product display map 173 includes an image (hereinafter referred to as a "node image") 181A indicating a node corresponding to the product of interest, and a node image (for example, a node image corresponding to a carrot) 181B corresponding to another product (hereinafter referred to as a "first related product") that is directly related to the product of interest.

[0124] Also, as shown in FIG. 16, the user can operate the cursor (or mouse over) on node image 181B corresponding to, for example, a carrot to display a pop-up screen 182 showing detailed information about the product corresponding to node image 181B.

[0125] In the product display map 173, a node image (here, node image 181A) corresponding to the product of interest is placed at the center, and node images (for example, node image 181B) indicating first related products are placed so as to surround it. The node image 181A indicating the product of interest and the node images indicating each of the first related products are connected to each other by an image indicating an edge (hereinafter referred to as an "edge image"). For example, the node image 181A and the node image 181B are connected to each other by the edge image 183A.

[0126] Furthermore, the product display map 173 includes node images corresponding to other products directly related to the first related product (hereinafter referred to as "second related products", excluding the featured product). For example, the product display map 173 includes stew roux, curry roux, and beef as second related products corresponding to carrots, and node images 181C, 181D, and 181E corresponding to them are displayed. However, in FIG. 16, the node image for breadcrumbs as the second related product corresponding to carrots is omitted.

[0127] In addition, the product display map 173 may further include node images corresponding to other products that are directly related to the second related product (hereinafter referred to as "third related products", excluding the product of interest and the first related product).

[0128] As shown in Fig. 17, the first related products of potatoes produced in A prefecture include beef, bread, eggs, butter, etc. in addition to carrots. Also, for example, product display map 173 includes carrots, stew roux, and thread konnyaku as second related products that are directly related to beef (see node image 181E in Fig. 16) as the first related product, and displays node images 181B, 181C, and 181G (see Fig. 16) corresponding to them.

[0129] As shown in FIG. 18, each node image 181 shown on the product display map 173 is composed of two concentric circles 191A and 191B (examples of a first circle and a second circle) having different diameters.

[0130] The size of the diameter of the outer circle 191A is set according to the degree of interest in the target product. The size of the inner circle 191B is set according to the proportion of customers who purchase the target product (hereinafter referred to as the "product purchase proportion"). For example, when a plurality of people pick up the target product and the target product is recognized as being related to a plurality of people, the product purchase proportion corresponds to the proportion of people who actually purchase the target product afterwards among the plurality of people. In the second embodiment, the node images 181 included in the product display map 173 are exemplified, but the node images 181 corresponding to one or more products arbitrarily selected by the store manager or the like may be generated and displayed in a list without considering the relationship between the products. In this case, the store manager or the like can intuitively grasp the degree of interest and purchase proportion of the products from the node images 181 limited to the products whose sales situation is of interest.

[0131] As shown in FIG. 16, the purchasing tendency confirmation screen 171 also includes a map type setting field 192 and a product status setting field 163 .

[0132] In the map type setting field 192, the user can set the type of related products to be displayed on the product display map 173. Fig. 16 shows an example in which the user selects "relevance". As a result, products that are highly related to the focused product are displayed as first related products on the product display map 173.

[0133] Similarly, in the product display map 173, a product that is highly related to the first related product is displayed as the second related product. The degree of relatedness between products is set according to the degree to which the two target products are purchased simultaneously. More specifically, the degree of relatedness between products can be set according to, for example, the number of times (or frequency) that the two target products were purchased simultaneously (i.e., together) during a predetermined period in the past. Such information on the degree of relatedness (an example of information on linked products) is stored in advance as part of the product management information 136.

[0134] In addition, in the product display map 173, the thickness (ie, the thickness of the line) of the edge image (eg, edge image 183A) connecting each product is set according to the degree of association between the two products connected by each edge image.

[0135] In the product status setting field 163, the user can set the status of the product to be displayed on the product display map 173. Fig. 16 shows an example in which the user has selected "On Sale". As a result, only products that are on sale are displayed on the product display map 173 as the first related product and the second related product.

[0136] Similarly, the user can display only products that have just been released as the first related product and the second related product by selecting "New product" in the product status setting field 163. Also, the user can display only products that are out of stock as the first related product and the second related product by selecting "Out of stock" in the product status setting field 163.

[0137] On the other hand, when the user selects "similarity" in the map type setting field 192, for example, as shown in FIG. 19, products that have a high similarity to the focused product are displayed as first related products in the product display map 173. Similarly, products that have a high similarity to the first related product are displayed as second related products in the product display map 173. The similarity between products is set, for example, depending on whether one product can be a substitute for the other product (i.e., a product that can be purchased instead). Such similarity information (an example of information on substitute products) is stored in advance as part of the product management information 136.

[0138] 19 shows an example in which the user selects Company A's curry roux as the product of interest. The product display map 173 includes a node image 281A corresponding to the product of interest and a node image corresponding to a first related product (for example, a node image 281B corresponding to Hayashi rice).

[0139] Also, the product display map 173 shown in Fig. 19 includes a node image corresponding to a second related product that is directly related to the first related product, as in the case of Fig. 16. Note that the product display map 173 may further include a node image corresponding to a third related product that is directly related to the second related product.

[0140] Also, as shown in Fig. 20, for example, the user can select "weeks" in the time setting field 174 of the purchase tendency confirmation screen 171 shown in Fig. 19. As a result, the interest level used in the product display map 173 becomes the difference value between the interest level acquired based on information from one week ago (more precisely, information acquired during a predetermined period of time in the past (e.g., one week) at the point in time one week ago) and the interest level acquired based on current information (more precisely, information acquired during a predetermined period of time in the past at the present time).

[0141] 20, the color of the inner area of ​​each node image is set according to the difference in interest level (i.e., the amount of change) as shown in the legend 290 in the figure. For example, node image 281A indicates that the current interest level value of the corresponding product (here, Company A's curry roux) has increased by 40 points compared to the value based on information from one week ago. This allows the user to easily grasp the change over time in the interest level for each product displayed on the product display map 173.

[0142] In addition, when the user selects "days" in the time setting field 174, the interest level used in the product display map 173 is the difference between the interest level acquired based on information from one day ago (more precisely, information acquired during a specified period of time in the past as of one day ago) and the interest level acquired based on the current information. Similarly, when the user selects "monthly" in the time setting field 174, the interest level used in the product display map 173 is the difference between the interest level acquired based on information from one month ago (more precisely, information acquired during a specified period of time in the past as of one month ago) and the interest level acquired based on the current information.

[0143] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above embodiments to create new embodiments.

[0144] The user can cause the administrator terminal 6 to display the behavior confirmation screen and the product confirmation screen shown in the above-described embodiment in real time (that is, based on information already collected at the present time). [Industrial Applicability]

[0145] The store operations support device and store operations support method disclosed herein are useful as a store operations support device and store operations support method that enable a user to objectively understand the relationship between a product that the user is interested in and other products, and that provide the user with store operations support information for supporting store operations. [Explanation of symbols]

[0146] 1: Store management support system 2: Store 3: Camera 4: Recorder 5: Store operation support device 6: Administrator terminal 11: Control section 12: Storage section 21: Video acquisition unit 22: Behavior recognition section 23: Person attribute estimation section 24: Product Identification Department 25: Screen generation / output section 31:Video data 32: Purchasing behavior recognition model 33:Purchasing behavior information 34: Person attribute estimation model 35:Product information 41: Processor 46: Display 51: Action confirmation screen 52: Behavioral observation footage 53: Period setting field 54: Playback Controller 61 :Person 62: Product 63: form 75: First display configuration field 76: Second display configuration field 77: Third display configuration field 78: Slider 126: Product management information acquisition department 136: Product management information 151: Sales management screen 152: Product sales display area 161: Category setting field 162: Sort setting field 163: Product status setting field 171: Purchasing trend confirmation screen 173: Product display map 174: Time setting field 175: Featured product display area 181: Node image 182: Pop-up screen 183: Edge image 192: Map type setting field

Claims

1. A store operations support device including a processor that executes a process for providing store operations support information to a user to support the operation of a store, The processor, Acquire in-store video obtained by photographing the inside of the store, performing a process of recognizing one or more people and a plurality of products existing around each of the people in the in-store video; storing the results of the recognition process, including the purchasing behavior of each person, in a storage device; generating a product display map showing a relationship between the attention product selected by the user from among the plurality of products and a plurality of other related products, the product display map including information regarding a result of the recognition process; The store operations support device displays the product display map on a display as the store operations support information.

2. The result of the recognition process includes: The store operation support device according to claim 1 , further comprising a degree of interest calculated based on the purchasing behavior of each person with respect to the corresponding product.

3. The processor, storing product management information relating to the plurality of products in a storage device; The store operation support device according to claim 1 , wherein the product display map is generated based on the product management information.

4. The store operation support device of claim 3, wherein the product management information includes at least one of information regarding sales figures for each of the products, information regarding whether each of the products is on sale, information regarding whether each of the products is a new product, and information regarding inventory for each of the products.

5. the processor generates a sales management screen on which information about the plurality of products is selectably displayed based on the product management information; The store operation support device according to claim 3 , wherein the product of interest is selected based on an operation of the user on the sales management screen.

6. The processor, storing, in a storage device, information on one or more linked products that may be purchased simultaneously with each of the products, and information on one or more substitute products that may be purchased instead of the products; The store operation support device according to claim 1 , wherein the product display map indicates a relationship between the focused product and the linked product or the alternative product as the related product.

7. The store operation support device of claim 1, wherein the product display map includes an image showing nodes corresponding to the product of interest and each of the related products, and an image showing edges respectively connecting the nodes corresponding to mutually related products.

8. At least a part of the image showing the node includes a first circle whose size is set according to a degree of interest calculated based on the behavior of each of the persons with respect to each of the products, and a second circle whose size is set according to a rate at which each of the persons has ended up purchasing each of the products; The store operations support device according to claim 7 , wherein the first and second circles are concentric and have different diameters.

9. The processor, The store operation support device according to claim 7 , further comprising: a step of changing a color of at least a part of the image showing the plurality of nodes in accordance with an interest level calculated based on a purchasing behavior of each person in the corresponding product.

10. The processor, Calculating the degree of interest for each of the results of the recognition processing accumulated in two different periods; The store operation support device according to claim 9 , wherein the color is changed depending on a magnitude of a difference between the two interest levels.

11. the image showing each edge includes an image of a line segment; The processor, The store operation support device according to claim 7 , wherein the thickness of the line segment is changed according to a degree to which the attended product and the related product are purchased simultaneously.

12. the image showing each edge includes an image of a line segment; The processor, The store operation support device according to claim 7 , wherein a thickness of the line segment is changed according to a degree of similarity between the attended product and the related products.

13. The processor, The store operation support device according to claim 7, further comprising: a display unit configured to display a pop-up screen including information about a product corresponding to the node that is the target of cursor operation based on the user's cursor operation on an image showing each node in the product display map.

14. A store operations support method using a store operations support device that provides store operations support information to a user to support the operation of a store, comprising: The store operation support device, Acquire in-store video obtained by photographing the inside of the store, performing a process of recognizing one or more people and a plurality of products existing around each of the people in the in-store video; storing the results of the recognition process, including the purchasing behavior of each person, in a storage device; generating a product display map showing a relationship between the attention product selected by the user from among the plurality of products and a plurality of other related products, the product display map including information regarding a result of the recognition process; The store operation support method further comprises displaying the product display map on a display as the store operation support information.

Images