Store operation support device and store operation support method

Abstract

To enable a user to fully understand a customer's commodity selection status, and to enable effective measures or the like for improving store operation to be taken immediately.SOLUTION: On the basis of a camera image obtained by imaging a person staying in front of a display area in a store, an analysis server 2 performs an analysis relating to a commodity selection status of the person, and presents the analysis results to a user. In particular, the analysis server 2 detects persons from the camera image, identifies persons to be analyzed, detects behaviors of the persons from the camera image, acquires behavior information for each person to be analyzed, in association with a commodity, accumulates the behavior information for each person in a storage unit, generates commodity selection information including at least the time required to select the commodity, on the basis of the behavior information accumulated in the storage unit, accumulates the commodity selection information for each person in the storage unit, and acquires an analysis result visualizing the commodity selection status corresponding to each commodity, on the basis of the commodity selection information accumulated in the storage unit.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a store operation support device and a store operation support method that analyzes the product selection behavior of people based on camera images of people standing in front of a display area in a store and supports a user's store operation by presenting the analysis results to the user. [Background technology]

[0002] In a store, when a customer purchases an item, he or she examines the item in front of the display shelf. By analyzing such customer examination behavior, useful information for considering improvement measures for inventory management and in-store layout can be presented to the user (such as a store manager) to support the user's work.

[0003] A known technology related to the analysis of customer shopping habits in such stores is to detect "changes due to the placement of products on the shelves" or "changes due to the positioning of products on the shelves being shifted" based on camera images of display shelves, identify products that customers were interested in but did not purchase, and obtain the frequency with which customers were interested in products but did not purchase them (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 171574 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional technology only focuses on the frequency with which customers were interested in products but did not purchase them. This means that users cannot fully grasp the customer's product selection status, specifically, the degree to which the customer struggled to select products. This causes a problem in that users cannot immediately take effective measures to improve the operation of the store based on the presented information.

[0006] Therefore, the main object of the present invention is to provide a store operation support device and a store operation support method that enable a user to fully understand the product selection status of customers and immediately take effective measures to improve the operation of the store. [Means for solving the problem]

[0007] The store operations support device of the present invention is a store operations support device including a processor that performs processing to analyze the shopping behavior of a person based on a camera image of a person standing in front of a display area in a store and present the analysis result to a user, and the processor performs processing to analyze the shopping behavior of a person based on a camera image of a person standing in front of a display area in a store and present the analysis result to a user, Products and Detect people and include them in the analysis Products and The system is configured to identify a person, detect the person's product holding behavior and the product gaze behavior associated with the product holding behavior from the camera image, obtain the detection results as behavioral information, associate the behavioral information for each person to be analyzed with a product and store it in a memory unit, generate evaluation information including at least the duration of the product gaze behavior based on the behavioral information stored in the memory unit, store the evaluation information for each person in the memory unit, and obtain the analysis result that visualizes the evaluation situation corresponding to each product based on the evaluation information stored in the memory unit.

[0008] A store operation support method of the present invention is a store operation support method that causes an information processing device to perform a process of analyzing a person's shopping behavior based on a camera image of a person standing in front of a display area in a store and presenting the analysis result to a user, Products and Detect people and include them in the analysis Products andThe system is configured to identify a person, detect the person's product holding behavior and the product gaze behavior associated with the product holding behavior from the camera image, obtain the detection results as behavioral information, associate the behavioral information for each person to be analyzed with a product and store it in a memory unit, generate evaluation information including at least the duration of the product gaze behavior based on the behavioral information stored in the memory unit, store the evaluation information for each person in the memory unit, and obtain the analysis result that visualizes the evaluation situation corresponding to each product based on the evaluation information stored in the memory unit. Effect of the Invention

[0009] According to the present invention, an analysis result is obtained that visualizes the shopping situation corresponding to each product based on shopping information including the time required to evaluate the product, and the analysis result is presented to the user. This allows the user to fully understand the shopping situation of the customers and immediately take effective measures to improve the operation of the store. [Brief description of the drawings]

[0010] [Figure 1] Overall configuration of a store operation support system according to the present embodiment. [Diagram 2] An explanatory diagram showing the transition of camera images when a customer is examining products [Diagram 3] Block diagram showing the schematic configuration of the analysis server [Figure 4] An explanatory diagram showing the registered contents of the camera image database managed by the analysis server. [Diagram 5] FIG. 1 is an explanatory diagram showing the registered contents of a behavioral information database managed by an analysis server. [Figure 6] A diagram showing the registered contents of the concern level information database managed by the analysis server. [Figure 7] Flow diagram showing the steps of person identification processing performed on the analysis server [Figure 8] Flow diagram showing the procedure of behavior detection processing performed on the analysis server [Figure 9] Flow diagram showing the procedure for distress level estimation processing performed on the analysis server [Figure 10] An explanatory diagram showing the in-store map screen displayed on the viewing terminal. [Figure 11] FIG. 13 is an explanatory diagram showing a camera image screen displayed on a viewing terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The first invention made to solve the above problems is a store operation support device including a processor that performs processing to analyze the shopping behavior of a person standing in front of a display area in a store based on a camera image of the person and presents the analysis result to a user, the processor Products and Detect people and include them in the analysis Products and The system is configured to identify a person, detect the person's product holding behavior and the product gaze behavior associated with the product holding behavior from the camera image, obtain the detection results as behavioral information, associate the behavioral information for each person to be analyzed with a product and store it in a memory unit, generate evaluation information including at least the duration of the product gaze behavior based on the behavioral information stored in the memory unit, store the evaluation information for each person in the memory unit, and obtain the analysis result that visualizes the evaluation situation corresponding to each product based on the evaluation information stored in the memory unit.

[0012] According to this system, the system obtains an analysis result that visualizes the shopping situation for each product based on the shopping information including the time required to check the product, and presents the analysis result to the user. This allows the user to fully understand the shopping situation of the customers, and immediately take effective measures to improve the operation of the store.

[0013] In addition, in a second aspect of the present invention, when the processor determines that a person detected from the camera image is a store clerk based on feature information of the person, the processor excludes the person from the analysis target.

[0014] This makes it possible to avoid including store clerks performing tasks such as stocking products in the analysis, thereby obtaining appropriate analysis results.

[0017] Also, Third The invention is configured such that the processor outputs the analysis results, which include a map image in which an image visualizing the product evaluation information for each display area is drawn on an image representing the layout of the store.

[0018] This allows the user to immediately grasp the state of customers' product selection in each display area. In this case, the map image at each time may be played back as a video.

[0019] Also, Fourth The invention is configured such that the processor outputs the analysis results including the camera image corresponding to the selected display area in response to a user's operation of selecting the display area on a screen displaying the map image.

[0020] According to this, the user can specifically grasp the customer's selection status by viewing the camera images for the display area that the user has focused on by viewing the map image. In this case, the camera images at each time may be played back as a video.

[0021] Also, Fifth The invention is configured so that the processor acquires the number of times an item was held, the time spent looking at items, and the number of items held as the evaluation information based on the behavioral information, and acquires an evaluation level which quantifies the degree to which the person had difficulty evaluating items based on the number of times an item was held, the time spent looking at items, and the number of items held.

[0022] According to this, the shopping status of the customer can be visualized using a heat map or graph based on the shopping degree (degree of concern), which allows the user to easily understand the shopping status of the customer.

[0023] A sixth aspect of the present invention is a store operation support method for causing an information processing device to perform a process of analyzing a person's shopping behavior based on a camera image of a person standing in front of a display area in a store and presenting the analysis result to a user, the method comprising: Products and Detect people and include them in the analysis Products and The system is configured to identify a person, detect the person's product holding behavior and the product gaze behavior associated with the product holding behavior from the camera image, obtain the detection results as behavioral information, associate the behavioral information for each person to be analyzed with a product and store it in a memory unit, generate evaluation information including at least the duration of the product gaze behavior based on the behavioral information stored in the memory unit, store the evaluation information for each person in the memory unit, and obtain the analysis result that visualizes the evaluation situation corresponding to each product based on the evaluation information stored in the memory unit.

[0024] According to this, similarly to the first invention, the user can fully grasp the state of the customer's product selection, and can immediately take effective measures for improving the operation of the store.

[0025] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0026] FIG. 1 is a diagram showing the overall configuration of a store operation support system according to this embodiment.

[0027] This store operations support system analyzes the behavior of customers browsing products in front of display shelves in a store and presents the analysis results to a user (store manager) to support the user's work. The store operations support system comprises a camera 1, an analysis server 2 (store operations support device, information processing device), and a viewing terminal 3. The camera 1, analysis server 2, and viewing terminal 3 are connected via a network.

[0028] Camera 1 is installed in an appropriate location within the store. Camera 1 photographs the display shelves (display area) within the store and the aisle in front of the shelves (stay area) where customers stay to browse products.

[0029] The analysis server 2 analyzes the state of customers' product selection in the store. The analysis server 2 is composed of a PC, etc. The analysis server 2 may be installed in the store or may be a cloud computer.

[0030] The viewing terminal 3 allows a user (such as a store manager) to view the analysis results of the analysis server 2. The viewing terminal 3 is configured as a PC, a tablet terminal, or the like.

[0031] In this embodiment, an analysis of the customer's selection status is performed for each display area (display shelf) corresponding to a product category (noodles, rice balls, etc.). Meanwhile, the camera 1 captures an image of the display area of ​​the target product category. This allows the analysis server 2 to perform an analysis of the customer's selection status for each display area (product category) based on the camera images. Note that a single camera 1 may capture images of multiple display areas, and the captured images of each display area may be extracted from the camera images captured by the camera 1.

[0032] Next, the behavior of customers in front of display shelves in a store will be described. Figure 2 is an explanatory diagram showing the transition of camera images when a customer is examining products.

[0033] Camera 1 photographs, from above, the display shelves (display area) and the aisle in front of the shelves where customers stay to browse (stay area). The camera images show products on the shelves and people (customers) browsing in front of the shelves. Camera 1 may also photograph the shelves and people from the side. Camera 1 periodically transmits camera images (frames) taken at each time at a specified frame rate to analysis server 2.

[0034] Here, we will explain what happens if the person does not purchase the product. In this case, first, as shown in Figures 2(A) and (B), the person appears in front of the display shelf. Next, as shown in Figure 2(C), the person reaches out to the shelf and picks up an item from the shelf. Next, as shown in Figure 2(D), the person stares at the item they have picked up. Next, as shown in Figure 2(E), the person returns the item they have picked up to the shelf. Next, as shown in Figures 2(F) and (G), the person disappears from in front of the display shelf.

[0035] On the other hand, when a person purchases a product, as shown in Fig. 2(D), the person stares at the product he or she picked up, and then, without returning the product to the shelf, disappears from in front of the display shelf as shown in Fig. 2(G). Therefore, when the action of the person returning the product he or she picked up to the shelf cannot be detected, it can be determined that the person has purchased a product.

[0036] In this embodiment, product holding behavior and product gazing behavior are detected as behaviors related to customer selection. Product holding behavior is a behavior in which a person holds a product in their hand, as shown in Figs. 2(C), (D), and (E). Based on the detection status of this product holding behavior, it is possible to detect whether a person has picked up a product from a product shelf or returned the picked up product to the product shelf. Product gazing behavior is a behavior in which a person gazes at a product, as shown in Fig. 2(D). The duration of this product gazing behavior (product gazing time) is the time it takes to select a product, and indicates the degree to which the customer is struggling to select a product. It is estimated that the longer the gazing time, the more struggling the customer is to select a product.

[0037] In this embodiment, an example in which products are displayed on a display shelf will be described, but the store fixtures on which the products are displayed are not limited to display shelves. For example, products may be displayed on display stands (wagons) other than display shelves.

[0038] Next, a description will be given of a schematic configuration of the analysis server 2. Fig. 3 is a block diagram showing a schematic configuration of the analysis server 2.

[0039] The analysis server 2 includes a communication unit 11, a storage unit 12, and a processor 13.

[0040] The communication unit 11 communicates between the camera 1 and the viewing terminal 3 .

[0041] The storage unit 12 stores a program executed by the processor 13, etc. The storage unit 12 also stores registered information of a camera image database (see FIG. 4), registered information of a behavior information database (see FIG. 5), and registered information of a distress level information database (see FIG. 6).

[0042] The processor 13 performs various processes by executing the programs stored in the storage unit 12. In this embodiment, the processor 13 performs an image acquisition process, a person identification process, a behavior detection process, a distress level estimation process, a distress level aggregation process, an analysis result presentation process, and the like.

[0043] In the image acquisition process, the processor 13 acquires the camera image received from the camera 1 via the communication unit 11. This camera image is registered in the camera image database (see FIG. 4) in association with the ID of the camera 1 and the shooting time.

[0044] In the person identification process, processor 13 identifies a person to be analyzed based on the camera image. At this time, a person is first detected from the camera image (person detection process), and a person ID is assigned as the analysis target to a person determined not to be a store clerk, i.e., a customer, based on the characteristic information of the person. On the other hand, if the detected person is a store clerk, the person is excluded from the analysis target (detection result) (store clerk exclusion process). Furthermore, if the person is the same person as a person previously detected based on the characteristic information of the person extracted from the camera image, processor 13 performs a process of matching the person (person tracking process).

[0045] In the behavior detection process, the processor 13 detects the behavior of the person analyzed in the person identification process from the camera images (frames) at each time. As a result of the behavior detection process, behavior information for each person in each camera is registered in the behavior information database (see FIG. 5).

[0046] In this embodiment, the processor 13 detects, as behaviors related to a customer's selection of a product, a behavior of a person holding a product in their hand (product holding behavior) and a behavior of a person gazing at a product (product gazing behavior).

[0047] In addition, here, the processor 13 associates the actions detected from the camera images (frames) at each time as a series of actions by the same person (action tracking process). Specifically, when a person's action is newly detected from the camera images, a new action ID is assigned to the action, and the same action ID is assigned to a series of actions by the same person detected from the subsequent camera images.

[0048] Here, the processor 13 detects the product picked up by the person from the camera image, and identifies the name of the product by image recognition (product detection process).

[0049] Moreover, here, the processor 13 measures the duration of the person's product gaze action, thereby acquiring the time during which the person gazes at the product (product gaze time) (gaze time measurement process). Specifically, the product gaze time is measured based on the number of camera images (frames) in which the product gaze action is detected, and the time for one cycle corresponding to the interval between camera images (frame interval).

[0050] Also, here, processor 13 judges whether a purchase has been made when the person's tracking period (the period from when the person enters the shooting area of ​​camera 1 to when the person leaves) ends (purchase determination process). At this time, by tracking the person's product holding behavior, it is detected whether the person has returned the product they picked up to the product shelf, and whether a purchase has been made is determined based on the result. Note that a purchase can be determined when the person leaves the display shelf without returning the product they picked up, but it may also be possible to detect the action of putting the product they picked up into a basket.

[0051] In the worry degree estimation process, the processor 13 estimates the worry degree of each person in each display area at each time based on the behavior information that is the detection result of the behavior detection process registered in the behavior information database (see FIG. 5). As a result of the worry degree estimation process, the worry degree of each person at each time of each camera 1 corresponding to each display area is registered in the worry degree information database (see FIG. 6).

[0052] The degree of worry (degree of shopping) is a numerical representation of the degree to which a person had trouble shopping for a product. In this embodiment, the number of times a product was held, the product gaze time, and the number of held products are obtained for each person, and the degree of worry for each person is calculated based on the number of times a product was held, the product gaze time, and the number of held products, using the following formula: Worry level=λ 1 × Product gaze time (seconds) + λ 2 ×Number of times product is held (times) +λ 3 ×Number of products held (pieces) For example, for each coefficient, λ 1 =1,λ 2 =5,λ 3 = 10, and if you pick up each of the two products once and stare at them for a total of 30 seconds, your level of distress will be 1 x 30 + 5 x 2 + 10 x 2 = 60.

[0053] Here, the number of product holding times is the number of times a person performs a product holding action in which the person picks up a product. The product gaze time is the duration of the product gaze action in which the person gazes at the product. The number of products held is the number of products that are the subject of the product holding action in which the person picks up the product. Note that overlapping pick-ups of the same product are not counted.

[0054] In the worry level counting process, the processor 13 counts up the worry levels of each person at each time for each camera acquired in the worry level estimation process, and calculates the worry level at each time for each display area corresponding to each camera.

[0055] In the analysis result presentation process, the processor 13 presents the user with the analysis results regarding the customer's shopping situation in the store. Specifically, in response to a request from the viewing terminal 3, the processor 13 displays on the viewing terminal 3 an in-store map screen 21 (see FIG. 10) including a concern degree heat map that visualizes the concern degree at each time for each display area acquired in the concern degree counting process. The processor 13 also displays on the viewing terminal 3 a camera image screen 51 (see FIG. 11) including a camera image for each display area and a concern degree graph that visualizes the concern degree at each time for each display area.

[0056] Next, a description will be given of the camera image database managed by the analysis server 2. Fig. 4 is an explanatory diagram showing the registered contents of the camera image database.

[0057] The analysis server 2 registers and manages the camera images (frames) received from the camera 1 at each time in a camera image database. In the camera image database, the camera images are registered in association with the name of the camera 1 (camera ID) and the time of shooting.

[0058] Next, a description will be given of the behavioral information database managed by the analysis server 2. Fig. 5 is an explanatory diagram showing the registered contents of the behavioral information database.

[0059] The analysis server 2 performs a process (behavior detection process) to detect the behavior of people from the camera images (frames) at each time, and the results of this behavior detection process, that is, the behavior information for each person in each camera, are registered in a behavior information database.

[0060] The behavioral information database registers the name of the camera 1 corresponding to the display area (camera ID), the person ID, the behavior ID, the name of the product (product ID), the time spent looking at the product, and purchase status information (information regarding whether or not a purchase was made) such as purchase (True) and non-purchase (False) as behavioral information for each person.

[0061] In the behavior detection process, a behavior ID is assigned to a series of actions in which a person picks up a product and puts it back, or picks up a product and leaves the display shelf without putting it back. Therefore, if a person picks up a product and puts it back multiple times in one display area, it is detected as a different behavior and a different behavior ID is assigned, even if the product picked up by the person is different or the product is the same.

[0062] Next, a description will be given of the distress level information database managed by the analysis server 2. Fig. 6 is an explanatory diagram showing the registered contents of the distress level information database.

[0063] The analysis server 2 performs a process of estimating the degree of worry for each person (distress degree estimation process), and the results of this distress degree estimation process, i.e., the distress degree for each person at each time for each camera 1 corresponding to each display area, are registered in a distress degree information database.

[0064] In the worry level information database, the name of the camera 1 (camera ID), the estimated time, the person ID, and the worry level are registered as the worry level information for each person.

[0065] In the distress level estimation process, a process of estimating a distress level for each person is periodically performed. Therefore, as the time that a person gazes at a product to evaluate the product (the duration of the product gaze behavior) becomes longer, the distress level value for that person at each time point gradually increases.

[0066] Next, a description will be given of the person identification process carried out by the analysis server 2. FIG 7 is a flow diagram showing the procedure of the person identification process carried out by the analysis server 2.

[0067] The analysis server 2 performs a process (person identification process) to identify a person to be analyzed based on the camera image. In this person identification process, the flow shown in Fig. 7 is executed every time a camera image (frame) periodically transmitted from the camera 1 is received.

[0068] First, the processor 13 acquires a camera image received from the camera 1 via the communication unit 11 (image acquisition process) (ST101).

[0069] Next, the processor 13 detects a person from the camera image (person detection process) (ST102). At this time, a rectangular person frame (person area) surrounding the person is set in the camera image, and position information of the person frame on the camera image is obtained.

[0070] Next, the processor 13 judges whether the person detected from the camera image is a store clerk or not (ST103). At this time, it is possible to judge whether the person is a store clerk or not based on the characteristics of the person's clothing. Specifically, it is possible to judge whether the person is a store clerk or not depending on whether the person is wearing the store's uniform or not. Note that store clerks perform tasks such as stocking products in front of the display shelves, and are therefore confused with customers browsing the products in front of the display shelves.

[0071] If the person detected from the camera image is a store clerk (Yes in ST103), the person is excluded from the analysis target (detection result) (store clerk exclusion process) (ST104), and the processing for this camera image (frame) is then terminated.

[0072] On the other hand, if the person detected from the camera image is not a store clerk, i.e., a customer (No in ST103), processor 13 then determines whether or not the person detected from the camera image is already being tracked (ST105). Note that if there are multiple people in the camera image, and a person that is not a store clerk (i.e., a customer) is detected, the process of ST105 is performed for each person.

[0073] Here, if the person detected from the camera image is not already being tracked, that is, if the person is detected for the first time in the current camera image (No in ST105), the person is added to the tracking targets and a person ID is assigned to the person (ST106).

[0074] Next, the processor 13 registers the current camera image in the camera image database (see FIG. 4) in association with the camera ID and the shooting time. The processor 13 also adds the detection result regarding the current camera image, i.e., the position information of the person frame on the camera image, in association with the person ID to the person tracking information (ST107).

[0075] On the other hand, if the person detected from the camera image is already being tracked (Yes in ST105), the process in ST106 is omitted.

[0076] Next, a description will be given of the behavior detection process performed by the analysis server 2. FIG 8 is a flow diagram showing the procedure of the behavior detection process performed by the analysis server 2.

[0077] The analysis server 2 performs a process (behavior detection process) to detect the behavior of customers browsing in front of the shelves based on the camera images received from the camera 1. In this behavior detection process, the flow shown in Fig. 8 is executed every time a camera image (frame) periodically transmitted from the camera 1 is received.

[0078] First, the processor 13 determines a person detected from the current camera image as a person of interest, and acquires the current camera image (frame) in which the person of interest appears, the person ID, and position information of the person frame on the current camera image (ST201).

[0079] Next, the processor 13 executes a predetermined behavior recognition process on the entire image including the person of interest, and detects the behavior of the person of interest, such as a product holding behavior, a product gazing behavior, etc. (ST202).

[0080] Next, the processor 13 extracts behavior information relating to behaviors that have been previously detected and set as tracking targets for the person of interest from the behavior information database (ST203).

[0081] Next, the processor 13 compares the currently detected behavior of the person of interest with previously detected behaviors, and determines whether or not the previously detected behavior of the person of interest has not been detected this time (ST204).

[0082] Here, if the behavior previously detected with respect to the person of interest is detected again this time (No in ST101), processor 13 then determines whether the behavior currently detected with respect to the person of interest has already been set as a tracking target (ST205).

[0083] Here, if the behavior currently detected with respect to the person of interest has not already been set as a tracking target (No in ST205), the behavior currently detected with respect to the person of interest is additionally set as a tracking target, and a behavior ID is assigned to the behavior currently detected (ST206).

[0084] Next, the processor 13 determines whether or not the behavior currently detected with respect to the person of interest is a product gaze behavior (ST207).

[0085] If the behavior currently detected for the person of interest is a product gaze behavior (Yes in ST207), one cycle of time corresponding to the interval between camera images (frame interval) is added to the cumulative value of gaze time for that behavior (ST208). Here, each time a product gaze behavior is detected from a camera image (frame), the cumulative value of gaze time is updated so that one cycle of time is added.

[0086] Next, the processor 13 updates the registered contents of the behavior information database (ST209). At this time, if the behavior of the target person detected this time is a product gaze behavior, the gaze time added this time for that behavior is registered in the behavior information database (see FIG. 5).

[0087] On the other hand, if a behavior previously detected with respect to the person of interest is not detected this time (Yes in ST204), processor 13 then determines a tracking period for the behavior of the person of interest and extracts the detection results for the camera images included in that tracking period, i.e., the behavior information (corresponding to the behavior ID) of the person of interest included in the tracking period (ST210).

[0088] Next, the processor 13 judges whether the target person has purchased a product based on the behavior information (corresponding to the behavior ID) of the target person included in the tracking period (purchase judgment process) (ST211). At this time, if it is detected that the target person has returned the product to the display shelf, it is judged that the target person has not purchased the product. On the other hand, if it is detected that the target person has put the product in the basket or has left the display shelf while holding the product, it is judged that the target person has purchased the product.

[0089] Next, the processor 13 excludes the behavior of the person of interest that was set as a tracking target from the tracking target (ST212).

[0090] Next, the processor 13 updates the registered contents of the behavior information database (ST209). At this time, the determination result of the purchase determination process, that is, information on the purchase status (whether or not a purchase was made), is registered in the behavior information database (see FIG. 5).

[0091] Next, a description will be given of the distress degree estimation process carried out by the analysis server 2. FIG.

[0092] The analysis server 2 performs a process (distress degree estimation process) of estimating the distress level of each person based on the behavior information of each person registered in the behavior information database (see FIG. 5) by the behavior detection process (see FIG. 8). In this distress level estimation process, the flow shown in FIG. 9 is repeated for each person whose behavior is detected in the behavior detection process, that is, each person whose person ID is registered in the behavior information database.

[0093] First, the processor 13 acquires behavior information related to a person of interest from the behavior information database (see FIG. 5) (ST301).

[0094] Next, the processor 13 acquires the number of times the product was held, the product gaze time, and the number of held products based on the behavior information related to the person of interest (ST302).

[0095] Next, the processor 13 calculates the distress level for the person of interest based on the number of times the product was held, the product gaze time, and the number of held products (ST303).

[0096] Next, the processor 13 registers the concern degree for the noted person in the concern degree information database (see FIG. 6) together with the camera name, the current time, and the person ID (ST304).

[0097] Next, a description will be given of the in-store map screen 21 displayed on the viewing terminal 3. FIG.

[0098] The viewing terminal 3 displays an in-store map screen 21 that visualizes the customer's product selection status for each display area (display shelf) and presents it to the user.

[0099] The in-store map screen 21 is provided with a map display section 22. The map display section 22 displays a distress level heat map 31 (map image) that visualizes the level of distress (item selection status) for each display area on an in-store map that shows the layout of the store.

[0100] Specifically, display area images 32 representing display areas (display shelves) for each product category (noodles, rice balls, etc.) are drawn on the concern level heat map 31, and the display form of these display area images 32 changes according to the level of concern level. In the example shown in Fig. 10, the level of concern level for each display area is expressed by the shade of color. Display area images 32 related to display areas with high concern levels are highlighted in dark colors.

[0101] In this manner, in this embodiment, the concern degree (item selection status) for each display area is visualized in the concern degree heat map 31. This allows a user (such as a store manager) to immediately grasp the concern degree (item selection status) for each display area. In the example shown in Fig. 10, the user can immediately grasp that the concern degree is highest in the noodle display area.

[0102] The in-store map screen 21 is also provided with a playback operation section 23. This playback operation section 23 is provided with a slider 42 that can be moved on a seek bar 41. The seek bar 41 corresponds to the business hours of the target store in one day (from opening time to closing time). The user can operate the slider 42 to specify the playback position (playback time) to display the concern level heat map 31 at any time during the business hours.

[0103] The concern degree heat map 31 is displayed as a video. The playback operation unit 23 is provided with a play button 43. By operating the play button 43, the user can play the concern degree heat map 31 as a video from the opening time or from any time specified by the slider 42. This allows the user to instantly grasp the changes in the customer concern degree for each display area.

[0104] Here, the analysis server 2 calculates the worry degree for each display area (camera 1) by tallying up the worry degree for each person. At this time, a tally period of a predetermined length (e.g., one minute) based on the display time is set. For example, a tally period of a predetermined length is set immediately before the display time. Then, the worry degrees for each person included in the tally period are tallied to calculate the worry degree for each display area at the display time. As a result, the tally period shifts as the display time progresses, so the worry degree for each display area changes over time, and in response to this change in worry degree, the display state (e.g., the shade of color) of the display area image 32 in the worry degree heat map 31 changes.

[0105] In this manner, in this embodiment, the concern degree heat map 31 is displayed as a video, which allows the user to immediately grasp the transition of the concern degree of the customers in each display area.

[0106] Furthermore, in the playback operation unit 23, the user can specify the analysis target period on the seek bar 41. Specifically, two section designation buttons 44 are provided on the playback operation unit 23 so as to be movable along the seek bar 41. The two section designation buttons 44 correspond to the start point and end point of the analysis target period. By operating the section designation buttons 44, the user can designate any period as the analysis target period. At this time, a portion 45 on the seek bar 41 that corresponds to the analysis target period is highlighted.

[0107] When the analysis period is specified in this way, the analysis server 2 performs an analysis based on the behavioral information of customers included in the specified analysis period, and displays the distress degree heat map 31 as the analysis result on the in-store map screen 21. This allows the user to check the distress degree of customers by narrowing down the time period.

[0108] In addition, tabs 25 for selecting a screen are provided on the store map screen 21. By operating the tabs 25, the user can switch between the store map screen 21 (see FIG. 10) and the camera image screen 51 (see FIG. 11).

[0109] Next, there will be described camera image screen 51 displayed on viewing terminal 3. FIG.

[0110] When an operation to select a display area is performed on the in-store map screen 21 (see FIG. 10), a camera image screen 51 relating to the selected display area is displayed. In addition, when the camera image tab 25 is operated on the in-store map screen 21, the screen transitions to the camera image screen 51.

[0111] Camera image screen 51 is provided with camera image display section 52. Camera image 61 is displayed on camera image display section 52. Camera 1 captures from above the display area (display shelves) and the area in front of the display shelves where customers stay to browse. Camera image 61 captures products in the display area and customers picking up and looking at the products. This allows a user to visually and specifically check the actual situation of customers browsing the products by viewing camera image 61.

[0112] Furthermore, in the camera image display section 52, when the user performs an operation to select a person or a product on the camera image 61, a speech bubble 62 (information display section) is displayed. In this speech bubble 62, the name of the product picked up by the person, the number of times the person picked up the product (product holding count), and the time the person inspected the product (product gaze time) are displayed. Note that when an operation to select a product is performed, the name of the product may be displayed in the speech bubble 62, and when an operation to select a person is performed, the person ID may be displayed in the speech bubble 62.

[0113] Also, the camera image screen 51 is provided with a playback operation section 23, similar to the in-store map screen 21 (see FIG. 10). In this playback operation section 23, the camera image at a desired time can be displayed by moving the slider 42 along the seek bar 41. Also, by operating the playback button 43, the camera image can be played back as a video from any time.

[0114] Furthermore, the camera image screen 51 is provided with a graph display section 53. The graph display section 53 displays a distress level graph 65 for the display area corresponding to the camera image. In the distress level graph 65, the horizontal axis represents time and the vertical axis represents distress level, and the transition of the distress level over time is expressed. This allows the user to immediately grasp the transition of the distress level in the target display area.

[0115] Furthermore, a plurality of area selection buttons 54 are provided for each display area on the camera image screen 51. When the user operates an area selection button 54, the camera image screen 51 relating to the display area corresponding to that area selection button 54 is displayed. This allows the user to easily switch to the camera image screen 51 relating to a desired display area and check the status of customers selecting items in the desired display area.

[0116] In this manner, in this embodiment, since the distress level graph 65 showing the change in distress level is displayed, the user can immediately know the time when the distress level was high. Furthermore, by operating the playback operation unit 23 to display the camera image 61 at the time when the distress level was high, the user can confirm the situation when the customer was distressed and can also know whether the customer purchased a product. Furthermore, by displaying the speech bubble 62 on the camera image 61, the user can confirm the name of the product the customer picked up, etc.

[0117] This allows the user to recognize, for example, whether the customer has purchased a product after careful deliberation. If the customer has purchased a product after careful deliberation, it is assumed that the customer has made a decision to purchase after thoroughly comparing it with other products, and therefore the product selected by the customer is considered to be superior to the other products. For this reason, sales can be increased by displaying the product in a position where it is easily seen by many customers. On the other hand, if the customer has not purchased anything after careful deliberation, it is assumed that the customer has purchased the product if the store clerk has provided support such as introducing the product. Thus, the user can consider sales promotion ideas such as the timing of the store clerk's support for the customer.

[0118] Although the in-store map screen 21 and the camera image screen 51 display the customer's shopping status in real time based on the customer's behavior information on the day, the in-store map screen 21 and the camera image screen 51 may display the customer's shopping status in the past. In this case, the user may specify conditions (such as a date, a day of the week, or a period), and the in-store map screen 21 and the camera image screen 51 may be generated based on the customer's behavior information in the past so as to meet the conditions.

[0119] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0120] The store operations support device and store operations support method of the present invention have the effect of enabling users to fully understand the product shopping behavior of customers and immediately take effective measures to improve store operations, and are useful as a store operations support device and store operations support method that analyzes the product shopping behavior of people based on camera images taken of people in front of the display area in a store and supports the user's work by presenting the analysis results to the user. [Explanation of symbols]

[0121] 1 Camera 2 Analysis server (store operation support device, information processing device) 3 Viewing device 12 Storage section 13 Processors 21 Store map screen 23 Playback operation section 31 Worry Heat Map 32 Display area images 41 Seek Bar 42 Slider 43 Play button 44 Section designation button 45 The portion corresponding to the period under analysis 51 Camera image screen 54 Area selection button 61 Camera Images 62 Speech bubble 65 Worry Level Graph

Claims

1. A store operation support device including a processor that performs processing to analyze a person's shopping behavior based on a camera image of a person standing in front of a display area in a store and present the analysis result to a user, The processor, Detecting products and people from the camera image and identifying products and people to be analyzed; detecting a product holding behavior of a person and a product gaze behavior associated with the product holding behavior from the camera image, acquiring the detection result as behavior information, and storing the behavior information for each person to be analyzed in a storage unit in association with a product; generating product evaluation information including at least a duration of the product gaze behavior based on the behavior information stored in the storage unit, and storing the product evaluation information for each person in the storage unit; A store operation support device characterized by acquiring the analysis results that visualize the shopping situation corresponding to each product based on the shopping information stored in the memory unit.

2. The processor, 2. The store operation support device according to claim 1, wherein when a person is determined to be a store clerk based on the characteristic information of the person detected from the camera image, the person is excluded from analysis targets.

3. The processor, The store operation support device according to claim 1, characterized in that the analysis result is outputted in such a way that an image visualizing the product evaluation information for each display area includes a map image drawn on an image representing the layout of the store.

4. The processor, The store operation support device according to claim 3, characterized in that, in response to a user's operation of selecting the display area on a screen displaying the map image, the analysis result including the camera image corresponding to the selected display area is output.

5. The processor, The store operation support device described in claim 1, characterized in that based on the behavioral information, the number of times a product is held, the time spent looking at products, and the number of products held are obtained as the evaluation information, and a evaluation level is obtained that quantifies the degree to which a person had difficulty evaluating products based on the number of times a product is held, the time spent looking at products, and the number of products held.

6. A store operation support method that causes an information processing device to perform a process of analyzing a person's shopping behavior based on a camera image of a person standing in front of a display area in a store and presenting the analysis result to a user, comprising: Detecting products and people from the camera image and identifying products and people to be analyzed; detecting a person's product holding behavior and a product gaze behavior associated with the product holding behavior from the camera image, acquiring the detection result as behavior information, and storing the behavior information for each person to be analyzed in a storage unit in association with a product; generating product evaluation information including at least a duration of the product gaze behavior based on the behavior information stored in the storage unit, and storing the product evaluation information for each person in the storage unit; A store operation support method characterized by obtaining the analysis results that visualize the shopping situation corresponding to each product based on the shopping information accumulated in the memory unit.

Images