Evaluation program, evaluation method, and information processing device

Abstract

To evaluate effectiveness of a sales-promotion program.SOLUTION: An information processing device acquires a video obtained by photographing an area containing a commodity shelf on which commodities are arranged. The information processing device specifies an action of a person for purchasing the commodity by analyzing the acquired video. The information processing device acquires purchasing interest information based on an action for purchasing a commodity, indicating purchasing interest of a person. The information processing device acquires timing information regarding timing where the sales-promotion program for the commodity is conducted. On the basis of the timing information, the information processing device analyzes a change of the acquired purchase interest information before and after the sales-promotion program. The information processing device evaluates the sales-promotion program on the basis of the analyzed change.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an evaluation program and the like. [Background technology]

[0002] In store management, various sales promotion measures are implemented, such as distributing product flyers and airing product commercial messages. In recent years, there has been an increasing need to evaluate the impact of such sales promotion measures on customer purchasing behavior.

[0003] In addition, there is a conventional technology that can identify what attributes a customer has, what products they have purchased, and when by analyzing the purchase history of each customer's membership card, which is recorded in a store's point of sale (POS) register system.

[0004] In addition, there is a conventional technology that tracks customers by analyzing video data in a store and identifies their behavior to determine whether the tracked customers purchased products or left without purchasing products. With this conventional technology, it is possible to analyze when customers who are not members also purchased products. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2023-094868 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, while the above-mentioned conventional technology can identify whether a customer has purchased a product, it cannot determine the reason for the purchase, and therefore cannot be said to be able to identify the effectiveness of a sales promotion measure.

[0007] In one aspect, the present invention aims to provide an evaluation program, an evaluation method, and an information processing device that can evaluate the effectiveness of a sales promotion measure. [Means for solving the problem]

[0008] In the first proposal, a computer executes the following process. The computer acquires video footage of an area including product shelves on which products are placed. The computer identifies a person's product purchasing behavior by analyzing the acquired video. The computer acquires purchasing psychology information based on product purchasing behavior and indicating a person's purchasing psychology. The computer acquires timing information related to when a promotional measure for the product was implemented. The computer analyzes changes in the acquired purchasing psychology information before and after the promotional measure based on the timing information. The computer evaluates the promotional measure based on the analyzed changes. Effect of the Invention

[0009] The effectiveness of promotional measures can be evaluated. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram (1) for explaining the process of the information processing device according to the first embodiment. [Diagram 3] FIG. 3 is a diagram (2) for explaining the process of the information processing device according to the first embodiment. [Figure 4] FIG. 4 is a functional block diagram of the information processing device according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a data structure of the execution frequency information DB according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a data structure of the conditional probability distribution table DB according to the first embodiment. [Figure 7]FIG. 7 is a diagram illustrating an example of a data structure of the imaging DB according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a skeletal model of a human body. [Figure 9] FIG. 9 is a diagram showing an example of a joint name. [Figure 10] FIG. 10 is a flowchart of a processing procedure of the information processing device according to the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a system according to the second embodiment. [Figure 12] FIG. 12 is a functional block diagram of an information processing device according to the second embodiment. As shown in FIG. [Figure 13] FIG. 13 is a diagram illustrating an example of the data structure of the reliability DB. [Figure 14] FIG. 14 is a diagram illustrating an example of a data structure of the conditional probability distribution table DB according to the second embodiment. [Figure 15] FIG. 15 is a flowchart of a processing procedure of the information processing device according to the second embodiment. [Figure 16] FIG. 16 is a functional block diagram of an information processing device according to the third embodiment. As shown in FIG. [Figure 17] FIG. 17 is a diagram illustrating an example of a data structure of the interest level DB according to the third embodiment. [Figure 18] FIG. 18 is a diagram illustrating an example of a data structure of the execution frequency information DB according to the third embodiment. [Figure 19] FIG. 19 is a diagram illustrating an example of a data structure of the conditional probability distribution table DB according to the third embodiment. [Figure 20] FIG. 20 is a diagram illustrating an example of a conditional probability distribution table according to the third embodiment. [Figure 21] FIG. 21 is a diagram for explaining the process of the update unit according to the third embodiment. [Figure 22] FIG. 22 is a flowchart of a processing procedure of the information processing device according to the third embodiment. [Figure 23]FIG. 23 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of an evaluation program, an evaluation method, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiment. EXAMPLES

[0012] An example of a system according to the present embodiment 1 will be described. Fig. 1 is a diagram showing an example of a system according to the present embodiment 1. As shown in Fig. 1, this system includes cameras 10a, 10b, 10c, and 10d, and an information processing device 100. The cameras 10a to 10d and the information processing device 100 are connected to each other via a network.

[0013] For convenience of explanation, in FIG. 1, cameras 10a to 10d are shown, and the system according to the first embodiment may further include other cameras.

[0014] The cameras 10a to 10d are installed at predetermined positions in the store. A plurality of products are arranged in the store. The positions (coordinates) at which the cameras 10a to 10d are installed are different positions. In the following description, the cameras 10a to 10d will be referred to as "camera 10" unless otherwise distinguished.

[0015] Camera 10 captures video of the inside of the store and transmits the captured video data to information processing device 100. It is assumed that the imaging range of cameras 10a to 10d includes customers and product shelves. In the following description, the video data transmitted by camera 10 to information processing device 100 is referred to as "video data."

[0016] The video data includes a plurality of image frames in a time series. A frame number is assigned to each image frame in ascending chronological order. One image frame is a still image captured by the camera 10 at a certain timing. Time data may be assigned to each image frame. Camera identification information for identifying the camera 10 that captured the video data is set in the video data.

[0017] The information processing device 100 repeatedly executes a process of identifying the behavior of a customer in purchasing a product by analyzing the video data captured by the camera 10 and identifying the level of interest according to the identified behavior. The information processing device 100 also acquires time information regarding the time when a promotional measure for a product was implemented, and evaluates the promotional measure based on the change in the level of interest before and after the promotional measure. This allows the effectiveness of the promotional measure to be evaluated.

[0018] Next, the processing of the information processing device 100 will be specifically described. Fig. 2 and Fig. 3 are diagrams for explaining the processing of the information processing device according to the first embodiment. The information processing device 100 holds implementation frequency information of each promotional measure. The implementation frequency information is information indicating, in terms of probability, how frequently the promotional measure is implemented. The example shown in Fig. 2 shows implementation frequency information 20a of the promotional measure "CM video" and implementation frequency information 20b of the promotional measure "POP (Point of Purchase Display)".

[0019] In the implementation frequency information 20a, the value of "X2" is "0 (False)" which indicates that the promotional measure "CM video" is not being implemented. The value of "X2" is "1 (True)" which indicates that the promotional measure "CM video" is being implemented. Furthermore, when the value of "X2" is "0 (False)", "P(X3)" indicates the probability that the promotional measure "CM video" is not being implemented. When the value of "X2" is "1 (True)", "P(X2)" indicates the probability that the promotional measure "CM video" is being implemented.

[0020] In the implementation frequency information 20b, the value of "X3" is "0 (False)" which indicates that the promotional measure "POP" is not being implemented. The value of "X3" is "1 (True)" which indicates that the promotional measure "POP" is being implemented. Furthermore, when the value of "X3" is "0 (False)", "P(X3)" indicates the probability that the promotional measure "POP" is not being implemented. When the value of "X3" is "1 (True)", "P(X3)" indicates the probability that the promotional measure "POP" is being implemented.

[0021] When the administrator of the information processing device 100 knows in advance the effectiveness of the sales promotion measures at other stores, etc., he / she specifies the initial values ​​of P(X2) in the implementation frequency information 20a and P(X3) in the implementation frequency information 20b based on the sales records of the other stores. The information processing device 100 may set the current frequency (0.5) as the initial values ​​of P(X2) in the implementation frequency information 20a and P(X3) in the implementation frequency information 20b.

[0022] The information processing device 100 estimates the implementation period of each promotional measure based on the implementation frequency information 20a and 20b, and sets the estimated implementation period in the time information 25. For example, based on the implementation frequency information 20a, the information processing device 100 sets in the time information 25 that the CM operation is turned on and off every hour. Based on the implementation frequency information 20b, the information processing device 100 sets in the time information 25 that POP is performed in the morning and in the afternoon.

[0023] 3. The information processing device 100 analyzes the video data captured by the camera 10 to identify the behavior of a customer purchasing a product, and registers information regarding the interest level in the interest level DB 141 during the process of repeatedly executing a process of identifying the interest level according to the identified behavior.

[0024] As shown in FIG. 3, interest level DB141 has date and time, customer attributes, X3, X2, and X1. The date and time is the date and time when the interest level was identified. The customer attributes are the attributes of the customer whose interest level was identified. X1 indicates the presence or absence of interest. For example, if the interest level is equal to or greater than a threshold, X1 is "True". If the interest level is less than the threshold, X1 is "False". The explanation for X3 and X2 is the same as that above.

[0025] The information processing device 100 specifies the values ​​of X3 and X2 based on the time information 25 and the date and time when the level of interest was specified. For example, if the date and time when the level of interest was specified based on the time information 25 is a date and time when a commercial video is being broadcast, the information processing device 100 sets X2 to "True." If the date and time when the level of interest was specified based on the time information is a date and time when a commercial video is not being broadcast, the information processing device 100 sets X2 to "False."

[0026] The information processing device 100 sets X3 to "True" if the date and time when the interest level was identified based on the time information 25 is a date and time when POP is implemented. The information processing device 100 sets X3 to "False" if the date and time when the interest level was identified based on the time information is a date and time when POP is not implemented.

[0027] In the first row of records in Figure 3, the promotional measures "POP" and "CM video" were not implemented at the date and time "2 / 3 10:04", indicating that the customer attribute "women in their 20s and 30s" had no interest in the product at that date and time.

[0028] The relationship between X1, X2, and X3 can be expressed by a Bayesian network 26. Node 26-1 is a node corresponding to X1. Node 26-2 is a node corresponding to X2. Node 26-3 is a node corresponding to X3.

[0029] The information processing device 100 calculates a conditional probability distribution table for each customer attribute based on the interest level DB 141. The conditional probability distribution table 30 in Fig. 3 is a conditional probability distribution table for the customer attribute "female in 20s-30s". When generating the conditional probability distribution table 30, the information processing device 100 uses the records of the customer attribute "female in 20s-30s" from among the records in the interest level DB 141.

[0030] The conditional probability distribution table 30 has X2, X3, and P(X1|X2, X3). The explanations for X1, X2, and X3 are the same as those above. P(X1|X2, X3) is the probability of the event X1 under the condition that the events X2 and X3 occur.

[0031] The record in the first row of the conditional probability distribution table 30 indicates that the probability that the customer's interest level will be less than the threshold when X2 is "False" and X3 is "False" is "0.6 (60%, hereafter the same)." It also indicates that the probability that the customer's interest level will be equal to or greater than the threshold when X2 is "False" and X3 is "False" is "0.4."

[0032] For example, the information processing device 100 identifies records where X2 is "False" and X3 is "False" among records with the customer attribute "Female in 20s-30s", and sets the proportion of identified records where X1=False to cells in the conditional probability distribution table 30 corresponding to X2=False, X3=False, and P(X1|X2, X3)=False. The information processing device 100 identifies records where X2 is "False" and X3 is "False" among records with the customer attribute "Female in 20s-30s", and sets the proportion of identified records where X1=True to cells in the conditional probability distribution table 30 corresponding to X2=False, X3=False, and P(X1|X2, X3)=True.

[0033] The record in the second row of the conditional probability distribution table 30 indicates that the probability that the customer's interest level will be less than the threshold value is "0.2" when X2 is "False" and X3 is "True." It also indicates that the probability that the customer's interest level will be equal to or greater than the threshold value is "0.8" when X2 is "False" and X3 is "True."

[0034] The record in the third row of the conditional probability distribution table 30 indicates that the probability that the customer's interest level will be less than the threshold is "0.5" when X2 is "True" and X3 is "False." It also indicates that the probability that the customer's interest level will be equal to or greater than the threshold is "0.5" when X2 is "True" and X3 is "False."

[0035] The record in the fourth row of the conditional probability distribution table 30 indicates that the probability that the customer's interest level will be less than the threshold value is "0.3" when X2 is "True" and X3 is "True". It also indicates that the probability that the customer's interest level will be equal to or greater than the threshold value is "0.7" when X2 is "True" and X3 is "False".

[0036] The information processing device 100 may calculate the conditional probability distribution table 30 when the number of records of the customer attribute "female in her 20s to 30s" is equal to or greater than a predetermined number (for example, 20 or more).

[0037] FIG. 3 shows the conditional probability distribution table 30 for the customer attribute "female in her 20s to 30s," but the information processing device 100 generates conditional probability distribution tables for other customer attributes in the same manner.

[0038] The conditional probability distribution table 30 generated by the information processing device 100 is an evaluation result of the effectiveness of promotional measures. For example, comparing the record in the first row of the conditional probability distribution table 30 with the record in the fourth row, it is found that the implementation of a promotional measure increases customer interest more than the implementation of no promotional measure. Also, comparing the record in the second row of the conditional probability distribution table 30 with the record in the third row, it is found that the promotional measure "POP" alone increases customer interest more than the promotional measure "CM video" alone.

[0039] Here, the information processing device 100 updates the implementation period of the promotional measures after generating the conditional probability distribution table. The information processing device 100 sets the implementation frequency of each promotional measure by calculating the ratio of P(X1|X2, X3) when X2 and X3 are each 1 (True). Here, the explanation will be given using the conditional probability distribution table 30 corresponding to the customer attribute "women in their 20s and 30s".

[0040] For example, the implementation frequency P'(X2) of the promotional measure "POP(X2)" after the update is calculated by formula (1). As a result of the calculation of formula (1), the value of the implementation frequency P'(X2) becomes "0.5 (probability of implementation)". Based on the calculation result, the information processing device 100 updates P(X2) in the implementation frequency information 20a by P'(X2).

[0041]

number

[0042] The updated implementation frequency P'(X3) of the promotional measure "CM video (X3)" is calculated by formula (2). As a result of the calculation of formula (2), the value of the implementation frequency P'(X3) becomes "0.625 (probability of implementation)". Based on the calculation result, the information processing device 100 updates P(X3) in the implementation frequency information 20b by P'(X3).

[0043]

number

[0044] In addition, in calculating the above ratios, the information processing device 100 can calculate the ratios by excluding values ​​for which P(X1|X2, X3) is less than a predetermined value, thereby completely excluding promotional measures that have a low probability of being selected and then calculating the implementation frequencies P'(X2) and P'(X3).

[0045] For example, when the information processing device 100 excludes values ​​for which P(X1|X2, X3) is 0.5 or less, it can calculate the implementation frequency P'(X2) as shown in formula (3). Also, when the information processing device 100 excludes values ​​for which P(X1|X2, X3) is 0.5 or less, it can calculate the implementation frequency P'(X3) as shown in formula (4).

[0046]

number

[0047]

number

[0048] For example, looking at the calculation results of equation (4), since the implementation frequency P'(X3) is "1.0 (probability that POP will be implemented is 100%)", the condition where X3 = 0 (POP is not implemented) can be completely excluded.

[0049] Based on the updated execution frequency information 20a and 20b, the information processing device 100 updates the timing information 25. After updating the timing information 25, the information processing device 100 repeatedly executes the above-mentioned process.

[0050] Next, a configuration example of the information processing device 100 that executes the above-mentioned processing will be described. Fig. 4 is a functional block diagram showing the configuration of the information processing device according to the embodiment 1. As shown in Fig. 4, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0051] The communication unit 110 executes data communication between the camera 10, an external device, etc., via a network. The communication unit 110 is a NIC (Network Interface Card), etc. For example, the communication unit 110 receives video data from the camera 10.

[0052] The input unit 120 is an input device that inputs various types of information to the control unit 150 of the information processing device 100 .

[0053] The display unit 130 is a display device that displays information output from the control unit 150.

[0054] The storage unit 140 includes an interest level DB 141, an implementation frequency information DB 142, a conditional probability distribution table DB 143, an imaging DB 144, and a model DB 145. The storage unit 140 is a storage device such as a memory.

[0055] The interest level DB 141 is the interest level DB 141 described in Fig. 3. The interest level DB 141 associates the date and time, the customer attribute, X3, X2, and X1. The explanation regarding the date and time, the customer attribute, X3, X2, and X1 is the same as the explanation above.

[0056] The implementation frequency information DB142 holds implementation frequency information of each promotional measure. FIG. 5 is a diagram showing an example of the data structure of the implementation frequency information DB according to the first embodiment. As shown in FIG. 5, the implementation frequency information DB142 associates promotional measures with implementation frequency information. For example, the implementation frequency information corresponding to the promotional measure "CM video" corresponds to the implementation frequency information 20a shown in FIG. 2. The implementation frequency information corresponding to the promotional measure "POP" corresponds to the implementation frequency information 20b shown in FIG. 2. Although not described in FIG. 5, the implementation frequency information DB142 may further hold implementation frequency information corresponding to other promotional measures.

[0057] The conditional probability distribution table DB143 holds a conditional probability distribution table for each customer attribute. FIG. 6 is a diagram showing an example of a data structure of the conditional probability distribution table DB according to the first embodiment. As shown in FIG. 6, the conditional probability distribution table DB143 associates customer attributes with conditional probability distribution tables. The customer attributes are attributes of customers who are targets for identifying the interest level, and are a set of age and gender. The conditional probability distribution table has X2, X3, and P(X1|X2, X3). The explanation regarding the conditional probability distribution table is the same as the explanation regarding the conditional probability distribution table given in FIG. 3.

[0058] The imaging DB 144 holds video data captured by the camera 10. FIG. 7 is a diagram showing an example of a data structure of the imaging DB according to the first embodiment. As shown in FIG. 7, the imaging DB 144 associates camera identification information with video data. The camera identification information is information that uniquely identifies the camera 10. For example, camera identification information C-10a, C-10b, C-10c, and C-10d correspond to the cameras 10a, 10b, 10c, and 10d, respectively. The video data is video data captured by the corresponding camera 10.

[0059] The model DB 145 has a plurality of machine learning models used by the control unit 150. For example, the model DB 145 has a first machine learning model, a second machine learning model, and a third machine learning model. The first machine learning model, the second machine learning model, and the third machine learning model are trained machine learning models, such as a Neural Network (NN).

[0060] The first machine learning model is a model that outputs skeletal data of a person when an image of a person is input. The skeletal data is data in which two-dimensional or three-dimensional coordinates are set for a plurality of joints defined in a skeletal model of the human body. FIG. 8 is a diagram showing an example of a skeletal model of the human body. For example, as shown in FIG. 8, the skeletal model of the human body is defined by 21 joints ar0 to ar20.

[0061] The relationship between each of the joints ar0 to ar20 shown in Fig. 8 and the joint names is as shown in Fig. 9. Fig. 9 is a diagram showing an example of joint names. For example, the joint name of the joint ar0 is "SPINE_BASE". The joint names of the joints ar0 to ar20 are as shown in Fig. 9, and description thereof will be omitted.

[0062] The second machine learning model is a model that outputs a person's behavior type when time-series skeletal data is input. For example, the person's behavior types include "walking," "stopping," "looking at the shelves," "looking at the product," "reaching out for the product," etc.

[0063] The third machine learning model is a model that outputs the attributes (age, gender) of a person when an image of a person is input. The third machine learning model may output other attributes as the attributes of the person. The other attributes include the age, clothing, etc. of the person.

[0064] Next, the control unit 150 will be described. The control unit 150 has an acquisition unit 151, a skeleton detection unit 152, a motion detection unit 153, a mental state estimation unit 154, an attribute estimation unit 155, an evaluation unit 156, and an update unit 157. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0065] The acquisition unit 151 acquires video data from the camera 10. The acquisition unit 151 stores the acquired video data in the imaging DB 144 in association with the camera identification information of the camera.

[0066] The skeleton detection unit 152 acquires video data from the imaging DB 144, and detects a person area (Bounding Box) from each of the time-series image frames included in the video data. The skeleton detection unit 152 generates time-series skeletal data by inputting images of the time-series person areas detected from the time-series image frames to a first machine learning model. The skeleton detection unit 152 outputs the time-series skeletal data to the motion detection unit 153. The skeleton detection unit 152 sets, in the skeletal data, information on the time (date and time) included in the image frames from which the skeletal data is generated. The skeleton detection unit 152 repeatedly executes the above process.

[0067] The movement detection unit 153 divides the time-series skeletal data into predetermined periods and inputs the time-series skeletal data for the predetermined periods into a second machine learning model to detect the person's behavior type for the predetermined period. The movement detection unit 153 repeatedly executes the above process for each predetermined period, thereby outputting the person's behavior type for each predetermined period to the mind estimation unit 154. The movement detection unit 153 sets information on the time (date and time) set in the skeletal data as the person's behavior type. The movement detection unit 153 repeatedly executes the above process.

[0068] The psychology estimation unit 154 estimates whether the level of interest is high or not based on the person's behavior type. For example, when the person's behavior type is a predetermined behavior type set in advance, the psychology estimation unit 154 estimates that the level of interest is high. On the other hand, when the person's behavior is not a predetermined behavior type set in advance, the psychology estimation unit 154 estimates that the level of interest is low.

[0069] For example, the preset predetermined behavior types are "stopping in front of a product shelf for 5 seconds or more," "looking at a product for 5 seconds or more," "reaching out for a product," etc. Note that the psychology estimation unit 154 may estimate that the behavior type of the person is a predetermined behavior type when the behavior type "stopping" is acquired a predetermined number of times in succession. Similarly, the psychology estimation unit 154 may estimate that the behavior type of the person is a predetermined behavior type when the behavior type "looking at a product" is acquired a predetermined number of times in succession.

[0070] The psychology estimation unit 154 may estimate whether the level of interest is high using an estimation policy different from the above. The psychology estimation unit 154 estimates whether the level of interest is high by focusing on the transition of the behavior type. For example, the psychology estimation unit 154 estimates that the level of interest is high when the customer's behavior type transitions in the order of "stopping in front of a product shelf for 5 seconds or more" → "looking at a product for 5 seconds or more" → "reaching out for a product".

[0071] The psychology estimation unit 154 generates interest level information based on the above estimation result. The interest level information includes a date and time, and information on whether the interest level is high or not. The psychology estimation unit 154 outputs the interest level information to the evaluation unit 156. The psychology estimation unit 154 repeatedly executes the above process. The interest level information is an example of "purchasing psychology information".

[0072] The attribute estimation unit 155 detects a person area (Bounding Box) from each of the time-series image frames included in the video data. The attribute estimation unit 155 estimates the customer attribute of the person by inputting the image of the person area to a third machine learning model. The attribute estimation unit 155 generates attribute information based on the estimation result. The attribute information includes date and time and customer attributes. The attribute estimation unit 155 outputs the attribute information to the evaluation unit 156. The attribute estimation unit 155 repeatedly executes the above process.

[0073] The evaluation unit 156 analyzes changes in interest level information before and after a promotional campaign based on the promotional campaign timing information, and evaluates the promotional campaign based on the analyzed changes. An example of the process of the evaluation unit 156 will be described below.

[0074] The evaluation unit 156 generates the time information 25 based on the implementation frequency information of each promotional measure stored in the implementation frequency information DB 142. The process by which the evaluation unit 156 generates the time information 25 based on the implementation frequency information corresponds to the process described in FIG. 2. The evaluation unit 156 identifies the period during which each promotional measure was implemented based on the time information 25. When the implementation frequency information of each promotional measure stored in the implementation frequency information DB 142 is updated by the update unit 157 described later, the evaluation unit 156 generates the time information 25 again.

[0075] The evaluation unit 156 acquires the interest level information from the mental state estimation unit 154 and acquires the attribute information from the attribute estimation unit 155. The evaluation unit 156 generates a record to be registered in the interest level DB 141 based on the period information 25, the interest level information, and the attribute information. For example, the evaluation unit 156 associates the interest level information with the attribute information using the date and time of the interest level information and the date and time of the attribute information as keys. The evaluation unit 156 specifies the values ​​of X2 and X3 based on the date and time of the interest level information and the period information 25. If information indicating that the interest level is high is set in the interest level information, the evaluation unit 156 sets X1 to "True". If information indicating that the interest level is low is set in the interest level information, the evaluation unit 156 sets X1 to "False".

[0076] The evaluation unit 156 repeatedly executes the above process to repeatedly register records in the interest level DB 141.

[0077] Furthermore, the evaluation unit 156 calculates a conditional probability distribution table for each customer attribute based on the interest level DB 141. The process by which the evaluation unit 156 generates a conditional probability distribution table corresponds to the process described in Fig. 3. The evaluation unit 156 registers (updates) the generated conditional probability distribution table in the conditional probability distribution table DB 143.

[0078] When there is a significant difference in the causality between the presence or absence of a promotional measure and the degree of interest, the evaluation unit 156 may output the generated conditional probability distribution table to the display unit 130 for display. For example, when the difference between the value of P(X1=1|X2=0, X3=0) and the value of P(X1=1|X2=1, X3=1) in the conditional probability distribution table 30 shown in Fig. 3 is equal to or greater than a threshold, the evaluation unit 156 determines that there is a significant difference in the causality between the presence or absence of a promotional measure and the degree of interest. Note that the above-mentioned determination criteria are merely examples, and such determination criteria are set in advance.

[0079] The update unit 157 updates the implementation frequency information of each promotional measure based on the conditional probability distribution table registered in the conditional probability distribution table DB 143. The update unit 157 calculates the implementation frequency of each promotional measure by calculating the ratio of P(X1|X2, X3) when X2 and X3 are each 1 (True). The process by which the update unit 157 calculates the ratio of P(X1|X2, X3) when X2 and X3 are each 1 (True) is the same as that described above.

[0080] The update unit 157 updates the performance frequency information in the performance frequency information DB 142 based on the calculation result.

[0081] Next, an example of a processing procedure of the information processing device 100 according to the first embodiment will be described. Fig. 10 is a flowchart showing the processing procedure of the information processing device according to the first embodiment. As shown in Fig. 10, the evaluation unit 156 of the information processing device 100 generates time information based on the implementation frequency information (step S101). The acquisition unit 151 of the information processing device 100 acquires video data from the camera 10 (step S102).

[0082] The skeleton detection unit 152 and the attribute estimation unit 155 of the information processing device 100 detect a human region from image data included in the video data (step S103). The skeleton detection unit 152 generates time-series skeleton data (step S105). The attribute estimation unit 155 generates attribute information of the person (step S104).

[0083] The action detection unit 153 of the information processing device 100 detects the person's action type based on the time-series skeletal data (step S106). The mind estimation unit 154 generates interest level information based on the person's action type (step S107).

[0084] The evaluation unit 156 of the information processing device 100 registers a record (information on date and time, customer attributes, X3, X2, and X1) in the interest level DB 141 based on the time information, interest level information, and attribute information (step S108). The evaluation unit 156 generates a conditional probability distribution table for each customer attribute based on the interest level DB 141 (step S109).

[0085] The update unit 157 of the information processing device 100 updates the implementation frequency information based on the conditional probability distribution table (step S110). If there is no significant difference between the presence or absence of a promotional measure and causality (step S111, No), the evaluation unit 156 proceeds to step S101.

[0086] On the other hand, if there is a significant difference between the presence or absence of a promotional measure and causality (step S111, Yes), the evaluation unit 156 outputs a conditional probability distribution table to the display unit 130 (step S112). Note that the store manager or the like refers to the conditional probability distribution table output to the display unit 130 and adopts the promotional measure for which a significant difference is recognized.

[0087] Next, the effects of the information processing device 100 according to the first embodiment will be described. The information processing device 100 repeatedly executes a process of identifying a customer's product purchasing behavior by analyzing video data captured by the camera 10 and identifying the level of interest according to the identified behavior. The information processing device 100 also acquires timing information regarding the time when a promotional measure for a product was implemented, and evaluates the promotional measure based on the change in the level of interest before and after the promotional measure. This allows the effectiveness of the promotional measure to be evaluated. Store managers and the like can carry out effective promotional activities by referring to such evaluation results.

[0088] The information processing device 100 generates interest level information based on the identified behavior type, thereby making it possible to generate interest level information based on psychological changes leading up to a purchase.

[0089] The information processing device 100 identifies the behavior of the customer staying in front of the product shelf based on the identified behavior, and generates interest level information based on the time the person stays in front of the product shelf. This makes it possible to generate interest level information according to the stay time.

[0090] The information processing device 100 generates a conditional probability distribution table for each customer attribute based on the interest level DB 141, and evaluates the sales promotion measures based on the conditional probability distribution table, thereby making it possible to evaluate the effectiveness of the sales promotion measures for each customer attribute. EXAMPLES

[0091] Next, a system according to the second embodiment will be described. Fig. 11 is a diagram showing an example of a system according to the second embodiment. As shown in Fig. 11, this system includes cameras 10a to 10d, a questionnaire input device 50, and an information processing device 200. The cameras 10a to 10d, the questionnaire input device 50, and the information processing device 200 are connected to each other via a network.

[0092] The camera 10 captures an image of the inside of the store, and transmits the captured image data to the information processing device 200. Other explanations regarding the camera 10 are the same as those regarding the camera 10 described in the first embodiment.

[0093] The questionnaire input device 50 is a device for a customer to input a questionnaire about a predetermined product. In the present embodiment 2, as an example, the content of the questionnaire is to select whether or not the customer likes the product A. The customer operates the questionnaire input device 50 to select whether or not the customer likes the product A, and presses a completion button or the like set in the questionnaire input device 50.

[0094] For example, a camera is installed in the questionnaire input device 50. When a completion button is pressed by a customer, the questionnaire input device 50 takes an image of the customer, generates questionnaire information including image information of the customer and the input result of the questionnaire, and transmits the questionnaire information to the information processing device 200.

[0095] The information processing device 200 analyzes the video data captured by the camera 10 to identify the behavior of the customer in purchasing the product for each customer, and identifies the reliability according to the identified behavior. The information processing device 200 also acquires questionnaire information from the questionnaire input device 50, and estimates the customer's subjective reliability of the product for each customer based on the questionnaire information. The information processing device 200 identifies the average value of the reliability according to the customer's behavior and the reliability based on the questionnaire information as the reliability of the product for each customer. The information processing device 200 then acquires timing information regarding the time when a promotional measure for the product was implemented, and evaluates the promotional measure based on the change in reliability before and after the promotional measure. This makes it possible to evaluate the effectiveness of the promotional measure.

[0096] Next, a configuration example of the information processing device 200 according to the present embodiment 2 will be described. Fig. 12 is a functional block diagram showing the configuration of the information processing device according to the present embodiment 2. As shown in Fig. 12, the information processing device 200 has a communication unit 210, an input unit 220, a display unit 230, a storage unit 240, and a control unit 250.

[0097] The communication unit 210 executes data communication between the camera 10, the questionnaire input device 50, an external device, etc., via the network. The communication unit 210 is a NIC, etc. For example, the communication unit 210 receives video data from the camera 10.

[0098] The input unit 220 is an input device that inputs various types of information to the control unit 250 of the information processing device 200 .

[0099] The display unit 230 is a display device that displays information output from the control unit 250.

[0100] The storage unit 240 includes a reliability DB 241, an implementation frequency information DB 242, a conditional probability distribution table DB 243, an imaging DB 244, and a model DB 245. The storage unit 240 is a storage device such as a memory.

[0101] The reliability DB241 holds various information related to the reliability of a customer. FIG. 13 is a diagram showing an example of the data structure of the reliability DB. As shown in FIG. 13, the reliability DB241 associates a date and time, a customer ID, X3, X2, and X1. The explanation of the date and time, X3, and X2 is the same as that explained in the first embodiment. The customer ID is information that uniquely identifies a customer. In the reliability DB241, X1 indicates whether the reliability is high or not. For example, if the reliability is equal to or higher than a threshold, X1 is "True". If the reliability is less than the threshold, X1 is "False".

[0102] The implementation frequency information DB 242 holds implementation frequency information of each promotional measure. The data structure of the implementation frequency information DB 242 is the same as the data structure of the implementation frequency information DB 142 described with reference to FIG.

[0103] The conditional probability distribution table DB243 holds a conditional probability distribution table for each customer ID. FIG. 14 is a diagram showing an example of a data structure of the conditional probability distribution table DB according to the second embodiment. As shown in FIG. 14, the conditional probability distribution table DB243 associates a customer ID with a conditional probability distribution table. The customer ID is information for identifying a customer whose reliability is specified. The conditional probability distribution table has X2, X3, and P(X1|X2, X3). The explanation regarding the conditional probability distribution table is the same as the explanation regarding the conditional probability distribution table in FIG. 3 of the first embodiment.

[0104] The imaging DB 244 holds video data captured by the camera 10. The data structure of the imaging DB 244 is similar to the data structure of the imaging DB 144 described with reference to FIG.

[0105] The model DB 245 has a plurality of machine learning models used by the control unit 250. For example, the model DB 245 has a first machine learning model and a second machine learning model. The description of the first machine learning model and the second machine learning model is the same as that described in the first embodiment.

[0106] Next, the control unit 250 will be described. The control unit 250 has an acquisition unit 251, a skeleton detection unit 252, a motion detection unit 253, a mental state estimation unit 254, a person identification unit 255, an evaluation unit 256, and an update unit 257. The control unit 250 is a CPU, a GPU, or the like.

[0107] The acquisition unit 251 acquires video data from the camera 10. The acquisition unit 251 stores the acquired video data in the imaging DB 244 in association with the camera identification information of the camera.

[0108] The skeleton detection unit 252 acquires video data from the imaging DB 244, and generates time-series skeleton data based on the video data. The skeleton detection unit 252 outputs the time-series skeleton data to the motion detection unit 253. Other details regarding the skeleton detection unit 252 are the same as those of the skeleton detection unit 152 in the first embodiment.

[0109] The movement detection unit 253 divides the time-series skeletal data into predetermined periods and inputs the time-series skeletal data for the predetermined periods into a second machine learning model to detect the behavior type of the person for the predetermined period. The movement detection unit 253 repeatedly executes the above process for each predetermined period to output the behavior type of the person for each predetermined period to the evaluation unit 256. Other explanations regarding the movement detection unit 253 are the same as those of the movement detection unit 153 in the first embodiment.

[0110] The person identification unit 255 acquires video data from the imaging DB 244, detects person areas from time-series image frames included in the video data, and assigns the same customer ID to the same customer by tracking or the like. The person identification unit 255 outputs the set customer ID to the mind estimation unit 254. The person identification unit 255 may output the set customer ID to the mind estimation unit 254 by associating the frame number of the image frame, the camera identification information, and the customer ID.

[0111] Furthermore, the person identification unit 255 acquires survey information including an image of a customer, and identifies the customer ID who entered the survey information based on the characteristics of the image in the survey information and the characteristics of the person area to which the customer ID is assigned. The person identification unit 255 sets the identified customer ID in the survey information, and outputs the survey information in which the customer ID is set to the evaluation unit 256. Note that, although the case where the person identification unit 255 identifies the customer ID corresponding to the survey information has been described in the second embodiment, the customer ID corresponding to the survey information may be set using other well-known techniques.

[0112] The psychology estimation unit 254 estimates whether the reliability is high or not based on the behavior type of the person corresponding to the customer ID. For example, the psychology estimation unit 254 counts the cumulative number of times that the person corresponding to the customer ID reaches out for product A during a predetermined period. The psychology estimation unit 254 estimates that the reliability is high when the counted cumulative number is equal to or greater than a threshold. On the other hand, the psychology estimation unit 254 estimates that the reliability is high when the cumulative number is less than the threshold.

[0113] The psychology estimation unit 254 generates reliability information based on the above estimation result. The reliability information includes a date and time, a customer ID, and information on whether the reliability is high or not. The psychology estimation unit 254 outputs the reliability information to the evaluation unit 256. The psychology estimation unit 254 repeatedly executes the above process. The reliability information is an example of "purchasing psychology information".

[0114] The evaluation unit 256 analyzes the change in the reliability information before and after the promotional campaign based on the timing information of the promotional campaign, and evaluates the promotional campaign based on the analyzed change. An example of the process of the evaluation unit 256 will be described below.

[0115] The evaluation unit 256 generates the timing information 25 based on the implementation frequency information of each promotional measure stored in the implementation frequency information DB 242. The process of generating the timing information 25 by the evaluation unit 256 based on the implementation frequency information is similar to the process performed by the evaluation unit 156 described in the first embodiment.

[0116] The evaluation unit 256 acquires reliability information from the psychology estimation unit 254 and acquires questionnaire information from the person identification unit 255. The evaluation unit 256 adjusts reliability for the same customer ID based on the reliability information and the questionnaire information. For example, when the reliability information indicates that the reliability is high and the questionnaire information indicates that the customer likes product A, the evaluation unit 256 sets the reliability corresponding to the corresponding customer ID to "True." On the other hand, when the reliability information indicates that the reliability is low or when the questionnaire information indicates that the customer does not like product A, the evaluation unit 256 sets the reliability corresponding to the corresponding customer ID to "False."

[0117] Note that the evaluation unit 256 may execute the following process when the reliability of the reliability information or the degree to which the customer likes product A in the questionnaire information is set as a numerical value. The evaluation unit 256 calculates the average value of the reliability and the numerical value of the degree to which the customer likes product A, and when the average value is equal to or greater than a threshold, sets the reliability corresponding to the corresponding customer ID to "True." When the average value is less than the threshold, the evaluation unit 256 sets the reliability corresponding to the corresponding customer ID to "False."

[0118] The evaluation unit 256 generates a record to be registered in the reliability DB 241 based on the time information 25, the adjusted reliability, and the reliability information. The evaluation unit 256 specifies the values ​​of X2 and X3 based on the date and time of the reliability information and the time information 25. The evaluation unit 256 sets a value corresponding to the adjusted reliability to X1.

[0119] The evaluation unit 256 repeatedly executes the above process to repeatedly register records in the reliability DB 241.

[0120] Furthermore, the evaluation unit 256 calculates a conditional probability distribution table for each customer ID based on the reliability DB 241. The process by which the evaluation unit 256 generates a conditional probability distribution table corresponds to the process described in Fig. 3. Note that the process in the evaluation unit 256 replaces each customer attribute described in Fig. 3 with each customer ID. The evaluation unit 256 registers (updates) the generated conditional probability distribution table in the conditional probability distribution table DB 243.

[0121] When there is a significant difference in the causality between the presence or absence of a promotional measure and the degree of interest, the evaluation unit 256 may output the generated conditional probability distribution table to the display unit 230 for display. For example, when the difference between the value of P(X1=1|X2=0, X3=0) and the value of P(X1=1|X2=1, X3=1) is equal to or greater than a threshold, the evaluation unit 256 determines that there is a significant difference in the causality between the presence or absence of a promotional measure and the degree of interest. Note that the above-mentioned criteria are merely examples, and such criteria are set in advance.

[0122] The update unit 257 updates the implementation frequency information of each promotional measure based on the conditional probability distribution table registered in the conditional probability distribution table DB 243. The processing of the update unit 257 corresponds to the processing of the update unit 157. Note that the processing in the update unit 257 replaces each customer attribute described in FIG. 3 with each customer ID.

[0123] Next, an example of a processing procedure of the information processing device 200 according to the second embodiment will be described. Fig. 15 is a flowchart showing the processing procedure of the information processing device according to the second embodiment. As shown in Fig. 15, the evaluation unit 256 of the information processing device 200 generates time information based on the implementation frequency information (step S201). The acquisition unit 251 of the information processing device 200 acquires video data from the camera 10 (step S202).

[0124] The skeleton detection unit 152 and the person identification unit 255 of the information processing device 200 detect a person area from image data included in the video data (step S203). The skeleton detection unit 252 generates time-series skeleton data (step S205). The person identification unit 255 sets a customer ID for the person (step S204).

[0125] The motion detection unit 253 of the information processing device 200 detects the type of behavior of the person based on the time-series skeleton data (step S206). The mental state estimation unit 254 generates reliability information based on the type of behavior of the person (step S207). The information processing device 200 acquires questionnaire information from the questionnaire input device 50 and sets a customer ID (step S208).

[0126] The evaluation unit 256 of the information processing device 200 registers a record (information on date and time, customer ID, X3, X2, and X1) in the reliability DB 241 based on the time information, reliability information, and questionnaire information (step S209). The evaluation unit 256 generates a conditional probability distribution table for each customer ID based on the reliability DB 241 (step S210).

[0127] The update unit 257 of the information processing device 200 updates the implementation frequency information based on the conditional probability distribution table (step S211). If there is no significant difference between the presence or absence of a promotional measure and causality (step S212, No), the evaluation unit 256 proceeds to step S201.

[0128] On the other hand, if there is a significant difference between the presence or absence of a promotional measure and causality (step S212, Yes), the evaluation unit 156 outputs a conditional probability distribution table to the display unit 130 (step S213). Note that the store manager or the like refers to the conditional probability distribution table output to the display unit 230 and adopts the promotional measure for which a significant difference is recognized.

[0129] Next, the effect of the information processing device 200 according to the second embodiment will be described. The information processing device 200 analyzes the video data captured by the camera 10 to identify the behavior of the customer purchasing the product for each customer ID, and identifies the reliability according to the identified behavior. The information processing device 200 also acquires questionnaire information from the questionnaire input device 50, and estimates the subjective reliability of the customer for each customer ID based on the questionnaire information. The information processing device 200 identifies, for each customer ID, a value adjusted based on the reliability according to the customer's behavior and the reliability based on the questionnaire information as the reliability of the product. Then, the information processing device 200 acquires time information related to the time when the promotional measures for the product were implemented, and evaluates the promotional measures based on the change in reliability before and after the promotional measures. This makes it possible to evaluate the effectiveness of the promotional measures for each customer. EXAMPLES

[0130] Next, a system according to the third embodiment will be described. The system according to the third embodiment is similar to the system described in FIG. 1, and therefore the description will be omitted. The information processing device according to the third embodiment will be referred to as "information processing device 300". The information processing device 300 is connected to cameras 10a to 10d via a network. In the following description, the cameras 10a to 10d will be collectively referred to as camera 10.

[0131] The camera 10 captures an image of the inside of the store, and transmits the captured image data to the information processing device 300. The description of the camera 10 is the same as that of the camera 10 above.

[0132] In the above-mentioned first embodiment, POP and commercial videos were used as promotional measures, but in the present embodiment, commercial videos of multiple products are used as promotional measures. The information processing device 300 repeatedly executes a process of identifying the behavior of customers when they purchase products by analyzing the video data captured by the camera 10, and identifying the level of interest according to the identified behavior. The information processing device 300 also acquires timing information regarding the time when the promotional measures for each commercial video were implemented, and evaluates the promotional measures based on the change in the level of interest before and after the promotional measures. This makes it possible to evaluate the effectiveness of the promotional measures by each commercial video, and compare which commercial video for which product is the most effective.

[0133] Next, a configuration example of an information processing device 300 according to the present embodiment 3 will be described. Fig. 16 is a functional block diagram showing a configuration of the information processing device according to the present embodiment 3. As shown in Fig. 16, the information processing device 300 has a communication unit 310, an input unit 320, a display unit 330, a storage unit 340, and a control unit 350.

[0134] The communication unit 310 executes data communication between the camera 10, an external device, etc., via a network. The communication unit 310 is a NIC, etc. For example, the communication unit 310 receives video data from the camera 10.

[0135] The input unit 320 is an input device that inputs various types of information to the control unit 350 of the information processing device 300 .

[0136] The display unit 330 is a display device that displays information output from the control unit 350.

[0137] The storage unit 340 has an interest level DB 341, an implementation frequency information DB 342, a conditional probability distribution table DB 343, an imaging DB 344, and a model DB 345. The storage unit 340 is a storage device such as a memory.

[0138] The interest level DB 341 holds information on the interest level of a customer. Fig. 17 is a diagram showing an example of a data structure of the interest level DB according to the third embodiment. As shown in Fig. 17, the interest level DB 341 has date and time, customer attributes, X2, and X1. In Fig. 17, the explanations regarding the date and time, customer attributes, and X1 are the same as those described in Fig. 3.

[0139] X2 is set to a value between 0 and 3. X2=0 indicates product A. On the date and time when a commercial video for product A is aired, the cell of the record corresponding to X2=0 is set to "True." On the date and time when a commercial video for product A is not aired, the cell of the record corresponding to X2=0 is set to "False." X2=1 indicates product B. On the date and time when a commercial video for product B is aired, the cell of the record corresponding to X2=1 is set to "True." On the date and time when a commercial video for product B is not aired, the cell of the record corresponding to X2=1 is set to "False."

[0140] X2=2 indicates product C. On the date and time when a commercial video for product C is aired, the cell of the record corresponding to X2=2 is set to "True." On the date and time when a commercial video for product C is not aired, the cell of the record corresponding to X2=2 is set to "False." X2=3 indicates product D. On the date and time when a commercial video for product D is aired, the cell of the record corresponding to X2=3 is set to "True." On the date and time when a commercial video for product D is not aired, the cell of the record corresponding to X2=3 is set to "False."

[0141] 17, the first record indicates that at the date and time "2 / 3 10:04", a CM video for sales promotion measure X2=0 (product A) is being implemented, and CM videos for sales promotion measures X2=1-3 (products B, C, D) are not being implemented. Also, at the date and time "2 / 3 10:04", it is indicated that the customer attribute "women in their 20s-30s" is interested in the product.

[0142] The implementation frequency information DB342 holds implementation frequency information of each promotional measure. FIG. 18 is a diagram showing an example of the data structure of the implementation frequency information DB according to the third embodiment. As shown in FIG. 18, the implementation frequency information DB342 associates X2 with P(X2) and a time period. The explanation regarding X2 is the same as that in FIG. 17. P(X2) indicates the probability that each promotional measure will be implemented. P(X1=0) is the probability that a commercial video for product A will be implemented. P(X1=1) is the probability that a commercial video for product B will be implemented. P(X1=2) is the probability that a commercial video for product C will be implemented. P(X1=3) is the probability that a commercial video for product D will be implemented. The time period indicates the time period during which the sales measure is implemented (recommended).

[0143] The conditional probability distribution table DB343 holds a conditional probability distribution table for each customer attribute. FIG. 19 is a diagram showing an example of a data structure of the conditional probability distribution table DB according to the third embodiment. As shown in FIG. 19, the conditional probability distribution table DB343 associates customer attributes with conditional probability distribution tables. The customer attributes are attributes of customers whose interest levels are identified, and are a combination of age and gender. The conditional probability distribution table has X2 and P(X1|X2). An example of the conditional probability distribution table is described below with reference to FIG. 20.

[0144] FIG. 20 is a diagram showing an example of a conditional probability distribution table according to the third embodiment. The conditional probability distribution table 35 is a conditional probability distribution table corresponding to a certain customer attribute (for example, women in their 20s and 30s). The conditional probability distribution table 35 has X2 and P(X1|X2). The explanation regarding X1 and X2 is the same as that above. P(X1|X2) is the probability of event X1 under the condition that event X2 occurs.

[0145] The record in the first row of the conditional probability distribution table 35 indicates that under the condition that X2=0 (product A) is "True (implemented)", the probability that the customer's interest level will be less than the threshold is "0.3 (30%, hereafter the same)". Also, it indicates that under the condition that X2=0 (product A) is "True (implemented)", the probability that the customer's interest level will be equal to or greater than the threshold is "0.7".

[0146] The record in the second row of the conditional probability distribution table 35 indicates that the probability that the customer's interest level will be less than the threshold value is "0.7" when X2=1 (product B) is "True (implemented)". It also indicates that the probability that the customer's interest level will be equal to or greater than the threshold value is "0.3" when X2=1 (product B) is "True (implemented)".

[0147] The record in the third row of the conditional probability distribution table 35 indicates that the probability that the customer's interest level will be less than the threshold value is "0.6" when X2=2 (product C) is "True (implemented)". It also indicates that the probability that the customer's interest level will be equal to or greater than the threshold value is "0.4" when X2=2 (product C) is "True (implemented)".

[0148] The record in the fourth row of the conditional probability distribution table 35 indicates that the probability that the customer's interest level will be less than the threshold value is "0.4" when X2=3 (product D) is "True (implemented)". It also indicates that the probability that the customer's interest level will be equal to or greater than the threshold value is "0.6" when X2=3 (product D) is "True (implemented)".

[0149] Returning to the description of Fig. 16, the imaging DB 344 holds video data captured by the camera 10. The data structure of the imaging DB 344 is similar to the data structure of the imaging DB 144 described with reference to Fig. 7.

[0150] The model DB 345 has a plurality of machine learning models used by the control unit 350. For example, the model DB 345 has a first machine learning model, a second machine learning model, and a third machine learning model. The explanation of the first machine learning model, the second machine learning model, and the third machine learning model is similar to the explanation of the first machine learning model, the second machine learning model, and the third machine learning model described in the first embodiment.

[0151] Next, the control unit 350 will be described. The control unit 350 has an acquisition unit 351, a skeleton detection unit 352, a motion detection unit 353, a mental state estimation unit 354, an attribute estimation unit 355, an evaluation unit 356, and an update unit 357. The control unit 350 is a CPU, a GPU, or the like.

[0152] The acquisition unit 351 acquires video data from the camera 10. The acquisition unit 351 stores the acquired video data in the imaging DB 344 in association with the camera identification information of the camera.

[0153] The skeleton detection unit 352 acquires video data from the imaging DB 344, and generates time-series skeleton data based on the video data. The skeleton detection unit 352 outputs the time-series skeleton data to the motion detection unit 353. Other details regarding the skeleton detection unit 352 are the same as those of the skeleton detection unit 152 in the first embodiment.

[0154] The movement detection unit 353 divides the time-series skeletal data into predetermined periods and inputs the time-series skeletal data for the predetermined periods to a second machine learning model to detect the behavior type of the person for the predetermined period. The movement detection unit 353 repeatedly executes the above process for each predetermined period to output the behavior type of the person for each predetermined period to the evaluation unit 356. Other explanations regarding the movement detection unit 353 are the same as those of the movement detection unit 153 in the first embodiment.

[0155] The psychology estimation unit 354 estimates whether the level of interest is high or not based on the person's behavior type. For example, if the person's behavior type is a predetermined behavior type set in advance, the psychology estimation unit 354 estimates that the level of interest is high. On the other hand, if the person's behavior is not a predetermined behavior type set in advance, the psychology estimation unit 354 estimates that the level of interest is low.

[0156] The psychology estimation unit 354 generates interest level information based on the estimation result. The interest level information includes a date and time, and information on whether the interest level is high or not. The psychology estimation unit 354 outputs the interest level information to the evaluation unit 356. The psychology estimation unit 354 repeatedly executes the above process. The interest level information is an example of "purchase psychology information". Other explanations regarding the psychology estimation unit 354 are the same as those regarding the psychology estimation unit 154 described in the first embodiment.

[0157] The attribute estimation unit 355 detects a person area from each of the time-series image frames included in the video data. The attribute estimation unit 355 estimates the customer attribute of the person by inputting the image of the person area to a third machine learning model. The attribute estimation unit 355 generates attribute information based on the estimation result. The attribute information includes date and time and the customer attribute. The attribute estimation unit 355 outputs the attribute information to the evaluation unit 356. The attribute estimation unit 355 repeatedly executes the above process. Other explanations regarding the attribute estimation unit 355 are the same as those regarding the attribute estimation unit 155 described in the first embodiment.

[0158] The evaluation unit 356 analyzes the change in interest level information before and after the promotional campaign based on the promotional campaign timing information of each CM video, and evaluates the promotional campaign based on the analyzed change. An example of the process of the evaluation unit 356 will be described below.

[0159] The evaluation unit 356 uses the time period of the implementation frequency information of each promotional measure stored in the implementation frequency information DB 342 as the time information. The evaluation unit 356 may generate the time information based on P(X2). The evaluation unit 356 identifies the period during which each promotional measure was implemented based on the time information 25. The evaluation unit 356 may generate the time information again when the implementation frequency information of each promotional measure stored in the implementation frequency information DB 342 is updated by the update unit 357 described later.

[0160] The evaluation unit 356 acquires the interest level information from the mental state estimation unit 354 and acquires the attribute information from the attribute estimation unit 355. The evaluation unit 356 generates a record to be registered in the interest level DB 341 based on the time information, the interest level information, and the attribute information. For example, the evaluation unit 356 associates the interest level information with the attribute information using the date and time of the interest level information and the date and time of the attribute information as keys. The evaluation unit 356 specifies a value of X2=0 to 3 (True or False) based on the date and time information of the interest level information. When information indicating that the interest level is high is set in the interest level information, the evaluation unit 356 sets X1 to "True". On the other hand, when information indicating that the interest level is low is set in the interest level information, the evaluation unit 356 sets X1 to "False".

[0161] The evaluation unit 356 repeatedly executes the above process to repeatedly register records in the interest level DB 341.

[0162] Furthermore, the evaluation unit 356 calculates a conditional probability distribution table for each customer attribute based on the interest level DB 341. For example, the conditional probability table for the customer attribute "female in her 20s to 30s" is the conditional probability distribution table 35 shown in Fig. 20. When generating the conditional probability distribution table 35, the evaluation unit 356 uses the records of the customer attribute "female in her 20s to 30s" from among the records of the interest level DB 341.

[0163] The evaluation unit 356 extracts records where X2=0 (product A) is "True" from among the records with the customer attribute "female in 20s-30s", and sets the ratio of records where X1=True among the extracted records to the cell of P(X1=1|X2) corresponding to X2=0 (0.7 in FIG. 20). The evaluation unit 356 extracts records where X2=0 (product A) is "True" from among the records with the customer attribute "female in 20s-30s", and sets the ratio of records where X1=False among the extracted records to the cell of P(X1=0|X2) corresponding to X2=0 (0.3 in FIG. 20).

[0164] The evaluation unit 356 extracts records where X2=1 (product B) is "True" from among the records with the customer attribute "female in 20s-30s", and sets the ratio of records where X1=True among the extracted records to the cell of P(X1=1|X2) corresponding to X2=1 (0.3 in FIG. 20). The evaluation unit 356 extracts records where X2=1 (product B) is "True" from among the records with the customer attribute "female in 20s-30s", and sets the ratio of records where X1=False among the extracted records to the cell of P(X1=0|X2) corresponding to X2=1 (0.7 in FIG. 20).

[0165] The evaluation unit 356 extracts records where X2=2 (product C) is "True" from among the records with the customer attribute "female in 20s-30s", and sets the ratio of records where X1=True among the extracted records to the cell of P(X1=1|X2) corresponding to X2=2 (0.4 in FIG. 20). The evaluation unit 356 extracts records where X2=2 (product C) is "True" from among the records with the customer attribute "female in 20s-30s", and sets the ratio of records where X1=False among the extracted records to the cell of P(X1=0|X2) corresponding to X2=2 (0.6 in FIG. 20).

[0166] The evaluation unit 356 extracts records where X2=3 (product D) is "True" from among the records with the customer attribute "female in 20s-30s", and sets the ratio of records where X1=True among the extracted records to the cell of P(X1=1|X2) corresponding to X2=3 (0.6 in FIG. 20). The evaluation unit 356 extracts records where X2=3 (product D) is "True" from among the records with the customer attribute "female in 20s-30s", and sets the ratio of records where X1=False among the extracted records to the cell of P(X1=0|X2) corresponding to X2=3 (0.4 in FIG. 20).

[0167] 20 shows the conditional probability distribution table 35 for the customer attribute "female in her 20s to 30s," but the evaluation unit 356 generates conditional probability distribution tables for other customer attributes in the same manner. The evaluation unit 356 registers (updates) the conditional probability distribution table for each customer attribute in the conditional probability distribution table DB 343.

[0168] The conditional probability distribution table 35 generated by the evaluation unit 356 is the evaluation result of the effectiveness of the sales promotion measures. For example, when comparing the values ​​of P(X1=1|X2) for X2=0 to 3 in the conditional probability distribution table 35, the value of P(X1=1|X2) for X2=0 is the maximum (0.7). Therefore, it can be seen that implementing X2=0 (product A) is the most effective.

[0169] If there is a significant difference in the causality between the presence or absence of a promotional measure and the degree of interest, the evaluation unit 356 may output the generated conditional probability distribution table to the display unit 330 for display. For example, the evaluation unit 356 compares the values ​​of P(X1=1|X2) in the conditional probability distribution table 35 shown in Fig. 20, and if the difference between the maximum value of P(X1=1|X2) and the minimum value of P(X1=1|X2) is equal to or greater than a threshold, it determines that there is a significant difference in the causality between the presence or absence of a promotional measure and the degree of interest. Note that the above-mentioned criteria are merely examples, and such criteria are set in advance.

[0170] After the evaluation unit 356 generates the conditional probability distribution table, the update unit 357 updates P(X2) and the time period in the performance frequency information DB.

[0171] For example, the update unit 357 calculates "0.35" for updated P'(X2=0) based on equation (5). The update unit 357 calculates "0.15" for updated P'(X2=1) based on equation (6). The update unit 357 calculates "0.20" for updated P'(X2=2) based on equation (7). The update unit 357 calculates "0.30" for updated P'(X2=3) based on equation (8).

[0172]

number

number

number

number

[0173] 21 is a diagram for explaining the process of the update unit according to the third embodiment. The execution frequency information DB 342 is the execution frequency information DB before the update. The execution frequency information DB 342a is the execution frequency information DB updated by the update unit 357.

[0174] Based on the above calculation results, the update unit 357 sets 0.35, 0.15, 0.20, and 0.30 in each cell of P'(X2) in the implementation frequency information DB 342a. The update unit 357 also sets a time period according to P'(X2). For example, the time period corresponding to P'(X2)=0.35 is set to "8:00-12:12". The time period corresponding to P'(X2)=0.15 is set to "12:12-14:00". The time period corresponding to P'(X2)=0.20 is set to "14:00-16:24". The time period corresponding to P'(X2)=0.30 is set to "16:24-20:00".

[0175] Next, an example of a processing procedure of the information processing device 300 according to the third embodiment will be described. Fig. 22 is a flowchart showing the processing procedure of the information processing device according to the third embodiment. As shown in Fig. 22, the evaluation unit 156 of the information processing device 300 generates time information based on the implementation frequency information DB 342 (step S301). The acquisition unit 351 of the information processing device 300 acquires video data from the camera 10 (step S302).

[0176] The skeleton detection unit 352 and the attribute estimation unit 355 of the information processing device 300 detect a human region from image data included in the video data (step S303). The skeleton detection unit 352 generates time-series skeleton data (step S305). The attribute estimation unit 355 generates attribute information of the person (step S304).

[0177] The action detection unit 353 of the information processing device 300 detects the type of behavior of the person based on the time-series skeleton data (step S306). The mind estimation unit 354 generates interest level information based on the type of behavior of the person (step S307).

[0178] The evaluation unit 356 of the information processing device 300 registers a record (information on date and time, customer attributes, X3, X2, and X1) in the interest level DB 341 based on the time information, interest level information, and attribute information (step S308). The evaluation unit 356 generates a conditional probability distribution table for each customer attribute based on the interest level DB 341 (step S309).

[0179] The update unit 357 of the information processing device 300 updates the implementation frequency information DB 342 based on the conditional probability distribution table (step S310). If there is no significant difference between the presence or absence of a promotional measure and causality (step S311, No), the evaluation unit 356 proceeds to step S301.

[0180] On the other hand, if there is a significant difference between the presence or absence of a promotional measure and causality (step S311, Yes), the evaluation unit 356 outputs a conditional probability distribution table to the display unit 330 (step S312). Note that the store manager or the like refers to the conditional probability distribution table output to the display unit 330 and adopts the promotional measure for which a significant difference is recognized.

[0181] Next, the effects of the information processing device 300 according to the third embodiment will be described. The information processing device 300 repeatedly executes a process of identifying the behavior of a customer in purchasing a product by analyzing the video data captured by the camera 10 and identifying the level of interest according to the identified behavior. The information processing device 300 also acquires time information regarding the time when a promotional measure for each CM video was implemented, and evaluates the promotional measure based on the change in the level of interest before and after the promotional measure. This makes it possible to evaluate the effectiveness of the promotional measure for each CM video.

[0182] Next, an example of a hardware configuration of a computer that realizes the same functions as the information processing device 100 (200, 300) described above will be described. Fig. 21 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0183] 21, the computer 400 includes a CPU 401 that executes various arithmetic processes, an input device 402 that accepts data input from a user, and a display 403. The computer 400 also includes a communication device 404 that transmits and receives data to and from external devices, etc., via a wired or wireless network, and an interface device 405. The computer 400 also includes a RAM 406 that temporarily stores various information, and a hard disk device 407. The devices 401 to 407 are connected to a bus 408.

[0184] The hard disk drive 407 has an acquisition program 407a, a skeleton detection program 407b, a motion detection program 407c, a mental state estimation program 407d, a specification program 407e, an evaluation program 407f, and an update program 407g. The CPU 401 reads out each of the programs 407a to 407f and develops them in the RAM 406.

[0185] The acquisition program 407a functions as an acquisition process 406a. The skeleton detection program 407b functions as a skeleton detection process 406b. The movement detection program 407c functions as a movement detection process 406c. The mental state estimation program 407d functions as a mental state estimation process 406d. The identification program 407e functions as an identification process 406e. The evaluation program 407f functions as an evaluation process 406f. The update program 407g functions as an update process 406g.

[0186] The processing of the acquisition process 406a corresponds to the processing of the acquisition unit 151, 251, 351. The processing of the skeleton detection process 406b corresponds to the processing of the skeleton detection unit 152, 252, 352. The processing of the movement detection process 406c corresponds to the processing of the movement detection unit 153, 253, 353. The processing of the mental state estimation process 406d corresponds to the processing of the mental state estimation unit 154, 254, 354. The identification process 407e corresponds to the processing of the attribute estimation unit 155, 355 or the person identification unit 255. The processing of the evaluation process 406e corresponds to the processing of the evaluation unit 156, 256, 356. The processing of the update process 406e corresponds to the processing of the update unit 157, 257, 357.

[0187] Each of the programs 407a to 407g does not necessarily have to be stored in the hard disk device 307 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 400. Then, the computer 400 may read and execute each of the programs 407a to 407g.

[0188] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0189] (Appendix 1) Obtain footage of an area including product shelves on which products are placed, Identifying a behavior of a person purchasing the product by analyzing the acquired video; acquiring purchasing psychology information based on the behavior of purchasing the product, the purchasing psychology information indicating the purchasing psychology of the person; Acquire timing information regarding the time when a promotional measure for the product was implemented; Analyzing changes in the acquired purchasing psychology information before and after the sales promotion measures based on the time information; Evaluating the promotional measures based on the analyzed changes. An evaluation program that causes a computer to execute a process.

[0190] (Appendix 2) The evaluation program according to appendix 1, further comprising causing a computer to execute a process of generating the purchasing psychology information based on the identified behavior.

[0191] (Appendix 3) The evaluation program described in Appendix 2, characterized in that the purchasing psychology information is information indicating whether or not the person is interested in the product, and the generating process identifies the behavior of the person staying in front of the product shelf based on the identified behavior, and generates the purchasing psychology information based on the time the person stays in front of the product shelf.

[0192] (Appendix 4) The evaluation program described in Appendix 2, characterized in that the purchasing psychology information is information indicating whether or not the person has trust in the product, and the generating process identifies the behavior of the person reaching for the product on the shelf based on the identified behavior, and generates the purchasing psychology information based on the cumulative number of times the person reaches for the product.

[0193] (Appendix 5) The evaluation program described in Appendix 1, characterized in that the evaluation process generates a conditional probability distribution table indicating the probability of the presence or absence of interest, conditional on whether or not the promotional measure is implemented, based on the purchasing psychology information obtained during the period when the promotional measure is implemented and the selling psychology information obtained during the period when the promotional measure is not implemented.

[0194] (Appendix 6) The evaluation program according to appendix 2, further comprising causing a computer to execute a process of correcting the purchasing psychology information based on questionnaire information regarding the product input by the person.

[0195] (Supplementary Note 7) The evaluation program according to Supplementary Note 2, characterized in that the promotional measures are promotional measures for a plurality of products, and the evaluation process compares the effectiveness of the promotional measures for each product.

[0196] (Appendix 8) Obtain a video of the area including the product shelves on which the products are placed, Identifying a behavior of a person purchasing the product by analyzing the acquired video; acquiring purchasing psychology information based on the behavior of purchasing the product, the purchasing psychology information indicating the purchasing psychology of the person; Acquire timing information regarding the time when a promotional measure for the product was implemented; Analyzing changes in the acquired purchasing psychology information before and after the sales promotion measures based on the time information; Evaluating the promotional measures based on the analyzed changes. An evaluation method characterized in that the processing is executed by a computer.

[0197] (Appendix 9) The evaluation method according to appendix 8, further comprising causing a computer to execute a process of generating the purchasing psychology information based on the identified behavior.

[0198] (Appendix 10) The evaluation method described in Appendix 9, characterized in that the purchasing psychology information is information indicating whether or not the person is interested in the product, and the generating process identifies the behavior of the person staying in front of the product shelf based on the identified behavior, and generates the purchasing psychology information based on the time the person stays in front of the product shelf.

[0199] (Appendix 11) The purchasing psychology information is information indicating whether or not the person has trust in the product, and the generating process is characterized in that it identifies the behavior of the person reaching for the product on the shelf based on the identified behavior, and generates the purchasing psychology information based on the cumulative number of times the person reaches for the product, in the evaluation method described in Appendix 9.

[0200] (Appendix 12) The evaluation method described in Appendix 8, characterized in that the evaluation process generates a conditional probability distribution table indicating the probability of the presence or absence of interest, conditional on the presence or absence of the promotional measure, based on the purchasing psychology information obtained during the period when the promotional measure was implemented and the selling psychology information obtained during the period when the promotional measure was not implemented.

[0201] (Appendix 13) The evaluation method according to appendix 9, further comprising causing a computer to execute a process of correcting the purchasing psychology information based on questionnaire information regarding the product entered by the person.

[0202] (Appendix 14) The evaluation method according to appendix 9, wherein the promotional measures are promotional measures for a plurality of products, and the evaluation process compares the effectiveness of the promotional measures for each product.

[0203] (Appendix 15) Obtain a video of the area including the product shelves on which the products are placed, Analyzing the acquired video to identify a behavior of a person purchasing the product; acquiring purchasing psychology information based on the behavior of purchasing the product, the purchasing psychology information indicating the purchasing psychology of the person; Acquire timing information regarding the time when a promotional measure for the product was implemented; Analyzing changes in the acquired purchasing psychology information before and after the sales promotion measures based on the time information; Evaluating the promotional measures based on the analyzed changes. An information processing device having a control unit that executes processing.

[0204] (Supplementary Note 16) The information processing device according to Supplementary Note 15, wherein the control unit further executes a process of generating the purchasing psychology information based on the identified behavior.

[0205] (Appendix 17) The information processing device described in Appendix 16, characterized in that the purchasing psychology information is information indicating whether or not the person is interested in the product, and the generating process identifies the behavior of the person staying in front of the product shelf based on the identified behavior, and generates the purchasing psychology information based on the time the person stays in front of the product shelf.

[0206] (Appendix 18) The purchasing psychology information is information indicating whether or not the person has trust in the product, and the generating process identifies the behavior of the person reaching for the product on the shelf based on the identified behavior, and generates the purchasing psychology information based on the cumulative number of times the person reaches for the product, according to the information processing device described in Appendix 16.

[0207] (Appendix 19) The information processing device described in Appendix 15, characterized in that the evaluation process generates a conditional probability distribution table indicating the probability of the presence or absence of interest, conditional on the presence or absence of the promotional measure, based on the purchasing psychology information obtained during the period when the promotional measure was implemented and the selling psychology information obtained during the period when the promotional measure was not implemented.

[0208] (Supplementary Note 20) The information processing device according to Supplementary Note 17, characterized in that the control unit further executes a process of correcting the purchasing psychology information based on questionnaire information regarding the product input by the person.

[0209] (Supplementary Note 21) The information processing device according to Supplementary Note 15, characterized in that the promotional measures are promotional measures for a plurality of products, and the evaluating process compares the effectiveness of the promotional measures for each product. [Explanation of symbols]

[0210] 100, 200, 300 Information processing equipment 110,210,310 Communications Department 120,220,320 Input section 130,230,330 Display section 140,240,340 storage section 141,341 Interest DB 142,242,342 Implementation frequency information DB 143,243,343 Conditional Probability Distribution Table DB 144,244,344 Imaging DB 150,250,350 Control section 151,251,351 Acquisition Department 152,252,352 Skeleton detection section 153,253,353 Motion detector 154,254,354 Psychological Estimation Department 155,355 Attribute estimation part 156,256,356 Evaluation section 157,257,357 Update Department 241 Trustworthiness DB 255 Person Identification Department

Claims

1. Obtaining video of an area including product shelves on which products are placed, Identifying a behavior of a person purchasing the product by analyzing the acquired video; acquiring purchasing psychology information based on the behavior of purchasing the product, the purchasing psychology information indicating the purchasing psychology of the person; Acquire timing information regarding the time when a promotional measure for the product was implemented; Analyzing changes in the acquired purchasing psychology information before and after the sales promotion measures based on the time information; Evaluating the promotional measures based on the analyzed changes. An evaluation program that causes a computer to execute a process.

2. 2. The evaluation program according to claim 1, further causing a computer to execute a process of generating the purchasing psychology information based on the identified behavior.

3. The evaluation program described in claim 2, characterized in that the purchasing psychology information is information indicating whether or not the person is interested in the product, and the generating process identifies the behavior of the person staying in front of the product shelf based on the identified behavior, and generates the purchasing psychology information based on the time the person stays in front of the product shelf.

4. The evaluation program described in claim 2, characterized in that the purchasing psychology information is information indicating whether or not the person has trust in the product, and the generating process identifies the behavior of the person reaching for a product on the shelf based on the identified behavior, and generates the purchasing psychology information based on the cumulative number of times the person reaches for a product.

5. The evaluation program described in claim 1, characterized in that the evaluation process generates a conditional probability distribution table indicating the probability of the presence or absence of interest, conditional on whether or not the promotional measure is implemented, based on the purchasing psychology information obtained during the period when the promotional measure was implemented and the sales psychology information obtained during the period when the promotional measure was not implemented.

6. 3. The evaluation program according to claim 2, further causing the computer to execute a process of correcting the purchasing psychology information based on questionnaire information regarding the product input by the person.

7. 3. The evaluation program according to claim 2, wherein the promotional measures are promotional measures for a plurality of products, and the evaluating process compares effectiveness of the promotional measures for each product.

8. Obtaining video of an area including product shelves on which products are placed, Identifying a behavior of a person purchasing the product by analyzing the acquired video; acquiring purchasing psychology information based on the behavior of purchasing the product, the purchasing psychology information indicating the purchasing psychology of the person; Acquire timing information regarding the time when a promotional measure for the product was implemented; Analyzing changes in the acquired purchasing psychology information before and after the sales promotion measures based on the time information; Evaluating the promotional measures based on the analyzed changes. An evaluation method characterized in that the processing is executed by a computer.

9. Obtaining video of an area including product shelves on which products are placed, Identifying a behavior of a person purchasing the product by analyzing the acquired video; acquiring purchasing psychology information based on the behavior of purchasing the product, the purchasing psychology information indicating the purchasing psychology of the person; Acquire timing information regarding the time when a promotional measure for the product was implemented; Analyzing changes in the acquired purchasing psychology information before and after the sales promotion measures based on the time information; Evaluating the promotional measures based on the analyzed changes. An information processing device having a control unit that executes processing.

Images