Information processing device, information processing method, and information processing program

Abstract

An information processing apparatus according to one embodiment of the present disclosure comprises: an acquisition unit that acquires behavior information of a user in a predetermined space; and a generation unit that uses a model that learns a correlation between the behavior information and an advertising effect of the content presented in the space, and generates optimized content, that is, content newly generated for the user, from the behavior information of the user to be inferred.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus, information processing method, and information processing procedure. Background Technology

[0002] Use technologies to analyze customer behavior in stores and other settings to improve sales and service quality.

[0003] For example, systems are known to analyze customer movement patterns, predict the customer's next location, and recommend products related to that location (e.g., Patent Document 1). Alternatively, techniques are known to switch the content displayed on a sign (electronic advertisement) after considering the customer's behavior before reaching the sign (e.g., Patent Document 2).

[0004] Reference List

[0005] Patent documents

[0006] Patent Document 1: JP 2014-232362 A

[0007] Patent Document 2: JP 2021-176061 A Summary of the Invention

[0008] Technical issues

[0009] However, the aforementioned conventional technologies only predict the customer's next action or display advertisements based on that action.

[0010] In other words, existing technologies struggle to obtain more in-depth information about measures to encourage the purchase of promotional items, such as customers' impressions of a store's goods or services. For instance, existing technologies make it difficult to accurately grasp customer intentions and precisely display what customers expect.

[0011] Therefore, this disclosure provides an information processing apparatus, information processing method, and information processing program capable of generating content by accurately estimating the user's behavior and psychological state.

[0012] Solution to the problem

[0013] To address the aforementioned problems, an information processing apparatus according to one aspect of this disclosure includes: an acquisition unit for acquiring customer behavior information in a predetermined space; and a generation unit for generating optimized content based on the user's behavior information by using a model that has learned the correlation between the behavior information and the advertising effectiveness of the content presented in the space, wherein the optimized content is newly generated content for the user to be estimated. Attached Figure Description

[0014] Figure 1 This is a diagram (1) showing an outline of information processing according to an implementation method.

[0015] Figure 2 This is a diagram (2) showing an outline of the information processing according to the implementation method.

[0016] Figure 3 This is a diagram (3) showing an outline of information processing according to the implementation method.

[0017] Figure 4 This is a diagram illustrating an example configuration of an information processing system and information processing apparatus according to an embodiment.

[0018] Figure 5 This is a diagram illustrating a summary of the processes performed by the state estimation system.

[0019] Figure 6 This is a diagram illustrating an example of learning data used by a state estimation system.

[0020] Figure 7 This is a flowchart (1) illustrating the learning process of the state estimation system.

[0021] Figure 8 This is a flowchart (2) showing the learning process of the state estimation system.

[0022] Figure 9 This is a flowchart (3) showing the learning process of the state estimation system.

[0023] Figure 10 This is a flowchart (1) showing the estimation process of the state estimation system.

[0024] Figure 11 This is a flowchart (2) showing the estimation process of the state estimation system.

[0025] Figure 12 This is a flowchart (3) showing the estimation process of the state estimation system.

[0026] Figure 13 This is a flowchart (4) showing the estimation process of the state estimation system.

[0027] Figure 14 This is a diagram (1) illustrating an example of the processing performed by the state estimation system using the estimation results.

[0028] Figure 15 This is a diagram (2) illustrating an example of the processing performed by the state estimation system using the estimation results.

[0029] Figure 16 This is a diagram illustrating an overview of the learning process in a state estimation system.

[0030] Figure 17This is a diagram showing an overview of the behavior detection by the behavior detection unit.

[0031] Figure 18 This is a diagram illustrating an example of the learning data used by the purchase probability estimation system.

[0032] Figure 19 This is a diagram (1) showing an example of behavior detected by a purchase probability estimation system.

[0033] Figure 20 This is a diagram (2) showing an example of behavior detected by the purchase probability estimation system.

[0034] Figure 21 This is a diagram (1) illustrating an example of behavioral information used by a purchase probability estimation system.

[0035] Figure 22 This is a flowchart (1) illustrating the learning process of the purchase probability estimation system.

[0036] Figure 23 This is a diagram (2) showing an example of behavioral information used by a purchase probability estimation system.

[0037] Figure 24 This is a flowchart (2) illustrating the learning process of the purchase probability estimation system.

[0038] Figure 25 This is a diagram (3) showing an example of behavioral information used by a purchase probability estimation system.

[0039] Figure 26 This is a flowchart (3) illustrating the learning process of the purchase probability estimation system.

[0040] Figure 27 This is a flowchart (1) illustrating the estimation process of the purchase probability estimation system.

[0041] Figure 28 This is a flowchart (2) showing the estimation process of the purchase probability estimation system.

[0042] Figure 29 This is a flowchart (3) showing the estimation process of the purchase probability estimation system.

[0043] Figure 30 This is a diagram illustrating an overview of the learning processes performed by the content generation system.

[0044] Figure 31 This is a diagram illustrating an overview of the generation process performed by the content generation system.

[0045] Figure 32 This is a flowchart illustrating the learning process of the content generation system.

[0046] Figure 33 This is a diagram illustrating an example of the model generation process in a content generation system.

[0047] Figure 34 This is a flowchart illustrating the content generation process of the content generation system.

[0048] Figure 35 This is a flowchart illustrating the relearning process of the content generation system.

[0049] Figure 36 This is a flowchart illustrating the second content generation process of the content generation system.

[0050] Figure 37 This is an illustration showing an example of optimizing content based on the user's gaze.

[0051] Figure 38 This is a diagram illustrating an example of optimizing content based on user movement.

[0052] Figure 39 This is a hardware configuration diagram illustrating an example of a computer that performs the functions of an information processing device. Detailed Implementation

[0053] In the following description, embodiments will be described in detail with reference to the accompanying drawings. In the examples below, the same parts are indicated by the same reference numerals, and redundant descriptions will be omitted.

[0054] This disclosure will be described in the following order of items.

[0055] 1. Implementation Method

[0056] 1-1. Overview of information processing according to the implementation method

[0057] 1-2. Configuration of the information processing device according to the embodiment

[0058] 1-3. Information processing performed by the state estimation system

[0059] 1-4. Information processing performed by the purchase probability estimation system

[0060] 1-5. Information processing performed by the content generation system

[0061] 2. Other implementation methods

[0062] 3. The effects of the information processing device according to this disclosure

[0063] 4. Hardware Configuration

[0064] (1. Implementation Method)

[0065] (1-1. Overview of information processing according to the implementation method)

[0066] First, refer to Figures 1 to 3 A summary of information processing according to embodiments of this disclosure is provided. Figure 1 This is a diagram (1) showing an outline of information processing according to an implementation method. Figure 1 The example shown illustrates an instance of information processing device 100 (not shown) performing processing to estimate a user's behavior or mental state in a store. It should be noted that, in the following examples, for distinction, the person whose behavior, mental state, etc., is being estimated is referred to as a "user," and the person used to generate the learning data for performing the estimation processing is referred to as a "customer." Furthermore, in the following text, a set of data processed by information processing device 100 is referred to as "information," and each piece of data is referred to as "data," but the distinction is not strict, and a single piece of data may be referred to as "information."

[0067] The information processing apparatus 100 is an apparatus that performs information processing according to the present disclosure, and is, for example, an information processing terminal, such as a server device or a personal computer (PC).

[0068] exist Figure 1 In the example shown, the information processing device 100 acquires the movement information of customers 10A and 10B who visit the store. Figure 1 The traffic flow information 21 shown indicates the customer 10A's movement path from entering the store, passing the checkout counter, and leaving the store. Similarly, Figure 1 The flow information 22 shown in the image indicates the flow of customer 10B from entering the store, passing the cashier to make payment, and leaving the store.

[0069] For example, the information processing device 100 acquires traffic flow information 21 and traffic flow information 22 based on top-view images and top-view motion images acquired by an imaging device (camera) installed on the ceiling of the store. Specifically, the information processing device 100 identifies customers included in the top-view images or top-view motion images through image recognition, assigns identification information (ID) to the identified customers, and tracks a series of movements of the designated customers to generate traffic flow information.

[0070] Note that the information processing device 100 can acquire movement information from information obtained by plotting location information acquired from terminal devices held by customers 10A and 10B, or from information acquired by sensor devices installed on merchandise shelves or the like in a store. In other words, the information processing device 100 can acquire movement information in any manner.

[0071] Note that the circulation information does not necessarily include all routes from entering the store to leaving the store, and the information processing device 100 can process a predetermined proportion (e.g., 50% or 80%) of the customer's route from entering the store to leaving the store as circulation information.

[0072] The information processing device 100 acquires traffic flow information 21 and 22, and obtains the behavioral results of customers 10A and 10B in the store. For example, the information processing device 100 acquires purchase history, such as whether customer 10A or customer 10B has visited the store's checkout area (cashier), whether they actually purchased goods or services, what kind of goods they purchased, and the amount purchased. Specifically, for each customer with an assigned ID, the information processing device 100 acquires the point of sale (POS) data at the checkout and obtains the customer's purchase history from that POS data. Furthermore, the information processing device 100 acquires the history of no purchase if a customer leaves the store without having visited the checkout. In addition, the information processing device 100 can acquire information such as the customer's dwell time in the store.

[0073] Furthermore, the information processing device 100 can acquire not only data such as purchase history, but also data indicating customers' psychological states as behavioral outcomes in the store. For example, the information processing device 100 acquires data indicating psychological states, such as whether customers who have finished shopping are satisfied with services such as customer service, whether they intend to purchase goods, whether they are satisfied with the product categorization, or whether the store layout is suitable. Specifically, the information processing device 100 acquires data indicating customers' psychological states through customer questionnaires when leaving the store or making payments.

[0074] Furthermore, the information processing device 100 can acquire not only data such as purchase history, but also data indicating customer attributes as behavioral outcomes in the store. For example, the information processing device 100 can acquire attributes such as the customer's age, gender, and whether they are accompanied by someone through image recognition. Alternatively, the information processing device 100 can acquire customer attributes by performing customer questionnaires and receiving reports of the customer's age, gender, etc.

[0075] Then, the information processing device 100 generates a state estimation model 50, which is a model that has learned the correlation between movement information and the user's behavioral outcomes in the store. For example, the information processing device 100 uses movement information, purchase history, and psychological state as a learning set to generate the state estimation model 50 through a predetermined machine learning method. That is, the state estimation model 50 receives the movement information of the user to be estimated as input and outputs the user's purchase probability, the user's psychological state, the user's attributes, etc.

[0076] In this way, the information processing device 100 acquires the user's movement information and inputs the acquired movement information into the state estimation model 50, thereby estimating at least one of the user's behavior, attributes, and psychology in the store. Specifically, the information processing device 100 estimates information such as the user's probability of purchasing goods, the user's satisfaction with the store, and the attributes the user possesses based on the user's movement information in the store.

[0077] Therefore, the information processing device 100 can estimate users' purchasing behavior, store satisfaction, etc., and thus, for example, can provide useful services such as proactively offering services to users when it is estimated that users are unaware of the location of goods. Furthermore, the information processing device 100 can execute appropriate retail measures by understanding users' psychological states, such as analyzing future customer service systems and store layout.

[0078] Furthermore, the information processing device 100 can analyze the behavior of customers viewing the merchandise and estimate in more detail which behaviors lead to a purchase. (See also...) Figure 2 Describe this. Figure 2 This is a diagram (2) showing an overview of information processing according to an implementation method.

[0079] exist Figure 2 In the example, such as in Figure 1 The image shows a scenario where customer behavior is captured and image recognition is performed using cameras installed in the store.

[0080] Specifically, the information processing device 100 detects each of the customer's behaviors in front of the merchandise using image recognition. Note that the behavior detection is achieved using known image recognition technology. For example, when the device detects a behavior of looking at a specific merchandise, the information processing device 100 acquires behavior information 25, which includes information about the behavior and the merchandise the customer is looking at. Furthermore, when the information processing device detects behaviors such as the customer looking at a smartphone instead of focusing on the merchandise, the information processing device 100 acquires behavior information 26 that indicates the behavior.

[0081] Subsequently, with Figure 1 Similarly, the information processing device 100 acquires POS data corresponding to customer 10C and acquires customer 10C's purchase history. Alternatively, the information processing device 100, based on movement information, acquires the behavioral result indicating that customer 10C left the store without purchasing any goods.

[0082] Then, the information processing device 100 generates a purchase estimation model 60 that has learned the correlation between customer 10C's behavior in front of the product and customer 10C's purchase outcome. That is, the purchase estimation model 60 is a model that receives user behavior information in front of the product as input and outputs estimation results of various purchase behaviors such as whether the user buys the product, which products the user buys, and how much the user pays.

[0083] By using the purchase estimation model 60, the information processing device 100 can estimate user behavior in a more diverse manner. For example, the information processing device 100 can estimate how a series of behaviors that seem unrelated to purchasing goods (such as a user in front of a product shelf looking at a price tag or advertisement on the shelf (first behavior) and then turning his / her gaze to his / her smartphone in his / her hand (second behavior)) affect the purchase.

[0084] For example, by using the information processing device 100, store managers can analyze how users' behavior of viewing smartphone screens affects purchasing outcomes, and whether this influence is due to information that can be searched on the smartphone. This allows store managers to consider strategies to promote purchases from more diverse perspectives.

[0085] Furthermore, the information processing device 100 can generate content (electronic advertisements, etc.) that is expected to have a higher advertising effect based on customer behavior. (See reference...) Figure 3 Describe this. Figure 3 This is a diagram (3) showing an overview of information processing according to an implementation method.

[0086] Figure 3 The example shows customer 10D behavioral information as such Figure 1 and Figure 2 In this case, the information processing device 100 can acquire information about the customer 10D using not only the camera but also any sensor such as a sensor that measures temperature and humidity. Furthermore, the information processing device 100 can acquire information not only from the sensors but also from external servers that provide weather information, etc.

[0087] For example, the information processing device 100 acquires attribute information (gender, age, etc.), gaze information (which product or information is being viewed), movement information, and environmental information of the customer 10D (weather, temperature, etc.) as information group 30.

[0088] In addition, with Figure 1 and Figure 2 Similarly, the information processing device 100 acquires result information, such as the purchase history of customer 10D.

[0089] Then, the information processing device 100 generates a content generation model 70, which is a model that learns the correlation between information sets related to customer 10D and the advertising effect of content presented in the store. For example, the information processing device 100 generates a model of customer 10D purchasing goods after viewing content, or learns the relationship between customer 10D's points of interest in content (price, product images, other information, etc.) and purchase results.

[0090] Subsequently, upon acquiring information about a target user, the information processing device 100 generates effective content (hereinafter referred to as "optimized content") and presents this content to the user as digital signage. For example, upon acquiring information about a user's activity level and attributes of users entering the store, the information processing device 100 generates optimized content for the user and displays it on a display device (such as a signage display) installed in the store. Thus, the information processing device 100 can present the most effective content to the user in real time, thereby promoting the user's willingness to purchase.

[0091] In this case, the information processing device 100 can further correct and optimize the content based on information such as the user's gaze at the optimized content being presented. For example, if the information processing device 100 obtains gaze information indicating that the user is paying attention to "price information" in the content, it can further perform generation processing to change the price information to a more prominent color or enlarge the text.

[0092] As described above, the information processing device 100 not only simply switches between digital signs for promotional items from a predetermined design, but also presents more effective signs for each user based on user characteristics (attributes, movement patterns, line of sight, etc.) or circumstances (weather, temperature, etc.). Furthermore, by tracking users' purchase history after viewing optimized content, the information processing device 100 can obtain detailed information about advertising design strategies, such as which types of sign designs promote purchases.

[0093] As per the above reference Figures 1 to 3 According to the information processing device 100, various information (such as whether a user intends to purchase goods or is satisfied with the service) can be estimated using behavioral information (such as movement patterns or user gestures). Furthermore, the information processing device 100 generates content such as effective signage tailored to each user, thereby enabling the promotion of purchase intentions and the evaluation of advertising effectiveness. In other words, the information processing device 100 can provide analytical results regarding useful retail measures.

[0094] (1-2. Configuration of the information processing device according to the embodiment)

[0095] Next, the configuration of the information processing system 1 and the information processing device 100 will be described. Figure 4This is a diagram showing an example configuration of the information processing system 1 and the information processing device 100 according to this embodiment.

[0096] like Figure 4 As shown, the information processing system 1 includes an information processing device 100, a sensor device 200, and a display device 300.

[0097] As described above, the information processing device 100 is a device that performs information processing using various models learned through artificial intelligence (AI).

[0098] Sensor device 200 is a device for detecting various types of information. For example, sensor device 200 is a camera (image sensor) installed in a store. That is, sensor device 200 is an information processing device with imaging capabilities, and is, for example, a digital camera or digital video camera installed in a store, etc. In this case, sensor device 200 includes a microcontroller unit (MCU) or microprocessor unit (MPU), and also includes a CMOS image sensor (CIS), and performs a series of information processing such as image capture, image storage, and image transmission / reception.

[0099] It should be noted that the sensor device 200 may have a pre-learned model to identify a predetermined target and determine whether the predetermined target is included in the captured image. For example, the sensor device 200 may detect a person, detect a person's specific behavior or gesture through image recognition, or estimate a person's age or gender by using a pre-learned AI model. That is, the sensor device 200 may be used as a terminal device (edge) in the information processing system 1.

[0100] Furthermore, the sensor device 200 is not limited to a camera and may include any sensor for detecting various types of information. For example, the sensor device 200 may include a laser sensor for detecting the presence of a customer or user, or a distance measurement sensor such as a ToF sensor. Additionally, the sensor device 200 may include a temperature sensor and a humidity sensor.

[0101] Display device 300 is a display that shows content such as digital signage. For example, display device 300 is installed in a store and displays advertising content related to the goods placed in the store. Note that display device 300 does not necessarily have to be installed in a store and can be, for example, a user terminal (smartphone, etc.) used by a user. In this case, information processing device 100 sends the generated content to the user terminal and controls the user terminal to display the sent content.

[0102] Notice, Figure 4Each device in the diagram conceptually illustrates the function of the information processing system 1 and can take various forms depending on the implementation. Furthermore, the number of each device included in the information processing system 1 is not limited to those shown.

[0103] Next, the internal configuration of the information processing device 100 will be described. For example... Figure 4 As shown, the information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. It should be noted that the information processing device 100 may include: an input unit (e.g., a touch screen, etc.) for receiving various operations from an administrator or other person managing the information processing device 100; and a display unit (e.g., a liquid crystal display, etc.) for displaying various types of information.

[0104] The communication unit 110 is implemented by, for example, a network interface card (NIC) or a network interface controller. The communication unit 110 is connected to the network N via a wired or wireless means, and transmits information to and receives information from the sensor device 200, display device 300, etc., via the network N. The network N is implemented, for example, through wireless communication standards or systems such as Bluetooth, the Internet, Wi-Fi, UWB (Ultra-Wideband), LPWA (Low Power Wide Area), and ELTRES (Registered Trademark).

[0105] Storage unit 120 is implemented by, for example, semiconductor storage elements (such as random access memory (RAM) or flash memory) or storage devices (such as hard disk or optical disk).

[0106] In this implementation, storage unit 120 includes a state estimation information storage unit 121, a purchase estimation information storage unit 122, and a content generation information storage unit 123. Each storage unit stores information related to... Figures 1 to 3 Information corresponding to each model described herein. A detailed description of each storage unit and the processing using each model will be given later.

[0107] The control unit 130 is implemented by, for example, a central processing unit (CPU), an MPU, a graphics processing unit (GPU), etc., which uses RAM or the like as a working area to execute programs (e.g., information processing programs according to this disclosure) stored in the information processing device 100. Furthermore, the control unit 130 is a controller and can be implemented by, for example, an integrated circuit (such as an application-specific integrated circuit (ASIC)), a field-programmable gate array (FPGA), or a microcontroller.

[0108] like Figure 4 As shown, the control unit 130 includes an acquisition unit 131, a state estimation system 132, a purchase probability estimation system 133, and a content generation system 134.

[0109] The acquisition unit 131 acquires various types of information that the processing unit will use in subsequent stages. For example, the acquisition unit 131 acquires movement information indicating the movement of customers in a predetermined space (e.g., a store). Specifically, the acquisition unit 131 acquires movement information from images acquired by an imaging device (sensor device 200, etc.) installed in the predetermined space.

[0110] The acquisition unit 131 also acquires result information corresponding to the traffic flow information. For example, the acquisition unit 131 acquires the customer's purchasing behavior and purchase history in the store corresponding to the traffic flow information. Specifically, the acquisition unit 131 acquires information based on POS data, such as whether the customer went to the cashier, whether the customer has already paid for goods or services in the store, and the amount paid by the customer for goods or services in the store.

[0111] In addition, the acquisition unit 131 can acquire various types of information, such as the average unit price of goods purchased by customers, the total quantity of goods, the total number of goods purchased, the type of goods purchased, the customer's payment method, and whether the customer has a loyalty card that can be used in the store.

[0112] Furthermore, the acquisition unit 131 can acquire behavioral information about customers along their movement path. For example, the acquisition unit 131 can acquire information such as whether a customer ultimately purchases a product, the number of times a customer picks up a product, and the frequency with which a customer puts a product back (whether the customer puts a product back). In addition, the acquisition unit 131 can acquire information such as the customer's dwell time, the number of times a customer views pop-up advertisements or signs in the store, and the number and composition of the customer's companions.

[0113] Acquisition unit 131 can acquire psychological information about customers corresponding to the traffic flow information. For example, after a customer has stayed in the store, acquisition unit 131 obtains the customer's evaluation information about the store. Acquisition unit 131 obtains the customer's evaluation information about the store through, for example, feedback obtained from the customer (questionnaire collection, etc.).

[0114] Specifically, the acquisition unit 131 acquires psychological information about the customer's purchase intentions (whether they decided to buy something when visiting the store, whether they decided to buy something but did not, etc.). In addition, the acquisition unit 131 can acquire psychological information such as whether the customer is satisfied with the store's service, whether the customer understands the arrangement of the goods in the store (whether the location of the goods is easy to understand), and whether the customer decides what to buy based on pop-up advertisements or signs in the store.

[0115] Furthermore, the acquisition unit 131 can acquire information about the behavior of customers located in front of the shelves where goods are placed. For example, the acquisition unit 131 acquires customer behavior information through image recognition or the like using a camera installed near the goods shelf.

[0116] The acquisition unit 131 acquires gaze information as behavioral information, such as whether a customer is paying attention to goods or services offered by the store. For example, the acquisition unit 131 can acquire gaze information by analyzing images captured by cameras in the store. Specifically, the acquisition unit 131 can use known eye-tracking methods, such as image recognition methods, to acquire information indicating whether a customer is paying attention to a specific product. The acquisition unit 131 can also acquire gaze information by analyzing the customer's gaze using a camera or sensor embedded in a sign such as a display device 300.

[0117] Furthermore, the acquisition unit 131 acquires gesture information taken by customers viewing the merchandise as behavioral information. For example, the acquisition unit 131 detects predetermined gestures such as looking at a smartphone, picking up a merchandise and putting it back on the shelf, scratching one's head, or crossing one's arms, and acquires the detected behavioral information. Note that the acquisition unit 131 can acquire multiple behaviors, such as paying attention to the merchandise and subsequent behaviors of the customer, as a series of behavioral information.

[0118] In addition, the acquisition unit 131 can acquire customer attribute information. For example, the acquisition unit 131 can acquire attribute information such as the customer's gender and age, and whether they are accompanied, through image recognition. Note that the acquisition unit 131 may acquire attribute information through customer questionnaires or input of attributes based on visual observations by the cashier manager, without using image recognition.

[0119] Furthermore, the acquisition unit 131 can acquire customer behavior information regarding content such as advertisements in a store. For example, the acquisition unit 131 can acquire information such as whether a customer has viewed the content, or eye contact information such as which parts or information of the content the customer has viewed.

[0120] In addition, the acquisition unit 131 can acquire predetermined external environmental information. For example, the acquisition unit 131 can acquire information such as the weather, temperature and time when a customer visits the store, and the area where the store is located.

[0121] That is, the acquisition unit 131 acquires all information that may serve as learning data for the state estimation system 132, the purchase probability estimation system 133, and the content generation system 134 when generating models in subsequent stages. Furthermore, the acquisition unit 131 acquires all information that may serve as input when the state estimation system 132, the purchase probability estimation system 133, and the content generation system 134 use the model to perform estimation and generation processes.

[0122] The following describes the processes performed by the state estimation system 132, the purchase probability estimation system 133, and the content generation system 134 in sequence. Note that the process performed by the state estimation system 132 corresponds to... Figure 1 The state estimation process is shown in the diagram. The process performed by the purchase probability estimation system 133 corresponds to... Figure 2 The purchase probability estimation process is shown in the diagram. Furthermore, the process performed by the content generation system 134 corresponds to... Figure 3 The content generation process is shown in the diagram. Furthermore, although the acquisition unit 131 has been described separately from each system for ease of explanation, the acquisition unit 131 can be integrated into each system.

[0123] (1-3. Information processing performed by the state estimation system)

[0124] The state estimation system 132 is described. For example... Figure 4 As shown, the state estimation system 132 includes a motion analysis unit 132A, a state estimation model generation unit 132B, and a state estimation unit 132C.

[0125] The movement analysis unit 132A analyzes the movement information acquired by the acquisition unit 131. For example, the movement analysis unit 132A assigns a corresponding customer or user ID to each piece of movement information acquired and stores the ID in the state estimation information storage unit 121.

[0126] The state estimation model generation unit 132B generates a state estimation model 50 by learning the correlation between movement information and the user's behavior in space.

[0127] The state estimation unit 132C estimates at least one of the user’s behavior, attributes, or psychology in the space based on the user’s movement information using the state estimation model 50.

[0128] Note that, in the following description, for the sake of simplicity, the processes performed by the acquisition unit 131, the movement analysis unit 132A, the state estimation model generation unit 132B, and the state estimation unit 132C are described as being performed by the state estimation system 132.

[0129] For example, state estimation system 132 estimates a user's purchasing behavior in the store based on the user's movement information by using a model that learns the correlation between movement information and the customer's purchase history in the store. Specifically, state estimation system 132 estimates whether the user has made a payment for goods or services in the store (i.e., whether they went to the checkout) as the user's purchasing behavior. Alternatively, state estimation system 132 may estimate the amount paid by the user for goods or services in the store as the user's purchasing behavior.

[0130] The state estimation system 132 can also estimate the psychological state of users in the store based on user movement information by using a model that has learned the correlation between customer movement information after they have stayed in the store and customer evaluations. For example, the state estimation system 132 estimates whether users in the store are satisfied with the services provided by the store based on user movement information.

[0131] Reference Figure 5 The following figures illustrate the processing of the aforementioned state estimation system 132. Figure 5 This is a diagram showing an outline of the processes performed by the state estimation system 132.

[0132] like Figure 5 As shown, when a user newly enters the store, the state estimation system 132 acquires the user's movement information. Then, the state estimation system 132 inputs the movement information into the state estimation model 50.

[0133] Then, the state estimation system 132 acquires various information output by the state estimation model 50. Note that the information output by the state estimation model 50 may vary depending on what information is used as the correct response data (label) during learning.

[0134] For example, after the state estimation model 50 has learned the relationship between movement information and purchase history, the state estimation system 132 estimates the user's predicted purchase behavior (whether the user will purchase goods, etc.) based on the user's movement information. Alternatively, after the state estimation model 50 has learned the relationship between movement information and customer behavior history, the state estimation system 132 estimates the user's predicted purchase behavior (whether to go to the checkout before leaving the store, etc.) based on the user's movement information. Alternatively, after the state estimation model 50 has learned the relationship between movement information and customer psychological information, the state estimation system 132 estimates the user's predicted psychological state (whether the user is satisfied with the service, etc.) based on the user's movement information. Note that the state estimation model 50, depending on the model's learning method or pattern, can also receive movement information as input and output all information such as purchase prediction, behavior prediction, and psychological prediction.

[0135] Reference Figure 6 Describe the labels learned by the state estimation system 132. Figure 6 This is a diagram illustrating an example of the learning data used by the state estimation system 132. Figure 6 Data table 121A is shown as an example of a data table in the state estimation information storage unit 121.

[0136] like Figure 6As shown, data table 121A includes items such as "Customer ID", "Traffic Flow Information", and "Label Information". Note that in the following text, the information stored in data table 121A, etc., may be conceptually represented as "A01" or "B01", but in reality, it stores specific information described later.

[0137] "Customer ID" is identification information used to identify customers. "Customer movement information" is information indicating customer movement patterns. "Label information" is information used as data for correct responses during learning. For example, label information includes sub-items such as "POS data," "behavioral data," and "psychological data." "POS data" includes information such as the time a customer enters or leaves the store, the items purchased, the amount spent, attributes, and payment method. "Behavioral data" includes information such as whether a customer has purchased an item and the number of times they returned items. "Psychological data" includes information such as the customer's purchase intention and satisfaction with the store's services.

[0138] The state estimation system 132 generates a state estimation model 50 by performing machine learning using the data shown in data table 121A as learning data.

[0139] Reference Figure 7 The following figures describe the processing procedure of the state estimation system 132. Figure 7 This is a flowchart (1) illustrating the learning process of the state estimation system 132. Figure 7 The process of state estimation system 132 generating a model for estimating user purchasing behavior as state estimation model 50 is shown.

[0140] First, the acquisition unit 131 acquires a top-view image from a camera or similar device installed in the store (step S10). The state estimation system 132 performs image recognition on the top-view image and detects people (step S11). The state estimation system 132 creates a movement history for each person (step S12).

[0141] State estimation system 132 determines whether the generated path history is above a predetermined threshold (step S13). The threshold is a value indicating how much path information in the entire path is used for processing. For example, when the threshold is 50%, state estimation system 132 performs processing using 50% of the entire path. (See later...) Figure 16 Describe the details of the threshold settings.

[0142] If the historical movement history is less than a threshold (step S13; No), the state estimation system 132 continues to acquire the person's movement history. On the other hand, if the historical movement history is equal to or greater than the threshold (step S13; Yes), the state estimation system 132 creates movement data (step S14). The movement data is data representing the customer's movement in a manner used for learning data.

[0143] Next, the state estimation system 132 tracks the customer's movements in the store (step S15). For example, the state estimation system 132 tracks the customer's movements in the store by using camera-based person detection and tracking processing, and determines whether the customer has left the store (step S16).

[0144] If the customer does not leave the store (step S16; No), the state estimation system 132 continues to track the customer. On the other hand, if the customer leaves the store (step S16; Yes), the state estimation system 132 records information used to confirm the customer's purchase history, such as the checkout counter number and time the customer passed before leaving the store (e.g., POS data) (step S17). Then, based on the confirmed information, the state estimation system 132 obtains the customer's POS data from an external server or other server that manages POS data (step S18).

[0145] The state estimation system 132 uses movement data and corresponding POS data (purchase history, etc.) as a learning set and generates an estimation model related to the POS data (step S19).

[0146] Next, we will refer to Figure 8 Describe the process when the state estimation system 132 generates a model for estimating user behavior as the state estimation model 50. Figure 8 This is a flowchart (2) showing the learning process of the state estimation system 132.

[0147] because Figure 8 The processing before step S16 in the middle and Figure 7 The processing is similar to that described in step S21, so its description will be omitted. After the customer leaves the store, the state estimation system 132 detects the customer's behavior based on images or motion images including the customer and creates behavior history data. Then, the state estimation system 132 assigns behavior labels based on the behavior history data (step S22). For example, the state estimation system 132 assigns labels for behaviors that the customer has performed (such as whether the customer has gone to the checkout or whether the customer has returned items).

[0148] Then, the state estimation system 132 uses the movement data and the corresponding behavior labels as a learning set to generate an estimation model about the behavior data (step S23).

[0149] Next, we will refer to Figure 9 Describe the process when the state estimation system 132 generates a model for estimating the user's mental state as the state estimation model 50. Figure 9 This is a flowchart (3) showing the learning process of the state estimation system 132.

[0150] because Figure 9 The processing before step S16 in the middle and Figure 7 The process is similar, so its description will be omitted. After a customer leaves the store, the state estimation system 132 records the time of the customer's departure (step S25). Then, the state estimation system 132 queries feedback such as a questionnaire to be obtained later and the customer's departure time, and determines the questionnaire results corresponding to that customer. The state estimation system 132 assigns psychological labels based on the determined questionnaire results, etc. (step S26). For example, the state estimation system 132 assigns labels indicating whether the customer is satisfied with the store's services.

[0151] Then, the state estimation system 132 uses the movement data and the mental labels corresponding to the movement data as a learning set to generate an estimation model related to the mental data (step S27).

[0152] Next, we will refer to Figure 10 Describe the process when the state estimation system 132 uses the state estimation model 50 to estimate the user's behavior and psychological state. Figure 10 This is a flowchart (1) showing the estimation process of the state estimation system 132.

[0153] Similar to the creation of learning data, acquisition unit 131 acquires top-view images from cameras or similar devices set up in the store (step S30). State estimation system 132 performs image recognition on the top-view images and detects people (step S31). State estimation system 132 creates a movement history for each person (step S33).

[0154] The state estimation system 132 determines whether the generated historical movement data exceeds a predetermined threshold (step S34). For example, when movement data is input into the state estimation model 50, the threshold is set based on whether the amount of movement data is sufficient to cause a predetermined accuracy to be exceeded.

[0155] If the user's movement history is less than the threshold (step S34; No), the state estimation system 132 continues to acquire the user's movement history. On the other hand, if the user's movement history is equal to or greater than the threshold (step S34; Yes), the state estimation system 132 creates the user's movement data (step S35). The movement data is data representing the user's movement in a form that can be input into the model.

[0156] The state estimation system 132 inputs the created movement data to... Figures 7 to 9 One of the estimation models shown (step S36). For example, the state estimation system 132 selects any model based on the information of the user to be estimated (e.g., whether they purchased goods, etc.) according to the request of the store manager.

[0157] The state estimation system 132 inputs the movement data into any model and outputs the inference results corresponding to that model (step S37). Thus, the store manager can know the inference results, such as whether a user purchased goods during their stay in the store.

[0158] Note that the state estimation system 132 also inputs movement data into multiple models, rather than selecting one of those models. (Refer to...) Figure 11 Describe this example. Figure 11 This is a flowchart (2) showing the estimation process of the state estimation system 132.

[0159] because Figure 11 The processing before step S35 in the middle and Figure 10 The processing is similar to that in the previous step, so its description will be omitted. After generating the movement data, the state estimation system 132 inputs the movement data into multiple models (step S40). Then, the state estimation system 132 obtains the output corresponding to each of the multiple models (step S41).

[0160] Therefore, during the time a user stays in the store, the store manager can obtain purchase information such as which products the user bought, behavioral information such as whether the user went to the cashier, and psychological information such as whether the user is satisfied with the service.

[0161] Furthermore, the state estimation system 132 can select the model from the input movement data based on the circumstances. (Refer to...) Figure 12 Describe this example. Figure 12 This is a flowchart (3) showing the estimation process of the state estimation system 132.

[0162] because Figure 12 The processing before step S35 in the middle and Figure 10 The processing is similar to that in the previous section, so its description will be omitted. After generating the movement data, the state estimation system 132 determines which model to input the movement data into (step S45).

[0163] For example, in a crowded store situation, there is a high probability that the store manager wants to know, for instance, how long multiple users in the store will remain. Therefore, for example, when it is determined that the number of people in the store exceeds a predetermined threshold (when the store is determined to be crowded), the state estimation system 132 inputs movement data into a model for estimating user behavior (step S46). Then, the state estimation system 132 outputs an estimate of how long users will remain in the store (step S47).

[0164] Alternatively, when the store is not crowded, the store manager may want to know the average transaction value of customers (e.g., what price range of goods they will purchase). In this case, the state estimation system 132 inputs the movement data into a model used to estimate customer purchase information. Then, the state estimation system 132 outputs an estimate of the price range of goods that customers in the store are likely to purchase.

[0165] As described above, the state estimation system 132 can determine whether to estimate user purchasing behavior or user psychological state based on the store's congestion level. The state estimation system 132 can estimate the information needed by store managers by selectively using the model based on the store's situation without increasing processing load.

[0166] In addition to estimation processing, the state estimation system 132 can perform processing to acquire user attributes and learn the acquired attributes. (See reference...) Figure 13 Describe this example. Figure 13 This is a flowchart (4) showing the estimation process of the state estimation system 132.

[0167] because Figure 13 The processing before step S37 in the middle and Figure 10 The processing is similar to that in the previous step, so its description will be omitted. When a person is detected, the state estimation system 132 uses known techniques such as image recognition to identify attributes such as the user's gender and age (step S50). Then, the state estimation system 132 assigns attribute labels associated with the movement data (step S51). Afterward, the state estimation system 132 relearns the existing model using the user's purchase and behavioral results, as well as the movement data with attribute labels.

[0168] As described above, the state estimation system 132 can estimate the future behavior or mental state of a user located in the space based on the user's movement information by using a model that has learned the correlation between movement information and attributes and the behavioral outcomes of customers (or users) in the space. That is, the state estimation system 132 can perform estimation processing in which users are further segmented by accumulating data including attributes. For example, when movement data is available, in addition to the movement data, the state estimation system 132 can input information indicating that "the customer attribute in the movement data is a woman in her 20s" into the model. In this case, compared to performing estimation using only movement data, the state estimation system 132 can predict user behavior with higher accuracy.

[0169] Next, a scenario is described using the estimation process results of the state estimation system 132. Figure 14 This is a diagram (1) illustrating an example of the processing performed by the state estimation system 132 using the estimation results.

[0170] like Figure 14 As shown, the state estimation system 132 acquires movement data 31 indicating the path of user 10 after entering the store. The state estimation system 132 inputs the movement data 31 into the state estimation model 50. For example, the state estimation system 132 estimates user 10's current psychological state and purchase intention in the store based on the movement data 31. Specifically, the state estimation system 132 estimates information such as the products user 10 might buy and the products user 10 is considering buying.

[0171] Then, the state estimation system 132 displays advertisements for items that user 10 is likely to purchase on a display device 300 installed in the store, indicating that user 10 has a purchase intention for those items. In this way, the state estimation system 132 can further promote user 10's purchase intention by displaying advertisements using the estimation results.

[0172] In addition, the estimation results of the state estimation system 132 can also be used in other scenarios. Figure 15 This is a diagram (2) illustrating an example of the processing performed by the state estimation system 132 using the estimation results.

[0173] Similar to Figure 14 The state estimation system 132 acquires movement data 32 indicating the path of user 10 after entering the store. The state estimation system 132 inputs the movement data 32 into the state estimation model 50. For example, the state estimation system 132 estimates the user 10's psychological state of not being able to find the product, or the area where the product the user 10 wants to buy is located, based on the movement data 32.

[0174] Then, when the state estimation system 132 estimates that the user 10 is in a state of searching for goods, it instructs the store clerk 33 to provide service to the user 10. In this way, the state estimation system 132 can use the estimation results to sense whether the user is lost in the store or searching for goods, and can appropriately arrange the store clerk 33. Therefore, the state estimation system 132 can improve user satisfaction. Note that the state estimation system 132 can estimate the user's psychological state to estimate the areas the user is likely to visit next in the store (such as the areas where the user intends to buy goods). Therefore, the state estimation system 132 can display advertisements for goods in the areas the user is likely to visit next, or instruct the store clerk to provide service.

[0175] Here, we will refer to Figure 16 The description describes the generation of an estimation model through the state estimation system 132. Figure 16 This is a diagram illustrating an outline of the learning process of the state estimation system 132.

[0176] As described above, the state estimation system 132 can perform learning and estimation by setting a specific proportion (threshold) during model generation instead of using all customer movement paths. In this case, due to higher processing efficiency, it is desirable to perform model generation and estimation with higher accuracy using less movement path data. Therefore, as... Figure 16 As shown, the state estimation system 132 can process data obtained by dividing the movement data to be processed by a predetermined threshold.

[0177] The model generation and estimation process in this case will be described. For example, Figure 16 The flow data 35 shown is data indicating all customer movement from entering to leaving the store.

[0178] The state estimation system 132 sets some thresholds, such as 30%, 40%, 60%, and 80%, and repeatedly performs model generation and estimation processing using movement data. The state estimation system 132 then compares the estimation processing with the correct response data and calculates the verification result.

[0179] The state estimation system 132 sets a threshold based on the validation results, resulting in a model with a low threshold (the movement data has less information) and high accuracy. Figure 16 In the example, the model achieves the highest accuracy and keeps the information content of the movement data relatively low when using movement data with a 60% threshold. Therefore, the state estimation system 132 sets the threshold to 60%.

[0180] As described above, in model generation, the state estimation system 132 can set an optimal threshold by temporarily setting a threshold, performing model generation and estimation processes, and performing verification. Therefore, the state estimation system 132 can generate a high-accuracy model while suppressing processing load.

[0181] (1-4. Information processing performed by the purchase probability estimation system)

[0182] return Figure 4 This will describe a purchase probability estimation system 133. For example... Figure 4 As shown, the purchase probability estimation system 133 includes a behavior detection unit 133A, a purchase estimation model generation unit 133B, and a purchase estimation unit 133C.

[0183] The behavior detection unit 133A detects the behavior or gestures of a customer or user from images or the like acquired by the acquisition unit 131. (See reference...) Figure 17 Describe this. Figure 17 This is a diagram showing an overview of the motion detection of the behavior detection unit 133A.

[0184] Figure 17An example of detecting customers 10A, 10B, 10C, and 10D in a top-view image is shown. Behavior detection unit 133A detects various behaviors, such as customer 10A gazing at a specific item or customer 10B pacing without noticing the item. Note that these types of detection processing are achieved by learning a behavior detection model to detect predefined specific behaviors. The behavior detection model may be mounted on a camera used as an edge terminal with the function of performing detection processing, or it may be included in information processing device 100.

[0185] The purchase estimation model generation unit 133B learns the correlation between customer behavior information detected by the behavior detection unit 133A in the store and the customer's purchase results in the store. Then, given the user's behavior information as input, the purchase estimation model generation unit 133B generates a purchase estimation model 60, which is a model used to output the user's purchase information in the store.

[0186] Purchase estimation unit 133C estimates a user's purchase information in a store based on the user's behavior using purchase estimation model 60. For example, purchase estimation unit 133C estimates the probability of a user purchasing an item.

[0187] At this point, the purchase estimation unit 133C can use the purchase estimation model 60 to estimate the probability of a user making a purchase in the store. This purchase estimation model 60 learns the correlation between customer behavior information after they have viewed a product or service and the customer's purchase outcome in the store. That is, the purchase estimation unit 133C can input information indicating that the user has viewed products, services, price tags, etc., presented in the store, or the user's actions after viewing products, services, price tags, etc., as a series of behavioral information into the purchase estimation model 60. Thus, the purchase estimation unit 133C performs estimation processing of user behaviors that are not obviously related to the purchase of the product, thereby enabling in-depth analysis of what behaviors lead to a purchase.

[0188] For the sake of simplicity, the processes performed by the acquisition unit 131, behavior detection unit 133A, purchase estimation model generation unit 133B, and purchase estimation unit 133C are described as being performed by the purchase probability estimation system 133.

[0189] For example, purchase probability estimation system 133 acquires customer behavior information after focusing on goods or services offered by a store, as well as the customer's purchase outcome in the store. Then, purchase probability estimation system 133 generates a model that learns the correlation between customer behavior information after focusing on goods or services and the customer's purchase outcome in the store. Using such a model, purchase probability estimation system 133 estimates the probability of a user making a purchase in the store based on the goods or services the user focuses on and the user's behavior information. Note that the user's behavior information used for estimation can be a single behavior or a combination of multiple behaviors.

[0190] Furthermore, the purchase probability estimation system 133 can acquire customer attributes and generate a model that has learned the correlation between behavioral information and attributes and the customer's purchase outcome. In this case, the purchase probability estimation system 133 estimates the probability of a user making a purchase in the store based on the user's behavioral information and attributes using such a model.

[0191] Note that, for example, the purchase probability estimation system 133 can associate a customer with a purchase outcome based on at least one of facial recognition processing obtained by imaging the customer, specific identification by the store equipment used by the customer, and image-based matching of the customer's clothing. For example, the purchase probability estimation system 133 can associate a customer with POS data based on the degree of matching between the customer's face and clothing captured by an in-store camera and the customer's face and clothing captured by a camera at the checkout counter. The store equipment used by the customer is, for example, a shopping basket. The purchase probability estimation system 133 can associate a customer with POS data by identifying a radio frequency identification (RFID) tag attached to the shopping basket at the checkout counter.

[0192] Reference Figure 18 The following figures illustrate the processing of the purchase probability estimation system 133. First, the information used as learning data in the model generation of the purchase probability estimation system 133 is explained. Figure 18 This is a diagram illustrating an example of the learning data used by the purchase probability estimation system 133. Figure 18 Data table 122A is shown as an example of a data table in the purchase estimate information storage unit 122.

[0193] like Figure 18 As shown, data table 122A includes items such as "Customer ID", "Behavioral Information", and "Purchase Results".

[0194] "Customer ID" is identification information used to identify the customer. "Behavioral Information" is information about the actions or gestures performed by the customer. Note that behavioral information can include multiple actions and gestures. "Purchase Result" indicates the customer's purchase outcome. Purchase outcome includes information about whether the customer who performed the action or gesture has purchased an item as a result, and if so, the time of purchase, the item purchased, etc. Additionally, purchase outcome can include attributes such as the customer's age and gender, and information such as whether they were accompanied.

[0195] Purchase probability estimation system 133 uses purchase outcomes associated with customer behavior information as correct response data to generate purchase estimation model 60. As an example, purchase probability estimation system 133 uses a series of behavioral information instructing customers to view products, such as smartphones, and results of customers having already purchased products as a training dataset to generate purchase estimation model 60.

[0196] The behaviors or gestures detected by the purchase probability estimation system 133 can include various things. See also Figure 19 and Figure 20 Describes the behavior detected by the purchase probability estimation system 133. Figure 19 This is a diagram (1) showing an example of behavior detected by the purchase probability estimation system 133.

[0197] Figure 19 Examples of behaviors based on customer gesture estimation (122B) and behavior based on behavior recognition (122C) are shown. For example, the purchase probability estimation system 133 uses known gesture estimation techniques for customers (e.g., skeleton estimation) to detect behaviors or gestures (such as a customer crossing their arms in front of their body or placing their hand on their chin) and stores the detected behaviors or gestures.

[0198] In addition, the purchase probability estimation system 133 uses known behavior recognition technologies (e.g., facial recognition, how many times a customer is detected within a certain imaging range) to detect behaviors or gestures such as changes in a customer's facial expressions or a customer returning to the same product shelf multiple times, and stores the detected behaviors or gestures.

[0199] Next, we will refer to Figure 20 Describe another behavior detected by the purchase probability estimation system 133. Figure 20 This is a diagram (2) showing an example of behavior detected by the purchase probability estimation system 133.

[0200] Figure 20Examples of behaviors based on customer gaze estimation (122C) and motion detection (122D) are shown. For example, a purchase probability estimation system 133 uses known gaze detection techniques (e.g., eye tracking) to detect behaviors or gestures such as customers looking at smartphones, glancing around, or looking at specific products, and stores the detected behaviors or gestures.

[0201] In addition, the purchase probability estimation system 133 uses known motion detection technology to detect customer behavior or gestures, such as a customer confirming that the bag is inside or that it is separated from the shelf where the goods are placed by more than a predetermined distance, and stores the detected behavior or gestures.

[0202] Notice, Figure 19 and Figure 20 The behaviors or gestures shown are examples, and the behaviors or gestures detected by the purchase probability estimation system 133 are not limited to those shown. Figure 19 and Figure 20 The example shown is shown in the image.

[0203] Next, Figure 21 An example of behavioral information stored in the purchase probability estimation system 133 is shown. Figure 21 This is a diagram (1) illustrating an example of behavioral information used by the purchase probability estimation system 133. Figure 21 Data table 122F is shown as an example of a data table in the purchase estimate information storage unit 122.

[0204] like Figure 21 As shown, Data Table 122F includes items such as "Customer ID," "Behavior," and "Purchase Result." "Behavior" indicates each of the customer's detected behaviors and gestures. Note that behavior items may include multiple series of behaviors. Each behavior or series of behaviors corresponds to customer behavioral information.

[0205] exist Figure 21 In the example shown, customer A11, who was instructed to browse a smartphone, did not purchase any items from the store. In another example, customer A12, who was instructed to cross his / her arms in front of an item, purchased that item from the store.

[0206] Purchase probability estimation system 133 uses the above-mentioned information to estimate the user's purchase probability. (See reference...) Figure 22 The following figures illustrate the processing procedure of the purchase probability estimation system 133. Figure 22 This is a flowchart (1) illustrating the learning process of the purchase probability estimation system 133.

[0207] First, the acquisition unit 131 acquires images of customers captured by cameras or similar devices installed near the merchandise shelves (step S60). The purchase probability estimation system 133 performs image recognition on the images and detects people (step S61). The purchase probability estimation system 133 detects the behavior of each person (step S62). The purchase probability estimation system 133 generates behavioral data for each person in a pattern that can be used as learning data (step S63).

[0208] Subsequently, the purchase probability estimation system 133 tracks the customer's movements in the store (step S64). For example, the purchase probability estimation system 133 tracks the customer's movements in the store by using person detection and tracking processing with a camera, and determines whether the customer has left the store (step S65).

[0209] If the customer does not leave the store (step S65; No), the purchase probability estimation system 133 continues to track the customer. On the other hand, if the customer leaves the store (step S65; Yes), the purchase probability estimation system 133 records information used to determine the customer's purchase history (e.g., POS data), such as the checkout counter number and time passed before leaving the store (step S66). Then, the purchase probability estimation system 133 obtains the customer's POS data from an external server or other entity that manages POS data based on the determined information (step S67).

[0210] Purchase probability estimation system 133 uses behavioral data and corresponding POS data (purchase history, etc.) as a learning set to learn an estimation model related to POS data (step S68).

[0211] Next, refer to Figure 23 and Figure 24 This section will describe an example of how the purchase probability estimation system 133 generates an estimation model that includes not only customer behavior but also the products that the customer has already considered prior to that behavior.

[0212] First, refer to Figure 23 Describe an example of the behavioral information used for model generation in this context. Figure 23 This is a diagram (2) illustrating an example of behavioral information used by the purchase probability estimation system 133. Figure 23 Data table 122G is shown as an example of a data table in the purchase estimate information storage unit 122.

[0213] like Figure 23As shown, in addition to Data Table 122F, Data Table 122G also includes the item "Products of Interest". Products of interest indicate products that customers are interested in. Note that a product of interest can be a specific product or a type of product. Furthermore, a product of interest can be an attribute of a product, such as a discounted or limited-edition item, rather than the type of product.

[0214] exist Figure 23 In the example shown, customer A11 looks at product E11 and then browses a smartphone, thus not purchasing the product in the store. In another example, customer A12 looks at product E12 and then crosses his / her arms in front of the product, thus purchasing the product in the store.

[0215] Reference Figure 24 Describe the process of generating the estimation model in this situation. Figure 24 This is a flowchart (2) illustrating the learning process of the purchase probability estimation system 133.

[0216] because Figure 24 The processing from step S60 to step S67 and Figure 22 The processing is similar to that in the previous section, so its description will be omitted. The purchase probability estimation system 133 obtains information about the products that the customer is interested in when detecting customer behavior (step S70).

[0217] Upon obtaining the customer's purchase results, the purchase probability estimation system 133 uses behavioral data, including the customer's interest in the products, as a learning dataset to learn the estimation model (step S71).

[0218] For example, by using such an estimation model, the purchase probability estimation system 133 can perform a more detailed analysis of goods with a low purchase probability.

[0219] Next, refer to Figure 25 and Figure 26 This describes an example of how the purchase probability estimation system 133 generates an estimation model that includes not only customer behavior but also customer attributes.

[0220] First, refer to Figure 25 Describe an example of the behavioral information used for model generation in this context. Figure 25 This is a diagram (3) showing an example of behavioral information used by the purchase probability estimation system 133. Figure 25 Data table 122H is shown as an example of a data table in the purchase estimate information storage unit 122.

[0221] like Figure 25As shown, in addition to data table 122G, data table 122H also includes items for "Age," "Gender," and "Companions." That is, data table 122H contains attribute information such as the customer's age and gender, companions (family structure), etc. Note that the customer's attributes are not limited to this example and can include various types of information. For example, attribute information can include all characteristic information of an individual, such as the customer's nationality. Furthermore, attribute information can include all characteristic information about people related to the customer, such as the number of companions and the companions' attribute information. Specifically, the purchase probability estimation system 133 detects customer attributes based on the customer's clothing and facial recognition technology and obtains the detected information.

[0222] exist Figure 25 In the example shown, customer A11 is a male in his 20s who is with a companion and takes the action of looking at a smartphone after paying attention to product E11, and therefore does not buy the product in the store.

[0223] Reference Figure 26 Describe the process of generating the estimation model in this situation. Figure 26 This is a flowchart (3) illustrating the learning process of the purchase probability estimation system 133.

[0224] because Figure 26 The processing from step S60 to step S67 and Figure 22 The processing is similar to that in the previous section, so its description will be omitted. The purchase probability estimation system 133 obtains information about the products that the customer is interested in and the customer's attribute information when detecting customer behavior (step S75).

[0225] Upon obtaining the customer's purchase result, the purchase probability estimation system 133 associates the customer's attributes, including behavioral data on the products the customer is interested in, with the purchase result as a learning dataset to learn the estimation model (step S76).

[0226] By using this estimation model, the purchase probability estimation system 133 can perform a more detailed analysis of, for example, which age group of customers has a low purchase probability.

[0227] Next, the inference process using the estimation model will be described. Figure 27 This is a flowchart (1) showing the estimation process of the purchase probability estimation system 133. Figure 27 The example shows a purchase probability estimation system 133 using a method derived from... Figure 22 An example of the estimated model generated by the process is shown in the figure.

[0228] First, the purchase probability estimation system 133 determines whether the user to be estimated has been detected in the store (step S80). If no user is detected (step S80; no), the purchase probability estimation system 133 waits until a user is detected.

[0229] Upon detecting a user (step S80; Yes), the purchase probability estimation system 133 acquires the user's behavioral data (step S81). Next, the purchase probability estimation system 133 inputs the acquired behavioral data into a model for estimating the purchase probability (step S82). Then, the purchase probability estimation system 133 outputs the user's purchase probability (step S83).

[0230] Next, the purchase probability estimation system 133 will be described using [the following]... Figure 24 An example of the estimated model generated by the process is shown in the figure. Figure 28 This is a flowchart (2) showing the estimation process of the purchase probability estimation system 133.

[0231] First, the purchase probability estimation system 133 determines whether the user to be estimated has been detected in the store (step S85). If no user is detected (step S85; no), the purchase probability estimation system 133 waits until a user is detected.

[0232] If a user is detected (step S85; Yes), the purchase probability estimation system 133 acquires information about the products the user is interested in (step S86). Then, the purchase probability estimation system 133 acquires behavioral data including the actions taken by the user after they have interested in the products (step S87).

[0233] Subsequently, the purchase probability estimation system 133 inputs information and behavioral data about the products the user is interested in into a model for estimating the purchase probability (step S88). Then, the purchase probability estimation system 133 outputs the user's purchase probability (step S89).

[0234] Next, the purchase probability estimation system 133 will be described using [the following]... Figure 26 An example of the estimated model generated by the process is shown in the figure. Figure 29 This is a flowchart (3) showing the estimation process of the purchase probability estimation system 133.

[0235] First, the purchase probability estimation system 133 determines whether the user to be estimated has been detected in the store (step S90). If no user is detected (step S90; No), the purchase probability estimation system 133 waits until a user is detected.

[0236] Upon detecting a user (step S90; Yes), the purchase probability estimation system 133 acquires the user's attributes (step S91). Furthermore, the purchase probability estimation system 133 acquires information about the products the user has shown interest in (step S92). Then, the purchase probability estimation system 133 acquires behavioral data, including actions taken by the user after showing interest in the products (step S93).

[0237] Next, the purchase probability estimation system 133 inputs the user's attributes, information about the products the user is interested in, and behavioral data into the model used to estimate the purchase probability (step S94). Then, the purchase probability estimation system 133 outputs the user's purchase probability (step S95).

[0238] (1-5. Information processing performed by the content generation system)

[0239] return Figure 4 The description content will be generated by system 134. For example... Figure 4 As shown, the content generation system 134 includes a learning unit 134A, a content generation unit 134B, and a display control unit 134C. Note that the data and information used in each process of the content generation system 134, as described later, are stored in the content generation information storage unit 123.

[0240] Learning Unit 134A learns the correlation between customer behavior information and the advertising effectiveness of content presented in the store, and generates a model for content generation.

[0241] The content generation unit 134B generates optimized content based on the user's behavior information to be estimated by using a model that learns the correlation between customer behavior information and the advertising effect of content presented in the store. This optimized content is newly generated for the user.

[0242] The display control unit 134C controls the display device 300, etc., to display optimized content generated by the content generation unit 134B.

[0243] In the following text, for the sake of simplicity, each process performed by the acquisition unit 131, the learning unit 134A, the content generation unit 134B, and the display control unit 134C will be described as being performed by the content generation system 134.

[0244] For example, content generation system 134 uses a model that has learned from customers' purchase history in stores as a basis for understanding advertising effectiveness to generate optimized content. Alternatively, content generation system 134 can use a model that has learned from results such as whether customers who have viewed the content have made payments for goods or services in the store as a basis for understanding purchase history to generate optimized content.

[0245] Furthermore, the content generation system 134 can use a model that has learned the features of location indicators corresponding to gaze information in the content and the customer's purchase history to generate optimized content. Specifically, the content generation system 134 can generate optimized content to emphasize the features of location indicators that the user is looking at in the content based on the user's gaze information.

[0246] Furthermore, the content generation system 134 can generate optimized content using a model that has learned the correlation between customer movement information and customer purchase history in the store. For example, the content generation system 134 obtains the settings information of goods or services in the store associated with the user's movement information (whether it is a discount area or a limited-time area). Then, the content generation system 134 can generate optimized content based on the settings information to emphasize features in the displayed content that indicate the relevance of the settings information.

[0247] Furthermore, the content generation system 134 can use a model that has learned the correlation between customer attribute information and the customer's purchase history in the store to generate optimized content. Additionally, the content generation system 134 can use a model that has learned the correlation between external environmental information (such as weather, temperature, time, and location) and the customer's purchase history when the customer views the content to generate optimized content.

[0248] Reference Figure 30 The following figures illustrate the processing of the above content generation system 134. Figure 30 This is a diagram illustrating an overview of the learning process performed by the content generation system 134.

[0249] like Figure 30 As shown, the content generation system 134 uses an information group 123A that includes various types of information during learning. For example, the content generation system 134 obtains external data from an information server 400 that stores information such as weather and temperature. Furthermore, the content generation system 134 obtains customer movement data from in-store cameras 200A installed in the store. Additionally, the content generation system 134 obtains customer attribute and gaze data from a sign-in camera 200B installed near signs displaying advertising content, etc.

[0250] Furthermore, the content generation system 134 controls the display of advertising data 123B to be displayed on the display device 300A installed in the store. The display of advertising data 123B is a predetermined advertising data image to be displayed on a sign. At this time, the content generation system 134 acquires various types of data (metadata) included in the content, such as what kind of product the displayed advertising content is advertising, what the price is, and what type of image is used in the advertisement at what location.

[0251] In addition, the content generation system 134 obtains purchase information 123D from the POS system 250, including, for example, information indicating the items purchased by customers in the store when they leave the store.

[0252] The content generation system 134 generates a model for generating optimized content based on the acquired information. Note that the following will refer to... Figure 32 Describe the details of the model's learning process.

[0253] Next, we will use Figure 31 This illustrates the process when the content generation system 134 uses a model to generate optimized content. Figure 31 This is a diagram illustrating an overview of the generation process performed by the content generation system 134.

[0254] As in Figure 30 In this process, the content generation system 134 acquires various types of information included in the information group 123A. Furthermore, the content generation system 134 acquires predetermined content data 123E, which serves as a source for optimizing content, and content information 123C, including information such as product prices.

[0255] Then, the content generation system 134 inputs the acquired information into the model and generates optimized content 123F for the user who is the viewer. The content generation system 134 sends the data of the optimized content 123F to the display device 300A and controls the display device 300A to display the optimized content 123F.

[0256] As described above, when displaying content, the content generation system 134 can generate and display the content in real time using a user-related information group, which is expected to have a higher appeal to the user.

[0257] For example, the content generation system 134 can generate content that reflects what types of advertisements users with a certain attribute have already paid attention to and ultimately purchased by using a model that has been trained based on customer attributes (gender, age, etc.). Thus, the content generation system 134 can generate content that has a high purchase propensity among customers with the same attributes, or it can generate content that emphasizes information corresponding to the attribute, and can make the content viewable by users with the same attributes.

[0258] Furthermore, the content generation system 134 acquires portions of the content that the customer is looking at based on the customer's gaze information, and can analyze which information-emphasized content is more effective by using a model that has learned the correlation between this gaze and subsequent purchase results.

[0259] Furthermore, the content generation system 134 obtains information about which product areas customers visit from customer movement information (movement within the store), thereby understanding customer preference trends and generating content that emphasizes those areas. Thus, when a user visits the store, the content generation system 134 can present content that reinforces information matching that user preference.

[0260] In addition, the content generation system 134 can generate content that emphasizes the customer's desired purchase based on the current situation or that matches the products the store wants to promote by acquiring external information (weather, temperature, time zone, region, etc.) and customer information.

[0261] The content generation system 134 can use all these information groups during content generation, or it can perform predetermined weighting based on the advertiser's intent, etc. Furthermore, the content generation system 134 can generate content using only a subset of the information.

[0262] Next, we will refer to Figure 32 Describe the learning process of the content generation system 134. Figure 32 This is a flowchart illustrating the learning process of the content generation system 134.

[0263] First, the content generation system 134 determines whether a customer has been detected in the store (step S101). If no customer has been detected (step S101; no), the content generation system 134 waits until a customer is detected.

[0264] On the other hand, when a customer is detected (step S101; Yes), the content generation system 134 assigns an ID to the customer and obtains each piece of data about the movement and behavior corresponding to that customer (step S102).

[0265] Subsequently, the content generation system 134 displays the predetermined content on the sign (step S103). Note that the predetermined content refers to the original advertising content that has not been optimized or otherwise processed by the content generation system 134.

[0266] Subsequently, the content generation system 134 determines whether a customer has been detected in front of the sign (step S104). If no customer has been detected (step S104; no), the content generation system 134 waits until a customer is detected.

[0267] On the other hand, when a customer is detected (step S104; Yes), the content generation system 134 associates the customer ID to associate each piece of data, such as the customer's past behavior in front of the sign, with the customer's subsequent behavior (step S105).

[0268] Then, the content generation system 134 acquires each piece of data associated with the customer's ID, including data about signage viewing (step S106). For example, the content generation system 134 acquires gaze information, such as which type of information (price information, product images, etc.) the customer is primarily viewing on the signage.

[0269] In addition, the content generation system 134 acquires data about customer purchases, such as POS data (step S107). Then, the content generation system 134 packages data about customer behavior and purchases that may become tags, and records this packaged data as learning data (step S108). When enough learning data has been accumulated, the content generation system 134 generates a model for content generation based on the learning data (step S109).

[0270] Reference Figure 33 Example of a method for generating models in the content generation system 134. Figure 33 This is a diagram illustrating an example of the model generation process of the content generation system 134.

[0271] For example, the content generation system 134 may employ a learning method called Generative Adversarial Network (GAN). In such a learning scenario, learning data 401, consisting of predetermined advertising data and customer data (including various information groups such as movement patterns, eye contact, and purchase results), is transmitted to an advertising generation model (generator) and to a person (advertising designer, etc.) who generates a real advertisement to compare with the advertisement generated by the advertising generation model.

[0272] The advertising designer manually generates an advertisement that is assumed to be most attractive to customers (step S121). On the other hand, the advertising generation model generates an advertisement based on the learning data 401 (step S122). Specifically, in the advertising generation model, the generator (advertising generation model) performs learning by using original advertising data, customer attributes, gaze areas, movement data, etc. as input to generate an advertisement created by the advertising designer (considered a real advertisement).

[0273] The two generated ads are input into the authenticity determination AI model (discriminator) (step S123). The authenticity determination AI model performs learning so that the ad created by the ad designer can be determined as "real" and the ad generated by the ad generation model can be determined as "fake" (step S124). Then, the ad generation model and the authenticity determination AI model share loss values ​​with each other (even though they compete) and perform learning so that the ad generation model can generate ads that are as close to "real" as possible.

[0274] Content generation system 134 uses this method to generate a content generation model for producing content deemed most effective for users. Note that... Figure 33 The model generation method shown is an example, and the content generation system 134 can generate content generation models using any known method.

[0275] Next, the process by which the content generation system 134 generates optimized content for users by using models will be described. Figure 34 This is a flowchart illustrating the content generation process of the content generation system 134.

[0276] First, the content generation system 134 determines whether a user has been detected in the store (step S201). If no user has been detected (step S201; no), the content generation system 134 waits until a user is detected.

[0277] On the other hand, when a user is detected (step S201; Yes), the content generation system 134 assigns an ID to the user and obtains each piece of data about the movement and behavior corresponding to that user (step S202).

[0278] Subsequently, the content generation system 134 displays the predetermined content on the sign (step S203). Then, the content generation system 134 determines whether a user has been detected in front of the sign (step S204). If no user is detected (step S204; No), the content generation system 134 waits until a user is detected.

[0279] On the other hand, when a user is detected (step S204; yes), the content generation system 134 associates the user ID with each piece of data such as the user's past behavior in front of the sign to associate the user ID with the user (step S205).

[0280] Then, the content generation system 134 acquires each piece of data associated with the user ID, including data about sign viewing (step S206). For example, the content generation system 134 acquires gaze information, such as which type of information (price information, product images, etc.) the user is primarily viewing on the sign.

[0281] The content generation system 134 inputs the acquired information into the content generation model and generates content (optimized content) corresponding to the user (step S207). Then, the content generation system 134 switches from the predetermined content to the optimized content and displays the optimized content on a display device 300 such as a sign (step S208).

[0282] Note that the content generation system 134 can further learn its content generation model by using information from users who have viewed optimized content. (Refer to...) Figure 35 Describe this. Figure 35 This is a flowchart illustrating the relearning process of the content generation system 134.

[0283] The content generation system 134 determines whether the user who has viewed the optimized content has left the store (step S211). If the user has not left the store (step S211; no), the content generation system 134 waits until the user leaves the store.

[0284] On the other hand, if the user has left the store (step S211; Yes), the content generation system 134 acquires POS data indicating the user's purchase results, etc. (step S212). The content generation system 134 records information about the user and the POS data in association as learning data (step S213).

[0285] Subsequently, the content generation system 134 relearns the content generation model using the recorded learning data (step S214).

[0286] As described above, by acquiring the user's purchase history after watching optimized content, the content generation system 134 can relearn the model based on the user's behavior information and purchase history after watching optimized content.

[0287] For example, even if the content generation system 134 generates optimized content, there are still cases where the optimized content fails to lead to a purchase or user satisfaction decreases. Therefore, the content generation system 134 can generate a model that further relearns by using user purchasing behavior after viewing optimized content, etc., to generate content with higher advertising effectiveness. For example, if users who have viewed optimized content emphasizing price information are more likely to purchase goods or make higher-priced purchases, the content generation system 134 can relearn to generate a model that emphasizes this information with a higher probability.

[0288] Furthermore, the content generation system 134 can, after displaying optimized content once, further utilize user information to perform control to change the content. For example, the content generation system 134 can generate optimized content (hereinafter referred to as "first content" for distinction) using information about the user before they arrive at the sign, and can generate modified content (hereinafter referred to as "second content" for distinction) using subsequent information about the user. (See reference...) Figure 36 Describe this. Figure 36 This is a flowchart illustrating the second content generation process of the content generation system 134.

[0289] Content generation system 134 Figure 34 After step S206 shown, the first content is generated (step S221). The content generation system 134 displays the first content on the display device 300 (step S222).

[0290] Subsequently, the content generation system 134 obtains information about the user who has viewed the first content (step S223). For example, the content generation system 134 obtains information indicating which area of ​​the first content the user is looking at.

[0291] The content generation system 134 generates second content based on the acquired data (step S224). For example, the content generation system 134 generates second content that emphasizes the area of ​​the first content that the user is looking at.

[0292] Then, the content generation system 134 switches the first content to the second content and displays the second content on the display device 300 (step S225).

[0293] As described above, after generating the first content, the content generation system 134 can generate second content based on the acquired behavioral information of users who have viewed the first content. For example, the content generation system 134 generates second content to emphasize information about the location corresponding to the user's gaze information (i.e., information about the portion of the first content that the user is looking at). Therefore, the content generation system 134 can provide users with more real-time content.

[0294] Will use Figure 37 and Figure 38 Examples of optimized content generated as the first or second content. Figure 37 This is an illustration showing an example of content optimized based on user 10's gaze.

[0295] Content 80 is advertising content used to promote a product. Content 80 includes multiple pieces of information such as the product name, product price, product image, and discount information.

[0296] Content generation system 134 detects the gaze information of user 10 viewing content 80. At this time, assume user 10 is looking at discount information 81 indicating a lower price in content 80. In this case, content generation system 134 modifies content 80 to emphasize the discount information 81 more than before. For example, content generation system 134 displays the discount information 81 in a larger size or with a more prominent color than before.

[0297] Alternatively, suppose user 10 is looking at product image 82 in content 80. In this case, content generation system 134 modifies content 80 to emphasize product image 82 more than before. For example, content generation system 134 displays product image 82 in a larger size than before, displays product image in a more prominent color, or changes the overall layout of content 80 to make the image of the product or product packaging more prominent.

[0298] Notice, Figure 37 The gaze information shown can include not only the area of ​​gaze but also the duration of gaze. For example, in the case where user 10 gazes at multiple areas of content 80, the area with the longest gaze duration can be emphasized.

[0299] Another example of optimized content generated by content generation system 134 will be described. Figure 38 This is a diagram illustrating an example of content optimized for user 10's movement.

[0300] Before user 10 views content 80, content generation system 134 acquires movement information indicating which areas of the store user 10 has already passed through. For example, suppose user 10 spends more time or lingers longer in the promotional area with many inexpensive items. In this case, content generation system 134 modifies content 80 to emphasize discount information 83 more than before.

[0301] Alternatively, suppose user 10 spends more time in the store, particularly in areas with many limited-time items. In this case, content generation system 134 modifies content 80 to emphasize the limited-time information 84 more than before.

[0302] As described above, the content generation system 134 can further improve the advertising effectiveness of the content by generating content in real time based on the user's behavior in the store.

[0303] (2. Other implementation methods)

[0304] In addition to each of the above embodiments, the processing according to each of the above embodiments can be carried out in various different modes.

[0305] In the processes described in the above embodiments, all or part of the processes described as automatically executed can be performed manually, or all or part of the processes described as manually executed can be performed automatically by known methods. Furthermore, unless otherwise stated, the processing procedures, specific names, and information including various data and parameters shown in the documents and figures can be arbitrarily changed. For example, the various types of information shown in each figure are not limited to the information shown.

[0306] Furthermore, each component of each device shown in the accompanying drawings is functionally conceptual and does not necessarily have to be physically configured as shown in the drawings. That is, the specific form of dispersion and integration of each device is not limited to the form shown, and all or part of it can be functionally or physically dispersed and integrated in any unit according to various loads, usage conditions, etc. For example, the processing performed by each functional unit (control unit 130) of the information processing device 100 of this disclosure can be performed by an edge terminal such as a sensor device 200. In addition, the information processing device 100 can be an edge terminal itself with shooting capabilities, such as a camera.

[0307] Furthermore, the above-described embodiments and variations can be appropriately combined without conflicting with the processing content.

[0308] Furthermore, the effects described in this specification are merely illustrative and not limited, and other effects may be provided.

[0309] (3. Effects of the information processing device according to this disclosure)

[0310] As described above, the information processing apparatus according to this disclosure (information processing apparatus 100 in the embodiments) includes an acquisition unit (acquisition unit 131 in the embodiments) and an estimation unit (state estimation system 132 in the embodiments). The acquisition unit acquires movement information indicating the movement of a customer in a predetermined space. The estimation unit uses a first model (state estimation model 50 in this example) that has learned the correlation between movement information and the outcome of customer behavior in the space to estimate the future behavior or psychological state of the user located in the space from the movement information of the user to be estimated. For example, the acquisition unit acquires movement information from images acquired by an imaging device installed in the predetermined space.

[0311] As described above, the information processing apparatus according to this disclosure can estimate what kind of behavior a user wants to perform in the future or what psychological state a user is in by analyzing the user's movement using a model that has learned the correlation between movement paths and behavioral outcomes. That is, the information processing apparatus can accurately estimate a user's behavior and psychological state by analyzing the possible purchasing activities and the user's psychological state that may occur along the movement path.

[0312] Furthermore, the designated space is a store, and the estimation unit uses a first model that has learned the correlation between movement information and customers' purchase history in the store to estimate the user's purchasing behavior in the store based on the user's movement information. For example, the estimation unit estimates whether the user has made a payment for goods or services in the store as the user's purchasing behavior. Additionally, the estimation unit can estimate the amount paid by the user for goods or services in the store as the user's purchasing behavior.

[0313] As described above, by using a model that learns the correlation between movement patterns and purchasing behavior to analyze user movement patterns, the information processing device can accurately estimate the purchasing behavior of a user's identity.

[0314] The estimation unit utilizes user movement information by employing a first model that learns the correlation between customer movement information after they remain in the store and customer evaluations to estimate the psychological state of users located in the store. For example, the estimation unit estimates whether users located in the store are satisfied with the services provided by the store based on user movement information.

[0315] As described above, the information processing device can accurately estimate the user's psychological state by using a model that learns the correlation between movement patterns and store ratings based on customer feedback.

[0316] In addition, the estimation unit determines whether to estimate the user's purchasing behavior or psychological state based on the store's crowding situation.

[0317] As described above, the information processing device can accurately estimate information useful for store operations by changing the analysis objectives in real time according to the store's situation.

[0318] The acquisition unit acquires user attributes and movement information corresponding to the movement information. The estimation unit, using a first model that has learned the correlation between movement information and attributes, estimates the user's future behavior or psychological state in the space, as well as the customer's behavioral outcome in the space, based on the movement information of the user to be estimated.

[0319] In this way, the information processing device can further improve the accuracy of the estimation by performing estimation processing including attributes.

[0320] The acquisition unit acquires customer behavior information and customer purchase results in the store. The estimation unit (purchase probability estimation system 133 in this embodiment) uses a second model (purchase estimation model 60 in this example) that has learned the correlation between behavioral information and customer purchase results to estimate the probability of a user making a purchase in the store based on the user's behavioral information.

[0321] As described above, information processing devices can accurately estimate whether a user will make a purchase by analyzing user behavior using not only movement patterns but also models that learn the correlation between customers' unintentional behaviors or gestures and purchase outcomes.

[0322] Additionally, the acquisition unit obtains customer behavior information after they have focused on the products or services offered by the store, as well as their purchase outcomes within the store. The estimation unit estimates the probability of a user making a purchase in the store based on the products or services the user has focused on and the user's behavioral information, using a second model that has learned the correlation between customer behavior information after they have focused on the products or services and their purchase outcomes within the store.

[0323] As described above, the information processing device learns a series of behaviors, including the first behavior of paying attention to a product and the second behavior such as subsequent behaviors or gestures, and therefore can accurately estimate the probability of whether a user will make a purchase based on user behaviors that seem unrelated to the purchase.

[0324] Additionally, the acquisition unit acquires customer behavior information and attributes within the store, as well as customer purchase outcomes. The estimation unit estimates the probability of a user making a purchase in the store based on the user's behavior information and attributes by using a second model that has learned the correlation between behavioral information and attributes and customer purchase outcomes. In this case, the acquisition unit can acquire the customer and the customer's purchase outcome in a way that is correlated with at least one of the following: facial recognition processing based on images of the customer, specifications of store fixtures used by the customer, and processing of matching the customer's clothing based on the images.

[0325] In this way, the information processing device can further improve the accuracy of the estimation by performing estimation processing including attributes.

[0326] Furthermore, the information processing apparatus according to this disclosure may be configured to include an acquisition unit (acquisition unit 131 in the embodiment) and a generation unit (content generation system 134 in the embodiment). The acquisition unit acquires behavioral information of customers in a predetermined space. The generation unit generates optimized content (newly generated content for the user) based on the user's behavioral information by using a model (content generation model 70 in the embodiment) that has learned the correlation between the behavioral information and the advertising effect of the content presented in the space.

[0327] As described above, the information processing apparatus according to this disclosure can present more effective digital signage and the like to users by generating optimized content based on information about the user, etc. That is, the information processing apparatus can generate content with higher advertising effectiveness for each individual user.

[0328] The acquisition unit acquires at least one of the following as behavioral information: movement information indicating the customer's movement within the space, gaze information indicating the customer's gaze history regarding the content, customer attribute information, and external environmental information when the customer views the content. For example, when acquiring movement information, the acquisition unit acquires movement information by analyzing images acquired by an imaging device installed in the space. When acquiring gaze information, the acquisition unit acquires gaze information by analyzing the user's gaze using sensors included in the display device displaying the content.

[0329] As described above, by acquiring various types of information about users, the information processing device can precisely perform optimizations for each user.

[0330] Furthermore, the designated space is a store, and the generation unit uses a model learned from customers' purchase history in the store as an indicator of advertising effectiveness to generate optimized content. For example, when viewing content, the acquisition unit acquires the customer's gaze information. The generation unit generates optimized content by using a model that has learned features of location indicators in the content corresponding to the gaze information and the customer's purchase history. Additionally, the acquisition unit acquires gaze information while the user views the content. The generation unit generates optimized content to emphasize features of location indicators in the content based on the user's gaze information.

[0331] In this way, information processing devices can further enhance the advertising effect on users by generating content such as prices and images that emphasize the user's gaze.

[0332] The acquisition unit gathers customer movement information within the store before viewing content. The generation unit uses a model that has learned the correlation between movement information and the customer's purchase history within the store to generate optimized content. For example, the acquisition unit gathers settings information for products or services in the store that are associated with the user's movement information. The generation unit generates optimized content based on this settings information to emphasize features in the displayed content that are relevant to those settings.

[0333] As mentioned above, because the information processing device optimizes content based on the user's movement within the store, it can generate content based on the user's interests and behaviors.

[0334] The acquisition unit gathers customer attribute information. The generation unit generates optimized content using a model that learns the correlation between customer attribute information in the store and their purchase history. Furthermore, while the customer views the content, the acquisition unit can gather external environmental information, including at least one of weather, temperature, time, and region. The generation unit generates optimized content using a model that learns the correlation between external environmental information and the customer's purchase history in the store.

[0335] As mentioned above, by optimizing content using user attributes and information about the user's external environment, the information processing device can generate more content based on the user's interests or generate content according to the situation.

[0336] Furthermore, after generating optimized content, if the generation unit further acquires behavioral information of users who have viewed the optimized content, it can generate second optimized content based on the acquired behavioral information. Additionally, after generating optimized content, if the generation unit further acquires gaze information of users who have viewed the optimized content, it can generate second optimized content based on the acquired gaze information to emphasize information in the optimized content corresponding to the gaze information.

[0337] In this way, the information processing device can perform phased generation processes, such as further modifying and optimizing the content. Thus, the information processing device can provide users with real-time content, such as advertisements.

[0338] Furthermore, when the purchase history of users who have viewed optimized content is obtained, the generation unit relearns the model based on the behavioral information of users who have viewed optimized content and their purchase history.

[0339] As described above, the information processing device can improve the accuracy of generating more user-friendly content by further learning the model in response to feedback on optimized content.

[0340] Furthermore, the generation unit generates optimized content using a model that learns from purchase history, such as whether customers who view the content have already made payments for goods or services in the store. Additionally, the generation unit can generate optimized content using a model that learns from purchase history, such as the amount customers who view the content have paid for goods or services in the store.

[0341] As described above, the information processing device uses purchase results as labels to learn the model, enabling the generation of advertising content that demonstrates a more positive effect in driving purchases.

[0342] (4. Hardware Configuration)

[0343] For example, by having such Figure 39 The computer 1000 configured as shown implements an information apparatus such as the information processing apparatus 100 according to each of the above embodiments. Hereinafter, the information processing apparatus 100 according to this disclosure will be described as an example. Figure 39This is a hardware configuration diagram illustrating an example of a computer 1000 that implements the functions of the information processing device 100. The computer 1000 includes a CPU 1100, RAM 1200, read-only memory (ROM) 1300, hard disk drive (HDD) 1400, communication interface 1500, and input / output interface 1600. Each unit of the computer 1000 is connected via a bus 1050.

[0344] The CPU 1100 operates based on programs stored in ROM 1300 or HDD 1400 and controls each unit. For example, the CPU 1100 loads programs stored in ROM 1300 or HDD 1400 into RAM 1200 and executes processing corresponding to various programs.

[0345] ROM 1300 stores boot programs, such as the Basic Input / Output System (BIOS) executed by CPU 1100 when computer 1000 is activated, and programs that depend on the hardware of computer 1000.

[0346] HDD 1400 is a computer-readable recording medium that non-transitoryly records a program executed by CPU 1100, data used by that program, etc. Specifically, HDD 1400 is a recording medium that records an information processing program according to this disclosure, exemplified as program data 1450.

[0347] Communication interface 1500 is an interface for connecting computer 1000 to an external network 1550 (e.g., the Internet). For example, CPU 1100 receives data from another device or sends data generated by CPU 1100 to another device via communication interface 1500.

[0348] Input / output interface 1600 is an interface for connecting input / output device 1650 and computer 1000. For example, CPU 1100 receives data from input devices such as keyboard and mouse via input / output interface 1600. Furthermore, CPU 1100 transmits data to output devices such as display, speaker, or printer via input / output interface 1600. Additionally, input / output interface 1600 can be used as a media interface for reading programs recorded on a predetermined recording medium. For example, the medium is an optical recording medium such as digital multifunction optical disc (DVD) or phase-change rewritable optical disc (PD), a magneto-optical recording medium such as magneto-optical disc (MO), magnetic tape, magnetic recording medium, semiconductor memory, etc.

[0349] For example, when the computer 1000 is used as the information processing apparatus 100 according to this embodiment, the CPU 1100 of the computer 1000 implements the functions of the control unit 130, etc., by executing the information processing program loaded on the RAM 1200. Furthermore, the HDD 1400 stores the information processing program and data according to this disclosure in the storage unit 120. Note that the CPU 1100 reads program data 1450 from the HDD 1400 and executes the program data; however, as another example, these programs can be obtained from another device via an external network 1550.

[0350] Note that this technology can also be configured as follows.

[0351] (1) An information processing device, comprising:

[0352] The acquisition unit acquires information about customer behavior within the designated space; and

[0353] The generation unit generates optimized content based on the user's estimated behavior information by using a model that learns the correlation between behavioral information and the advertising effectiveness of content presented in space. The optimized content is newly generated content for the user.

[0354] (2) The information processing device according to (1), wherein

[0355] The acquisition unit acquires at least one of the following information as behavioral information: movement information indicating the customer's movement in the space, gaze information indicating the customer's gaze history about the content, customer attribute information, and external environment information when the customer views the content.

[0356] (3) The information processing device according to (2), wherein

[0357] The acquisition unit obtains movement information by analyzing images acquired by imaging devices installed in the space, while acquiring movement information itself.

[0358] (4) The information processing device according to (2) or (3), wherein

[0359] When acquiring gaze information, the acquisition unit acquires gaze information by analyzing the customer's gaze using sensors included in the display device displaying the content.

[0360] (5) An information processing device according to any one of (1) to (4), wherein

[0361] The reserved space is a shop, and

[0362] The generation unit uses a model that has been learned from customers’ purchase history in stores as a basis for advertising effectiveness to generate optimized content.

[0363] (6) The information processing device according to (5), wherein

[0364] The acquisition unit captures information about the customer's gaze while viewing content, and

[0365] The generation unit generates optimized content by using a model that learns the features of location indicators corresponding to gaze information in the content and the customer's purchase history.

[0366] (7) The information processing device according to (6), wherein

[0367] The acquisition unit acquires gaze information when the user is viewing content, and

[0368] The generation unit generates optimized content based on the user's gaze information to emphasize features in the content indicated by the position the user is looking at.

[0369] (8) An information processing apparatus according to any one of (5) to (7), wherein

[0370] The acquisition unit obtains information about the customer's movement within the store before viewing the content, and

[0371] The generating unit generates optimized content by using a model that learns the correlation between traffic flow information and customers' purchase history in the store.

[0372] (9) The information processing device according to (8), wherein

[0373] The acquisition unit acquires settings information for goods or services in the store associated with the user's movement information, and

[0374] The generation unit generates optimized content based on the settings information to emphasize features in the content that are relevant to the settings information.

[0375] (10) An information processing apparatus according to any one of (5) to (9), wherein

[0376] The acquisition unit obtains customer attribute information, and

[0377] The generation unit uses a model that has learned the correlation between attribute information and customers’ purchase history in the store to generate optimized content.

[0378] (11) An information processing apparatus according to any one of (5) to (10), wherein

[0379] The acquisition unit acquires external environmental information, including at least one of the following: weather, temperature, time, and region when the customer is viewing the content:

[0380] The generating unit uses a model that has learned the correlation between external environmental information and customers’ purchase history in the store to generate optimized content.

[0381] (12) An information processing apparatus according to any one of (5) to (11), wherein

[0382] After generating optimized content, the generation unit further obtains the behavioral information of users who have viewed the optimized content, and then generates second optimized content based on the obtained behavioral information.

[0383] (13) The information processing device according to (12), wherein

[0384] After generating optimized content, the generation unit further obtains the gaze information of users who have viewed the optimized content, and generates second optimized content based on the obtained gaze information to emphasize the information in the optimized content that corresponds to the gaze information.

[0385] (14) An information processing apparatus according to any one of (5) to (13), wherein

[0386] The generation unit relearns the model based on the user's behavior information and purchase history after obtaining the purchase history of users who have viewed the optimized content.

[0387] (15) An information processing apparatus according to any one of (5) to (14), wherein

[0388] The generation unit generates optimized content by using a model that learns from the purchase history of customers who have viewed the content, such as whether they have made payments for goods or services in the store.

[0389] (16) An information processing apparatus according to any one of (5) to (15), wherein

[0390] The generation unit generates optimized content by using a model that has learned from the purchase history of customers who have viewed the content and paid for goods or services in the store.

[0391] (17) An information processing method that enables a computer to perform:

[0392] Obtain information about customer behavior within the reserved space; and

[0393] By using a model that learns the correlation between behavioral information and the advertising effectiveness of content presented in space, optimized content is generated based on the user's behavioral information to be estimated. This optimized content is newly generated for the user.

[0394] (18) An information processing program that enables a computer to be used as an information processing apparatus, the information processing apparatus comprising:

[0395] The acquisition unit acquires information about customer behavior within the designated space; and

[0396] The generation unit generates optimized content based on the user's estimated behavior information by using a model that learns the correlation between behavioral information and the advertising effectiveness of content presented in space. This optimized content is newly generated for the user.

[0397] Reference number list

[0398] 1. Information Processing System

[0399] 100 Information Processing Device

[0400] 110 Communication Unit

[0401] 120 storage units

[0402] 121 State estimation information storage unit

[0403] 122 Purchase estimated information storage unit

[0404] 123 Content Generation Information Storage Unit

[0405] 130 Control Unit

[0406] 131 Acquisition Unit

[0407] 132 State Estimation System

[0408] 133 Purchase Probability Estimation System

[0409] 134 Content Generation System

[0410] 200 sensor devices

[0411] 300 Display device.

Claims

1. An information processing apparatus, comprising: The acquisition unit acquires information about customer behavior within the designated space. as well as The generation unit generates optimized content based on the user's estimated behavior information by using a model that has learned the correlation between the behavioral information and the advertising effect of the content presented in the space. The optimized content is newly generated content for the user.

2. The information processing apparatus according to claim 1, wherein, The acquisition unit acquires at least one of the following information as the behavioral information: movement information indicating the customer's movement path in the space, gaze information of the customer's gaze history regarding the content, attribute information of the customer, and external environment information when the customer views the content.

3. The information processing apparatus according to claim 2, wherein, The acquisition unit acquires the movement information by analyzing images acquired by an imaging device installed in the space.

4. The information processing apparatus according to claim 2, wherein, When acquiring the gaze information, the acquisition unit acquires the gaze information by analyzing the customer's gaze using sensors included in the display device displaying the content.

5. The information processing apparatus according to claim 1, wherein, The reserved space is a shop, and The generation unit uses a model that has been learned from the customer's purchase history in the store as a basis for the effectiveness of the advertising to generate the optimized content.

6. The information processing apparatus according to claim 5, wherein, The acquisition unit acquires the customer's gaze information while viewing the content, and the generation unit generates the optimized content by using a model that has learned the features of the position indicators in the content corresponding to the gaze information and the customer's purchase history.

7. The information processing apparatus according to claim 6, wherein, The acquisition unit acquires the user's gaze information while viewing the content, and The generation unit generates the optimized content based on the user's gaze information to emphasize the features of the position indicated by the user's gaze in the content.

8. The information processing apparatus according to claim 5, wherein, The acquisition unit acquires the customer's movement information in the store before viewing the content, and The generation unit generates the optimized content by using a model that learns the correlation between the movement information and the customer's purchase history in the store.

9. The information processing apparatus according to claim 8, wherein, The acquisition unit acquires the settings information of the goods or services in the store associated with the user's movement information, and The generation unit generates the optimized content based on the setting information to emphasize the features in the content that are related to the setting information.

10. The information processing apparatus according to claim 5, wherein, The acquisition unit acquires the customer's attribute information, and The generation unit uses a model that has learned the correlation between the attribute information and the customer's purchase history in the store to generate the optimized content.

11. The information processing apparatus according to claim 5, wherein, The acquisition unit acquires external environment information, which includes at least one of the following: weather, temperature, time, and region when the customer is viewing the content: The generation unit uses a model that has learned the correlation between the external environment information and the customer's purchase history in the store to generate the optimized content.

12. The information processing apparatus according to claim 5, wherein, After generating the optimized content, the generation unit further obtains the behavior information of users who have viewed the optimized content, and then generates a second optimized content based on the obtained behavior information.

13. The information processing apparatus according to claim 12, wherein, After generating the optimized content, the generation unit further obtains the gaze information of users who have viewed the optimized content, and generates the second optimized content based on the obtained gaze information to emphasize the information in the optimized content that corresponds to the gaze information.

14. The information processing apparatus according to claim 5, wherein, The generation unit, upon acquiring the purchase history of users who have viewed the optimized content, relearns the model based on the behavioral information of the users who have viewed the optimized content and their purchase history.

15. The information processing apparatus according to claim 5, wherein, The generation unit generates the optimized content by using a model that learns from the purchase history of customers who have viewed the content, based on whether they have made payments for goods or services in the store.

16. The information processing apparatus according to claim 5, wherein, The generation unit generates the optimized content by using a model that has learned from the purchase history of customers who have viewed the content and paid for goods or services in the store.

17. An information processing method that causes a computer to perform: Obtain information about customer behavior within the reserved space; and By using a model that learns the correlation between the behavioral information and the advertising effectiveness of the content presented in the space, optimized content is generated based on the user's behavioral information to be estimated, the optimized content being newly generated for the user.

18. An information processing program that enables a computer to be used as an information processing device, the information processing device comprising: The acquisition unit acquires information about customer behavior within the designated space. as well as The generation unit generates optimized content based on the user's estimated behavior information by using a model that has learned the correlation between the behavioral information and the advertising effect of the content presented in the space. The optimized content is newly generated content for the user.

Images