Policy decision-making device, policy decision-making method, and policy decision-making program

Abstract

In a technology that uses reinforcement learning to differentiate content output to a content output device based on a person's attributes, we aim to achieve relatively high accuracy in content differentiation even when the amount of data available for training the trained model is relatively small, by utilizing external information. [Solution] The policy decision device 100, which uses reinforcement learning to differentiate content output to a content output device according to the attributes of the target person, comprises a weight calculation unit 171 and a policy decision unit 173. The weight calculation unit 171 calculates policy adjustment weights based on attribute distribution information that shows a prediction of the distribution of attributes of the appearing person, so that exploration is prioritized for attributes with a relatively high observation probability, and utilization is prioritized for attributes with a relatively low observation probability. The policy decision unit 173 determines the content to be presented to the target person based on the expected reward corresponding to each content, the confidence interval corresponding to each expected reward, and the policy adjustment weights.

Description

Technical Field

[0001] The present disclosure relates to a policy decision-making apparatus, a policy decision-making method, and a policy decision-making program.

Background Art

[0002] There is known a technique for selecting content to be displayed on a digital signage according to the attributes of a person viewing the digital signage or the situation when a person is present in front of the digital signage. In online advertising, there is known a technique for sorting content by a user, as a technique utilizing reinforcement learning that utilizes log data such as click-through rates. Patent Document 1 discloses a technique for identifying a person in an image captured by an imaging unit attached to a signage device and playing predetermined content based on the position of the signage, the date and time, and the age and gender of the person.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the online marketing field where many technologies similar to the technology for promoting in-facility circulation by sorting optimal content on digital signage according to attributes and conditions are used, content is sorted using user information and operation log data such as clicks. On the other hand, in an offline medium such as digital signage, since the number of viewers is relatively small, there is little data that can be used for learning. Therefore, it is highly likely that content cannot be appropriately sorted by an approach similar to that of online advertising.

[0005] This disclosure aims to achieve relatively high accuracy in content delivery to a content output device by utilizing external information, even when there is relatively little data available for training a pre-trained model, by leveraging reinforcement learning to deliver content to a content output device according to a person's attributes. [Means for solving the problem]

[0006] The policy decision-making mechanism related to this disclosure is: A policy decision device that utilizes reinforcement learning to differentiate the content output of a content output device according to the attributes of the target person of the policy, so that the output of the content output device can be an output of interest, A weight calculation unit calculates policy adjustment weights based on attribute distribution information showing a prediction of the distribution of attributes of persons whose output can be the output of interest, which weights correspond to the confidence interval of the expected reward corresponding to each content shown in the policy list information, and are weights corresponding to the probability that a person having the attributes of the policy target person can make the output of interest, such that exploration is prioritized for attributes that have a relatively high probability of being observed as attributes of a person whose output can be the output of interest, and utilization is prioritized for attributes that have a relatively low probability of being observed as attributes of a person whose output can be the output of interest, A policy decision unit determines the content to be presented to the policy target person based on the expected rewards corresponding to each content shown in the policy list information derived as each expected reward in the expected reward group corresponding to the attributes of the policy target person, the confidence interval corresponding to each expected reward in the expected reward group, and the policy adjustment weight. It is equipped with. [Effects of the Invention]

[0007] The policy decision-making device described in this disclosure utilizes reinforcement learning to output content to a content output device according to the attributes of the target individual. In this process, a weight calculation unit calculates policy adjustment weights based on attribute distribution information, which act to prioritize exploration for attributes with a relatively high observation probability and prioritize utilization for attributes with a relatively low observation probability. The policy decision-making unit then determines the content to present to the target individual based on each expected reward in the expected reward group, the confidence interval corresponding to each expected reward, and the policy adjustment weights. Therefore, according to this disclosure, in a technology that uses reinforcement learning to differentiate content output to a content output device according to a person's attributes, by utilizing external information, it is possible to differentiate content with relatively high accuracy even when there is relatively little data available to train the trained model. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example configuration of the policy decision system 90 according to Embodiment 1. [Figure 2] A diagram illustrating the policy result storage unit 111 according to Embodiment 1. [Figure 3] A diagram illustrating the policy list storage unit 113 according to Embodiment 1. [Figure 4] A diagram illustrating the attribute distribution storage unit 115 according to Embodiment 1. [Figure 5] A diagram showing an example of the hardware configuration of the policy decision-making device 100 according to Embodiment 1. [Figure 6] A diagram illustrating the overview of the operation of the policy decision system 90 according to Embodiment 1. [Figure 7] This diagram illustrates the outline of the policy decision algorithm according to Embodiment 1, where (a) is a diagram illustrating the bandit algorithm, (b) is a diagram illustrating the conventional method and the proposed method, and (c) is a diagram illustrating the confidence interval. [Figure 8] A diagram illustrating the policy decision algorithm according to Embodiment 1. [Figure 9] A diagram illustrating the fairness of the policy decision algorithm according to Embodiment 1. [Figure 10] A diagram for explaining the policy decision algorithm according to Embodiment 1. [Figure 11] A diagram for explaining the policy decision algorithm according to Embodiment 1, where (a) is a diagram for explaining the case where emphasis is placed on "search", (b) is a diagram for explaining the case where emphasis is placed on "utilization", and (c) is a diagram for explaining the case where adjustment is made based on prior knowledge. [Figure 12] A flowchart showing the operation of the policy decision algorithm according to Embodiment 1. [Figure 13] A flowchart showing the operation of the learning step of the policy decision device 100 according to Embodiment 1. [Figure 14] A flowchart showing the operation of the inference step of the policy decision device 100 according to Embodiment 1. [Figure 15] A diagram for explaining the processing of the inference processing unit 172 according to Embodiment 1. [Figure 16] A diagram for explaining the processing of the weight calculation unit 171 according to Embodiment 1. [Figure 17] A diagram for explaining the processing of the policy decision unit 173 according to Embodiment 1. [Figure 18] A diagram showing an example of the hardware configuration of the policy decision device 100 according to a modification of Embodiment 1.

Modes for Carrying Out the Invention

[0009] In the description of the embodiments and the drawings, the same elements and corresponding elements are denoted by the same reference numerals. The description of the elements denoted by the same reference numerals may be omitted or simplified as appropriate. The arrows in the drawings mainly indicate the flow of data or the flow of processing. Also, "section" may be appropriately read as "circuit", "device", "equipment", "process", "step", "procedure", "processing", or "circuitry". The functions of each part included in each device may be realized by firmware, software, hardware, or a combination thereof.

[0010] Embodiment 1. Hereinafter, this embodiment will be described in detail with reference to the drawings.

[0011] ***Explanation of the structure*** Figure 1 shows an example configuration of the policy decision system 90 according to this embodiment. As shown in Figure 1, the policy decision system 90 comprises a policy decision device 100, a policy result collection device 200, an attribute information collection device 300, and an attribute distribution collection device 400. The multiple devices comprising the policy decision system 90 may be configured as an integrated unit as appropriate.

[0012] The policy decision-making device 100 utilizes reinforcement learning that employs expected rewards to output content to the content output device according to the attributes of the person targeted by the policy. The policy decision-making device 100 comprises a storage unit 110 and a control unit 150. The policy decision-making device 100 and the content output device may be configured as an integrated unit. A content output device is a device that has the function of outputting content. Specific examples of content output devices include digital signage, smartphones, PCs (personal computers), or speakers. The following explanation will primarily assume that the content output device is digital signage. If the content output device is a device other than digital signage, the same processing as when the content output device is digital signage will be performed. The expected reward is the expected value of the reward corresponding to each content item shown in the policy list information. In the initial state, when a trained model has not yet been generated, each expected reward may be a reward randomly generated based on a set of prior distributions. The prior distribution group consists of multiple prior distributions, each corresponding to a different piece of content shown in the policy list information, and each corresponding to a reward corresponding to the attributes of the target individual of the policy. A person targeted by the measures is a person who can consider the output of the content output device as an output of interest, and is a person targeted by the measures of the measures decision device 100. Specific examples of a person targeted by the measures include a person who is viewing the content output device, a person who may view the content output device, a person who is near the content output device, or a person who is expected to reach the vicinity of the content output device within a certain period of time. A person targeted by the measures does not necessarily have to actually view the content output device. A person targeted by the measures may also be a person who is located within the area targeted by the measures. A person targeted by the measures may consist of multiple people. A person targeted by the measures may also be called a user. Specific examples of a person targeted by the measures are detected based on images captured by cameras installed within the facility or by cameras equipped on the measures presentation device. Whether or not a person targeted by the measures perceived the output of the content output device, and the duration for which a person targeted by the measures perceived the output of the content output device, may be determined based on images captured by cameras equipped on the measures presentation device. An output of interest is, as a concrete example, the output of a device located within the area targeted by the policy, or an audible sound within the area targeted by the policy. An output of interest corresponding to a person is an output that the person can actually be interested in, or an output that is considered to be of interest to the person. If it is determined that a person can consider a certain output of interest, as a concrete example, the person can perceive that output for at least a portion of the time elapsed from the time of determination until a certain period of time has elapsed. The area targeted by the measures is the area corresponding to the output of the content output device. The scope of the area targeted by the measures may be defined in any way. When the content output device is digital signage, the area targeted by the measures is the area in which the digital signage is installed. The area targeted by the measures may be an area in which the sound output by the speaker can be heard, a specific area within a web page, multiple web pages as a whole, or an area within a virtual space. Reinforcement learning is exemplified by the bandit algorithm. Content can be images, videos, or audio, for example. Each piece of content shown in the campaign list information may be an advertisement. The expected reward corresponding to each piece of content shown in the campaign list information may be a value corresponding to the effectiveness of each piece of content. If each piece of content shown in the campaign list information is a still image, the expected reward corresponding to each piece of content may be determined based on whether or not the target person views each piece of content. If each piece of content shown in the campaign list information is a video, the expected reward corresponding to each piece of content may be determined based on the amount of time the target person continues to view each piece of content. The memory unit 110 includes a policy result memory unit 111, an attribute memory unit 112, a policy list memory unit 113, a trained model memory unit 114, and an attribute distribution memory unit 115. The control unit 150 comprises a learning unit 160 and an inference unit 170.

[0013] The policy result storage unit 111 stores information indicating the policy results collected by the policy result collection device 200. The results of the initiative are data obtained by displaying content on the signage. The results of the initiative may also be the reward corresponding to the initiative. The results of the initiative may also be the total number of seconds that the target person viewed the signage, information indicating whether or not a coupon corresponding to the initiative was acquired or used, the amount of coupons used corresponding to the initiative, sales, number of units sold, or number of purchasers of the product corresponding to the initiative, sales or number of users of the service corresponding to the initiative, or sales, number of purchasers, or number of visitors at the store corresponding to the initiative. If the content is a still image, the results of the initiative may also be information indicating whether or not the target person viewed the signage, or information indicating whether or not the target person viewed the signage for a certain period of time or longer. If the content is a video, the results of the initiative may also be the ratio of the target person's viewing time to the video playback time. The upper limit of this ratio may be 100%. In this specification, digital signage may also be simply referred to as "signage". The timing for recording sales revenue for goods or services is typically after the end of business hours on each business day. This means that the results of a measure may not be obtained immediately after its implementation.

[0014] Figure 2 shows an example of the results of the measures. In Figure 2, each person's identifier, the date and time when each person viewed the signage, each person's attributes, the content presented to each person, and the total time each person viewed the signage are shown.

[0015] The attribute storage unit 112 stores information indicating the attributes of the target person of the policy, which has been collected by the attribute information collection device 300. The attributes of the target individual are determined by AI (Artificial Intelligence) based on the characteristics of the target individual's image, for example. If information is registered by the target individual upon entry to the facility, or if the policy presentation device is linked to a smartphone application and can utilize the information of the target individual registered in the application, the registered information may be used when determining the attributes of the target individual. When the target group of a policy consists of multiple individuals, the attributes of the target group may be determined according to the attributes of each individual constituting the target group. Specific examples of the target group's attributes include being elderly and children, adults and children, multiple children, or adult men and adult women. Attribute classifications may be determined according to a combination of multiple factors related to each individual, such as age and gender, or according to the classification of the importance of the target in a store or facility. Classifications of relatively low importance may be grouped together as "other attributes." If the attributes of the target individual fall under "other attributes," the measures may not be implemented, or optional measures may be implemented.

[0016] The policy list storage unit 113 stores policy list information that shows a list of policies. The measure involves displaying content on digital signage. Content, as a concrete example, is an advertisement. Content may be a still image or a video. Advertisements may include coupon offers. Coupon offers may display a link to a website where coupons can be obtained. Furthermore, there may be multiple advertisements with similar purposes or target audiences, and multiple advertisements for the same store may exist. Figure 3 shows an example of a list of policies. In Figure 3, the identifier of each content, the type of content, the display time of each content, and the start and end dates of publication for each content are shown. If the content is a video, "Display time" indicates the length of the video.

[0017] The trained model storage unit 114 stores the trained model. A trained model is a model that has learned the correspondence between a person's attributes and the expected reward and confidence interval corresponding to each piece of content (arm). The trained model takes attributes (such as a combination of age and gender) as input and outputs the expected reward and the confidence interval corresponding to the expected reward for each piece of content, depending on the input attributes. The expected reward is sometimes called the estimated reward. A concrete example of a trained model is using it to display different content based on the attributes of a person viewing a digital sign. The trained model can be any type of machine learning model. The accuracy of a trained model can be improved through reinforcement learning. A concrete example of a trained model is not simply a supervised AI that selects content based on collected data, but rather a reinforcement learning AI. In a reinforcement learning AI, the AI ​​considers how to collect data and how to effectively display advertisements. The trained model outputs the central value and interval for each content, for example, "Content A: 20±10, Content B: 20±5 (95% confidence interval)". Confidence intervals other than the 95% confidence interval may also be used.

[0018] The attribute distribution storage unit 115 stores the attribute distribution information collected by the attribute distribution collection device 400. Attribute distribution information is information that predicts the distribution of attributes of individuals whose output from a content output device could be of interest. When the content output device is digital signage, attribute distribution information is information that predicts the distribution of attributes of individuals appearing within the target area of ​​the campaign. Attribute distribution information may also be information that predicts the distribution of attributes of individuals whose output from a content output device could be of interest for each time period. The attribute distribution is a distribution for each time period and represents the distribution of attributes of the individuals targeted by the intervention. The attribute distribution indicates the tendencies or number of users receiving the intervention. The attribute distribution may show the predicted proportion of each attribute in each time period, or it may show the predicted number of occurrences of each attribute in each time period. The attribute distribution may be calculated based on actual results, calculated based on a model, or it may be based on attribute distributions from other stores or facilities. The results may be the results of analyzing video footage from cameras installed in a facility or store, or they may be pedestrian flow analysis information or location information analysis services provided by mobile phone carriers, etc. Time divisions are determined according to at least one of the time periods and their characteristics. Each time period may be divided in any way. The characteristics of a time of day are determined by one or more factors related to the time of day that can influence the behavior of many people. These factors may include, specifically, those tied to a calendar (weekday or holiday), season, weather, or events or sales held within or near facilities where signage is installed during each time period.

[0019] Figure 4 shows an example of attribute distribution information. In Figure 4, the area covered by the attribute distribution, the day divisions, the time periods, the number of people staying in the area, and the proportion of each attribute are shown.

[0020] The learning unit 160 comprises a data linking unit 161 and a model learning unit 162. The data linking unit 161 generates training data by linking attributes with the results of the measures based on the measure result information. The results of the measures include the reward corresponding to each measure and the confidence interval corresponding to the reward for each measure. The model learning unit 162 trains the pre-trained model using the data linked by the data linking unit 161.

[0021] The inference unit 170 comprises a weight calculation unit 171, an inference processing unit 172, a policy decision unit 173, and a transmission unit 174.

[0022] The weight calculation unit 171 calculates policy adjustment weights based on attribute distribution information. The policy adjustment weight is a weight corresponding to the confidence interval of the expected reward for each content item shown in the policy list information, and is a weight corresponding to the probability that a person with the attributes of a policy target person can make the output of the content output device an output of interest. The weight acts so that exploration is prioritized for attributes that have a relatively high probability of being observed as attributes of a person who can make the output of the content output device an output of interest, and utilization is prioritized for attributes that have a relatively low probability of being observed as attributes of a person who can make the output of the content output device an output of interest. The policy adjustment weight may also be a weight corresponding to the probability that a person with the attributes of a policy target person can make the output of the content output device an output of interest in a time segment that includes the time when the policy target person makes the output of the content output device an output of interest. The probability of making the output of the content output device an output of interest may also be the probability of the person appearing in the policy target area. Note that the policy adjustment weight may also be simply referred to as "weight". The weights represent the importance of each user. By assigning weights to each attribute based on its frequency of appearance, it becomes less likely that incorrect policy decisions will be made. The weight corresponding to the main target of the advertisement may be a relatively large value. A coefficient may be introduced for the weights to adjust the scale of the weights.

[0023] The inference processing unit 172 uses attribute information, policy list information, and a trained model to infer which content to display.

[0024] The policy decision unit 173 determines the content to present to the target person based on the expected rewards in the expected reward group corresponding to the attributes of the target person, the confidence intervals corresponding to each expected reward in the expected reward group, and the policy adjustment weights. Specifically, the policy decision unit 173 determines the policy based on the weights calculated by the weight calculation unit 171 and the inference results from the inference processing unit 172. The policy decision unit 173 may calculate the sum of each expected reward in the expected reward group and the value obtained by multiplying the upper bound of the confidence interval corresponding to each expected reward in the expected reward group by the policy adjustment weight, as the policy score in the policy score group, and determine the content to present to the target person based on each policy score in the policy score group. Each policy score in the policy score group is a score corresponding to the attributes of the target person and is a score corresponding to each content shown in the policy list information. For the purpose of ensuring fairness, the policy decision unit 173 may relatively increase the proportion of new content being determined as content to present to the target person during the effect verification time when new content is added as new content to the policy list information. The effect verification time may be defined in any way. Each expected reward in the expected reward group corresponding to the attributes of the target individual is an expected reward derived as the expected reward corresponding to the attributes of the target individual, and is the expected reward corresponding to each content shown in the policy list information. The upper bound of the confidence interval for the expected reward corresponds to the upper limit of the width of the confidence interval that corresponds to the expected reward. Furthermore, multiple measures may be decided upon for a single target individual. When multiple measures are implemented for a single target individual, for example, each measure may be executed for a certain period of time in order of corresponding estimated effect or priority, while the target individual is able to view the signage. In this case, a single measure may be executed multiple times. When new content is added to the list of policies, the policy decision unit 173 may adjust the policies so that the added content is displayed a predetermined number of times.

[0025] The transmitting unit 174 transmits information indicating the decided policy to the policy presentation device. The policy presentation device includes a digital signage that displays the decided policy, and displays content on the digital signage according to the policy decided by the policy decision device 100. If no policy has been decided, the policy presentation device does not have to display anything, or it may display any content.

[0026] The policy result collection device 200 collects policy result information. The attribute information collection device 300 collects information indicating the attributes of the person targeted by the policy. The attribute distribution collection device 400 collects information showing the attribute distribution in the area targeted by the policy.

[0027] Figure 5 shows an example of the hardware configuration of the policy decision-making device 100 according to this embodiment. The policy decision-making device 100 consists of a computer. The policy decision-making device 100 may consist of multiple computers.

[0028] As shown in this figure, the policy decision-making device 100 is a computer equipped with hardware such as a processor 11, memory 12, auxiliary storage device 13, input / output interface 14, and communication device 15. These hardware components are connected as appropriate via signal lines 19.

[0029] The processor 11 is an integrated circuit (IC) that performs arithmetic operations and controls the hardware of the computer. Specific examples of the processor 11 include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit). The policy decision-making device 100 may include multiple processors that replace the processor 11. The multiple processors share the role of the processor 11.

[0030] Memory 12 is typically a volatile storage device, specifically RAM (Random Access Memory). Memory 12 is also called main memory. Data stored in memory 12 is saved to auxiliary storage device 13 as needed.

[0031] The auxiliary storage device 13 is typically a non-volatile storage device, specifically a ROM (Read Only Memory), an HDD (Hard Disk Drive), or flash memory. Data stored in the auxiliary storage device 13 is loaded into memory 12 as needed. The memory 12 and the auxiliary storage device 13 may be configured as a single unit.

[0032] Input / Output IF14 is a port to which input and output devices are connected. A specific example of an input / output IF14 is a USB (Universal Serial Bus) terminal. Specific examples of input devices include a keyboard and mouse. Specific examples of output devices include a display.

[0033] The communication device 15 consists of a receiver and a transmitter. A specific example of the communication device 15 is a communication chip or a NIC (Network Interface Card).

[0034] Each part of the policy decision-making device 100 may use the input / output IF 14 and the communication device 15 as appropriate when communicating with other devices.

[0035] The auxiliary storage device 13 stores the policy decision program. The policy decision program is a program that enables the computer to implement the functions of each part of the policy decision device 100. The policy decision program is loaded into memory 12 and executed by the processor 11.

[0036] Data used when executing the policy decision program, and data obtained by executing the policy decision program, are appropriately stored in the memory device. Each part of the policy decision device 100 utilizes the memory device as appropriate. The memory device consists of, specifically, memory 12, auxiliary memory device 13, registers in the processor 11, and at least one of the cache memory in the processor 11. Note that the terms data and information are sometimes synonymous. The memory device may be independent of the computer. The functions of memory 12 and auxiliary storage device 13 may be implemented by other storage devices.

[0037] The policy decision program may be recorded on a computer-readable non-volatile recording medium. Specific examples of non-volatile recording media include optical discs or flash memory. The policy decision program may also be provided as a program product.

[0038] ***Explanation of operation*** The operating procedure of the policy decision-making device 100 corresponds to the policy decision-making method. Furthermore, the program that implements the operation of the policy decision-making device 100 corresponds to the policy decision-making program.

[0039] Figure 6 shows an overview of the operation of the policy decision system 90. The goal of the policy decision system 90 is to display content on the signage that is tailored to the attributes of the target individuals of the policy. In the policy decision system 90, when content determined according to the attributes of the target person is displayed on signage, information indicating the viewing time of the content is collected as a policy result. Subsequently, reinforcement learning is performed based on the policy results and external information, and the policy is decided again. The external information is extracted and processed as appropriate. When a sufficient amount of data is collected, ads with a relatively high estimated effect will have a relatively high impression rate, and ads with a relatively low estimated effect will have a relatively low impression rate. On the other hand, when a relatively small amount of data is collected, the estimated effect of the ads is unknown, and the impression rate of the ads will be relatively high. The estimated effect is the effect of the ad that has been estimated.

[0040] Figure 7 is a diagram illustrating the outline of the policy decision algorithm according to this embodiment. A specific example of a policy decision-making algorithm is the bandit algorithm shown in Figure 7(a).

[0041] As shown in Figure 7(b), with conventional technology, it is a black box which user groups will benefit from the measures. On the other hand, in this embodiment, as shown in Figure 7(b), the objective is to reduce the amount of data required to train the trained model until it becomes a trained model with sufficient performance, by efficiently performing reinforcement learning using external information (prior knowledge). Specifically, in cases where the attribute distribution differs for each time segment, the attribute distribution for each time segment is provided in advance as prior knowledge.

[0042] As shown in Figure 7(c), the bandit algorithm becomes more sensitive to changes in the confidence interval as the amount of data decreases. In other words, the wider the confidence interval (i.e., the larger the value of c), the more the exploration is promoted. In this embodiment, weighting is performed by taking advantage of the characteristics of the bandit algorithm. Here, "exploration" means presenting various content, and "utilization" means presenting content for which the reward is known. Specifically, for attributes with a relatively high probability of future appearance in each time period, exploration will be prioritized, meaning rewards will be obtained steadily without failure. On the other hand, for attributes with a relatively low probability of future appearance in each time period, utilization will be prioritized, meaning rewards will be harvested as early as possible.

[0043] The policy decision algorithm according to this embodiment is, as a specific example, an algorithm based on a bandit algorithm (reinforcement learning) called Upper Confidence Bound (UCB). The policy decision algorithm basically aims to maximize effectiveness by automatically determining the display ratio (balance between data utilization and exploration) from the following three perspectives.

[0044] Perspective 1: Increase the display rate of advertisements that are likely to be highly effective (utilize them). Perspective 2: Reduce the display rate of ads that are likely to be ineffective (utilize them). Perspective 3: Increase the display rate of ads whose effectiveness is unknown (new stores) (exploration).

[0045] The basic idea of ​​the policy decision algorithm is to determine the UCB score and probability using the expected value (mean) and the upper bound of the confidence interval for the expected value. The UCB score is calculated using the expected value + upper bound, as shown in [Equation 1].

[0046]

number

[0047] To facilitate understanding, the following is a specific example where the continuous value "viewing time" is used as the UCB score. Figure 8 is a diagram illustrating the UCB score. In Figure 8, the number of data points (number of recommendations), the average viewing time, the upper limit of the 95% confidence interval (UCB score), and the softmax value are shown for each store. As shown in the example for store A, the more data there is, the narrower the confidence interval becomes. On the other hand, as shown in the example for store C, the less data there is, the wider the confidence interval becomes.

[0048] This explains the fairness of the policy algorithm. Basically, fairness is maintained by determining which ads to display based on the following rules (specifically, the UCB score), that is, the basic principles of ad differentiation. Rule 1: For ads that have been recommended in the past and have had a relatively long viewing time, the probability of showing them will be relatively higher. Rule 2: For ads that have been recommended in the past and had a relatively short viewing time, the probability of showing them again will be relatively low. Rule 3: For ads that have never been recommended before, or have been recommended infrequently, the probability of recommending them will be relatively higher, regardless of the conversion rate.

[0049] Based on the rules above, when presenting target advertisements, the probability of a target advertisement being presented is as follows: If the target advertisement is displayed and the viewing time of the advertisement is relatively long, the likelihood of the target advertisement being displayed increases relatively. If the time spent viewing the target advertisement is relatively short after it has been presented, the likelihood of the target advertisement being presented decreases. If an ad is displayed relatively infrequently, the probability of it being displayed will be relatively increased, regardless of the viewing time of the ad. This ensures fairness for new ads. Furthermore, advertisers are encouraged to strive to improve the quality of their ads to increase the likelihood of them being displayed.

[0050] Figure 9 shows a specific example of the process by which each advertisement is presented based on the UCB score. Initially, because the recommendation probabilities (≒UCB scores) are equal, advertisements for store A and store B are displayed equally. Subsequently, because the viewing time for store A's advertisement was relatively longer, the recommendation probability for store A increased. On the other hand, the recommendation probability for store B decreased. Subsequently, when store C's advertisement enters the market, its initial recommendation probability is set relatively higher because it is a new store. Consequently, the recommendation probabilities for stores A and B decrease. Subsequently, because the viewing time for store C's advertisement was shorter than that for store A's advertisement, the recommendation probability for store C becomes lower than that of store A, and converges to that level.

[0051] The policy decision algorithm is explained using Figure 10. In the standard Bandit problem, it is unclear how each element that influences viewing time (reward) specifically affects viewing time. Furthermore, the elements are not hierarchical. On the other hand, in this embodiment, as shown in the lower part of Figure 10, the attribute distribution may change depending on the time segment. Therefore, information showing the attribute distribution in each time segment is obtained from other services and utilized. Here, even in the Causal Bandit problem, it is unclear how each element affects the viewing time. However, in this embodiment, each element is classified into either a controllable element or an uncontrollable element, and the elements are hierarchically structured. As a specific example, as shown in the upper part of Figure 10, although the actual attribute distribution in each time segment cannot be controlled, the prediction of the attribute distribution in each time segment is known in advance. Also, the timing of when to display an advertisement and which advertisement to display can be controlled. Therefore, in this embodiment, controllable elements are controlled by weighting. The Bandit problem according to this embodiment is the Causal Bandit problem.

[0052] Figure 11 illustrates the differences between the conventional technology and this embodiment. In Figure 11, blacked-out individuals represent target individuals with attributes that appear relatively frequently. Whited-out individuals represent target individuals with attributes that appear relatively infrequently. A circle (○) indicates that an effective advertisement was presented to the target individual. An "X" indicates that an ineffective advertisement was presented to the target individual. As shown in Figure 11(a), when LinTS (Linear Thompson Sampling) prioritizes "exploration," it may end without being able to utilize data on audiences with rare attributes. Conversely, as shown in Figure 11(b), when LinTS prioritizes "utilization," insufficient exploration can lead to misjudgments, resulting in the continuous display of incorrect or ineffective advertisements to heavy users. On the other hand, as shown in Figure 11(c), in this embodiment, the balance between "exploration" and "utilization" is adjusted in Causal LinTS (C-LinTS) based on prior knowledge, so that the ratio of "exploration" to "utilization" is adjusted according to the probability of appearance of each attribute.

[0053] As an example of a policy algorithm related to this embodiment, we will describe an algorithm that incorporates improvements to LinTS. The following assumptions are made to the model below. Assumption 1: Assume that the reward for each arm (choice) follows a Gaussian distribution. Assumption 2: Assume that the parameter vector representing the true reward of each arm is μ, which follows a Gaussian distribution. Assumption 3: The true reward of arm i is μ i Let μ be the mean of the prior distribution for arm i. i Let the covariance matrix of the prior distribution be Σ i Therefore, it can be expressed as [Equation 2]. Here, i is an integer between 1 and n (inclusive), and n is the total number of arms. Furthermore, instead of LinTS, other algorithms that output confidence intervals or other variances may be used as the policy algorithm.

[0054]

number

[0055] Figure 12 is a flowchart illustrating an example of the operation of the policy algorithm. This operation will be explained using Figure 12. Note that steps S1 to S3 are executed repeatedly.

[0056] (Step S1) As shown in [Equation 3], the expected reward is randomly generated from the probability distribution. In [Equation 3], the weight G is multiplied by the confidence interval Σ. In this algorithm, the variance of a Gaussian distribution is multiplied by a context-dependent weight G, and the expected reward is determined by sampling the reward from that Gaussian distribution. Weight G is a weight corresponding to the predicted frequency of the attributes receiving intervention. For example, if it is predicted that men in their 30s will receive frequent intervention, a relatively small weight will be assigned to them, and "cautious exploratory activities" will be conducted. On the other hand, if it is predicted that women in their 70s will receive infrequent intervention, a relatively large weight will be assigned to them, and "bold utilization activities" will be conducted. The frequency of intervention for each attribute may be calculated based on location information analysis services provided by mobile phone carriers, etc. The weight G is determined based on the average proportion per cell. If there are 2 cells for gender and 5 cells for age, for a total of 10 cells (=2×5), the average is 1 / 10. For example, if men in their 30s account for 25% of the intervention, the weight corresponding to men in their 30s is 2.5 (25% / 10%).

[0057]

number

[0058] (Step S2) In each round, the arm with the highest sampled expected reward is selected, as shown in [Equation 4].

[0059]

number

[0060] (Step S3) Arm i's reward r i ^(t) is observed, r i The model parameters are updated using ^(t).

[0061] Figure 13 is a flowchart illustrating an example of the operation of the model learning unit 160. This operation will be explained using Figure 13.

[0062] (Step S101) The data linking unit 161 acquires policy result information from the policy result storage unit 111, links the attributes indicated by the policy result information with the policy, and generates training data.

[0063] (Step S102) The model learning unit 162 builds or updates a trained model using the training data.

[0064] (Step S103) The model learning unit 160 stores the trained model in the trained model storage unit 114.

[0065] Figure 14 is a flowchart illustrating an example of the operation of the inference unit 170. This operation will be explained using Figure 14.

[0066] (Step S111) The inference processing unit 172 retrieves the trained model from the trained model storage unit 114.

[0067] (Step S112) The inference processing unit 172 obtains a list of measures from the measure list storage unit 113, obtains attribute information to be inferred from the attribute storage unit 112, and infers the effect (mean and confidence interval) of each measure on the target of inference based on the trained model and the acquired data. Figure 15 shows a concrete example of inference results based on a trained model. In Figure 15, for each target individual, the area, date and time, attributes, time category, estimated value (average) of viewing time for each content, and confidence interval for viewing time for each content are shown. C001 to C003 represent each content.

[0068] (Step S113) The weight calculation unit 171 obtains attribute distribution information from the attribute distribution storage unit 115 and calculates the reciprocal of the frequency of occurrence of each attribute as a weight based on the attribute distribution information. Figure 16 shows a specific example of weights corresponding to the attribute distribution information shown in Figure 4. In Figure 16, the weights of each attribute are shown for each time segment.

[0069] (Step S114) The policy decision unit 173 multiplies the confidence interval of the policy's effect by the weight calculated by the weight calculation unit 171 for each of the policies for that attribute. The policy decision unit 173 calculates the policy score by adding the value obtained by multiplying the weight by the upper bound of the confidence interval and the mean value of the policy's effect. The policy score is, in particular, the UCB score.

[0070] (Step S115) The policy decision unit 173 either selects the policy with the highest calculated policy score as the policy, or probabilistically determines the policy based on the ratio of each policy score to the total policy score. Figure 17 shows a concrete example of the policy decision-making process corresponding to the inference results shown in Figure 15 and the weights shown in Figure 16. In Figure 17, for each target individual, the policy score corresponding to each content and the policy decided based on the policy score are shown. The curves in Figure 17 indicate that a single table has been separated due to space limitations.

[0071] ***Explanation of the effects of Embodiment 1*** As described above, according to this embodiment, in a technology that uses reinforcement learning to display content on digital signage according to a person's attributes, the accuracy is improved compared to conventional technology, especially when the amount of data is small, by utilizing weights corresponding to the frequency of occurrence of each attribute. Therefore, according to this embodiment, the amount of data (amount of observation results) required to generate a trained model with sufficient accuracy for policy decisions is reduced, thus solving the conventional problem of not being able to appropriately execute policy decisions (specifically, differentiating advertisements) due to insufficient data collection. Furthermore, according to this embodiment, policy decisions (specifically, which advertisements to display to which attributes, which coupons to issue) can be made efficiently by using external information.

[0072] ***Other configurations*** <Example 1> Figure 18 shows an example of the hardware configuration of the policy decision-making device 100 according to this modified example. The policy decision-making device 100 includes a processor 11, a processor 11 and memory 12, a processor 11 and auxiliary storage device 13, or a processing circuit 18 instead of a processor 11, memory 12 and auxiliary storage device 13. The processing circuit 18 is hardware that implements at least some of the components of the policy decision-making device 100. The processing circuit 18 may be dedicated hardware, or it may be a processor that executes the program stored in memory 12.

[0073] When the processing circuit 18 is dedicated hardware, specific examples of the processing circuit 18 include a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The policy decision-making device 100 may include multiple processing circuits that replace the processing circuit 18. The multiple processing circuits share the role of the processing circuit 18.

[0074] In the policy decision-making device 100, some functions may be implemented by dedicated hardware, while the remaining functions may be implemented by software or firmware.

[0075] The processing circuit 18 can be implemented, in specific examples, by hardware, software, firmware, or a combination thereof. The processor 11, memory 12, auxiliary storage device 13, and processing circuit 18 are collectively referred to as the "processing circuitry." In other words, the functions of each functional component of the policy decision-making device 100 are realized by the processing circuitry.

[0076] ***Other Embodiments*** Although Embodiment 1 has been described, multiple parts of this embodiment may be combined and implemented. Alternatively, this embodiment may be implemented partially. Furthermore, this embodiment may be modified in various ways as needed, and may be implemented as a whole or in parts in any combination. The embodiments described above are essentially preferred examples and are not intended to limit the scope of this disclosure, its applications, or its uses. The procedures described using flowcharts, etc., may be modified as appropriate.

[0077] The various aspects of this disclosure are summarized below as an appendix.

[0078] (Note 1) A policy decision device that utilizes reinforcement learning to differentiate the content output of a content output device according to the attributes of the target person of the policy, so that the output of the content output device can be an output of interest, A weight calculation unit calculates policy adjustment weights based on attribute distribution information showing a prediction of the distribution of attributes of persons whose output can be the output of interest, which weights correspond to the confidence interval of the expected reward corresponding to each content shown in the policy list information, and are weights corresponding to the probability that a person having the attributes of the policy target person can make the output of interest, such that exploration is prioritized for attributes that have a relatively high probability of being observed as attributes of a person whose output can be the output of interest, and utilization is prioritized for attributes that have a relatively low probability of being observed as attributes of a person whose output can be the output of interest, A policy decision unit determines the content to be presented to the policy target person based on the expected rewards corresponding to each content shown in the policy list information derived as each expected reward in the expected reward group corresponding to the attributes of the policy target person, the confidence interval corresponding to each expected reward in the expected reward group, and the policy adjustment weight. A policy decision-making device equipped with the following features.

[0079] (Note 2) The attribute distribution information indicates a prediction of the distribution of the attributes of individuals whose output can be the output of interest for each time segment determined according to at least one of the time period and the characteristics of the time period. The policy adjustment weight is a weight corresponding to the probability that a person having the attributes of the policy target person can make the output the output of interest in a time segment that includes the time when the policy target person makes the output the output of interest, as described in Appendix 1 of the policy decision device.

[0080] (Note 3) The aforementioned policy decision-making unit, For each policy score in the policy score group, the sum of each expected reward in the expected reward group and the value obtained by multiplying the upper bound of the confidence interval corresponding to each expected reward in the expected reward group by the policy adjustment weight is calculated. Based on the policy score of each policy score in the aforementioned policy score group, the content to be presented to the target person of the policy is determined. The policy decision device described in Appendix 1 or 2, wherein each policy score in the policy score group is a score corresponding to the attributes of the person targeted by the policy, and is a score corresponding to each content indicated in the policy list information.

[0081] (Note 4) The policy decision unit is a policy decision device according to any one of the appendices 1 to 3, wherein when new content is added as new content to the policy list information, the proportion of cases in which the new content is decided to be presented to the policy target person during the effect confirmation period is relatively high.

[0082] (Note 5) The aforementioned reinforcement learning is a bandit algorithm, which is a policy decision-making device as described in any one of the appendices 1 to 4.

[0083] (Note 6) Each piece of content shown in the aforementioned list of measures is an advertisement. The expected reward corresponding to each content shown in the aforementioned list of measures is a measure decision device described in one of the appendices 1 to 5, which is a value corresponding to the effect of each content.

[0084] (Note 7) In cases where each piece of content shown in the aforementioned list of measures is a still image, the expected reward corresponding to each piece of content is determined depending on whether or not the person targeted by the measure views each piece of content. In cases where each piece of content shown in the aforementioned list of measures is a video, the expected reward corresponding to each piece of content is determined according to the amount of time the person targeted by the measure continues to view each piece of content, as described in one of the measures determination devices described in any one of the appendices 1 to 6.

[0085] (Note 8) The content output device is a digital signage policy decision device as described in any one of Appendix 1 to 7. [Explanation of Symbols]

[0086] 11 Processor, 12 Memory, 13 Auxiliary storage device, 14 Input / Output IF, 15 Communication device, 18 Processing circuit, 19 Signal line, 90 Policy decision system, 100 Policy decision device, 110 Storage unit, 111 Policy result storage unit, 112 Attribute storage unit, 113 Policy list storage unit, 114 Trained model storage unit, 115 Attribute distribution storage unit, 150 Control unit, 160 Learning unit, 161 Data linking unit, 162 Model learning unit, 170 Inference unit, 171 Weight calculation unit, 172 Inference processing unit, 173 Policy decision unit, 174 Transmission unit, 200 Policy result collection device, 300 Attribute information collection device, 400 Attribute distribution collection device.

Claims

1. A policy decision device that utilizes reinforcement learning to differentiate the content output of a content output device according to the attributes of the target person of the policy, so that the output of the content output device can be an output of interest, A weight calculation unit calculates policy adjustment weights based on attribute distribution information showing a prediction of the distribution of attributes of persons whose output can be the output of interest, which weights correspond to the confidence interval of the expected reward corresponding to each content shown in the policy list information, and are weights corresponding to the probability that a person having the attributes of the policy target person can make the output of interest, such that exploration is prioritized for attributes that have a relatively high probability of being observed as attributes of a person whose output can be the output of interest, and utilization is prioritized for attributes that have a relatively low probability of being observed as attributes of a person whose output can be the output of interest, A policy decision unit determines the content to be presented to the policy target person based on the expected rewards corresponding to each content shown in the policy list information derived as each expected reward in the expected reward group corresponding to the attributes of the policy target person, the confidence interval corresponding to each expected reward in the expected reward group, and the policy adjustment weight. A policy decision-making device equipped with the following features.

2. The attribute distribution information indicates a prediction of the distribution of the attributes of individuals whose output can be the output of interest for each time segment determined according to at least one of the time period and the characteristics of the time period. The policy adjustment weight is a weight corresponding to the probability that a person having the attributes of the policy target person can make the output the output of interest in a time segment that includes the time when the policy target person makes the output the output of interest.

3. The aforementioned policy decision-making unit, For each policy score in the policy score group, the sum of each expected reward in the expected reward group and the value obtained by multiplying the upper bound of the confidence interval corresponding to each expected reward in the expected reward group by the policy adjustment weight is calculated. Based on the policy score of each policy score in the aforementioned policy score group, the content to be presented to the target person of the policy is determined. The policy decision device according to claim 1 or 2, wherein each policy score in the policy score group is a score corresponding to the attributes of the person targeted by the policy, and is a score corresponding to each content indicated in the policy list information.

4. The policy decision device according to claim 1 or 2, wherein when new content is added as new content to the policy list information, the policy decision unit relatively increases the proportion in which the new content is decided to be presented to the policy target person during the effect confirmation period.

5. The policy decision-making apparatus according to claim 1 or 2, wherein the reinforcement learning is a bandit algorithm.

6. Each piece of content shown in the aforementioned list of measures is an advertisement. The policy decision device according to claim 1 or 2, wherein the expected reward corresponding to each content shown in the policy list information is a value corresponding to the effect of each content.

7. In cases where each piece of content shown in the aforementioned list of measures is a still image, the expected reward corresponding to each piece of content is determined depending on whether or not the person targeted by the measure views each piece of content. In the case where each piece of content shown in the aforementioned list of measures is a video, the expected reward corresponding to each piece of content is determined according to the amount of time the person targeted by the measure continues to view each piece of content, as described in claim 1 or 2.

8. The policy decision device according to claim 1 or 2, wherein the content output device is digital signage.

9. A policy decision method executed by a policy decision device, which is a computer that uses reinforcement learning to differentiate the content output of a content output device according to the attributes of the target person of the policy, the output of the content output device can be made into an output of interest, The policy decision-making device calculates policy adjustment weights based on attribute distribution information that shows a prediction of the distribution of attributes of persons whose output can be the output of interest, which weights correspond to the confidence interval of the expected reward corresponding to each content shown in the policy list information, and are weights corresponding to the probability that a person having the attributes of the policy target person can make the output of interest, such that exploration is prioritized for attributes that have a relatively high probability of being observed as attributes of a person whose output can be the output of interest, and utilization is prioritized for attributes that have a relatively low probability of being observed as attributes of a person whose output can be the output of interest. A policy decision method in which the policy decision device determines the content to be presented to the policy target person based on the expected rewards corresponding to each content shown in the policy list information derived as each expected reward in the expected reward group corresponding to the attributes of the policy target person, the confidence interval corresponding to each expected reward in the expected reward group, and the policy adjustment weight.

10. A policy decision program executed by a policy decision device, which is a computer that uses reinforcement learning to differentiate the content output of a content output device according to the attributes of the target person of the policy, so that the output of the content output device can be an output of interest, A weight calculation process calculates policy adjustment weights based on attribute distribution information showing a prediction of the distribution of attributes of persons whose output can be the output of interest, which are weights corresponding to the confidence interval of the expected reward corresponding to each content shown in the policy list information, which are weights corresponding to the probability that a person having the attributes of the policy target person can make the output of interest, and which act so that exploration is prioritized for attributes that have a relatively high probability of being observed as attributes of a person whose output can be the output of interest, and utilization is prioritized for attributes that have a relatively low probability of being observed as attributes of a person whose output can be the output of interest, A policy decision process that determines the content to present to the target person based on the expected rewards corresponding to each content shown in the policy list information derived as each expected reward in the expected reward group corresponding to the attributes of the target person, the confidence interval corresponding to each expected reward in the expected reward group, and the policy adjustment weight. A policy decision program that causes the policy decision device to execute the aforementioned policy decision program.

Images