Systems and methods for simulating complex reinforcement learning environments

Abstract

A computing system, method, and computer readable medium for training a machine learning recommendation system are provided. The computing system includes one or more processors; one or more non-transitory computer-readable media collectively storing instructions executable by the one or more processors to cause the computing system to perform operations including providing an entity profile of a simulated entity to a machine learning recommendation system, wherein the entity profile includes interpretable interest features; obtaining, from the machine learning recommendation system, a recommendation for a resource for consumption by the simulated entity; determining, based on the entity profile, one or more engagement metrics associated with the consumption; and updating one or more parameters of the machine learning recommendation system to increase a reward determined based on the one or more engagement metrics.

Description

[0001] This application is a divisional application of the invention patent application filed on April 29, 2019, with application number 201910354811.4 and invention title "System and Method for Simulating Complex Reinforcement Learning Environments". Technical Field

[0002] This disclosure generally relates to systems and methods for simulating reinforcement learning environments. More specifically, this disclosure relates to systems and methods for simulating systems capable of testing or otherwise learning a variety of different reinforcement learning strategies or models. Background Technology

[0003] Various techniques can be used to train reinforcement learning agents in simulated environments. Typically, reinforcement learning agents are rewarded based on their actions in the simulated environment. Over time, the agent learns a policy to maximize its rewards. However, real-world environments are usually much more complex than the simulated environments currently used in reinforcement learning systems. Summary of the Invention

[0004] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0005] One example aspect of this disclosure relates to a computing system for simulating the allocation of resources to multiple entities. The computing system may include one or more processors and a reinforcement learning agent model configured to receive entity profiles describing at least one of the preferences or needs of the simulated entities. In response to receiving the entity profile, the reinforcement learning agent model may output an allocation output describing the resource allocation to the simulated entities. The computing system may include an entity model configured to receive data describing at least one resource, and in response to receiving the data describing the at least one resource, simulate a simulated response output describing the responses of the simulated entities to the data describing the at least one resource. The computing system may include one or more non-transitory computer-readable media that share instructions, which, when executed by one or more processors, cause the computing system to perform operations. The operation may include inputting an entity profile into a reinforcement learning agent model; receiving an allocation output as the output of the reinforcement learning agent model, which describes the allocation of resources to a simulated entity; selecting at least one resource to be provided to the entity model based on the resource allocation described by the allocation output; providing at least one resource to the entity model; receiving a simulation response output as the output of the entity model, which describes the response of the simulated entity to at least one resource; and updating at least one of the resource profiles describing at least one resource or the entity profile based on the simulation response output.

[0006] Another exemplary aspect of this disclosure relates to a method for simulating the allocation of resources to multiple entities. The method may include inputting entity profiles, described by one or more computing devices, to a reinforcement learning agent model. The entity profiles describe at least one of the preferences or needs of the simulated entities. The reinforcement learning agent model may be configured to receive the entity profiles and, in response to the received entity profiles, output an allocation output describing the resource allocation to the simulated entities. The method may include receiving the allocation output, as an output of the reinforcement learning agent model, describing the resource allocation to the simulated entities, by one or more computing devices; selecting at least one resource by one or more computing devices to simulate providing an entity model configured to receive data describing at least one resource, and, in response to receiving the data describing at least one resource, simulating a simulation response output describing the simulated entity's response to the data describing at least one resource; providing the entity model with the data describing at least one resource by one or more computing devices; receiving the simulation response output, as an output of the entity model, describing the simulated entity's response to at least one resource, by one or more computing devices; and updating at least one of the resource profiles or entity profiles describing at least one resource by the one or more computing devices based on the simulation response output.

[0007] Another example aspect of this disclosure relates to a computational system for training a machine learning recommender system. The computational system includes: one or more processors; one or more non-transitory computer-readable media that collectively store instructions executable by the one or more processors to cause the computational system to perform operations including: providing the machine learning recommender system with entity profiles of simulated entities, wherein the entity profiles include interpretable interest features; obtaining recommendations from the machine learning recommender system for resources to be consumed through the simulated entities; determining one or more engagement metrics associated with the consumption based on the entity profiles; and updating one or more parameters of the machine learning recommender system to increase the reward determined based on the one or more engagement metrics.

[0008] Another example aspect of this disclosure relates to a computer-implemented method. The method includes: providing an entity profile of a simulated entity to a machine learning recommender system, wherein the entity profile includes interpretable interest features; obtaining recommendations from the machine learning recommender system for resources to be consumed through the simulated entity; determining one or more engagement metrics associated with the consumption based on the entity profile; and updating one or more parameters of the machine learning recommender system to increase the reward determined based on the one or more engagement metrics.

[0009] Other aspects of this disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0010] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. Exemplary embodiments of the disclosure are illustrated in conjunction with the accompanying drawings, which are incorporated in and constitute a part of this specification, and serve to illustrate the relevant principles. Attached Figure Description

[0011] A detailed discussion of embodiments for those skilled in the art is set forth in the description with reference to the accompanying drawings, wherein:

[0012] Figure 1A A block diagram of an example computing system for simulating resource allocation to multiple entities using a reinforcement learning agent model, according to an example embodiment of the present disclosure, is depicted.

[0013] Figure 1B A block diagram of an example computing device for simulating resource allocation to multiple entities using a reinforcement learning agent model, according to an example embodiment of the present disclosure, is depicted.

[0014] Figure 1C A block diagram of an example computing device for simulating resource allocation to multiple entities using a reinforcement learning agent model, according to an example embodiment of the present disclosure, is depicted.

[0015] Figure 2 A reinforcement learning agent model for machine learning according to an example embodiment of the present disclosure is described.

[0016] Figure 3 An embodiment of a system for simulating resource allocation to multiple entities, according to an example embodiment of the present disclosure, is described.

[0017] Figure 4 Another embodiment of a system for simulating resource allocation to multiple entities is described according to an example embodiment of the present disclosure.

[0018] Figure 5A An embodiment of a system for simulating a recommendation system according to an example embodiment of the present disclosure is described.

[0019] Figure 5B An example embodiment of the present disclosure is shown for use with Figure 5A Example graph of the update association function for the user conversion model of the system.

[0020] Figure 6 A flowchart is depicted for an example method for simulating resource allocation to multiple entities using a reinforcement learning agent model, according to an example embodiment of the present disclosure.

[0021] Figures 7A to 7C It shows Figure 5AThe system's simulation data includes average episode length, predicted click-through rate (pCTR), and average return as a function of the training steps of various models.

[0022] Figures 8A to 8U The above reference is shown. Figures 7A to 7C The simulation data described for the experiment includes the proportion of each recommended slate and cluster in the agent that was viewed over time.

[0023] Figures 9A to 9C It shows the use of Figure 5A The system's simulation data shows that the model parameters are chosen so that stochastic strategies of multinomial proportional and exponential cascade models produce similar returns.

[0024] Figures 10A to 10C The results of using a cascaded model for user selection and multiple proportions within the CSDQN model are shown.

[0025] Figures 11A to 11U The above reference is shown. Figures 10A to 10C The simulation data described for the experiment includes the proportion of each recommended section / cluster in the agent that was viewed over time.

[0026] The repeated reference numerals in multiple figures are intended to identify the same features in various implementations.

[0027] Specific implementation

[0028] Generally, this disclosure relates to systems and methods for simulating systems capable of testing or otherwise learning a variety of different reinforcement learning policies or models. Thus, environments can be simulated where policies, rules, settings, or other reinforcement learning properties can be tested prior to (or not implemented in) a real-world environment. As an example, a simulation system may include different components operating to provide a simulated environment, where a reinforcement learning agent can learn to allocate resources to multiple entities, such as resource allocation in an industrial setting, allocation of computational resources for a competing computational task, and / or selection of documents or other content to be offered to a user via a recommendation system. Specifically, according to one aspect of this disclosure, systems and methods can model resource-consuming entities and / or resources that change over time based on their interactions with and / or results observed in the simulated environment. Therefore, the disclosed systems and methods are particularly useful for simulating environments that allow reinforcement learning agents to learn policies that prioritize long-term benefits at the expense of short-term negative effects in such dynamic environments. Thus, policies or other reinforcement learning properties can be simulated prior to implementation in a real-world environment where experimentation may be too expensive or impractical.

[0029] In some implementations, the system and methods can be offered as a cloud-based service, where users can provide pre-trained or pre-configured reinforcement learning agent models. Users can set or adjust inputs and / or settings to customize the simulation environment, such as simulating a real-world environment where the user intends to deploy the reinforcement learning agent model. Users can then simulate the performance of the reinforcement learning agent model over time in the simulation environment to predict and / or optimize the performance of the agent model or several different variants thereof in a real-world environment.

[0030] According to one aspect of this disclosure, the computing system may include a reinforcement learning agent model and an entity model that models entities within a simulated environment. The reinforcement learning agent model may be configured to receive an entity profile describing at least one of the preferences or needs of the simulated entity (e.g., an industrial process). In response to receiving the entity profile, the reinforcement learning agent model may output an allocation output describing the allocation of resources for the simulated entity (e.g., inputs to an industrial process, such as raw materials, fuel, setup, and / or the like). For example, according to various example configurations, the reinforcement learning agent model may apply a learned policy to generate the allocation output in an attempt to maximize the cumulative reward received by the agent model over time.

[0031] The entity model can be configured to receive data describing resource allocations generated by a reinforcement learning agent model. In response to the received data, the entity model can be configured to simulate a simulated response output (e.g., an updated state or performance metric of an industrial process) describing the simulated entity's response to the data-described resource allocations.

[0032] Therefore, a computing system can use entity models to simulate the environment of a reinforcement learning agent model, where one or more simulated entities respond to one or more resource allocations generated by the reinforcement learning agent model for the entities.

[0033] More specifically, the computational system can input entity profiles into a reinforcement learning agent model and receive an allocation output as the output of the reinforcement learning agent model, which describes the resource allocation to the simulated entity. The computational system can select at least one resource to provide to the entity model based on the resource allocation described by the allocation output. The computational system can provide resources to the entity model and receive a simulated response output as the output of the entity model, which describes the simulated entity's response to at least one resource.

[0034] According to one aspect of this disclosure, a computational system can update at least one of a resource profile or entity profile describing at least one resource based on the simulated response output. For example, after simulating an entity's response to resource allocation, various features or states of the entity model can be updated or otherwise transformed. Some or all of the above steps can be performed iteratively to simulate the learning of a reinforcement learning agent model over time in a simulated environment. Furthermore, updating the entity profile and / or resource profile can allow the corresponding states or characteristics of the entity and / or resource to change over time in the simulation to simulate dynamic entities and / or resources. Thus, the ability of the simulated environment to simulate changes in entity features, behavior, or state enables reinforcement learning agents to learn strategies that explicitly consider and are based on the fact that entities may have dynamic and changing responses to resource allocation over time, and further, that such dynamic and changing responses may be a function of the resource allocation provided to the entity over time. In this way, aspects of this disclosure enable the learning of reinforcement learning agents with improved performance relative to dynamically changing resource-consuming entities in a simulated environment.

[0035] The disclosed systems and methods can be used to simulate a variety of real-world entities and environments. As described above, in some implementations, the simulated entity may include an industrial process (e.g., manufacturing, power generation, etc.). Resources may include inputs to the industrial process, such as raw materials, fuel, settings (e.g., temperature, processing rate, production rate), and / or the like. The simulation response output may include updated states, state changes, or other data describing the industrial process or its response to changes in received resources.

[0036] As another example, the simulated entity may include a computational task or a source of computational tasks. Resources may include computational resources used to run the computational task, such as workers (e.g., server computing devices, processor cores, physical computing devices, virtual machines, etc.). The simulated response output may include updated states, state changes, or other data describing responses or changes in response to the computational task or the source of the computational task that received the resources.

[0037] As another example, the systems and methods disclosed herein can be used to simulate recommendation systems for recommending content to human users. Simulated entities may include simulated human users. Resources may include content for viewing or engagement by simulated human users. Example resources include text, audio, or graphical content (e.g., images, videos, articles, or other media content). Such resources may be collectively referred to as “documents.” Simulated response outputs may include engagement metrics such as whether a document was viewed (e.g., “clicked”), interaction time, user ratings, etc.

[0038] In some implementations, the agent model may include a reinforcement learning agent learned based on a reward that is a function of the simulated response output. The reward may be positively correlated with a desired characteristic of the simulated response output. Examples include output or performance metrics associated with industrial or computational processes. In another example, the reward may be positively correlated with one or more engagement metrics describing the simulated human user's participation or positive feedback regarding resources.

[0039] In some implementations, entity profiles can describe a “stylized” model of an entity, where some or all of the entity’s features have interpretable meaning. Employing features with interpretable meaning can provide insights into how a particular entity’s responses over time affect a reinforcement learning agent model and / or how the actions of the reinforcement learning agent model affect the entity. Entity profiles can include or describe the requirements of industrial processes and / or computational processes (e.g., temperature, rate, etc.). As another example, entity profiles can include user profiles describing the interests and / or preferences of simulated human users.

[0040] For example, an entity profile may include one or more "interest" features. Interest features may include elements that describe the simulated human user's association with various topics (e.g., sports, music, etc.). Interest features can range from a negative lower bound representing a strong dislike for the corresponding topic to a positive upper bound representing a strong liking for the corresponding topic.

[0041] As another example, an entity profile may include one or more "budget" characteristics. The budget characteristic can describe the amount of time a simulated human user has available to interact with the content. Resources can be provided to the simulated human user until the budget reaches a minimum threshold (e.g., zero). Another simulated human user can then be selected for simulation. However, in some implementations, resources can be provided to multiple simulated human users (or other entities) simultaneously.

[0042] In some implementations, the computational system may include a user transition model configured to generate an updated set of user hidden state features in response to receiving data describing the simulated response output. The computational system can provide data describing the simulated response output to the user transition model and update entity profiles based on the user hidden state features. Some or all of the user hidden state features can be hidden from the reinforcement learning agent. The entity profile may include user-observable features accessible to the reinforcement learning agent (e.g., input). User-observable features can be updated based on the user hidden state features. Therefore, the reinforcement learning agent may not immediately discover certain information about the entity. The reinforcement learning agent can be trained to select resources to provide to the entity, allowing information about the entity to be discovered during simulation. Therefore, the reinforcement learning agent can be trained to balance the development and exploration of information about entities in the "multi-armed slot" context.

[0043] In some implementations, the computational system may include a resource model configured to receive data describing multiple resources and, in response to receiving the data describing the multiple resources, output resource observable features. A reinforcement learning agent model can be trained to select an allocation output based at least in part on the resource observable features. The resource observable features may describe the resources and be accessible to the reinforcement learning agent model (e.g., as input). More specifically, the computational system may input data describing multiple resources into the resource model and receive the resource observable features as the output of the resource model. The computational system may also input the resource observable features into the reinforcement learning agent model. The resource profile may also include hidden features that are inaccessible to the reinforcement learning agent model (e.g., as input). Therefore, a reinforcement learning agent can be trained to balance the development and exploration of information about resources within a "multi-armed slot machine" context.

[0044] Features of a resource profile (e.g., observable and / or hidden features) can describe a resource. As an example, in a recommender system application, a resource profile may include "attribute" features, which include one or more elements describing the document's topic. "Attribute" features may include elements corresponding to individual topics. The range of elements can be from minimum to maximum values ​​to indicate the relevance of the document to the corresponding topic.

[0045] As another example, a resource profile may include a "length" feature. The length feature can describe the duration associated with document engagement. For example, the length feature can describe the length of a video.

[0046] As another example, resource profiles can include a "quality" feature. A "quality feature" can describe the extent to which a document contains high-quality content, rather than content that initially appears interesting but doesn't provide meaningful information or relevant details about further engagement (e.g., "clickbait"). "Quality" features can also describe more objective quality measures, such as the video quality of a video, the writing quality of an article, etc.

[0047] In some implementations, the computational system can simulate providing multiple resources to a simulated entity, from which the simulated entity can "select" resources to engage with or consume. More specifically, the simulated response output can include selections of fewer than all the resources provided to the simulated entity. For example, in a recommender system example, a reinforcement learning agent can select multiple documents (e.g., videos, articles, images, advertisements, etc.) and present them to a simulated human user in a "forum." A "forum" can include a list of recommended documents for viewing or otherwise engaging with. In such an implementation, the simulated response output can describe the entity's selections of fewer than all of the multiple resource items. For example, a simulated human user could select one document from multiple documents presented in a forum.

[0048] For example, an entity model can include a discrete selection model. A discrete selection model is typically configured to select one item from a finite group of items. A discrete selection model can be configured to select one resource from multiple resources (e.g., sections of a document). Discrete selection models can employ a variety of suitable functions, including multinomial scaling functions, multinomial logit functions, exponential cascade functions, and / or similar functions.

[0049] The systems and methods disclosed herein define specific technical implementations for simulating the allocation of resources to multiple entities. The implementation of the described techniques thus provides technical functionality that allows for virtual experimentation, serving as a practical and practice-oriented part of the toolkit for skilled technicians. Furthermore, the systems and methods of this disclosure offer numerous additional technical effects and benefits. As an example, the systems and methods described herein can aid in the development and / or optimization of reinforcement learning agents for controlling industrial processes such as power generation. Improving the efficiency of controlling and / or monitoring these processes can reduce waste and save energy.

[0050] As an example, the systems and methods of this disclosure may be included or otherwise adopted in the context of an application, browser plugin, or other environment. Therefore, in some implementations, the model of this disclosure may be included, or otherwise stored and implemented in a user computing device such as a laptop, tablet computer, or smartphone. As yet another example, the model may be included, or otherwise stored and implemented in a server computing device that communicates with the user computing device according to a client-server relationship. For example, the model may be implemented by the server computing device as part of a web service (e.g., a web email service).

[0051] Exemplary embodiments of this disclosure will now be discussed in more detail with reference to the accompanying drawings.

[0052] Example devices and systems

[0053] Figure 1A A block diagram of an example computing system 100 for simulating a reinforcement learning environment according to an example embodiment of the present disclosure is depicted. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0054] User computing device 102 can be any type of computing device, such as a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0055] User computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be one processor or multiple processors operatively connected. Memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, and combinations thereof. Memory 114 can store data 116 and instructions 118 executed by processor 112 to cause user computing device 102 to perform operations.

[0056] User computing device 102 may store or include one or more reinforcement learning agent models 120, entity models 122, and / or resource models 124. For example, reinforcement learning agent models 120, entity models 122, and / or resource models 124 may be or may include various machine learning models, such as neural networks (e.g., deep neural networks) or other multi-layer nonlinear models. Neural networks may include recurrent neural networks (e.g., long short-term memory recurrence neural networks), feedforward neural networks, or other forms of neural networks. Reference Figure 2 Figure 5 discusses an example reinforcement learning agent model 120.

[0057] In some implementations, one or more reinforcement learning agent models 120 may be received from server computing system 130 via network 180, stored in user computing device memory 114, and used or otherwise implemented by one or more processors 112. In some implementations, user computing device 102 may implement multiple parallel instances of a single reinforcement learning agent model.

[0058] Additionally or alternatively, one or more reinforcement learning agent models 140, entity models 142, and / or resource models 144 may be included in or otherwise stored and implemented in server computing system 130, which communicates with user computing device 102 according to a client-server relationship. For example, reinforcement learning agent models 140, entity models 142, and / or resource models 144 may be implemented by server computing system 140 as part of a web service (e.g., a reinforcement learning simulation service). Thus, one or more models 120, 122, 124 may be stored and implemented at user computing device 102 and / or one or more models 140, 142, 144 may be stored and implemented at server computing system 130.

[0059] User computing device 102 may also include one or more user input components 122 for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). Touch-sensitive components can be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which the user can input communication.

[0060] Server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 can store data 136 and instructions 138 executed by processor 132 to cause server computing system 130 to perform operations.

[0061] In some implementations, server computing system 130 includes one or more server computing devices or is otherwise implemented using one or more server computing devices. Where server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0062] As described above, the server computing system 130 may store or otherwise include one or more reinforcement learning agent models 140, entity models 142, and / or resource models 144. For example, model 140 may be, or may otherwise include, various machine-learning models, such as neural networks (e.g., deep recurrent neural networks) or other multi-layered nonlinear models. (See reference...) Figure 2 See Figure 5 for example models 140, 142, and 144.

[0063] In some implementations, the system and methods can be offered as a cloud-based service (e.g., via server computing system 130). Users can provide pre-trained or pre-configured reinforcement learning agent models. Users can set or adjust inputs and / or settings to customize the simulation environment, such as simulating a real-world environment where the user intends to deploy the reinforcement learning agent model. Users can then simulate the performance of the reinforcement learning agent model over time in the simulation environment to predict and / or optimize the performance of the agent model or several different variants thereof in a real-world environment.

[0064] Server computing system 130 can train model 140 via interaction with training computing system 150, which is communicatively coupled to network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0065] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processors 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0066] Training computing system 150 may include model trainer 160, which trains models 140, 142, 144 stored at server computing system 130 using various training or learning techniques, such as backpropagation of errors. In some implementations, performing backpropagation of errors may include performing truncated backpropagation over time. Model trainer 160 may perform various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained models.

[0067] Specifically, the model trainer 160 can train or pre-train the reinforcement learning agent model 140, entity model 142, and / or resource model 144 based on training data 162. Training data 162 may include labeled and / or unlabeled data. For example, training data 162 may include resource allocation data associated with a real-world environment (e.g., an industrial process, a recommender system, etc.).

[0068] In some implementations, if the user has already provided consent, the training examples can be provided by the user computing device 102 (e.g., based on communications previously provided by the user of user computing device 102). Therefore, in such an implementation, the model 120 provided to user computing device 102 can be trained by training computing system 150 on user-specific communication data received from user computing device 102. In some cases, this process can be referred to as a personalized model.

[0069] Model trainer 160 includes computer logic for providing the required functionality. Model trainer 160 can be implemented using hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium, such as RAM, hard disk, or optical or magnetic media.

[0070] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Typically, communication over network 180 can be carried via any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0071] Figure 1A An example computing system that can be used to implement this disclosure is shown. Other computing systems may also be used. For example, in some implementations, user computing device 102 may include model trainer 160 and training data 162. In such an implementation, model 120 may be trained and used locally at user computing device 102. In some such implementations, user computing device 102 may implement model trainer 160 to personalize model 120 based on user-specific data.

[0072] Figure 1B A block diagram depicts an example computing device 10 implemented according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0073] The computing device 10 includes multiple applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0074] like Figure 1B As shown, each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0075] Figure 1C A block diagram depicts an example computing device 50 implemented according to an example embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0076] Computing device 50 includes multiple applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using APIs (e.g., a common API across all applications).

[0077] The central intelligence layer comprises multiple machine learning models. For example, such as... Figure 1C As shown, each application can have its own machine learning model (e.g., a model) managed by a central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included within the operating system of computing device 50 or otherwise implemented by the operating system of computing device 50.

[0078] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for computing device 50. For example... Figure 1C As shown, the central device data layer can communicate with multiple other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0079] Example model layout

[0080] Figure 2 A block diagram of an example reinforcement learning model 200 according to an example embodiment of the present disclosure is depicted. In some implementations, the reinforcement learning model 200 is trained to receive an entity profile 202 describing at least one preference or demand of a simulated entity (e.g., an industrial process). In response to receiving the entity profile 202, an allocation output 204 describing the allocation of resources to the simulated entity (e.g., inputs to an industrial process, such as raw materials, fuel, setup, etc.) is provided. For example, according to various example configurations, the reinforcement learning agent model 200 may apply a learning strategy to generate the allocation output 204 in an attempt to maximize the cumulative reward received by the agent model over time.

[0081] Figure 3A block diagram of an example reinforcement learning simulation system 300 according to an exemplary embodiment of the present disclosure is depicted. The reinforcement learning simulation system 300 may include a reinforcement learning agent model 302, such as those referenced above. Figure 2 As stated above.

[0082] The reinforcement learning simulation system 300 can select at least one resource 308 to provide to the entity model 310 based on the resource allocation described in the allocation output 306. The entity model 308 can be configured to receive data describing the resource 308, and in response to receiving the data describing the resource 308, simulate a simulation response output 312, which describes the simulated entity's response to the data describing the at least one resource 308.

[0083] The reinforcement learning simulation system 300 can update at least one of the resource profiles or entity profiles 304 describing at least one resource based on the simulation response output 312.

[0084] Therefore, the reinforcement learning simulation system 300 can use entity model 310 to simulate the environment for reinforcement learning agent model 302, wherein one or more simulated entities respond to one or more resource allocations generated by reinforcement learning agent model 302 for the entities.

[0085] More specifically, the computing system can input entity profile 304 into reinforcement learning agent model 302 and receive allocation output 306 as the output of reinforcement learning agent model 302, which describes resource allocation to the simulated entity. The computing system can select at least one resource 308 to provide to entity model 310 based on the resource allocation described in allocation output 306. The computing system can provide resource 308 to entity model 310 and receive simulated response output 310 as the output of entity model, which describes the simulated entity's response to at least one resource 308.

[0086] According to one aspect of this disclosure, the computing system can update at least one of a resource profile or entity profile 304 describing at least one resource based on the simulated response output 312. For example, after simulating an entity's response to resource allocation, various characteristics or states of the entity model 310 can be updated or otherwise transformed. Some or all of the above steps can be performed iteratively to simulate the learning of the reinforcement learning agent model 302 over time in the simulated environment. Furthermore, updating the entity profile 304 and / or the resource profile can allow the corresponding states or characteristics of the entities and / or resources to change over time in the simulation to simulate dynamic entities and / or resources. Thus, the ability of the simulated entity characteristics, behaviors, or states to change in the simulated environment enables the reinforcement learning agent to learn strategies that explicitly consider and are based on the fact that entities may have dynamic and changing responses to resource allocation over time, and that such dynamic and changing responses may be a function of the resource allocation provided to the entity over time. In this way, aspects of this disclosure enable the learning of reinforcement learning agents with improved performance relative to dynamically changing resource-consuming entities in a simulated environment.

[0087] The disclosed systems and methods can be used to simulate various real-world entities and environments. As described above, in some implementations, the simulated entity may include an industrial process (e.g., manufacturing, power generation, etc.). Resource 308 may include inputs to the industrial process, such as raw materials, fuel, settings (e.g., temperature, processing rate, productivity, etc.). Simulation response output 312 may include updated states, state changes, or other data describing the industrial process or its changes in response to received resources.

[0088] As another example, the simulated entity may include a computing task or a source of computing tasks. Resource 308 may include computing resources used to run the computing task, such as workers (e.g., server computing devices, processor cores, physical computing devices, virtual machines, etc.). Simulated response output 312 may include updated status, status changes, or other data describing the response or change in response to receiving resources, computing tasks, or sources of computing tasks.

[0089] Figure 4 A block diagram of an example reinforcement learning simulation system 400 according to an exemplary embodiment of the present disclosure is depicted. The reinforcement learning simulation system 400 may include a reinforcement learning agent model 402 and an entity model 403, as referenced above. Figure 2 and Figure 3 As stated above.

[0090] The reinforcement learning simulation system 400 can select at least one resource 404 to provide to the entity model 403 based on the resource allocation described by the allocation output 406 output by the reinforcement learning agent model 402. The entity model 403 can be configured to receive data describing the resource 404, and in response to receiving the data describing the resource 404, simulate a simulated response output 408 describing the response of the simulated entity to the data describing the at least one resource 404.

[0091] According to one aspect of this disclosure, the computing system can update the resource profile 410 and / or entity profile 412 describing at least one resource based on the simulated response output 408 (e.g., using an entity transformation model 414). For example, after simulating an entity's response to a resource allocation, various characteristics or states of the entity model 403 can be updated or otherwise transformed. The computing system can update the resource profile 410 or entity profile 412 describing resource 404 based on the simulated response output 408. For example, system 400 may include a resource model 411 configured to receive data describing a plurality of resources including at least one resource (e.g., including resource profile 410). Resource model 411 may be configured to output resource observable features 413 in response to receiving data describing the plurality of resources (e.g., resource profile 410). A reinforcement learning agent model 402 can be trained to select an allocation output 406 based at least in part on the resource observable features 413. Thus, the computing system can simulate an environment in which resource characteristics change over time.

[0092] In some implementations, the reinforcement learning simulation system 400 may include an entity transformation model 414 configured to generate an updated set of entity hidden state features 416 in response to receiving data describing the simulation response output 408. The computational system may provide the data describing the simulation response output 408 to the entity transformation model 414 and update the entity profile 412 based on the entity hidden state features 416. For example, the entity profile 412 may include the entity hidden state features 416. Some or all of the entity hidden state features 416 may be hidden from the reinforcement learning agent model 402. The entity profile 412 may include entity observable features 418 that are accessible (e.g., input) to the reinforcement learning agent model 402. The entity observable features 418 may be updated based on the entity hidden state features 416. Therefore, some information about the entity may not be immediately discovered by the reinforcement learning agent model 402. The reinforcement learning agent model 402 may be trained to select resources to provide to the entity, allowing information about the entity to be discovered during simulation. Therefore, the reinforcement learning agent may be trained to balance the development and exploration of information about the entity in the context of a "multi-armed slot machine".

[0093] In some implementations, agent model 402 may include a reinforcement learning agent learned based on a reward that is a function of the simulated response output 408. The reward may be positively correlated with a desired characteristic of the simulated response output 408. Examples include output or performance metrics associated with industrial or computational processes. In another example, the reward may be positively correlated with one or more engagement metrics describing the human user's participation or positive feedback regarding the resources in the simulation.

[0094] In some implementations, entity profile 412 can describe a “stylized” model of the entity, where some or all of the entity’s features have interpretable meaning. Employing features with interpretable meaning can provide insights into how a particular entity response 408 affects the reinforcement learning agent model 402 and / or how the actions of the reinforcement learning agent model 402 affect the entity (e.g., as described in entity profile 412). For example, entity profile 412 may include or describe the requirements (e.g., temperature, rate, etc.) of industrial and / or computational processes. As another example, entity profile 412 may include a user profile describing the interests and / or preferences of a simulated human user.

[0095] Therefore, the computational system can use entity model 403, resource model 411, and / or entity transformation model 414 to simulate the environment of reinforcement learning agent model 402, wherein one or more simulated entities respond to one or more resource allocations generated for the entities by reinforcement learning agent model 402. Additionally, the system can simulate resource-consuming entities and / or resources that transform over time based on their interactions with the simulated environment and / or results observed in the simulated environment. For example, updating entity profiles 412 and / or resource profiles 410 can allow the respective states or characteristics of entities and / or resources to change over time in the simulation to simulate dynamic entities and / or resources.

[0096] Figure 5A A block diagram of system 500, according to aspects of this disclosure, is depicted for simulating a recommender system for recommending content to human users. Simulated entities may include simulated human users. Resources may include content for viewing or engaging with by simulated human users. Example resources include text, audio, or graphical content (e.g., images, videos, articles, or other media content). These resources may be collectively referred to as “documents.” Although system 500 is described below with reference to “simulated human users” and “documents,” it should be understood that aspects of system 500 can be used in other contexts, including industrial processes and / or computational processes.

[0097] The above is for reference only. Figure 4 Some elements of the described system 400 may correspond to elements of the system 500 in Figure 5. For example, Figure 4The simulated response output 408 can correspond to the simulated user response 508 in Figure 5. The user response 508 can include engagement metrics such as whether the document was viewed (e.g., "click"), interaction time, user rating, etc.

[0098] In some implementations, agent model 502 may include a reinforcement learning agent (e.g., simulated user response 508) learned based on a reward as a function of the simulated response output. The reward may be positively correlated with a desired feature of the simulated response output (e.g., simulated user response 508). For example, the reward may be positively correlated with one or more engagement metrics describing the simulated human user's participation or positive feedback regarding the resource.

[0099] In some implementations, the user profile 512 can describe a “stylized” model of the simulated human user, where some or all of the simulated human user’s characteristics have interpretable meaning. Employing features with interpretable meaning can provide insights into how a specific simulated user response 508 over time affects the reinforcement learning agent model 502 and / or how the actions of the reinforcement learning agent model 502 affect the simulated human user (e.g., the user profile 512). The user profile 512 can describe the simulated human user’s interests and / or preferences.

[0100] User profile 512 may include user observable features and / or context 518, user hidden state features 516, and / or a basic user profile 515, which may initially describe user hidden state features 516 before any updates occur. User transition model 514 may be configured to receive user hidden state features 516 from user profile 512, simulated user responses 508 from user selection model 503, and / or resources 504 (e.g., document sections) from document model 511. In response, user transition model 514 may be configured to output the next user hidden state feature 517. System 500 may then update existing user hidden state features 516 with the next user hidden state feature 517. All user hidden state features 516 may be hidden from reinforcement learning agent 502. User observable features 518 may be accessed by reinforcement learning agent 502 (e.g., input). User observable features 518 may be updated based on user hidden state features 516. Therefore, reinforcement learning agent 502 may not immediately discover some information about the user. The reinforcement learning agent 502 can be trained to select resources 504 (e.g., select documents contained in a forum section) to provide to the simulated human user, enabling information about the simulated human user to be discovered during the simulation. Therefore, the reinforcement learning agent 502 can be trained to balance the development and exploration of information about the simulated human user in a "multi-armed slot machine" context.

[0101] Therefore, the computational system can use user selection model 503, document model 511, and / or user transition model 514 to simulate the environment of reinforcement learning agent model 502, where one or more simulated human users respond to one or more additional resources (e.g., documents) selected by reinforcement learning agent model 502 for presentation to the simulated human users. Furthermore, the system can model the simulated human users and / or documents consuming resources, which change over time based on their interactions with the simulated environment and / or the outcomes observed in the simulated environment. For example, updating user profiles 512 and / or resource profiles 510 can allow the corresponding states or characteristics of the simulated human users and / or resources to change over time in the simulation to simulate dynamic human users and / or resources.

[0102] User profile 512 may include various information about the simulated human user. For example, user profile 512 may include one or more "interest" characteristics. Interest characteristics may include elements describing the simulated human user's associations with various topics (e.g., sports, music, etc.). The range of interest characteristics can be from a negative lower bound representing a strong dislike for the corresponding topic to a positive upper bound representing a strong liking for the corresponding topic.

[0103] As another example, user profile 512 may include one or more "budget" features. The budget feature can describe the amount of time available for simulated human users to interact with the document (e.g., for watching a video). The document can be provided to simulated human users until the budget reaches a minimum threshold (e.g., zero). Another simulated human user can then be selected for simulation. However, in some implementations, resources can be provided to multiple simulated human users (or other entities) simultaneously.

[0104] As described above, the user transition model 514 can update the user state based on the presented section 504 and the selected items as indicated in the simulated human user response 508.

[0105] Interest Updates

[0106] Interest can be updated only for documents that simulate user engagement (e.g., clicks, views):

[0107]

[0108] The above update can be scaled using the following update correlation function, where y represents the maximum possible update score and x represents the maximum point where the update should be 0.

[0109]

[0110] Figure 5BThe update correlation function for user transformation model 514 is shown when x = 1 and y = 0.3. Example diagram. For neutral interests, updates can be larger, while for more specific interests, updates can be smaller.

[0111] Masks can be applied so that only interests related to document attributes (e.g., matching) are updated. For example, an attribute could be a one-hot vector code.

[0112]

[0113] Based on the simulated user interest in document F(u, d), updates can be positive or negative with a certain probability. Therefore, the final update rule can be expressed as follows:

[0114]

[0115] In some implementations, the computational system may include a document model 511 configured to receive data describing multiple resources and, in response to receiving data describing multiple resources (e.g., including a document profile 510). The document model 511 may be configured to output document observable features 513. A reinforcement learning agent model 502 may be trained to assign an output 506 at least partially based on the document observable features 513. The document observable features 513 may describe a document accessible (e.g., as input) to the reinforcement learning agent model 502. More specifically, the computational system may input data describing multiple resources (e.g., document profile 510) into the document model 511 and receive the document observable features 513 as the output of the document model 511. The computational system may input the document observable features 513 into the reinforcement learning agent model 502. The document profile 510 may also include hidden features inaccessible (e.g., as input) to the reinforcement learning agent model 502. Therefore, the reinforcement learning agent 502 may be trained to balance the development and exploration of information about the document within the context of a "multi-armed slot machine".

[0116] Features of a document profile (e.g., observable and / or hidden features) can describe the document. As an example, document profile 510 and / or document observable features 513 may include "attribute" features, which include one or more elements describing the document's topic. "Attribute" features may include elements corresponding to their respective topics. The range of elements can be from minimum to maximum values ​​to indicate the relevance of the document to its respective topic.

[0117] As another example, document profile 510 and / or document observable features 513 may include a "length" feature. A length feature can describe the duration of time associated with a document engagement. For example, a length feature can describe the length of a video.

[0118] As another example, document profile 510 and / or document observable features 513 may include “quality” features. “Quality features” can describe the extent to which a document contains high-quality content, rather than content that initially appears interesting but does not provide meaningful or further engagement information (e.g., “clickbait”). “Quality” features can describe more objective quality measures, such as the video quality of a video, the writing quality of an article, etc.

[0119] In some implementations, the computational system may simulate providing a simulated human user with multiple documents 504 (e.g., "sections" of documents), from which the simulated human user can "select" resources to engage with or consume. More specifically, a simulated user response 508 may include a selection of fewer documents than all available to the simulated human user. For example, a reinforcement learning agent 502 may select multiple videos, articles, images, advertisements, etc., and provide documents to the simulated human user in section 504. Section 504 may include a list of recommended documents for viewing or otherwise engaging with. In such an implementation, a simulated user response 508 may describe a simulated human user's selection of fewer than all documents in section 504. For example, a simulated human user may select one document from multiple documents presented in a section.

[0120] For example, user selection model 503 may include a discrete selection model. A discrete selection model is typically configured to select one item from a finite group of items. A discrete selection model may be configured to select one document from multiple documents 504 (e.g., forums). Discrete selection models can employ various suitable functions, including multinomial proportional functions, multinomial logit functions, exponential cascade functions, and / or similar functions.

[0121] Discrete choice models can compute unstandardized scores. As user interests and document properties dot product between:

[0122]

[0123] Given a vector of unstandardized scores F for different documents provided to simulated human users, user selection model 503 can sample / select documents based on a "probability function" of selection model 503. Example probability functions include multinomial proportional functions, multinomial logit functions, and exponential cascade functions.

[0124] Multiple ratios

[0125] This model calculates the probability of selecting document d within the forum as follows:

[0126]

[0127] because The score can be negative, so the score can be shifted to the minimum possible score (in the example shown here, it's -1) to ensure a valid probability. Additionally, a "noclick" score can be added to F to account for results where no item was selected.

[0128] Multiple Logit

[0129] The multinomial logit model can calculate the probability of selecting document d in the selected forum, as shown below:

[0130]

[0131] Using this model, no additional shifting is typically required. The same "no-click" score can be used to simulate no selection.

[0132] Exponential cascade

[0133] Both multi-proportional and multi-logit models typically do not consider item location within a forum when assigning scores. In contrast, the exponential cascade model assumes that attention is given to one item at a time, with attention decreasing exponentially for items further away in the forum. The exponential cascade model also assumes a base probability P(u,d) of having sufficient attention for each item. The click probability of an item at position i can be calculated as:

[0134] for i = 0, 1, 2, … slate_size

[0135] in Represents the probability of the basic choice; `d` represents the decay factor; and `slate_size` represents the number of documents in the forum. `P(u, d)` can represent the probability of selecting an item regardless of position (e.g., using one of the two models mentioned above). Items can be considered in order from `i = 0` to `slate_size`. Once an item is selected, the process can be terminated. The Conditional State Deep Network (CSDQN) algorithm described in this paper assumes that the selection model is of the multinomial scaling type.

[0136] Budget Update

[0137] Budget updates are taken from a utility perspective. We assume that simulated human users receive utility based on their expected utility. e Select a document.

[0138]

[0139] However, the utility actually received by simulated human users is...r It is its expected utility, Utility e The weighted sum of document quality:

[0140]

[0141] The results below assign all quality to document quality to show a larger gap, but using smaller amounts also works. The budget will be updated based on the received utility and the length of the video. If simulated human users are “watching” the video, the budget will be updated as follows:

[0142]

[0143] in It is a portion of the video length used to extend the session, multiplied by a normalization constant to make the Utility r The range is between [-1, 1]. Therefore, when simulated users watch higher quality videos, simulated human users are willing to extend the session, while displaying low quality content will shorten the session.

[0144] If the video is not clicked, a constant step penalty (0.5 time units in our example) will be applied:

[0145]

[0146] Response Model

[0147] Simulated human users provide responses for each element on the forum. There are currently two response variables:

[0148] 1. "Click": Whether the document was clicked; and

[0149] 2. "Viewing Time": The simulated length of time a human user views a document.

[0150] For this experiment, we assume that the clicked video has been fully consumed or viewed. The agent is trained to optimize the total viewing time for the entire session.

[0151] The systems and methods disclosed herein define specific technical implementations for simulating the allocation of resources to multiple entities. Therefore, the implementations of the described techniques provide technical functionality that allows for virtual experimentation, which is part of the practical and practice-oriented toolkit of a skilled technician. However, the systems and methods of this disclosure offer numerous additional technical effects and benefits. As an example, the systems and methods described herein can aid in the development and / or optimization of reinforcement learning agents for controlling industrial processes, such as power generation. Improving the efficiency of controlling and / or monitoring these processes can reduce waste and save energy.

[0152] Example Method

[0153] Figure 6 A flowchart depicts an example method according to an exemplary embodiment of this disclosure. Although for purposes of illustration and discussion, Figure 6 The steps are described in a specific order, but the method of this disclosure is not limited to the order or arrangement specifically shown. The steps of method 600 may be omitted, rearranged, combined, and / or modified in various ways without departing from the scope of this disclosure.

[0154] At point 602, the computational system can input an entity profile describing at least one of the preferences or needs of the simulated entity into the reinforcement learning agent model, as shown in the reference above. Figures 2 to 5B As stated above.

[0155] At 604, the computational system can receive an allocation output as the output of a reinforcement learning agent model, which describes the resource allocation for the simulated entity, such as the one referenced above. Figures 2 to 5B As stated above.

[0156] At 606, the computing system can select at least one resource to simulate the provision to the entity model based on the resource allocation described by the allocation output, for example, as referenced above. Figures 3 to 5B As stated above.

[0157] At point 608, the computing system can provide data describing at least one resource to the entity model, as shown in the reference above. Figures 3 to 5B As stated above.

[0158] At 610, the computing system can receive a simulated response output as the output of an entity model, which describes the simulated entity's response to at least one resource, such as as referenced above. Figure 3 As shown in Figure 5B.

[0159] At point 612, the computing system can update at least one of the resource profiles or entity profiles describing at least one resource based on the simulation response output, for example, as referenced above. Figures 3 to 5B As stated above.

[0160] Simulation results

[0161] First, we consider the case where simulated human users follow a multinomial proportional model. This model is matched with the predicted click-through rate (pCTR) model used in Conditional State Deep Q Network (CSDQN) models. Figure 7A , Figure 7B and Figure 7CThe figure shows the average program length, pCTR on the segment, and average return of the training step (non-program) function for each of the CSDQN Lifetime Value (LTV) model, CSDQN myopic model (gamma = 0), perfect greedy model (always selecting the top K pCTR items), and random multinomial scaling model. As shown, the LTV method provides the highest return (2 time units, approximately 1% of the budget, a 1.2% increase over myopic / greedy). As expected, the myopic model converges to the same return as the perfect greedy model. The effects on the cluster shown over time indicate that the high degree of randomness is the reason for the “small” gain.

[0162] Figures 8A to 8U The above reference is shown. Figures 7A to 7C The proportion of times each recommended section in the agent of the experiment is viewed over time. Figures 8A to 8M It is a low-quality cluster; Figure 8N and Figure 8O It is of medium quality; and Figures 8P to 8U It is high quality. As shown in the figure, the CSDQN LTV model learns over time to suggest higher quality clusters. For low-quality clusters, the CSDQN LTV model is generally lower than other clusters, while for high-quality clusters, it is higher than other clusters. Since there are a total of 20 clusters, and only 10 clusters are displayed at a time, high-quality clusters are not always selected. Therefore, sometimes the agent must select low-quality clusters. The proportion of times the video was not watched is... Figure 8U As shown, the monitoring ratio of greedy agents on all clusters labeled “AverageClusterWatch_None” (except for “AverageClusterWatch_None”) converges to approximately the same value (~0.035-0.036). Since the interests of simulated human users are also uniformly distributed, it is recommended to choose clusters uniformly.

[0163] Figures 9A to 9C This illustrates what happens if the underlying user choice model differs. The CSDQN model assumes user choices are performed in multinomial proportions, but the actual user model is an exponential concatenation with multinomial proportion base probabilities. To make this experiment comparable to the last one, the parameters of the choice model are adjusted so that the stochastic policies of both the multinomial proportion and exponential concatenation models produce similar rewards.

[0164] Figures 10A to 10C The results are shown using a cascaded model for user selection and multiple scaling in the CSDQN model. Using this selection model, an increase in viewing time of approximately 2 to 2.5% is expected. The results show a similar effect to the first case. Finally, Figures 11A to 11U The above reference is shown. Figures 10A to 10CThe data described in the experiment. These graphs were generated from a single run, with each point evaluated 50 times. More specifically, Figures 11A to 11U The above reference is shown. Figures 10A to 10C The proportion of times each recommended section in the experiment's agent was viewed over time. Figures 11A to 11M It is a low-quality cluster; Figure 11N and Figure 11O It is of medium quality; and Figures 11P to 11U It is of high quality.

[0165] Additional disclosure

[0166] This paper discusses technical reference servers, databases, software applications, and other computer-based systems, as well as the actions taken by these systems and the information sent to and from them. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions within and between components. For example, the processes discussed in this paper can be implemented using a single device or component, or a combination of multiple devices or components. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can run sequentially or in parallel.

[0167] Although the subject matter has been described in detail with respect to various specific exemplary embodiments of the invention, each example is provided by way of illustration and not limitation. Changes, variations, and equivalents of these embodiments will be readily apparent to those skilled in the art upon gaining an understanding of the foregoing. Therefore, this disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter, as will be apparent to those skilled in the art. For example, features shown or described as part of one embodiment may be used with another embodiment to produce yet another embodiment. Therefore, this disclosure is intended to cover such changes, variations, and equivalents.

Claims

1. A computing system for training a machine learning recommendation system, the computing system comprising: One or more processors; One or more non-transitory computer-readable media, collectively storing instructions executable by the one or more processors to cause the computing system to perform operations, including: Provide machine learning recommendation systems with entity profiles of simulated entities, wherein the entity profiles include interpretable interest features; Obtain recommendations for resources to be consumed by simulated entities from machine learning recommendation systems; Based on the entity profile, identify one or more engagement metrics associated with consumption; and Update one or more parameters of the machine learning recommendation system to increase the reward determined based on the one or more participation metrics.

2. The computing system according to claim 1, wherein, A simulated entity is a simulated human user, and the entity profile includes a user profile describing topics associated with the simulated human user.

3. The computing system according to claim 1, wherein, Explainable interest characteristics change over time.

4. The computing system according to claim 2, wherein, The resources include documents, and the recommendations are based on document features that indicate the relevance of a document to at least one of the topics.

5. The computing system according to claim 1, wherein, The one or more participation metrics describe at least one of the following: interaction time, consumption, number of participations, or rating of the simulated entity relative to the resource.

6. The computing system according to claim 1, wherein, The one or more participation metrics are determined based on the budgetary characteristics of the entity profile.

7. The computing system according to claim 6, wherein, Budget characteristics are based on the utility of resources to simulated entities.

8. The computing system according to claim 1, wherein, The operation includes updating the entity profile based on the one or more participation metrics.

9. A computer-implemented method, comprising: Provide machine learning recommendation systems with entity profiles of simulated entities, wherein the entity profiles include interpretable interest features; Obtain recommendations for resources to be consumed by simulated entities from machine learning recommendation systems; Based on the entity profile, identify one or more engagement metrics associated with consumption; and Update one or more parameters of the machine learning recommendation system to increase the reward determined based on the one or more participation metrics.

10. The method according to claim 9, wherein, A simulated entity is a simulated human user, and the entity profile includes a user profile describing topics associated with the simulated human user.

11. The method according to claim 9, wherein, Explainable interest characteristics change over time.

12. The method according to claim 10, wherein, The resources include documents, and the recommendations are based on document features that indicate the relevance of a document to at least one of the topics.

13. The method according to claim 9, wherein, The one or more participation metrics describe at least one of the following: interaction time, consumption, number of participations, or rating of the simulated entity relative to the resource.

14. The method according to claim 9, wherein, The one or more participation metrics are determined based on the budgetary characteristics of the entity profile.

15. The method according to claim 14, wherein, Budget characteristics are based on the utility of resources to simulated entities.

16. The method of claim 9, comprising: Update the entity profile based on one or more of the participating metrics.

17. One or more non-transitory computer-readable media, commonly storing instructions executable by one or more processors to cause a computing system to perform operations, said operations including: Provide machine learning recommendation systems with entity profiles of simulated entities, wherein the entity profiles include interpretable interest features; Obtain recommendations for resources to be consumed by simulated entities from machine learning recommendation systems; Based on the entity profile, identify one or more engagement metrics associated with consumption; and Update one or more parameters of the machine learning recommendation system to increase the reward determined based on the one or more participation metrics.

18. One or more non-transitory computer-readable media according to claim 17, wherein, A simulated entity is a simulated human user, and the entity profile includes a user profile describing topics associated with the simulated human user.

19. One or more non-transitory computer-readable media according to claim 18, wherein, The resources include documents, and the recommendations are based on document features that indicate the relevance of a document to at least one of the topics.

20. One or more non-transitory computer-readable media according to claim 17, wherein, The operation includes updating the entity profile based on the one or more participation metrics.

Images