Material delivery strategy determination method and device, and electronic device

Abstract

The application discloses a material putting strategy determination method and device and electronic equipment. It is related to the field of artificial intelligence. The method comprises the following steps: obtaining expected conversion information and environment state information corresponding to a target material; using a first prediction model in a strategy prediction model to predict according to the expected conversion information and the environment state information, obtaining a first candidate putting action and environment state information at a next time step; using a second prediction model in the strategy prediction model to predict according to the environment state information and the environment state information at the next time step, obtaining a second candidate putting action; and using an evaluation model in the strategy prediction model to determine a target putting strategy corresponding to the target material according to the first candidate putting action and the second candidate putting action. The application solves the technical problem of low accuracy in determining the material putting strategy in the related art.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method, apparatus, and electronic device for determining a material delivery strategy. Background Technology

[0002] Automatically determining the creative delivery strategy is a key technology to help creative advertisers improve recommendation effectiveness. Currently, related technologies mainly rely on the current environment to make decisions when determining the creative delivery strategy, which has relatively low accuracy.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for determining a content delivery strategy, in order to at least solve the technical problem of low accuracy in determining a content delivery strategy in related technologies.

[0005] According to one aspect of the embodiments of this application, a method for determining a creative delivery strategy is provided, comprising: obtaining expected conversion information and environmental state information corresponding to a target creative; making a prediction based on the expected conversion information and environmental state information using a first prediction model in a strategy prediction model to obtain a first candidate delivery action and environmental state information for the next time step; making a prediction based on the environmental state information and environmental state information for the next time step using a second prediction model in the strategy prediction model to obtain a second candidate delivery action; and determining a target delivery strategy corresponding to the target creative based on the first candidate delivery action and the second candidate delivery action using an evaluation model in the strategy prediction model.

[0006] Furthermore, the first prediction model in the strategy prediction model makes predictions based on the expected conversion information and environmental state information to obtain the first candidate delivery action and the environmental state information of the next time step, including: encoding the expected conversion information and environmental state information respectively through the first encoder in the first prediction model to obtain the first prediction vector and the second prediction vector; processing the first prediction vector and the second prediction vector respectively through the attention layer in the first prediction model to obtain the first intermediate representation vector and the second intermediate representation vector; and converting and decoding the first intermediate representation vector and the second intermediate representation vector respectively through the output layer in the first prediction model to obtain the first candidate delivery action and the environmental state information of the next time step.

[0007] Furthermore, the second prediction model in the strategy prediction model makes predictions based on the environmental state information and the environmental state information of the next time step, and obtains the second candidate delivery action by: performing information fusion processing on the environmental state information and the environmental state information of the next time step through the input layer of the second prediction model to obtain the fused state representation vector; performing feature extraction processing on the fused state representation vector through the hidden layer of the second prediction model to obtain the target state representation vector; and performing transformation processing on the target state representation vector through the output layer of the second prediction model to obtain the second candidate delivery action.

[0008] Furthermore, the target delivery strategy corresponding to the target material is determined by the evaluation model in the strategy prediction model based on the first candidate delivery action and the second candidate delivery action. This includes: receiving the first candidate delivery action, the second candidate delivery action, and environmental state information through the input layer of the evaluation model; generating a first representation vector based on the first candidate delivery action and the environmental state information; generating a second representation vector based on the second candidate delivery action and the environmental state information; performing feature extraction and mapping processing on the first and second representation vectors through the hidden layer of the evaluation model to obtain a first value and a second value; and determining the target delivery action from the first and second candidate delivery actions based on the magnitude of the first and second values ​​through the output layer of the evaluation model, and determining the target delivery strategy based on the target delivery action.

[0009] Furthermore, the strategy prediction model is trained using the following steps: A training sample set is obtained, comprising sequence information of multiple sample materials, including at least historical expected conversion information, historical delivery action information, and historical environmental state information; the training sample set is processed using the first prediction model in the initial prediction model to obtain the first predicted sample candidate delivery actions and the predicted sample environmental state information for the next time step corresponding to each of the multiple sample materials; the historical environmental state information and the predicted sample environmental state information for the next time step corresponding to each of the multiple sample materials are processed using the second prediction model in the initial prediction model to obtain the second predicted sample candidate delivery actions corresponding to each of the multiple sample materials; the initial prediction model is trained based on the target loss function, the first predicted sample candidate delivery actions, the predicted sample environmental state information for the next time step, the second predicted sample candidate delivery actions, and the training sample set until preset conditions are met, thus obtaining the strategy prediction model.

[0010] Furthermore, the initial prediction model is trained based on the target loss function, the first predicted sample candidate delivery action, the predicted sample environment state information at the next time step, the second predicted sample candidate delivery action, and the training sample set until the preset conditions are met, resulting in a policy prediction model. This includes: training the first prediction model in the initial prediction model based on the first loss function in the target loss function, the first predicted sample candidate delivery action, historical delivery action information, the predicted sample environment state information at the next time step, and historical environment state information to obtain a third prediction model; training the second prediction model in the initial prediction model based on the second loss function in the target loss function, the second predicted sample candidate delivery action, and historical delivery action information to obtain a fourth prediction model; determining the third loss function based on the first loss function, the second loss function, and the preset action value function, and jointly optimizing the third and fourth prediction models based on the third loss function and the training sample set until the preset conditions are met to obtain a policy prediction model.

[0011] Furthermore, after determining the target delivery strategy corresponding to the target material, the method also includes: executing the target delivery strategy, collecting feedback information of the target material after the execution of the target delivery strategy; updating the environmental status information based on the feedback information to obtain the updated environmental status information.

[0012] According to another aspect of the embodiments of this application, a method for determining a material delivery strategy is also provided, comprising: obtaining expected conversion information and environmental state information corresponding to a target material uploaded by a client; performing prediction in a cloud server using a first prediction model in a strategy prediction model based on the expected conversion information and environmental state information to obtain a first candidate delivery action and environmental state information for the next time step; performing prediction in a second prediction model in the strategy prediction model based on the environmental state information and environmental state information for the next time step to obtain a second candidate delivery action; determining a target delivery strategy corresponding to the target material using an evaluation model in the strategy prediction model based on the first candidate delivery action and the second candidate delivery action; and feeding back the target delivery strategy corresponding to the target material to the client.

[0013] According to another aspect of the embodiments of this application, a device for determining a material delivery strategy is also provided, comprising: a first acquisition unit, configured to acquire expected conversion information and environmental state information corresponding to a target material; a first processing unit, configured to perform prediction based on the expected conversion information and environmental state information using a first prediction model in a strategy prediction model to obtain a first candidate delivery action and environmental state information at the next time step; a second processing unit, configured to perform prediction based on the environmental state information and environmental state information at the next time step using a second prediction model in a strategy prediction model to obtain a second candidate delivery action; and a first determination unit, configured to determine a target delivery strategy corresponding to a target material based on the first candidate delivery action and the second candidate delivery action using an evaluation model in a strategy prediction model.

[0014] Further, the first processing unit includes: a first processing subunit, used to encode the expected transformation information and the environmental state information respectively through the first encoder in the first prediction model to obtain a first prediction vector and a second prediction vector; a second processing subunit, used to process the first prediction vector and the second prediction vector respectively through the attention layer in the first prediction model to obtain a first intermediate representation vector and a second intermediate representation vector; and a third processing subunit, used to convert and decode the first intermediate representation vector and the second intermediate representation vector respectively through the output layer in the first prediction model to obtain the first candidate delivery action and the environmental state information of the next time step.

[0015] Furthermore, the second processing unit includes: a fourth processing subunit, used to perform information fusion processing on the environmental state information and the environmental state information of the next time step through the input layer in the second prediction model to obtain a fused state representation vector; a fifth processing subunit, used to perform feature extraction processing on the fused state representation vector through the hidden layer in the second prediction model to obtain a target state representation vector; and a sixth processing subunit, used to perform transformation processing on the target state representation vector through the output layer in the second prediction model to obtain a second candidate delivery action.

[0016] Further, the first determining unit includes: a first determining subunit, used to receive a first candidate delivery action, a second candidate delivery action, and environmental state information through the input layer of the evaluation model, generate a first representation vector based on the first candidate delivery action and the environmental state information, and generate a second representation vector based on the second candidate delivery action and the environmental state information; a second determining subunit, used to perform feature extraction and mapping processing on the first representation vector and the second representation vector through the hidden layer of the evaluation model to obtain a first value and a second value; and a third determining subunit, used to determine the target delivery action from the first candidate delivery action and the second candidate delivery action based on the magnitude of the first value and the second value through the output layer of the evaluation model, and determine the target delivery strategy based on the target delivery action.

[0017] Furthermore, the device also includes: a second acquisition unit for acquiring a training sample set, wherein the training sample set includes sequence information of multiple sample materials, and the sequence information includes at least historical expected conversion information, historical delivery action information, and historical environmental state information; a third processing unit for processing the training sample set using the first prediction model in the initial prediction model to obtain the first predicted sample candidate delivery action and the predicted sample environmental state information for the next time step corresponding to the multiple sample materials; a fourth processing unit for processing the historical environmental state information and the predicted sample environmental state information for the next time step corresponding to the multiple sample materials using the second prediction model in the initial prediction model to obtain the second predicted sample candidate delivery action corresponding to the multiple sample materials; and a fifth processing unit for training the initial prediction model based on the target loss function, the first predicted sample candidate delivery action, the predicted sample environmental state information for the next time step, the second predicted sample candidate delivery action, and the training sample set until a preset condition is met to obtain a strategy prediction model.

[0018] Furthermore, the fifth processing unit includes: a first training subunit, used to train the first prediction model in the initial prediction model based on the first loss function in the target loss function, the first predicted sample candidate delivery action, historical delivery action information, the prediction sample environment state information of the next time step, and the historical environment state information, to obtain a third prediction model; a second training subunit, used to train the second prediction model in the initial prediction model based on the second loss function in the target loss function, the second predicted sample candidate delivery action, and the historical delivery action information, to obtain a fourth prediction model; and a third training subunit, used to determine the third loss function based on the first loss function, the second loss function, and the preset action value function, and to jointly optimize the third prediction model and the fourth prediction model based on the third loss function and the training sample set until the preset conditions are met, to obtain a policy prediction model.

[0019] Furthermore, the device also includes: an execution unit, used to execute the target delivery strategy after determining the target material, and collect feedback information of the target material after the execution of the target delivery strategy; and an update unit, used to update the environmental state information based on the feedback information to obtain the updated environmental state information.

[0020] According to another aspect of the present invention, an electronic device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the method for determining the material delivery strategy described above during runtime.

[0021] According to another aspect of the present invention, a computer-readable storage medium is also provided, the storage medium storing a program, wherein, when the program is running, a method for determining the material delivery strategy that controls the device where the storage medium is located to execute any of the above-mentioned methods is provided.

[0022] According to another aspect of the present invention, a computer program product is also provided, including a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement a method for determining a material delivery strategy as described above.

[0023] In this embodiment, the expected conversion information and environmental state information corresponding to the target material are obtained; the first prediction model in the strategy prediction model predicts based on the expected conversion information and environmental state information to obtain the first candidate delivery action and the environmental state information of the next time step; the second prediction model in the strategy prediction model predicts based on the environmental state information and the environmental state information of the next time step to obtain the second candidate delivery action; the evaluation model in the strategy prediction model determines the target delivery strategy corresponding to the target material based on the first candidate delivery action and the second candidate delivery action. The first prediction model and the second prediction model are used to generate dual-path delivery actions, and the evaluation model is introduced to evaluate and select the candidate actions generated by the two paths. This achieves accurate determination of the target delivery strategy, improves the quality of decision-making, and thus improves the accuracy of the material delivery strategy. This solves the technical problem of low accuracy in determining the material delivery strategy in related technologies. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 This is a schematic diagram of a computer terminal provided according to Embodiment 1 of this application;

[0026] Figure 2 This is a flowchart of the method for determining the material delivery strategy according to Embodiment 1 of this application;

[0027] Figure 3 This is a flowchart illustrating an optional model inference stage according to Embodiment 1 of this application;

[0028] Figure 4 This is a flowchart illustrating an optional model training phase provided according to Embodiment 1 of this application;

[0029] Figure 5This is a flowchart of the method for determining the material delivery strategy according to Embodiment 2 of this application;

[0030] Figure 6 This is a schematic diagram of the device for determining the material delivery strategy according to Embodiment 3 of this application;

[0031] Figure 7 This is a structural block diagram of an electronic device provided according to Embodiment 4 of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0035] Example 1

[0036] According to an embodiment of this application, a method for determining a material delivery strategy is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a method for determining a content delivery strategy is shown. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include, but is not limited to, processing devices such as microprocessors (MCUs) or field-programmable gate arrays (FPGAs), and the processor set 102 may include a processor set, Figure 1 The data is illustrated using 102a, 102b, ..., 102n. A memory 104 is used for storing data, and a transmission device 106 is used for communication functions. In addition, it may include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0038] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be wholly or partially embodied in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element within the computer terminal 10 (or mobile device).

[0039] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the material delivery strategy determination method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned material delivery strategy determination method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0040] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0041] The display may be a touchscreen LCD display that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0042] In the field of creative placement (such as e-commerce advertising and video recommendations), automatically determining the creative placement strategy is a key technology to help creative placement users improve recommendation results. It can help creative placement users win traffic while meeting multiple constraints such as resources and costs.

[0043] Currently, when determining the material delivery strategy, related technologies typically make decisions based solely on the current delivery environment, lacking long-term insights into historical information, or modeling only a single dimension. This leads to a strong reliance on data enhancement quality and strategies, posing a risk of statistical distribution bias and resulting in relatively low accuracy.

[0044] Against the above-mentioned technical background, this application provides as follows Figure 2 The method for determining the material delivery strategy is shown. Figure 2 This is a flowchart of a method for determining a material delivery strategy according to Embodiment 1 of this application. The method includes:

[0045] Step S201: Obtain the expected conversion information and environmental status information corresponding to the target material.

[0046] Optionally, the expected conversion information and environmental status information corresponding to the target material can be obtained through the material delivery strategy determination system. The target material can be a product advertisement, short video, or other material waiting to be delivered and displayed. The expected conversion information is the current expected future return, such as the expected number of clicks, conversions, etc. The environmental status information represents the current environmental status, such as advertiser identification information, current delivery resources, remaining delivery resources, historical click-through rate and conversion rate, time step, etc. This information reflects the immediate conditions and background of material delivery.

[0047] Step S202: Based on the expected conversion information and environmental state information, the first prediction model in the strategy prediction model makes a prediction to obtain the first candidate delivery action and the environmental state information of the next time step.

[0048] Optionally, the strategy prediction model includes a first prediction model, which can be a trained Decision Transformer (DT) model. The DT model makes predictions based on the current expected future return and the current environmental state, and can obtain the first candidate action and the environmental state information of the next time step.

[0049] Step S203: The second prediction model in the strategy prediction model makes a prediction based on the environmental state information and the environmental state information of the next time step to obtain the second candidate delivery action.

[0050] Optionally, the policy prediction model also includes a second prediction model, which can be a trained Inverse Dynamics Module (IDM), such as a multilayer perceptron or other types of neural network structures, to achieve the inverse mapping from state to action. The IDM predicts based on the current environmental state (i.e., the environmental state at the current time step) and the environmental state information at the next time step, and can obtain the second candidate action.

[0051] Optionally, the first and second candidate placement actions can be bid amounts or adjustment factors relative to the previous action, such as increasing by 5%, keeping it unchanged, or decreasing by 10%.

[0052] Step S204: Based on the first candidate delivery action and the second candidate delivery action, the evaluation model in the strategy prediction model determines the target delivery strategy corresponding to the target material.

[0053] Optionally, the strategy prediction model also includes an evaluation model, which can be a trained reinforcement learning model that can calculate the evaluation values ​​corresponding to the first candidate delivery action and the second candidate delivery action respectively, and take the candidate delivery action with the higher evaluation value as the target delivery strategy.

[0054] In an alternative embodiment, the following can be employed: Figure 3 The diagram shown illustrates how the target delivery strategy is determined. Figure 3 This is a flowchart illustrating the optional model inference stage provided in Embodiment 1 of this application, as follows: Figure 3 As shown, the current expected future return and current environmental status Input strategy prediction model, the DT model (first prediction model) in the strategy prediction model is based on the current expected future return. and current environmental status Make predictions to obtain the first candidate deployment action. and the environmental state at the next time step The IDM (Second Prediction Model) in the strategy prediction model is based on the current environmental state. and the environmental state at the next time step Make a prediction to obtain the second candidate deployment action. The Q-value prediction module (i.e., the evaluation model) in the strategy prediction model calculates the first candidate deployment action. Second candidate delivery action Based on the corresponding evaluation value, candidate actions with higher evaluation values ​​will be selected as target actions. , means as follows:

[0055]

[0056] Optionally, the material delivery strategy determination system can serve as an intelligent material delivery hosting system, adopting a daily rolling periodic deployment mode. The system uses historical bidding trajectory data from the previous week to train and update the model, and then deploys the new model to the online delivery platform to execute real-time bidding tasks. The new data generated in this process will be collected and used for the next round of model iteration, effectively improving the intelligence level, real-time decision-making, and resource stability of automatic bidding.

[0057] In this solution, a first prediction model and a second prediction model are used to generate dual-path delivery actions. At the same time, an evaluation model is introduced to evaluate and select the best candidate actions generated by the two paths, thereby achieving accurate determination of the target delivery strategy and improving the quality of decision-making. This achieves the technical effect of improving the accuracy of material delivery strategy, and solves the technical problem of low accuracy in determining material delivery strategy in related technologies.

[0058] To accurately predict the first candidate delivery action and the environmental state information of the next time step, the method for determining the material delivery strategy provided in Embodiment 1 of this application includes the following steps: the first prediction model in the strategy prediction model makes predictions based on the expected conversion information and the environmental state information to obtain the first candidate delivery action and the environmental state information of the next time step. This includes: encoding the expected conversion information and the environmental state information using the first encoder in the first prediction model to obtain a first prediction vector and a second prediction vector; processing the first prediction vector and the second prediction vector using the attention layer in the first prediction model to obtain a first intermediate representation vector and a second intermediate representation vector; and converting and decoding the first intermediate representation vector and the second intermediate representation vector using the output layer in the first prediction model to obtain the first candidate delivery action and the environmental state information of the next time step.

[0059] Optionally, the DT model includes at least an encoding layer (i.e., a first encoder), an attention layer, and an output layer. The first encoder encodes the current expected future reward into a first prediction vector and the current environment state into a second prediction vector. The attention layer uses a multi-layer self-attention mechanism to model the sequence dependencies of the input vectors (i.e., the first and second prediction vectors) respectively, capturing the global dependencies between elements in the input vectors. Through a forward propagation process (e.g., using a feedforward neural network to perform a nonlinear transformation on the modeled vectors), it generates intermediate representation vectors containing actions and intermediate representation vectors containing state prediction information (i.e., the first and second intermediate representation vectors). The output layer is used to transform and decode the first and second intermediate representation vectors respectively to obtain the first candidate delivery action and the environment state information of the next time step. For example, the output layer predicts the first candidate delivery action through a fully connected neural network and a suitable activation function, and predicts the environment state information of the next time step through another independent set of fully connected neural networks and a suitable activation function.

[0060] By using the first encoder, attention layer, and output layer, accurate prediction of the first candidate delivery action and the environmental state information of the next time step is achieved, thereby improving the accuracy of the material delivery strategy.

[0061] To improve the accuracy of action generation, in the method for determining the material delivery strategy provided in Embodiment 1 of this application, the second prediction model in the strategy prediction model makes predictions based on environmental state information and environmental state information at the next time step to obtain a second candidate delivery action. This includes: performing information fusion processing on the environmental state information and environmental state information at the next time step through the input layer of the second prediction model to obtain a fused state representation vector; performing feature extraction processing on the fused state representation vector through the hidden layer of the second prediction model to obtain a target state representation vector; and performing transformation processing on the target state representation vector through the output layer of the second prediction model to obtain the second candidate delivery action.

[0062] Optionally, an IDM (Integrated Device Model) includes at least an input layer, hidden layers, and an output layer. In the first few layers of the neural network, the IDM extracts and transforms features from the input state data through a series of linear layers and non-linear activation functions. The output layer of the IDM is responsible for predicting the action to be delivered. Taking a multilayer perceptron as an example, the environmental state at the current time step and the environmental state at the next time step are input into the first layer of the neural network for initial information fusion processing, resulting in a fused state representation vector. This fused state representation vector is then passed to the subsequent hidden layer neural network units. Each neural network unit receives the output vector from the previous layer, processes it through an activation function, and outputs a new vector representation until the hidden layer neural network unit outputs the target state representation vector. This target state representation vector is then input into the model's output layer, where it undergoes a linear transformation to obtain the second candidate action to be delivered.

[0063] By introducing IDM, an alternative safe path is provided for action generation, thereby improving the accuracy of action generation.

[0064] To accurately determine the target delivery strategy, the method for determining the delivery strategy of the material provided in Embodiment 1 of this application, which uses an evaluation model in the strategy prediction model to determine the target delivery strategy corresponding to the target material based on a first candidate delivery action and a second candidate delivery action, includes: receiving the first candidate delivery action, the second candidate delivery action, and environmental state information through the input layer of the evaluation model; generating a first representation vector based on the first candidate delivery action and the environmental state information; generating a second representation vector based on the second candidate delivery action and the environmental state information; performing feature extraction and mapping processing on the first and second representation vectors through the hidden layer of the evaluation model to obtain a first value and a second value; and determining the target delivery action from the first and second candidate delivery actions based on the magnitude of the first and second values ​​through the output layer of the evaluation model, and determining the target delivery strategy based on the target delivery action.

[0065] Optionally, such as Figure 3As shown, the evaluation model first receives two candidate delivery actions (i.e., the first candidate delivery action). Second candidate delivery action and the current environmental status The input layer performs preliminary processing on this information, such as normalization and encoding, to ensure that all features can be processed on the same scale, that is, converting states and actions into corresponding vector representations, thus obtaining... The corresponding first representation vector and The corresponding second representation vector.

[0066] Optionally, the current environment state can be calculated through a series of neural network operations in the hidden layers. The Q values ​​corresponding to the next two candidate actions are used to obtain the first Q value and the second Q value (i.e., the first numerical value and the second numerical value). That is, features are extracted from the first representation vector and the second representation vector respectively, and mapped to a continuous space Q value, which represents the expected future reward of taking the action in the current state.

[0067] Optionally, the output layer compares the magnitudes of the first Q-value and the second Q-value, and selects the candidate delivery action with the higher Q-value as the target delivery action, thereby obtaining the target delivery strategy.

[0068] The evaluation model enables the reasonable selection of candidate actions from different model perspectives, thereby improving the accuracy of the material delivery strategy.

[0069] To improve the accuracy of model action generation, in the method for determining the material delivery strategy provided in Embodiment 1 of this application, the strategy prediction model is trained using the following steps: A training sample set is obtained, wherein the training sample set includes sequence information of multiple sample materials, and the sequence information includes at least historical expected conversion information, historical delivery action information, and historical environmental state information; the training sample set is processed by the first prediction model in the initial prediction model to obtain the first predicted sample candidate delivery actions and the predicted sample environmental state information for the next time step corresponding to each of the multiple sample materials; the historical environmental state information and the predicted sample environmental state information for the next time step corresponding to each of the multiple sample materials are processed by the second prediction model in the initial prediction model to obtain the second predicted sample candidate delivery actions corresponding to each of the multiple sample materials; the initial prediction model is trained based on the target loss function, the first predicted sample candidate delivery actions, the predicted sample environmental state information for the next time step, the second predicted sample candidate delivery actions, and the training sample set until preset conditions are met, thus obtaining the strategy prediction model.

[0070] Optionally, a training sample set is first obtained. To effectively capture historical information and long-term dependencies, bid-related data from each placement within the last N days are collected and processed. Based on this, historical trajectories (i.e., sequence information) are constructed. Each trajectory can be represented as: ,in, This is for the expected future returns (i.e., historical expected conversion information). This refers to bidding actions (i.e., historical bidding action information). This refers to the environmental status (i.e., historical environmental status information), including advertiser identification information, current delivery resources, remaining delivery resources, historical click-through rate and conversion rate, time step, and other information.

[0071] Optionally, a decision transformer (i.e., the first prediction model in the initial prediction model) is used to train a unified model of the historical sequence containing actions and states, thereby achieving stronger generalization ability in complex material delivery environments. To jointly model actions and environmental states in the historical trajectory, at each time step, the DT model uses expected reward, historical state, and action as inputs to simultaneously predict the action of the current step end-to-end. and the state at the next time step For example, the prediction process of the model at time step t is represented as follows:

[0072]

[0073] Optionally, during training, mean squared error is used as the supervised learning objective, and the loss function of the DT model is... (i.e., the first loss function) is defined as follows:

[0074]

[0075] in, and These represent historical actions and states, respectively.

[0076] Optionally, by jointly generating the current action and the subsequent environmental state, more supervisory signals can be introduced into the training of the DT model, thereby explicitly guiding the model to capture state transitions. Simultaneously, the predicted... It will also serve as supplementary information to help the subsequent inverse dynamics module (IDM) better model the instantaneous state transition process.

[0077] Optionally, to enhance the diversity and robustness of the deployment strategy, after modeling the historical trajectory, an Inverse Dynamic Model (IDM) is further introduced to learn the inverse mapping from state to action, i.e., to infer the action corresponding to the instantaneous state transition, providing another safe path for action generation. For example, given two consecutive environmental states... and IDM uses a neural network (e.g., using a multilayer perceptron architecture) to estimate the actions that may lead to this state transition. ,Right now:

[0078]

[0079] To ensure consistency between the training and inference phases, the input to the IDM is the current true state. and the next state predicted by the DT model Together they constitute. During training, IDM uses supervised learning to minimize the mean squared error between its output action and the real action in the dataset. (That is, the second loss function) is defined as follows:

[0080]

[0081] This loss function encourages the IDM's action predictions to be as close as possible to the actual actions in the real trajectory, thereby ensuring that its behavior conforms to the historical policy distribution and improving the accuracy and stability of the model's action generation within the distribution.

[0082] Optionally, the target loss function includes a first loss function and a second loss function. The initial prediction model is trained based on the target loss function and the training sample set until the preset conditions are met, and the policy prediction model can be obtained.

[0083] To obtain a strategy prediction model, the method for determining the material delivery strategy provided in Embodiment 1 of this application involves training an initial prediction model based on a target loss function, first prediction sample candidate delivery actions, prediction sample environmental state information at the next time step, second prediction sample candidate delivery actions, and a training sample set until preset conditions are met. This process includes: training a first prediction model in the initial prediction model based on the first loss function in the target loss function, first prediction sample candidate delivery actions, historical delivery action information, prediction sample environmental state information at the next time step, and historical environmental state information to obtain a third prediction model; training a second prediction model in the initial prediction model based on the second loss function in the target loss function, second prediction sample candidate delivery actions, and historical delivery action information to obtain a fourth prediction model; determining a third loss function based on the first loss function, the second loss function, and a preset action value function, and jointly optimizing the third and fourth prediction models based on the third loss function and the training sample set until preset conditions are met to obtain a strategy prediction model.

[0084] Optionally, to improve the model's ability to model complex material delivery environments, a two-stage training strategy is adopted. The first stage is module pre-training (decoupled training), where the DT model and IDM are trained independently, thus laying a stable representation foundation for subsequent joint optimization. For example, based on the aforementioned first loss function, the first predicted sample candidate delivery actions, historical delivery action information, the predicted sample environment state information for the next time step, and the historical environment state information, the DT model in the initial prediction model is trained to obtain the third prediction model (i.e., the DT model trained in the first stage). Based on the aforementioned second loss function, the second predicted sample candidate delivery actions, and historical delivery action information, the IDM in the initial prediction model is trained to obtain the fourth prediction model (i.e., the IDM trained in the first stage).

[0085] Optionally, the second stage is joint optimization (end-to-end training). After obtaining the trained DT model and IDM, the overall policy is jointly optimized through a collaborative mechanism of action paths and state paths. The loss term of the IDM is explicitly incorporated into the training objective of the DT model, allowing gradients to propagate freely between the two models, thus achieving overall collaborative optimization of the state prediction path and the inverse dynamics action path. In this stage, the overall training loss function is expressed as follows:

[0086]

[0087] Among them, the two models predict the next state through the output of DT. Gradient connections are established. Under the combined effect of state prediction and inverse dynamic constraints, the policy can achieve a collaborative optimization mechanism of action paths and state paths through two-stage training, while ensuring the stability of behavior within the distribution.

[0088] Optionally, to achieve better policy optimization, a value evaluation network is introduced in the second stage, employing an actor and critic structure. By adding a Q-value regularization term (i.e., a pre-defined action value function) to the actor's loss function, the actor's training process is constrained and guided, achieving an effective balance between imitating existing behavioral strategies from offline data and exploring new strategies with better performance. At this point, the actor's composite loss function... (That is, the third loss function) is expressed as follows:

[0089]

[0090] Optionally, the DT model trained in the first stage and the IDM trained in the first stage are jointly optimized based on the third loss function and the training sample set until preset conditions are met, such as reaching a set number of iterations or converging to a stable state, to obtain the policy prediction model.

[0091] By introducing a value assessment network module, it is possible to make reasonable selections among candidate actions from different model perspectives, enabling the model to maintain strong robustness and good environmental adaptability in dynamic or even unseen material delivery environments.

[0092] To maintain model performance, in the method for determining the material delivery strategy provided in Embodiment 1 of this application, after determining the target delivery strategy corresponding to the target material, the method further includes: executing the target delivery strategy, collecting feedback information of the target material after the execution of the target delivery strategy; updating the environmental state information based on the feedback information, and obtaining the updated environmental state information.

[0093] Optionally, a target delivery strategy is executed, which involves performing one or a series of delivery operations on the target creatives according to the target delivery strategy, collecting feedback information on the target creatives after the target delivery strategy is executed, such as click-through rate, conversion rate, and actual return information, and then updating the environmental status information based on the feedback information to obtain the updated environmental status information for determining the next delivery strategy.

[0094] Optionally, such as Figure 3 As shown, a simulated / online system provides a creative delivery environment, which includes creative delivery providers corresponding to the target delivery strategy determined by the strategy prediction model and other creative delivery providers. These creative delivery providers bid for the requests corresponding to the traffic according to their respective delivery strategies, and update the environment status based on the statistical data of the bidding results to obtain the updated environment status information. This is used to determine the next deployment strategy.

[0095] In an alternative embodiment, the following can be employed: Figure 4 The diagram shown illustrates the process of model training. Figure 4 This is a flowchart illustrating an optional model training phase according to Embodiment 1 of this application, as shown below. Figure 4 As shown, input the historical trajectory ( , , First, embedding and positional encoding are performed to obtain the corresponding input vector, that is, the discrete or continuous input is converted into a form that the model can understand. Then, the input is fed into the DT model for prediction, and the state decoder decodes the state prediction result into the environment state of the next time step. The action decoder decodes the action prediction results into candidate delivery actions output by DT. Loss function based on DT model This allows the predicted values ​​to continuously approach the true values. An Inverse Dynamic Model (IDM) is introduced to learn the inverse mapping from state to action, based on the current true state. and the next state predicted by the DT model Predict candidate delivery actions from the IDM output The model is trained using the two-stage training strategy described in the foregoing embodiments until training is complete. Specific implementation details can be found in the foregoing embodiments and will not be repeated here.

[0096] Optionally, in this embodiment, a generative automatic determination framework for content delivery strategies is provided. Centered on a decision transformer, it uniformly models and predicts future bidding action sequences and environmental state sequences given an expected return target. An inverse dynamics module is introduced, which can reverse-engineer the optimal action to achieve the state transition based on the current state and the model's predicted future state. A value evaluation network is introduced: during the training phase, it enhances the model's ability to explore out-of-distribution trajectories through regularization constraints; during the inference phase, it evaluates two candidate actions generated by the decision transformer and the inverse dynamics module, and intelligently selects the bidding action with higher value for execution. This optimization mechanism combining exploration and utilization allows the framework to more flexibly adapt to complex content delivery scenarios, thereby making more accurate and effective content delivery decisions.

[0097] In this embodiment, the expected conversion information and environmental state information corresponding to the target material are obtained; the first prediction model in the strategy prediction model predicts based on the expected conversion information and environmental state information to obtain the first candidate delivery action and the environmental state information of the next time step; the second prediction model in the strategy prediction model predicts based on the environmental state information and the environmental state information of the next time step to obtain the second candidate delivery action; the evaluation model in the strategy prediction model determines the target delivery strategy corresponding to the target material based on the first candidate delivery action and the second candidate delivery action. The first prediction model and the second prediction model are used to generate dual-path delivery actions, and the evaluation model is introduced to evaluate and select the candidate actions generated by the two paths. This achieves accurate determination of the target delivery strategy, improves the quality of decision-making, and thus improves the accuracy of the material delivery strategy. This solves the technical problem of low accuracy in determining the material delivery strategy in related technologies.

[0098] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0100] Example 2

[0101] According to embodiments of this application, a method for determining a material delivery strategy is also provided, such as... Figure 5 As shown, the method includes:

[0102] Step S501: Obtain the expected conversion information and environmental status information corresponding to the target material uploaded by the client;

[0103] Step S502: In the cloud server, the first prediction model in the strategy prediction model makes a prediction based on the expected conversion information and environmental status information to obtain the first candidate delivery action and the environmental status information of the next time step; the second prediction model in the strategy prediction model makes a prediction based on the environmental status information and the environmental status information of the next time step to obtain the second candidate delivery action; the evaluation model in the strategy prediction model determines the target delivery strategy corresponding to the target material based on the first candidate delivery action and the second candidate delivery action.

[0104] Step S503: Feed back the target delivery strategy corresponding to the target material to the client.

[0105] The above scheme uses a first prediction model and a second prediction model to generate dual-path delivery actions. At the same time, an evaluation model is introduced to evaluate and select the best candidate actions generated by the two paths, which realizes the accurate determination of the target delivery strategy and achieves the goal of improving decision-making quality. This achieves the technical effect of improving the accuracy of material delivery strategy, and solves the technical problem of low accuracy in determining material delivery strategy in related technologies.

[0106] The specific method for determining the material delivery strategy on the cloud server is the same as that in Example 1, and will not be repeated here.

[0107] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0109] Example 3

[0110] According to an embodiment of this application, a device for determining a material delivery strategy for implementing the above-described method for determining a material delivery strategy is also provided, such as... Figure 6 As shown, the device includes: a first acquisition unit 601, a first processing unit 602, a second processing unit 603, and a first determination unit 604.

[0111] The first acquisition unit 601 is used to acquire the expected conversion information and environmental status information corresponding to the target material;

[0112] The first processing unit 602 is used to make predictions based on expected conversion information and environmental state information through the first prediction model in the strategy prediction model to obtain the first candidate delivery action and the environmental state information of the next time step.

[0113] The second processing unit 603 is used to make predictions based on environmental state information and environmental state information at the next time step through the second prediction model in the strategy prediction model to obtain the second candidate delivery action.

[0114] The first determining unit 604 is used to determine the target delivery strategy corresponding to the target material based on the first candidate delivery action and the second candidate delivery action through the evaluation model in the strategy prediction model.

[0115] In the material delivery strategy determination device provided in Embodiment 3 of this application, the first acquisition unit 601 acquires the expected conversion information and environmental state information corresponding to the target material; the first processing unit 602 predicts based on the expected conversion information and environmental state information using the first prediction model in the strategy prediction model to obtain the first candidate delivery action and the environmental state information of the next time step; the second processing unit 603 predicts based on the environmental state information and the environmental state information of the next time step using the second prediction model in the strategy prediction model to obtain the second candidate delivery action; and the first determination unit 604 determines the target delivery strategy corresponding to the target material based on the first candidate delivery action and the second candidate delivery action using the evaluation model in the strategy prediction model. In this solution, the first prediction model and the second prediction model are used to generate dual-path delivery actions, and an evaluation model is introduced to evaluate and select the candidate actions generated by the two paths, thereby achieving accurate determination of the target delivery strategy, improving the quality of decision-making, and thus achieving the technical effect of improving the accuracy of material delivery strategy. This solves the technical problem of low accuracy in determining material delivery strategies in related technologies.

[0116] Optionally, in the material delivery strategy determination device provided in Embodiment 3 of this application, the first processing unit 602 includes: a first processing subunit, used to encode the expected conversion information and the environmental state information respectively through the first encoder in the first prediction model to obtain a first prediction vector and a second prediction vector; a second processing subunit, used to process the first prediction vector and the second prediction vector respectively through the attention layer in the first prediction model to obtain a first intermediate representation vector and a second intermediate representation vector; and a third processing subunit, used to convert and decode the first intermediate representation vector and the second intermediate representation vector respectively through the output layer in the first prediction model to obtain a first candidate delivery action and the environmental state information of the next time step.

[0117] Optionally, in the material delivery strategy determination device provided in Embodiment 3 of this application, the second processing unit 603 includes: a fourth processing subunit, used to perform information fusion processing on environmental state information and environmental state information of the next time step through the input layer in the second prediction model to obtain a fused state representation vector; a fifth processing subunit, used to perform feature extraction processing on the fused state representation vector through the hidden layer in the second prediction model to obtain a target state representation vector; and a sixth processing subunit, used to perform transformation processing on the target state representation vector through the output layer in the second prediction model to obtain a second candidate delivery action.

[0118] Optionally, in the material delivery strategy determination device provided in Embodiment 3 of this application, the first determination unit 604 includes: a first determination subunit, configured to receive a first candidate delivery action, a second candidate delivery action, and environmental state information through the input layer of the evaluation model, generate a first representation vector based on the first candidate delivery action and the environmental state information, and generate a second representation vector based on the second candidate delivery action and the environmental state information; a second determination subunit, configured to perform feature extraction and mapping processing on the first representation vector and the second representation vector through the hidden layer of the evaluation model to obtain a first value and a second value; and a third determination subunit, configured to determine a target delivery action from the first candidate delivery action and the second candidate delivery action based on the magnitude of the first value and the second value through the output layer of the evaluation model, and determine a target delivery strategy based on the target delivery action.

[0119] Optionally, in the material delivery strategy determination device provided in Embodiment 3 of this application, the device further includes: a second acquisition unit, used to acquire a training sample set, wherein the training sample set includes sequence information of multiple sample materials, and the sequence information includes at least historical expected conversion information, historical delivery action information, and historical environmental state information; a third processing unit, used to process the training sample set through the first prediction model in the initial prediction model to obtain the first predicted sample candidate delivery action and the predicted sample environmental state information of the next time step corresponding to the multiple sample materials respectively; a fourth processing unit, used to process the historical environmental state information and the predicted sample environmental state information of the next time step corresponding to the multiple sample materials respectively through the second prediction model in the initial prediction model to obtain the second predicted sample candidate delivery action corresponding to the multiple sample materials respectively; and a fifth processing unit, used to train the initial prediction model based on the target loss function, the first predicted sample candidate delivery action, the predicted sample environmental state information of the next time step, the second predicted sample candidate delivery action, and the training sample set until a preset condition is met to obtain a strategy prediction model.

[0120] Optionally, in the material delivery strategy determination device provided in Embodiment 3 of this application, the fifth processing unit includes: a first training subunit, used to train the first prediction model in the initial prediction model based on the first loss function in the target loss function, the first prediction sample candidate delivery action, historical delivery action information, the prediction sample environmental state information of the next time step, and the historical environmental state information, to obtain a third prediction model; a second training subunit, used to train the second prediction model in the initial prediction model based on the second loss function in the target loss function, the second prediction sample candidate delivery action, and the historical delivery action information, to obtain a fourth prediction model; and a third training subunit, used to determine the third loss function based on the first loss function, the second loss function, and the preset action value function, and to jointly optimize the third prediction model and the fourth prediction model based on the third loss function and the training sample set until the preset conditions are met, to obtain a strategy prediction model.

[0121] Optionally, in the material delivery strategy determination device provided in Embodiment 3 of this application, the device further includes: an execution unit, configured to execute the target delivery strategy after determining the target material, and collect feedback information of the target material after the execution of the target delivery strategy; and an update unit, configured to update the environmental state information based on the feedback information to obtain the updated environmental state information.

[0122] It should be noted that the first acquisition unit 601, the first processing unit 602, the second processing unit 603, and the first determination unit 604 mentioned above correspond to steps S201 to S204 in Embodiment 1. The instances and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0123] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0124] Example 4

[0125] Embodiments of this application may provide an electronic device, which may be any one of a group of electronic devices. Optionally, in this embodiment, the aforementioned electronic device may also be replaced by a terminal device such as a mobile terminal.

[0126] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0127] In this embodiment, the aforementioned electronic device can execute the program code for the following steps in the method for determining the material delivery strategy: obtaining the expected conversion information and environmental state information corresponding to the target material; making a prediction based on the expected conversion information and environmental state information using the first prediction model in the strategy prediction model to obtain a first candidate delivery action and environmental state information for the next time step; making a prediction based on the environmental state information and environmental state information for the next time step using the second prediction model in the strategy prediction model to obtain a second candidate delivery action; and determining the target delivery strategy corresponding to the target material based on the first candidate delivery action and the second candidate delivery action using the evaluation model in the strategy prediction model.

[0128] The aforementioned electronic device can also execute the following steps in the method for determining the material delivery strategy: encoding the expected conversion information and the environmental state information respectively through the first encoder in the first prediction model to obtain the first prediction vector and the second prediction vector; processing the first prediction vector and the second prediction vector respectively through the attention layer in the first prediction model to obtain the first intermediate representation vector and the second intermediate representation vector; and converting and decoding the first intermediate representation vector and the second intermediate representation vector respectively through the output layer in the first prediction model to obtain the first candidate delivery action and the environmental state information of the next time step.

[0129] The aforementioned electronic device can also execute the following steps in the method for determining the material delivery strategy: fusing environmental state information and environmental state information at the next time step through the input layer of the second prediction model to obtain a fused state representation vector; extracting features from the fused state representation vector through the hidden layer of the second prediction model to obtain a target state representation vector; and transforming the target state representation vector through the output layer of the second prediction model to obtain a second candidate delivery action.

[0130] The aforementioned electronic device can also execute the following steps in the method for determining the material delivery strategy: receiving a first candidate delivery action, a second candidate delivery action, and environmental state information through the input layer of the evaluation model; generating a first representation vector based on the first candidate delivery action and environmental state information; and generating a second representation vector based on the second candidate delivery action and environmental state information; performing feature extraction and mapping processing on the first and second representation vectors through the hidden layer of the evaluation model to obtain a first value and a second value; and determining the target delivery action from the first and second candidate delivery actions based on the magnitude of the first and second values ​​through the output layer of the evaluation model, and determining the target delivery strategy based on the target delivery action.

[0131] The aforementioned electronic device can also execute the program code for the following steps in the method for determining the material delivery strategy: obtaining a training sample set, wherein the training sample set includes sequence information of multiple sample materials, and the sequence information includes at least historical expected conversion information, historical delivery action information, and historical environmental state information; processing the training sample set through the first prediction model in the initial prediction model to obtain the first predicted sample candidate delivery action and the predicted sample environmental state information for the next time step corresponding to each of the multiple sample materials; processing the historical environmental state information and the predicted sample environmental state information for the next time step corresponding to each of the multiple sample materials through the second prediction model in the initial prediction model to obtain the second predicted sample candidate delivery action corresponding to each of the multiple sample materials; training the initial prediction model based on the target loss function, the first predicted sample candidate delivery action, the predicted sample environmental state information for the next time step, the second predicted sample candidate delivery action, and the training sample set until the preset conditions are met to obtain the strategy prediction model.

[0132] The aforementioned electronic device can also execute the following steps in the method for determining the material delivery strategy: Based on the first loss function in the target loss function, the first predicted sample candidate delivery actions, historical delivery action information, the predicted sample environmental state information for the next time step, and historical environmental state information, train the first prediction model in the initial prediction model to obtain a third prediction model; based on the second loss function in the target loss function, the second predicted sample candidate delivery actions, and historical delivery action information, train the second prediction model in the initial prediction model to obtain a fourth prediction model; determine the third loss function based on the first loss function, the second loss function, and the preset action value function, and jointly optimize the third prediction model and the fourth prediction model based on the third loss function and the training sample set until the preset conditions are met, thus obtaining a strategy prediction model.

[0133] The aforementioned electronic device can also execute the program code for the following steps in the method for determining the material delivery strategy: after determining the target delivery strategy corresponding to the target material, execute the target delivery strategy, collect feedback information of the target material after the execution of the target delivery strategy; update the environmental status information based on the feedback information to obtain the updated environmental status information.

[0134] Optionally, Figure 7 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 7 As shown, the electronic device 70 may include: one or more ( Figure 7(Only one is shown in the image) Processor 702 and memory 704. The electronic device 70 may also include a memory controller to control and manage the memory 704; the electronic device 70 may also include a peripheral interface to connect to a radio frequency module, an audio module, and a display screen, etc.

[0135] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and apparatus for determining the material delivery strategy in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned method for determining the material delivery strategy. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0136] The processor can access the information and application programs stored in the memory via the transmission device to perform the following steps: obtain the expected conversion information and environmental state information corresponding to the target material; make a prediction based on the expected conversion information and environmental state information using the first prediction model in the strategy prediction model to obtain the first candidate delivery action and the environmental state information of the next time step; make a prediction based on the environmental state information and the environmental state information of the next time step using the second prediction model in the strategy prediction model to obtain the second candidate delivery action; and determine the target delivery strategy corresponding to the target material based on the first candidate delivery action and the second candidate delivery action using the evaluation model in the strategy prediction model.

[0137] Optionally, the processor may also execute program code for the following steps: encoding the expected transformation information and the environmental state information through the first encoder in the first prediction model to obtain a first prediction vector and a second prediction vector; processing the first prediction vector and the second prediction vector through the attention layer in the first prediction model to obtain a first intermediate representation vector and a second intermediate representation vector; and converting and decoding the first intermediate representation vector and the second intermediate representation vector through the output layer in the first prediction model to obtain the first candidate delivery action and the environmental state information of the next time step.

[0138] Optionally, the processor may also execute program code for the following steps: performing information fusion processing on the environmental state information and the environmental state information of the next time step through the input layer in the second prediction model to obtain a fused state representation vector; performing feature extraction processing on the fused state representation vector through the hidden layer in the second prediction model to obtain a target state representation vector; and performing transformation processing on the target state representation vector through the output layer in the second prediction model to obtain a second candidate delivery action.

[0139] Optionally, the processor may also execute program code with the following steps: receiving a first candidate delivery action, a second candidate delivery action, and environmental state information through the input layer of the evaluation model; generating a first representation vector based on the first candidate delivery action and the environmental state information; and generating a second representation vector based on the second candidate delivery action and the environmental state information; performing feature extraction and mapping processing on the first and second representation vectors through the hidden layer of the evaluation model to obtain a first value and a second value; and determining the target delivery action from the first and second candidate delivery actions based on the magnitude of the first and second values ​​through the output layer of the evaluation model, and determining the target delivery strategy based on the target delivery action.

[0140] Optionally, the processor may also execute program code for the following steps: acquiring a training sample set, wherein the training sample set includes sequence information of multiple sample materials, and the sequence information includes at least historical expected conversion information, historical delivery action information, and historical environmental state information; processing the training sample set through the first prediction model in the initial prediction model to obtain the first predicted sample candidate delivery action and the predicted sample environmental state information for the next time step corresponding to the multiple sample materials respectively; processing the historical environmental state information and the predicted sample environmental state information for the next time step corresponding to the multiple sample materials through the second prediction model in the initial prediction model to obtain the second predicted sample candidate delivery action corresponding to the multiple sample materials respectively; training the initial prediction model based on the target loss function, the first predicted sample candidate delivery action, the predicted sample environmental state information for the next time step, the second predicted sample candidate delivery action, and the training sample set until the preset conditions are met to obtain the policy prediction model.

[0141] Optionally, the processor may also execute program code with the following steps: training the first prediction model in the initial prediction model based on the first loss function in the target loss function, the first predicted sample candidate delivery action, historical delivery action information, the predicted sample environment state information of the next time step, and the historical environment state information to obtain the third prediction model; training the second prediction model in the initial prediction model based on the second loss function in the target loss function, the second predicted sample candidate delivery action, and the historical delivery action information to obtain the fourth prediction model; determining the third loss function based on the first loss function, the second loss function, and the preset action value function, and jointly optimizing the third prediction model and the fourth prediction model based on the third loss function and the training sample set until the preset conditions are met to obtain the policy prediction model.

[0142] Optionally, the processor may also execute program code for the following steps: after determining the target delivery strategy corresponding to the target material, execute the target delivery strategy, collect feedback information of the target material after the execution of the target delivery strategy; update the environmental status information based on the feedback information to obtain the updated environmental status information.

[0143] Those skilled in the art will understand that Figure 7 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 7 This does not limit the structure of the aforementioned electronic device. For example, electronic device 70 may also include components that are more... Figure 7 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 7 The different configurations shown.

[0144] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0145] Example 5

[0146] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the method for determining the material delivery strategy provided in Embodiment 1.

[0147] Optionally, in this embodiment, the storage medium may be located in any one of the electronic devices in the group of electronic devices in the computer network, or in any one of the mobile terminals in the group of mobile terminals.

[0148] Example 6

[0149] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the method for determining the material delivery strategy provided in Embodiment 1.

[0150] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0151] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0153] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0155] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0156] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for determining a material delivery strategy, characterized in that, include: Obtain the expected conversion information and environmental status information corresponding to the target material; The first prediction model in the strategy prediction model makes a prediction based on the expected conversion information and the environmental state information to obtain the first candidate delivery action and the environmental state information of the next time step. The second prediction model in the strategy prediction model makes a prediction based on the environmental state information and the environmental state information of the next time step to obtain the second candidate delivery action; The evaluation model in the strategy prediction model determines the target delivery strategy corresponding to the target material based on the first candidate delivery action and the second candidate delivery action.

2. The method according to claim 1, characterized in that, The first prediction model in the strategy prediction model predicts based on the expected conversion information and the environmental state information to obtain the first candidate delivery action and the environmental state information for the next time step, including: The expected transformation information and the environmental state information are encoded by the first encoder in the first prediction model to obtain the first prediction vector and the second prediction vector. The first prediction vector and the second prediction vector are processed by the attention layer in the first prediction model to obtain the first intermediate representation vector and the second intermediate representation vector. The first intermediate representation vector and the second intermediate representation vector are transformed and decoded by the output layer of the first prediction model to obtain the first candidate delivery action and the environmental state information of the next time step.

3. The method according to claim 1, characterized in that, The second prediction model in the strategy prediction model predicts based on the environmental state information and the environmental state information at the next time step, resulting in the following second candidate deployment actions: The environmental state information and the environmental state information of the next time step are fused through the input layer of the second prediction model to obtain the fused state representation vector. The target state representation vector is obtained by performing feature extraction processing on the fused state representation vector through the hidden layer in the second prediction model. The target state representation vector is transformed by the output layer of the second prediction model to obtain the second candidate delivery action.

4. The method according to claim 1, characterized in that, The evaluation model in the strategy prediction model determines the target delivery strategy corresponding to the target material based on the first candidate delivery action and the second candidate delivery action, including: The evaluation model receives the first candidate delivery action, the second candidate delivery action, and the environmental state information through its input layer. It generates a first representation vector based on the first candidate delivery action and the environmental state information, and generates a second representation vector based on the second candidate delivery action and the environmental state information. The first and second representation vectors are subjected to feature extraction and mapping processes through the hidden layer of the evaluation model to obtain the first and second numerical values. The output layer of the evaluation model determines the target delivery action from the first candidate delivery action and the second candidate delivery action based on the magnitude of the first value and the second value, and determines the target delivery strategy based on the target delivery action.

5. The method according to claim 1, characterized in that, The strategy prediction model is trained using the following steps: Obtain a training sample set, wherein the training sample set includes sequence information of multiple sample materials, and the sequence information includes at least historical expected conversion information, historical delivery action information, and historical environmental state information; The training sample set is processed by the first prediction model in the initial prediction model to obtain the first prediction sample candidate delivery action and the prediction sample environment state information of the next time step corresponding to the multiple sample materials. The second prediction model in the initial prediction model processes the historical environmental state information corresponding to the multiple sample materials and the predicted sample environmental state information corresponding to the next time step of the multiple sample materials to obtain the second predicted sample candidate delivery actions corresponding to the multiple sample materials. The initial prediction model is trained based on the target loss function, the first predicted sample candidate delivery action, the predicted sample environment state information at the next time step, the second predicted sample candidate delivery action, and the training sample set until the preset conditions are met, thus obtaining the policy prediction model.

6. The method according to claim 5, characterized in that, The initial prediction model is trained based on the target loss function, the first predicted sample candidate delivery action, the predicted sample environment state information at the next time step, the second predicted sample candidate delivery action, and the training sample set until preset conditions are met, resulting in the policy prediction model, which includes: Based on the first loss function in the target loss function, the first predicted sample candidate delivery action, the historical delivery action information, the predicted sample environmental state information at the next time step, and the historical environmental state information, the first prediction model in the initial prediction model is trained to obtain the third prediction model. Based on the second loss function in the target loss function, the second predicted sample candidate delivery action, and the historical delivery action information, the second prediction model in the initial prediction model is trained to obtain the fourth prediction model; A third loss function is determined based on the first loss function, the second loss function, and the preset action value function. The third prediction model and the fourth prediction model are then jointly optimized based on the third loss function and the training sample set until the preset conditions are met, thus obtaining the policy prediction model.

7. The method according to claim 1, characterized in that, After determining the target delivery strategy corresponding to the target material, the method further includes: Execute the target delivery strategy and collect feedback information on the target creative after the target delivery strategy is executed; The environmental status information is updated based on the feedback information to obtain the updated environmental status information.

8. A method for determining a material delivery strategy, characterized in that, include: Obtain the expected conversion information and environmental status information corresponding to the target materials uploaded by the client; In the cloud server, the first prediction model in the strategy prediction model makes predictions based on the expected conversion information and the environmental state information to obtain a first candidate delivery action and the environmental state information of the next time step; the second prediction model in the strategy prediction model makes predictions based on the environmental state information and the environmental state information of the next time step to obtain a second candidate delivery action; the evaluation model in the strategy prediction model determines the target delivery strategy corresponding to the target material based on the first candidate delivery action and the second candidate delivery action. The target delivery strategy corresponding to the target material is fed back to the client.

9. A device for determining a material delivery strategy, characterized in that, include: The first acquisition unit is used to acquire the expected conversion information and environmental status information corresponding to the target material; The first processing unit is used to make predictions based on the expected conversion information and the environmental state information through the first prediction model in the strategy prediction model, and to obtain the first candidate delivery action and the environmental state information of the next time step. The second processing unit is used to make a prediction based on the environmental state information and the environmental state information of the next time step through the second prediction model in the strategy prediction model to obtain the second candidate delivery action; The first determining unit is used to determine the target delivery strategy corresponding to the target material based on the first candidate delivery action and the second candidate delivery action through the evaluation model in the strategy prediction model.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the storage medium is located to perform the method for determining the material delivery strategy according to any one of claims 1 to 8.

11. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, executes the method for determining the material delivery strategy according to any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the method for determining the material delivery strategy according to any one of claims 1 to 8.

Images