Model training, information processing method and device, medium and electronic equipment

Abstract

The present disclosure relates to a model training, information processing method, device, medium and electronic equipment. The method comprises: obtaining training data, including first state information of reference carrying capacity in a first period, first reward information and second state information in a second period; predicting first order response information in the first period and second reward information in the second period through a target model according to the first state information and the first reward information; predicting second order response information in the second period through a model according to the second state information and the second reward information; parameter updating of the model is performed with the target confidence parameter and the product of the first joint probability distribution as the target; if the training termination condition is not met, return to the step of obtaining the training data; otherwise, obtain a reward prediction model. In this way, a reward prediction model with good generalization ability in counterfactual data can be reconstructed in a dataset with strong certainty and selection bias, so that the downstream solves the correct decision action.

Description

Technical Field

[0001] This disclosure relates to the field of information processing technology, specifically to a model training, information processing method, apparatus, medium, and electronic device. Background Technology

[0002] Regional time-slot cost allocation addresses the challenge of rationally allocating cost parameters across different time slots within a given budget constraint. This involves adjusting subsidy parameters for different time slots throughout the day to improve user experience (measured by the five-minute order acceptance rate). The current regional time-slot cost allocation scheme employs a "peak-shaving and valley-filling" strategy based on the principle of diminishing marginal returns: time slots are divided into three categories—top, middle, and bottom—based on their five-minute order acceptance rates from highest to lowest. Adjustments are made according to the rule of reducing fees for top-performing slots, keeping them unchanged for middle-performing slots, and increasing fees for bottom-performing slots, thereby optimizing the overall user experience. Essentially, the "peak-shaving and valley-filling" strategy shifts postage from high-performing time slots to low-performing ones. The underlying principle of this strategy is that "when the elasticity difference is small, the marginal return on price decreases." However, in some scenarios, the difference in elasticity curves between time periods is greater than the diminishing marginal returns effect. A reasonable decision is to "add money to periods with high cost efficiency and reduce money to periods with low cost efficiency." If the cost efficiency remains high during periods with good experience (i.e., the return on increasing unit cost is high), while the cost efficiency is low during periods with poor experience, then the empirical adjustment direction of "peak shaving and valley filling" is wrong, which will affect the user experience. Summary of the Invention

[0003] The purpose of this disclosure is to provide a model training, information processing method, apparatus, medium, and electronic device to rationally allocate rewards for different time periods throughout the day, thereby increasing the overall willingness to accept orders for transportation capacity throughout the day and improving user experience.

[0004] To achieve the above objectives, firstly, this disclosure provides a model training method, comprising:

[0005] Acquire training data, which includes first state information and first reward information of reference capacity in a first time period, and second state information of reference capacity in a second time period, wherein the second time period is the next time period after the first time period;

[0006] Based on the first status information and the first reward information, the first order acceptance response information of the reference capacity in the first time period is predicted by the target model, and based on the first order acceptance response information and the first status information, the second reward information of the reference capacity in the second time period is predicted.

[0007] Based on the second status information and the second reward information, the target model is used to predict the second order acceptance response information of the reference capacity in the second time period;

[0008] The target model is updated with the goal of minimizing the product of the target confidence parameter and the first joint probability distribution. The target confidence parameter is used to reflect the confidence of the second state information, the second reward information and the second order response information in representing the true capacity trajectory. The first joint probability distribution is the joint probability distribution of the second state information, the second reward information and the second order response information.

[0009] In response to the failure to meet the training cutoff condition, the step of acquiring training data is re-executed until the step of updating the model parameters of the target model with the goal of minimizing the product of the target confidence parameter and the first joint probability distribution.

[0010] In response to the satisfaction of the training cutoff condition, a reward prediction model is obtained.

[0011] Optionally, the target model includes a capacity response prediction sub-model and a reward prediction sub-model;

[0012] The capacity response prediction sub-model is used to predict the first order acceptance response information based on the first state information and the first reward information.

[0013] The reward prediction sub-model is connected to the capacity response prediction sub-model and is used to predict the second reward information based on the first order response information and the first status information.

[0014] The capacity response prediction sub-model is also used to predict the second order acceptance response information based on the second state information and the second reward information.

[0015] Optionally, the target model is a generative adversarial network including a generator and a discriminator, wherein the generator includes the capacity response prediction sub-model and the reward prediction sub-model.

[0016] Optionally, the method further includes:

[0017] The discriminator determines the probability that the second state information, the second reward information, and the second order response information represent the true trajectory.

[0018] The target confidence parameter is determined based on the probability.

[0019] Optionally, determining the target confidence parameter based on the probability includes:

[0020] The target confidence parameter is determined based on the probability and each probability determined by the discriminator during the previous training process.

[0021] Optionally, the training data may also include third reward information and third order response information of the reference capacity during the second time period;

[0022] The step of updating the model parameters of the target model with the objective of minimizing the product of the target confidence parameter and the first joint probability distribution includes:

[0023] The generator's model parameters are updated with the objective of minimizing the product of the target confidence parameter and the first joint probability distribution.

[0024] The discriminator's model parameters are updated using the third reward information, the third order response information, the second status information, the second reward information, the second order response information, and the probability.

[0025] Optionally, updating the model parameters of the discriminator using the third reward information, the third order response information, the second status information, the second reward information, the second order response information, and the probability includes:

[0026] The discriminator's model parameters are updated with the goal of maximizing the sum of the outputs of the first loss function and the second loss function. The first loss function is a function related to the second state information, the second reward information, the second order response information, and the probability. The second loss function is a function related to the second state information, the third reward information, the third order response information, and the probability.

[0027] Secondly, this disclosure provides an information processing method, including:

[0028] Obtain target status information and target reward information for the target capacity during the target time period;

[0029] The target state information and the target reward information are input into the reward prediction model to obtain the target order acceptance response information of the target capacity in the target time period and the predicted reward information of the target capacity in the next time period in the target time period. The reward prediction model is trained by the model training method provided in the first aspect of this disclosure.

[0030] Thirdly, this disclosure provides a model training apparatus, comprising:

[0031] The first acquisition module is used to acquire training data, the training data including first state information and first reward information of reference capacity in a first time period, and second state information of reference capacity in a second time period, wherein the second time period is the next time period after the first time period;

[0032] The first prediction module is used to predict the first order response information of the reference capacity in the first time period based on the first status information and the first reward information obtained by the first acquisition module, and to predict the second reward information of the reference capacity in the second time period based on the first order response information and the first status information.

[0033] The second prediction module is used to predict the second order response information of the reference capacity in the second time period based on the second status information obtained by the first acquisition module and the second reward information predicted by the first prediction module, using the target model.

[0034] The update module is used to update the model parameters of the target model with the goal of minimizing the product of the target confidence parameter and the first joint probability distribution. The target confidence parameter is used to reflect the confidence of the second state information, the second reward information and the second order response information in representing the real capacity trajectory. The first joint probability distribution is the joint probability distribution of the second state information, the second reward information and the second order response information.

[0035] The triggering module is configured to: trigger the first acquisition module to acquire training data in response to the failure to meet the training cutoff condition; and obtain a reward prediction model in response to the satisfaction of the training cutoff condition.

[0036] Fourthly, this disclosure provides an information processing apparatus, comprising:

[0037] The second acquisition module is used to acquire target status information and target reward information of the target capacity during the target time period;

[0038] The third prediction module is used to input the target state information and the target reward information obtained by the second acquisition module into the reward prediction model to obtain the target order acceptance response information of the target capacity in the target time period and the predicted reward information of the target capacity in the next time period of the target time period. The reward prediction model is trained by the model training method provided in the first aspect of this disclosure.

[0039] Fifthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the model training method provided in the first aspect of this disclosure or the steps of the information processing method provided in the second aspect of this disclosure.

[0040] Sixthly, this disclosure provides an electronic device, comprising:

[0041] A memory on which computer programs are stored;

[0042] A processor is configured to execute the computer program in the memory to implement the steps of the model training method provided in the first aspect of this disclosure or the steps of the information processing method provided in the second aspect of this disclosure.

[0043] In the above technical solution, the target model's parameters are updated by minimizing the product of the target confidence parameter and the first joint probability distribution. The target confidence parameter reflects the confidence level of the second state information, second reward information, and second order response information in representing the true capacity trajectory. The first joint probability distribution is the combined probability distribution of the second state information, second reward information, and second order response information. It is evident that the lower the prediction accuracy of the target model (i.e., the worse the target model's learning), the smaller the target confidence parameter, and the closer the product of the target confidence parameter and the first joint probability distribution is to the target, the better the reward prediction model can be reconstructed from datasets with strong determinism and selection bias, demonstrating good generalization ability in counterfactual data. This allows downstream solutions to arrive at more accurate decisions, ensuring the accuracy of the reward prediction model's predictions of the current period's order response information and the next period's reward information. Furthermore, this approach not only avoids the complex modeling process of manually abstracting and characterizing capacity but also avoids prediction errors caused by human cognitive deficiencies and biases, resulting in a more accurate reward prediction model. Therefore, rewards can be reasonably divided into different time periods throughout the day, increasing the overall willingness to accept orders throughout the day and improving the user experience.

[0044] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0045] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0046] Figure 1 This is a flowchart illustrating a model training method according to an exemplary embodiment.

[0047] Figure 2 This is a flowchart illustrating an information processing method according to an exemplary embodiment.

[0048] Figure 3 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment.

[0049] Figure 4 This is a block diagram illustrating an information processing apparatus according to an exemplary embodiment.

[0050] Figure 5This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0051] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0052] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0053] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0054] As discussed in the background section, the empirical adjustment approach of "peak shaving and valley filling" is flawed, negatively impacting user experience. The key to solving this problem lies in using historical data to personalize capacity profiles. However, current observational data exhibits strong determinism (i.e., subsidy parameters are almost constant for a given time period) and selection bias (different time periods show preferences for subsidy parameters). This results in numerous counterfactual points in the state-action-feedback space (i.e., only a few subsidy parameters are visible across a large number of time periods). This makes it difficult to guarantee the generalization accuracy of predictions based on counterfactual data when building reward prediction models, leading to misleading subsequent strategy evaluations and negatively impacting user experience.

[0055] In view of this, this disclosure provides a model training, information processing method, apparatus, medium and electronic device to reasonably divide the rewards for different time periods of the day, increase the total willingness to accept orders for the entire day's transportation capacity, and improve the user experience.

[0056] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0057] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0058] Figure 1 This is a flowchart illustrating a model training method according to an exemplary embodiment. Figure 1 As shown, the model training method may include the following S101 to S106.

[0059] In S101, acquire the training data.

[0060] In this disclosure, the training data includes first state information and first reward information of the reference capacity in a first time period, and second state information of the reference capacity in a second time period, wherein the second time period is the period following the first time period, and the first reward information is the actual reward information of the reference capacity in the first time period. The reference capacity can be a delivery person, a delivery robot, an unmanned delivery vehicle, a drone, or other delivery equipment, or a taxi, freight truck, etc.

[0061] The first and second status information can include capacity information and waybill information. Capacity information can include static information such as capacity type and capacity level, and dynamic information such as total delivery time over 7 days, number of orders completed over 7 days, on-time delivery rate over 7 days, and number of orders received over 7 days. When the capacity is a delivery person, the static information may also include gender and age. Waybill information can include time-slot waybill information, information about the area covered by the waybill (e.g., area type (e.g., commercial area, residential area), area base price, etc.). The first reward information can include time-slot subsidies.

[0062] In S102, based on the first state information and the first reward information, the first order acceptance response information of the reference capacity in the first time period is predicted by the target model, and based on the first order acceptance response information and the first state information, the second reward information of the reference capacity in the second time period is predicted.

[0063] Specifically, the first state information and the first reward information can be input into the target model to predict the first order response information of the reference capacity in the first time period through the target prediction model, and the second reward information of the reference capacity in the second time period can be predicted based on the first order response information and the first state information.

[0064] In S103, based on the second state information and the second reward information, the target model predicts the second order response information of the reference capacity in the second time period.

[0065] In this disclosure, the first order response information may include a first preset time order acceptance rate (e.g., 5-minute order acceptance rate), a second preset time order completion volume (e.g., 1-hour order completion volume), etc. The first preset time order acceptance rate is equal to the number of orders accepted within the first preset time period divided by the total number of orders accepted within the first time period.

[0066] In one implementation, the target model may include a capacity response prediction sub-model and a reward prediction sub-model. The capacity response prediction sub-model is used to predict first order acceptance response information based on first state information and first reward information. The reward prediction sub-model, connected to the capacity response prediction sub-model, is used to predict second reward information based on the first order acceptance response information and the first state information. Additionally, the capacity response prediction sub-model is also used to predict second order acceptance response information for reference capacity in a second time period based on the second state information and the second reward information.

[0067] In S104, the target model is updated with the goal of minimizing the product of the target confidence parameter and the first joint probability distribution.

[0068] In this disclosure, the target confidence parameter is used to reflect the confidence level of the second state information, the second reward information, and the second order response information in representing the true capacity trajectory, and the first joint probability distribution is the joint probability distribution of the second state information, the second reward information, and the second order response information.

[0069] For example, the target model may include a capacity response prediction sub-model and a reward prediction sub-model. In this case, the model parameters of the target model can be updated according to the objective function shown in the following formula (1):

[0070]

[0071] in, The output of the reward prediction sub-model; Z(s,a,y) is the output of the capacity response prediction sub-model; Z(s,a,y) is the target confidence parameter; s is the second state information; a is the second reward information; y is the second order acceptance response information; p(s) is the distribution of the first state information.

[0072] In S105, determine whether the training cutoff condition is met.

[0073] In this disclosure, the training cutoff condition can be either reaching a preset number of training iterations or the model's loss being less than a preset loss threshold.

[0074] If the training cutoff condition is not met, return to step S101 above and continue execution until the training cutoff condition is met. If the training cutoff condition is met, execute step S106 below.

[0075] In S106, the reward prediction model is obtained.

[0076] Additionally, it should be noted that the training data obtained in each loop is new training data.

[0077] In the above technical solution, the target model's parameters are updated by minimizing the product of the target confidence parameter and the first joint probability distribution. The target confidence parameter reflects the confidence level of the second state information, second reward information, and second order response information in representing the true capacity trajectory. The first joint probability distribution is the combined probability distribution of the second state information, second reward information, and second order response information. It is evident that the lower the prediction accuracy of the target model (i.e., the worse the target model's learning), the smaller the target confidence parameter, and the closer the product of the target confidence parameter and the first joint probability distribution is to the target, the better the reward prediction model can be reconstructed from datasets with strong determinism and selection bias, demonstrating good generalization ability in counterfactual data. This allows downstream solutions to arrive at more accurate decisions, ensuring the accuracy of the reward prediction model's predictions of the current period's order response information and the next period's reward information. Furthermore, this approach not only avoids the complex modeling process of manually abstracting and characterizing capacity but also avoids prediction errors caused by human cognitive deficiencies and biases, resulting in a more accurate reward prediction model. Therefore, rewards can be reasonably divided into different time periods throughout the day, increasing the overall willingness to accept orders throughout the day and improving the user experience.

[0078] The specific implementation method for obtaining training data in S101 described above will be explained in detail below.

[0079] In this disclosure, a training dataset can be created, and then, during model training, one training data point is extracted from the training dataset each time for model training.

[0080] Specifically, the training dataset can be built in the following ways:

[0081] First, for each of the multiple reference capacities, obtain the status information, reward information, and order acceptance response information of that reference capacity for each time period (e.g., every half hour) throughout the entire day of the preset date. Then, for each time period, aggregate the average values ​​of the status information, reward information, and order acceptance response information of that reference capacity for that time period. Finally, organize the aggregated average status information, aggregated average reward information, and aggregated average order acceptance response information of the reference capacity for each time period throughout the entire day of the preset date into a capacity trajectory set according to a Markov decision process. The length t of the trajectory is the number of time periods within a whole day on the aforementioned preset date (for example, if each half-hour period is a time period, then t = 48), τ i Let represent the capacity trajectory of reference capacity i; finally, normalize the capacity trajectories of the above multiple reference capacities (e.g., normalize to values ​​between 0 and 1) to obtain multiple training data points, forming a training dataset (τ1, τ2, ..., τ). n ),in, This constitutes a training dataset. Data normalization can eliminate the mutual influence between features, making it easier for model training to converge.

[0082] In one implementation, the target model can be a generative adversarial network including a generator and a discriminator, wherein the generator includes a capacity response prediction sub-model and a reward prediction sub-model.

[0083] At this point, the above method may also include the following steps:

[0084] The discriminator determines the probability that the second state information, the second reward information, and the second order response information represent the true trajectory; the target confidence parameter is determined based on this probability.

[0085] Specifically, after S103 and before S104, the second state information, the second reward information, and the second order response information can be input into the discriminator so that the discriminator can determine the probability that the second state information, the second reward information, and the second order response information represent the true trajectory; then, the target confidence parameter is determined based on the probability.

[0086] In this disclosure, the target confidence parameter can be determined based on the aforementioned probabilities in various ways. In one embodiment, the aforementioned probabilities can be directly determined as the target confidence parameter.

[0087] In another implementation, the target confidence parameter can be determined based on the aforementioned probabilities and each probability determined by the discriminator during previous training. This avoids the accuracy of the target confidence parameter being affected by accumulated errors during model training, thereby improving the efficiency of model training and the accuracy of model prediction.

[0088] For example, the target confidence parameter can be determined by the negative of the sum of the logarithms of the above probabilities and the logarithms of each probability determined by the discriminator during previous training.

[0089] In addition, the training data mentioned above may also include the third reward information and the third order response information of the reference capacity in the second time period, wherein the third reward information is the actual reward information of the reference capacity in the second time period, and the third order response information is the actual order response information of the reference capacity in the second time period. At this time, S104 can update the model parameters of the target model by minimizing the product of the target confidence parameter and the first joint probability distribution in the following way:

[0090] The generator's model parameters are updated with the objective of minimizing the product of the target confidence parameter and the first joint probability distribution; the discriminator's model parameters are updated using the third reward information, the third order response information, the second state information, the second reward information, the second order response information, and the probability.

[0091] Specifically, the discriminator's model parameters can be updated with the goal of maximizing the sum of the outputs of the first loss function and the second loss function. The first loss function is a function related to the second state information, the second reward information, the second order response information, and the probability. The second loss function is a function related to the second state information, the third reward information, the third order response information, and the probability.

[0092]

[0093] in, Let be the first loss function; D(s,a,y) be the above probability; ∫p(s)∫μ(a|s)∫m * (y|s,a)log(D(s,a,y))dy da ds is the second loss function; μ(a|s) is the third reward information, m * (y|s,a) represents the third order response information.

[0094] Figure 2 This is a flowchart illustrating an information processing method according to an exemplary embodiment. For example... Figure 2 As shown, the information processing method may include the following steps S201 to S202.

[0095] In S201, the target status information and target reward information of the target capacity during the target time period are obtained.

[0096] In S202, the target status information and target reward information are input into the reward prediction model to obtain the target order response information of the target capacity in the target time period and the predicted reward information of the target capacity in the next time period of the target time period.

[0097] In this disclosure, the above-mentioned reward prediction model can be trained using the model training method provided in this disclosure.

[0098] In the above technical solution, the target model's parameters are updated by minimizing the product of the target confidence parameter and the first joint probability distribution. The target confidence parameter reflects the confidence level of the second state information, second reward information, and second order response information in representing the true capacity trajectory. The first joint probability distribution is the combined probability distribution of the second state information, second reward information, and second order response information. It is evident that the lower the prediction accuracy of the target model (i.e., the worse the target model's learning), the smaller the target confidence parameter, and the closer the product of the target confidence parameter and the first joint probability distribution is to the target, the better the reward prediction model can be reconstructed from datasets with strong determinism and selection bias, demonstrating good generalization ability in counterfactual data. This allows downstream solutions to arrive at more accurate decisions, ensuring the accuracy of the reward prediction model's predictions of the current period's order response information and the next period's reward information. Furthermore, this approach not only avoids the complex modeling process of manually abstracting and characterizing capacity but also avoids prediction errors caused by human cognitive deficiencies and biases, resulting in a more accurate reward prediction model. Therefore, rewards can be reasonably divided into different time periods throughout the day, increasing the overall willingness to accept orders throughout the day and improving the user experience.

[0099] Figure 3 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment. Figure 3 As shown, the model training device 300 may include:

[0100] The first acquisition module 301 is used to acquire training data, the training data including first state information and first reward information of reference capacity in a first time period, and second state information of reference capacity in a second time period, wherein the second time period is the next time period after the first time period;

[0101] The first prediction module 302 is used to predict the first order acceptance response information of the reference capacity in the first time period based on the first status information and the first reward information obtained by the first acquisition module 301, and to predict the second reward information of the reference capacity in the second time period based on the first order acceptance response information and the first status information.

[0102] The second prediction module 303 is used to predict the second order response information of the reference capacity in the second time period based on the second status information obtained by the first acquisition module 301 and the second reward information predicted by the first prediction module 302, through the target model.

[0103] The update module 304 is used to update the model parameters of the target model with the goal of minimizing the product of the target confidence parameter and the first joint probability distribution. The target confidence parameter is used to reflect the confidence of the second state information, the second reward information and the second order acceptance response information in representing the real capacity trajectory. The first joint probability distribution is the joint probability distribution of the second state information, the second reward information and the second order acceptance response information.

[0104] Trigger module 305 is configured to: trigger the first acquisition module 301 to acquire training data in response to the failure to meet the training cutoff condition; and obtain a reward prediction model in response to the satisfaction of the training cutoff condition.

[0105] In the above technical solution, the target model's parameters are updated by minimizing the product of the target confidence parameter and the first joint probability distribution. The target confidence parameter reflects the confidence level of the second state information, second reward information, and second order response information in representing the true capacity trajectory. The first joint probability distribution is the combined probability distribution of the second state information, second reward information, and second order response information. It is evident that the lower the prediction accuracy of the target model (i.e., the worse the target model's learning), the smaller the target confidence parameter, and the closer the product of the target confidence parameter and the first joint probability distribution is to the target, the better the reward prediction model can be reconstructed from datasets with strong determinism and selection bias, demonstrating good generalization ability in counterfactual data. This allows downstream solutions to arrive at more accurate decisions, ensuring the accuracy of the reward prediction model's predictions of the current period's order response information and the next period's reward information. Furthermore, this approach not only avoids the complex modeling process of manually abstracting and characterizing capacity but also avoids prediction errors caused by human cognitive deficiencies and biases, resulting in a more accurate reward prediction model. Therefore, rewards can be reasonably divided into different time periods throughout the day, increasing the overall willingness to accept orders throughout the day and improving the user experience.

[0106] Optionally, the target model includes a capacity response prediction sub-model and a reward prediction sub-model;

[0107] The capacity response prediction sub-model is used to predict the first order acceptance response information based on the first state information and the first reward information.

[0108] The reward prediction sub-model is connected to the capacity response prediction sub-model and is used to predict the second reward information based on the first order response information and the first status information.

[0109] The capacity response prediction sub-model is also used to predict the second order acceptance response information based on the second state information and the second reward information.

[0110] Optionally, the target model is a generative adversarial network including a generator and a discriminator, wherein the generator includes the capacity response prediction sub-model and the reward prediction sub-model.

[0111] Optionally, the device 300 further includes:

[0112] The first determining module is used to determine the probability that the second state information, the second reward information, and the second order acceptance response information represent the true trajectory through the discriminator;

[0113] The second determining module is used to determine the target confidence parameter based on the probability determined by the first determining module.

[0114] Optionally, the second determining module is used to determine the target confidence parameter based on the probability and each probability determined by the discriminator during the previous training process.

[0115] Optionally, the training data may also include third reward information and third order response information of the reference capacity during the second time period;

[0116] The update module 304 includes:

[0117] The first update submodule is used to update the model parameters of the generator with the goal of minimizing the product of the target confidence parameter and the first joint probability distribution.

[0118] The second update submodule is used to update the model parameters of the discriminator using the third reward information, the third order response information, the second status information, the second reward information, the second order response information, and the probability.

[0119] Optionally, the second update submodule is used to update the model parameters of the discriminator with the goal of maximizing the sum of the outputs of the first loss function and the second loss function, wherein the first loss function is a function related to the second state information, the second reward information, the second order response information and the probability, and the second loss function is a function related to the second state information, the third reward information, the third order response information and the probability.

[0120] Figure 4 This is a block diagram illustrating an information processing apparatus according to an exemplary embodiment. Figure 4 As shown, the information processing device 400 may include:

[0121] The second acquisition module 401 is used to acquire the target status information and target reward information of the target capacity during the target time period;

[0122] The third prediction module 402 is used to input the target status information and the target reward information obtained by the second acquisition module 401 into the reward prediction model to obtain the target order acceptance response information of the target capacity in the target time period and the predicted reward information of the target capacity in the next time period of the target time period. The reward prediction model is trained by the model training method provided in this disclosure.

[0123] In the above technical solution, the target model's parameters are updated by minimizing the product of the target confidence parameter and the first joint probability distribution. The target confidence parameter reflects the confidence level of the second state information, second reward information, and second order response information in representing the true capacity trajectory. The first joint probability distribution is the combined probability distribution of the second state information, second reward information, and second order response information. It is evident that the lower the prediction accuracy of the target model (i.e., the worse the target model's learning), the smaller the target confidence parameter, and the closer the product of the target confidence parameter and the first joint probability distribution is to the target, the better the reward prediction model can be reconstructed from datasets with strong determinism and selection bias, demonstrating good generalization ability in counterfactual data. This allows downstream solutions to arrive at more accurate decisions, ensuring the accuracy of the reward prediction model's predictions of the current period's order response information and the next period's reward information. Furthermore, this approach not only avoids the complex modeling process of manually abstracting and characterizing capacity but also avoids prediction errors caused by human cognitive deficiencies and biases, resulting in a more accurate reward prediction model. Therefore, rewards can be reasonably divided into different time periods throughout the day, increasing the overall willingness to accept orders throughout the day and improving the user experience.

[0124] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0125] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the model training method or information processing method provided in this disclosure.

[0126] Figure 5 This is a block diagram illustrating an electronic device 500 according to an exemplary embodiment. For example... Figure 5 As shown, the electronic device 500 may include a processor 501 and a memory 502. The electronic device 500 may also include one or more of a multimedia component 503, an input / output (I / O) interface 504, and a communication component 505.

[0127] The processor 501 controls the overall operation of the electronic device 500 to complete all or part of the steps in the model training method described above. The memory 502 stores various types of data to support the operation of the electronic device 500. This data may include, for example, instructions for any application or method operating on the electronic device 500, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 502 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 503 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 502 or transmitted via communication component 505. The audio component also includes at least one speaker for outputting audio signals. I / O interface 504 provides an interface between processor 501 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 505 is used for wired or wireless communication between the electronic device 500 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 505 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0128] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the model training method described above.

[0129] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the model training method described above. For example, the computer-readable storage medium may be the memory 502 including the program instructions, which may be executed by the processor 501 of the electronic device 500 to complete the model training method described above.

[0130] Figure 6 This is a block diagram illustrating an electronic device 600 according to an exemplary embodiment. For example, the electronic device 600 may be provided as a server. (Refer to...) Figure 6 The electronic device 600 includes a processor 622, which may be one or more, and a memory 632 for storing computer programs executable by the processor 622. The computer programs stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 622 may be configured to execute the computer program to perform the model training method described above.

[0131] Additionally, the electronic device 600 may also include a power supply component 626 and a communication component 650. The power supply component 626 can be configured to perform power management of the electronic device 600, and the communication component 650 can be configured to enable communication of the electronic device 600, such as wired or wireless communication. Furthermore, the electronic device 600 may also include an input / output (I / O) interface 658. The electronic device 600 can operate on an operating system, such as Windows Server, stored in memory 632. TM Mac OSX TM Unix TM Linux TM etc.

[0132] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the model training method described above. For example, the computer-readable storage medium may be the memory 632 including the program instructions described above, which may be executed by the processor 622 of the electronic device 600 to complete the model training method described above.

[0133] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the model training method described above when executed by the programmable device.

[0134] Figure 7 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. Figure 7 As shown, the electronic device 700 may include a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0135] The processor 701 controls the overall operation of the electronic device 700 to complete all or part of the steps in the information processing method described above. The memory 702 stores various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 703 may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. I / O interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0136] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the information processing method described above.

[0137] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the information processing method described above. For example, the computer-readable storage medium may be the memory 702 including program instructions described above, which may be executed by the processor 701 of the electronic device 700 to complete the information processing method described above.

[0138] Figure 8 This is a block diagram illustrating an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be provided as a server. (Refer to...) Figure 8 The electronic device 800 includes a processor 822, which may be one or more, and a memory 832 for storing computer programs executable by the processor 822. The computer program stored in the memory 832 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 822 may be configured to execute the computer program to perform the information processing methods described above.

[0139] Additionally, the electronic device 800 may also include a power supply component 826 and a communication component 850. The power supply component 826 can be configured to perform power management of the electronic device 800, and the communication component 850 can be configured to enable communication of the electronic device 800, such as wired or wireless communication. Furthermore, the electronic device 800 may also include an input / output (I / O) interface 858. The electronic device 800 can operate on an operating system, such as Windows Server, stored in memory 832. TM Mac OSX TM Unix TM Linux TM etc.

[0140] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the information processing method described above. For example, the computer-readable storage medium may be the memory 832 including the program instructions described above, which may be executed by the processor 822 of the electronic device 800 to complete the information processing method described above.

[0141] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a programmable device, the computer program having a code portion for performing the information processing method described above when executed by the programmable device.

[0142] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0143] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0144] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A model training method, characterized in that, include: Acquire training data, which includes first state information and first reward information of reference capacity in a first time period, and second state information of reference capacity in a second time period, wherein the second time period is the next time period after the first time period; Based on the first status information and the first reward information, the first order acceptance response information of the reference capacity in the first time period is predicted by the target model, and based on the first order acceptance response information and the first status information, the second reward information of the reference capacity in the second time period is predicted. Based on the second status information and the second reward information, the target model is used to predict the second order acceptance response information of the reference capacity in the second time period; The probability that the second state information, the second reward information, and the second order acceptance response information represent the true trajectory is determined by the discriminator, and the target confidence parameter is determined based on the probability. The target model is updated with the goal of minimizing the product of the target confidence parameter and the first joint probability distribution. The first joint probability distribution is the joint probability distribution of the second state information, the second reward information, and the second order acceptance response information. The target model includes a capacity response prediction sub-model and a reward prediction sub-model. The capacity response prediction sub-model is used to predict the first order acceptance response information based on the first status information and the first reward information. The reward prediction sub-model is connected to the capacity response prediction sub-model and is used to predict the second reward information based on the first order acceptance response information and the first status information. The capacity response prediction sub-model is also used to predict the second order acceptance response information based on the second status information and the second reward information. The target model is a generative adversarial network that includes a generator and a discriminator. The generator includes the capacity response prediction sub-model and the reward prediction sub-model. In response to the failure to meet the training cutoff condition, the step of acquiring training data is re-executed until the step of updating the model parameters of the target model with the goal of minimizing the product of the target confidence parameter and the first joint probability distribution. In response to the satisfaction of the training cutoff condition, a reward prediction model is obtained.

2. The method according to claim 1, characterized in that, Determining the target confidence parameter based on the probability includes: The target confidence parameter is determined based on the probability and each probability determined by the discriminator during the previous training process.

3. The method according to claim 1, characterized in that, The training data also includes the third reward information and the third order response information of the reference capacity in the second time period; The step of updating the model parameters of the target model with the objective of minimizing the product of the target confidence parameter and the first joint probability distribution includes: The generator's model parameters are updated with the objective of minimizing the product of the target confidence parameter and the first joint probability distribution. The discriminator's model parameters are updated using the third reward information, the third order response information, the second status information, the second reward information, the second order response information, and the probability.

4. The method according to claim 3, characterized in that, The step of updating the model parameters of the discriminator using the third reward information, the third order acceptance response information, the second status information, the second reward information, the second order acceptance response information, and the probability includes: The discriminator's model parameters are updated with the goal of maximizing the sum of the outputs of the first loss function and the second loss function. The first loss function is a function related to the second state information, the second reward information, the second order response information, and the probability. The second loss function is a function related to the second state information, the third reward information, the third order response information, and the probability.

5. An information processing method, characterized in that, include: Obtain target status information and target reward information for the target capacity during the target time period; The target state information and the target reward information are input into the reward prediction model to obtain the target order acceptance response information of the target capacity in the target time period and the predicted reward information of the target capacity in the next time period of the target time period. The reward prediction model is trained by the model training method of any one of claims 1-4.

6. A model training device, characterized in that, include: The first acquisition module is used to acquire training data, the training data including first state information and first reward information of reference capacity in a first time period, and second state information of reference capacity in a second time period, wherein the second time period is the next time period after the first time period; The first prediction module is used to predict the first order response information of the reference capacity in the first time period based on the first status information and the first reward information obtained by the first acquisition module, and to predict the second reward information of the reference capacity in the second time period based on the first order response information and the first status information. The second prediction module is used to predict the second order response information of the reference capacity in the second time period based on the second status information obtained by the first acquisition module and the second reward information predicted by the first prediction module, using the target model. The first determining module is used to determine the probability that the second state information, the second reward information, and the second order acceptance response information represent the true trajectory through a discriminator; The second determining module is used to determine the target confidence parameter based on the probability determined by the first determining module; The update module is used to update the model parameters of the target model with the goal of minimizing the product of the target confidence parameter and the first joint probability distribution, wherein the first joint probability distribution is the joint probability distribution of the second state information, the second reward information and the second order acceptance response information; The target model includes a capacity response prediction sub-model and a reward prediction sub-model. The capacity response prediction sub-model is used to predict the first order acceptance response information based on the first status information and the first reward information. The reward prediction sub-model is connected to the capacity response prediction sub-model and is used to predict the second reward information based on the first order acceptance response information and the first status information. The capacity response prediction sub-model is also used to predict the second order acceptance response information based on the second status information and the second reward information. The target model is a generative adversarial network that includes a generator and a discriminator. The generator includes the capacity response prediction sub-model and the reward prediction sub-model. The triggering module is used to: trigger the first acquisition module to acquire training data in response to the failure to meet the training cutoff condition; In response to the satisfaction of the training cutoff condition, a reward prediction model is obtained.

7. An information processing device, characterized in that, include: The second acquisition module is used to acquire target status information and target reward information of the target capacity during the target time period; The third prediction module is used to input the target state information and the target reward information obtained by the second acquisition module into the reward prediction model to obtain the target order acceptance response information of the target capacity in the target time period and the predicted reward information of the target capacity in the next time period in the target time period. The reward prediction model is trained by the model training method of any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-5.

9. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-5.

Images