Information distribution model determination method, device, server and storage medium

Abstract

The present disclosure is about a method, device, server and storage medium for determining an information distribution model, including: training a first information distribution model according to sample data, training a second information distribution model according to data other than the first sample data and the second sample data in the sample data; generating a first information operation index comparison result according to the operation data of the first test account on the first information and the operation data of the first control account on the second information, and generating a second information operation index comparison result according to the operation data of the second test account on the first information and the operation data of the second control account on the second information; determining the first information distribution model as the target information distribution model when the first information operation index comparison result meets the corresponding first preset index comparison result and the second information operation index comparison result meets the corresponding second preset index comparison result. The present disclosure improves the accuracy of determining the information distribution model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, server, storage medium and computer program product for determining an information distribution model. Background Art

[0002] With the development of Internet technology, various information distribution strategies or information distribution models are used when distributing information; however, before using a new information distribution strategy or information distribution model, it is necessary to experiment with it.

[0003] In the related art, AB experiments are generally conducted by setting up a basic group and an experimental group, and then judging the experimental effects of the basic group and the experimental group to determine which information distribution model is better; however, in this process, there will be some overlap in the data between the experimental group and the basic group, and the experimental effects of the basic group and the experimental group are slightly different, resulting in inaccurate experimental effects of the AB experiment, making it difficult to accurately determine the iterative process of the information distribution model. Therefore, the accuracy of the determination of the existing information distribution model is still relatively low. Summary of the invention

[0004] The present disclosure provides a method, device, server, storage medium and computer program product for determining an information distribution model, so as to at least solve the problem that the accuracy of determining the information distribution model in the related art is still low. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for determining an information distribution model is provided, comprising:

[0006] Acquire sample data; the sample data at least includes first sample data, second sample data, third sample data and fourth sample data; the first sample data and the second sample data are respectively operation data of the first distribution information including the initial distribution information received by the first test account and the second test account; the third sample data and the fourth sample data are respectively operation data of the second distribution information not including the initial distribution information received by the first control account corresponding to the first test account and the second control account corresponding to the second test account;

[0007] A first information distribution model is trained based on the sample data, and a second information distribution model is trained based on data in the sample data other than the first sample data and the second sample data; the first information distribution model is used to determine first information, which is information sent to the first test account and the second test account; the second information distribution model is used to determine second information, which is information sent to the first control account and the second control account;

[0008] generating a first information operation index comparison result based on the operation data of the first test account on the first information and the operation data of the first control account on the second information, and generating a second information operation index comparison result based on the operation data of the second test account on the first information and the operation data of the second control account on the second information;

[0009] When the first information operation indicator comparison result satisfies the corresponding first preset indicator comparison result, and the second information operation indicator comparison result satisfies the corresponding second preset indicator comparison result, the first information distribution model is determined to be the target information distribution model.

[0010] In an exemplary embodiment, generating a first information operation indicator comparison result according to the operation data of the first test account on the first information and the operation data of the first control account on the second information includes:

[0011] Determine first indicator data of the first information operation indicator according to the operation data of the first test account on the first information, and determine second indicator data of the first information operation indicator according to the operation data of the first control account on the second information;

[0012] The first indicator data of the first information operation indicator and the second indicator data of the first information operation indicator are compared to obtain a comparison result of the first information operation indicator.

[0013] In an exemplary embodiment, generating a comparison result of the second information operation indicator according to the operation data of the second test account on the first information and the operation data of the second control account on the second information includes:

[0014] Determine first indicator data of the second information operation indicator according to the operation data of the second test account on the first information, and determine second indicator data of the second information operation indicator according to the operation data of the second control account on the second information;

[0015] The first indicator data of the second information operation indicator and the second indicator data of the second information operation indicator are compared to obtain a comparison result of the second information operation indicator.

[0016] In an exemplary embodiment, before training a first information distribution model based on the sample data and training a second information distribution model based on data other than the first sample data and the second sample data in the sample data, the method further includes:

[0017] generating a first operation indicator comparison result according to the operation data of the first test account on the first distribution information and the operation data of the first control account on the second distribution information, and generating a second operation indicator comparison result according to the operation data of the second test account on the first distribution information and the operation data of the second control account on the second distribution information;

[0018] The training of obtaining a first information distribution model based on the sample data, and the training of obtaining a second information distribution model based on data other than the first sample data and the second sample data in the sample data, include:

[0019] When the first operation indicator comparison result satisfies the corresponding first preset comparison result, and the second operation indicator comparison result satisfies the corresponding second preset comparison result, a first information distribution model is trained based on the sample data, and a second information distribution model is trained based on the data in the sample data other than the first sample data and the second sample data.

[0020] In an exemplary embodiment, after determining that the first information distribution model is a target information distribution model, the method further includes:

[0021] The account information of the first test account is input into the target information distribution model to obtain the distribution information for the first test account determined by the target information distribution model as new initial distribution information.

[0022] In an exemplary embodiment, the training of the first information distribution model according to the sample data further includes:

[0023] Randomly sorting the sample data to obtain randomly sorted sample data;

[0024] The randomly sorted sample data is divided into a plurality of sample data sets, and each sample data set is input into the first information distribution model to be trained in turn for training, so as to obtain the first information distribution model.

[0025] In an exemplary embodiment, before acquiring the sample data, the method further includes:

[0026] In response to a test request carrying a test instruction identifier, obtaining a test instruction corresponding to the test instruction identifier;

[0027] According to the test instruction, information that meets the test instruction is screened out from the information to be distributed as the initial distribution information.

[0028] In an exemplary embodiment, the test request also carries information of the account to be tested;

[0029] Before getting the sample data, also include:

[0030] According to the account information to be tested, a first preset number of account information is obtained from all the account information as the first test accounts, and a second preset number of account information is obtained as the second test accounts.

[0031] In an exemplary embodiment, after obtaining a first preset number of account information from all account information as the first test account and obtaining a second preset number of account information as the second test account according to the account information to be tested, the method further includes:

[0032] The first preset amount of account information excluding the first test account and the second test account is obtained from the entire account information as the first control account, and the second preset amount of account information excluding the first test account and the second test account is obtained from the entire account information as the second control account.

[0033] According to a second aspect of an embodiment of the present disclosure, there is provided an information distribution model determination device, including:

[0034] A data acquisition unit is configured to acquire sample data; the sample data includes at least first sample data, second sample data, third sample data and fourth sample data; the first sample data and the second sample data are respectively operation data of first distribution information including initial distribution information received by a first test account and a second test account; the third sample data and the fourth sample data are respectively operation data of second distribution information not including the initial distribution information received by a first control account corresponding to the first test account and a second control account corresponding to the second test account;

[0035] a model training unit, configured to execute training to obtain a first information distribution model based on the sample data, and to train to obtain a second information distribution model based on data in the sample data other than the first sample data and the second sample data; the first information distribution model is used to determine first information, the first information being information sent to the first test account and the second test account; the second information distribution model is used to determine second information, the second information being information sent to the first control account and the second control account;

[0036] A result generating unit is configured to generate a first information operation indicator comparison result according to the operation data of the first test account on the first information and the operation data of the first control account on the second information, and to generate a second information operation indicator comparison result according to the operation data of the second test account on the first information and the operation data of the second control account on the second information;

[0037] The model determination unit is configured to determine that the first information distribution model is the target information distribution model when the first information operation indicator comparison result meets the corresponding first preset indicator comparison result and the second information operation indicator comparison result meets the corresponding second preset indicator comparison result.

[0038] In an exemplary embodiment, the result generating unit is further configured to determine the first indicator data of the first information operation indicator based on the operation data of the first test account on the first information, and to determine the second indicator data of the first information operation indicator based on the operation data of the first control account on the second information; and to compare the first indicator data of the first information operation indicator with the second indicator data of the first information operation indicator to obtain the comparison result of the first information operation indicator.

[0039] In an exemplary embodiment, the result generating unit is further configured to determine the first indicator data of the second information operation indicator based on the operation data of the second test account on the first information, and to determine the second indicator data of the second information operation indicator based on the operation data of the second control account on the second information; and to compare the first indicator data of the second information operation indicator with the second indicator data of the second information operation indicator to obtain a comparison result of the second information operation indicator.

[0040] In an exemplary embodiment, the apparatus further comprises a result determination module configured to generate a first operation indicator comparison result according to the operation data of the first test account on the first distribution information and the operation data of the first control account on the second distribution information, and to generate a second operation indicator comparison result according to the operation data of the second test account on the first distribution information and the operation data of the second control account on the second distribution information;

[0041] The model training unit is further configured to execute, when the first operation indicator comparison result satisfies the corresponding first preset comparison result and the second operation indicator comparison result satisfies the corresponding second preset comparison result, training a first information distribution model according to the sample data and training a second information distribution model according to data in the sample data other than the first sample data and the second sample data.

[0042] In an exemplary embodiment, the apparatus further includes an information updating unit configured to input the account information of the first test account into the target information distribution model to obtain distribution information for the first test account determined by the target information distribution model as new initial distribution information.

[0043] In an exemplary embodiment, the model training unit is further configured to perform random sorting of the sample data to obtain randomly sorted sample data; divide the randomly sorted sample data into multiple sample data sets, and input each sample data set into the first information distribution model to be trained in turn for training to obtain the first information distribution model.

[0044] In an exemplary embodiment, the device also includes an information acquisition unit, which is configured to execute a test request in response to a test instruction identifier, and obtain a test instruction corresponding to the test instruction identifier; according to the test instruction, filter out information that meets the test instruction from the information to be distributed as the initial distribution information.

[0045] In an exemplary embodiment, the test request also carries information of the account to be tested;

[0046] The information acquisition unit is further configured to acquire a first preset amount of account information from all account information as the first test accounts, and acquire a second preset amount of account information as the second test accounts according to the account information to be tested.

[0047] In an exemplary embodiment, the information acquisition unit is further configured to acquire the first preset amount of account information excluding the first test account and the second test account from the entire account information as the first control account, and acquire the second preset amount of account information excluding the first test account and the second test account from the entire account information as the second control account.

[0048] According to a third aspect of an embodiment of the present disclosure, a server is provided, including:

[0049] processor;

[0050] a memory for storing instructions executable by the processor;

[0051] The processor is configured to execute the instructions to implement the information distribution model determination method as described in any one of the embodiments of the first aspect.

[0052] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which, when instructions in the computer-readable storage medium are executed by a processor of a server, enables the server to execute the information distribution model determination method as described in any one of the embodiments of the first aspect.

[0053] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product includes instructions, and when the instructions are executed by a processor of a server, the server is able to execute the information distribution model determination method as described in any one of the embodiments of the first aspect.

[0054] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0055] By acquiring sample data; the sample data at least includes first sample data, second sample data, third sample data and fourth sample data; the first sample data and the second sample data are respectively operation data of first distribution information including initial distribution information received by a first test account and a second test account; the third sample data and the fourth sample data are respectively operation data of second distribution information not including initial distribution information received by a first control account corresponding to the first test account and a second control account corresponding to the second test account; then, according to the sample data, a first information distribution model is trained, and according to the data in the sample data except the first sample data and the second sample data, a second information distribution model is trained; the first information The distribution model is used to determine the first information, which is information sent to the first test account and the second test account; the second information distribution model is used to determine the second information, which is information sent to the first control account and the second control account; finally, the first information operation index comparison result is generated according to the operation data of the first test account on the first information and the operation data of the first control account on the second information, and the second information operation index comparison result is generated according to the operation data of the second test account on the first information and the operation data of the second control account on the second information; and when both the first information operation index comparison result and the second information operation index comparison result meet the corresponding threshold, the first information distribution model is determined to be the target information distribution model. In this way, when training the information distribution model, the first sample data and the second sample data that play an effect in the sample data are isolated, and then the second information distribution model to be trained is trained, so as to avoid the first sample data and the second sample data used for training the first information distribution model from being learned by the second information distribution model to be trained, so that the trained second information distribution model also has the experimental effect of the trained first information distribution model, so that the experimental effect of the trained first information distribution model cannot be truly judged, resulting in the information distribution model being difficult to be accurately determined in the iterative process, resulting in a low accuracy rate of determining the information distribution model. At the same time, after the first information distribution model is trained according to the sample data, and the second information distribution model is trained according to the data other than the first sample data and the second sample data in the sample data, the comparison results of the first information operation indicators generated according to the operation data of the first test account on the first information and the operation data of the first control account on the second information, as well as the comparison results of the second information operation indicators generated according to the operation data of the second test account on the first information and the operation data of the second control account on the second information are compared with the corresponding thresholds, so that the experimental effect of the first information distribution model and the experimental effect of the second information distribution model can be accurately judged, thereby achieving the purpose of accurately determining the information distribution model when isolating the sample data, so that the information distribution model can be accurately determined in the iterative process, and the accuracy of determining the information distribution model is further improved.

[0056] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0058] Figure 1 The present invention is an application environment diagram of a method for determining an information distribution model according to an exemplary embodiment.

[0059] Figure 2 The invention is a flowchart of a method for determining an information distribution model according to an exemplary embodiment.

[0060] Figure 3 The present invention is a flowchart showing a step of generating a first information operation indicator comparison result according to an exemplary embodiment.

[0061] Figure 4 The present invention is a flowchart showing a step of generating a second information operation indicator comparison result according to an exemplary embodiment.

[0062] Figure 5 The figure is a flowchart of another method for determining an information distribution model according to an exemplary embodiment.

[0063] Figure 6 is a schematic diagram of a hill climbing experiment according to an exemplary embodiment.

[0064] Figure 7 is a schematic diagram of an incremental training strategy according to an exemplary embodiment.

[0065] Figure 8 It is a schematic diagram showing a hill climbing experiment verification based on an LTR (Learning to Ranking) model according to an exemplary embodiment.

[0066] Fig. 9 It is a block diagram of a device for determining an information distribution model according to an exemplary embodiment.

[0067] Fig.10 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0068] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0069] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar accounts, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0070] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0071] The information distribution model determination method provided by the present disclosure can be applied to Figure 1 In the application environment shown. Among them, the terminal 110 interacts with the server 120 through the network; the server 120 obtains sample data in response to the information distribution model determination request sent by the terminal 110; the sample data at least includes the first sample data, the second sample data, the third sample data and the fourth sample data; the first sample data and the second sample data are respectively the operation data of the first distribution information containing the initial distribution information received by the first test account and the second test account; the third sample data and the fourth sample data are respectively the operation data of the second distribution information not containing the initial distribution information received by the first control account corresponding to the first test account and the second control account corresponding to the second test account; according to the sample data, the first information distribution model is trained, and according to the data other than the first sample data and the second sample data in the sample data, the to the second information distribution model; the first information distribution model is used to determine the first information, the first information is the information sent to the first test account and the second test account; the second information distribution model is used to determine the second information, the second information is the information sent to the first control account and the second control account; according to the operation data of the first test account on the first information and the operation data of the first control account on the second information, a first information operation indicator comparison result is generated, and according to the operation data of the second test account on the first information and the operation data of the second control account on the second information, a second information operation indicator comparison result is generated; when the first information operation indicator comparison result meets the corresponding first preset indicator comparison result, and the second information operation indicator comparison result meets the corresponding second preset indicator comparison result, the first information distribution model is determined to be the target information distribution model.

[0072] The terminal 110 may be, but is not limited to, various smart phones, tablet computers, or laptop computers, and the server 120 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0073] Figure 2 is a flow chart of a method for determining an information distribution model according to an exemplary embodiment. Figure 2 As shown, the information distribution model determination method is used in the server 120, and includes the following steps:

[0074] In step S210, sample data is obtained; the sample data includes at least first sample data, second sample data, third sample data and fourth sample data; the first sample data and the second sample data are respectively operation data of first distribution information including initial distribution information received by a first test account and a second test account pair; the third sample data and the fourth sample data are respectively operation data of second distribution information not including initial distribution information received by a first control account corresponding to the first test account and a second control account corresponding to the second test account pair.

[0075] The first test account refers to a test account selected from a certain number of accounts for receiving test data, and the test data refers to initial distribution information, such as video, live broadcast, advertisement, etc. The second test account also refers to a test account selected from a certain number of accounts for receiving initial distribution information. It should be noted that the first test account and the second test account refer to different test accounts that receive the first distribution information containing the initial distribution information.

[0076] The first control account corresponding to the first test account refers to account information having the same account characteristics as the first test account; the second control account corresponding to the second test account refers to account information having the same account characteristics as the second test account; the account characteristics include account quantity, account type, account composition, etc. The first control account and the second control account refer to different control accounts that receive the second distribution information that does not include the initial distribution information.

[0077] It should be noted that the control account is used to receive distribution information that is different from the test account; for example, the test account receives the first distribution information containing the initial distribution information, and the control account receives the second distribution information that does not contain the initial distribution information. In this way, the difference in effects brought about by different distribution information can be determined based on the difference in distribution information, and then valid sample data can be isolated to train the information distribution model.

[0078] The initial distribution information is pre-screened distribution information, that is, the initial distribution information has certain characteristics, such as high content quality and high content popularity, and can be used to determine the information distribution effect after being distributed to the first test account and the second test account. The first distribution information refers to the distribution information that contains a preset number of initial distribution information, such as videos, live broadcasts, advertisements, etc.; the second distribution information refers to the distribution information in the first distribution information other than the initial distribution information, such as videos, live broadcasts, advertisements, etc.

[0079] It should be noted that among the distribution information included in the first distribution information, the initial distribution information accounts for a certain proportion; for example, the first distribution information includes 100 distribution information, and among these 100 distribution information, 5 initial distribution information that have been screened for testing are included. In addition, among these 100 distribution information, 95 distribution information other than the 5 initial distribution information are the second distribution information.

[0080] The operation data refers to the specific operation behavior information of the account on the received distribution information, such as video click behavior, video like behavior, video collection behavior, etc. The first sample data refers to the operation data of the first test account on the received first distribution information containing the initial distribution information, specifically refers to the operation behavior information of the first test account on the received first distribution information containing the initial distribution information. The second sample data refers to the operation data of the second test account on the received first distribution information containing the initial distribution information, specifically refers to the operation behavior information of the second test account on the received first distribution information containing the initial distribution information.

[0081] The third sample data refers to the operation data of the first control account on the second distribution information received without the initial distribution information, specifically refers to the operation behavior information of the first control account on the second distribution information received without the initial distribution information. The fourth sample data refers to the operation data of the second control account on the second distribution information received without the initial distribution information, specifically refers to the operation behavior information of the second control account on the second distribution information received without the initial distribution information.

[0082] It should be noted that, in addition to the first sample data, the second sample data, the third sample data and the fourth sample data, the sample data may also include other sample data, which is not specifically limited in the present disclosure.

[0083] Specifically, the server obtains initial distribution information from a local database, and determines, based on the initial distribution information, first distribution information that includes the initial distribution information and second distribution information that does not include the initial distribution information; determines, from the authorized accounts, a first test account, a first control account corresponding to the first test account, a second test account, and a second control account corresponding to the second test account; then recommends the first distribution information to the first test account and the second test account, and recommends the second distribution information to the first control account and the second control account; extracts, from the operation behavior log of the first test account, operation data of the first test account on the received first distribution information as first sample data; extracts, from the operation behavior log of the second test account, operation data of the second test account on the received first distribution information as second sample data; extracts, from the operation behavior log of the first control account, operation data of the first control account on the received second distribution information as third sample data; extracts, from the operation behavior log of the second control account, operation data of the second control account on the received second distribution information as fourth sample data; obtains sample data based on the first sample data, the second sample data, the third sample data, and the fourth sample data.

[0084] In step S220, a first information distribution model is trained based on the sample data, and a second information distribution model is trained based on data in the sample data other than the first sample data and the second sample data; the first information distribution model is used to determine the first information, which is information sent to the first test account and the second test account; the second information distribution model is used to determine the second information, which is information sent to the first control account and the second control account.

[0085] The first information distribution model and the second information distribution model both refer to trained neural network models used to determine the distribution information, such as a video recommendation model, an advertisement recommendation model, etc. It should be noted that the training data of the first information distribution model and the second information distribution model are different. The training data of the first information distribution model refers to sample data, such as Figure 6 log_all in; the training data of the second information distribution model refers to the data in the sample data except the first sample data and the second sample data, such as Figure 6 In the example, the first sample data is log_P1, and the second sample data is log_M1, then the data in the sample data other than the first sample data and the second sample data is log_all-log_P1-log_M1.

[0086] Among them, the first information distribution model is obtained by training with sample data, so the first information distribution model carries some features of the initial distribution information; the second information distribution model is obtained by training with data other than the first sample data and the second sample data in the sample data, so the second information distribution model does not carry some features of the initial distribution information; therefore, the comparison results can be obtained by comparing the operation data of the distribution information determined by the two information distribution models with the test account and the control account, and the difference in the effects of the two information distribution models can be judged based on the comparison results.

[0087] The first information distribution model is used to determine the distribution information for the first test account and the second test account, and the first information refers to the distribution information for the first test account and the second test account determined by the first information distribution model. The second information distribution model is used to determine the distribution information for the first control account and the second control account, and the second information refers to the distribution information for the first control account and the second control account determined by the second information distribution model.

[0088] It should be noted that after the first information distribution model and the second information distribution model are obtained through training, the first test account and the second test account receive the first information, and the first control account and the second control account receive the second information.

[0089] Specifically, the server trains the neural network model to be trained based on the sample data to obtain a trained neural network model as a first information distribution model; trains the neural network model to be trained based on data other than the first sample data and the second sample data in the sample data to obtain a trained neural network model as a second information distribution model; determines distribution information for the first test account and the second test account through the first information distribution model as first distribution information, and recommends the first distribution information to the first test account and the second test account respectively; determines distribution information for the first control account and the second control account through the second information distribution model as second distribution information, and recommends the second distribution information to the first control account and the second control account respectively.

[0090] For example, the server inputs sample data into the neural network model to be trained to obtain a prediction result of the sample data; determines a first loss value based on the difference between the actual result of the sample data and the prediction result; when the training end condition is not met, such as the first loss value has not reached convergence, the number of training times has not been reached, etc., the model parameters of the neural network model to be trained are adjusted according to the first loss value, and the neural network model after the model parameters are adjusted is trained until the training end condition is met; when the training end condition is met, the trained neural network model that meets the training end condition is used as the first information distribution model. Similarly, the server uses the data in the sample data except the first sample data and the second sample data as target sample data, and inputs them into the neural network model to be trained to obtain the prediction result of the target sample data; determines the second loss value according to the difference between the actual result of the target sample data and the prediction result; when the training end condition is not met, such as the first loss value does not reach convergence, the number of training times is not reached, etc., adjust the model parameters of the neural network model to be trained according to the second loss value, and train the neural network model after the model parameters are adjusted until the training end condition is met; when the training end condition is met, the trained neural network model that meets the training end condition is used as the second information distribution model.

[0091] In step S230, a first information operation indicator comparison result is generated based on the operation data of the first test account on the first information and the operation data of the first control account on the second information, and a second information operation indicator comparison result is generated based on the operation data of the second test account on the first information and the operation data of the second control account on the second information.

[0092] Among them, operation data also refers to the specific operation behavior information of the account on the received information, such as video click behavior, video like behavior, video collection behavior, etc. Information operation indicators refer to indicators used to measure the effect of information distribution, such as video viewing time, video exposure times, video like times, etc. It should be noted that the information operation indicators used to obtain the first information operation indicator comparison result and the information operation indicators used to obtain the second information operation indicator comparison result can be the same or different, and can be adjusted according to actual conditions, which is not limited in this disclosure.

[0093] The first information operation indicator comparison result refers to the comparison result between the indicator data of the information operation indicator obtained based on the operation data of the first test account on the first information and the indicator data of the same information operation indicator obtained based on the operation data of the first control account on the second information, specifically referring to the degree of improvement of the information operation indicator, such as a significant improvement in video viewing time, a significant improvement in video exposure times, a significant improvement in video likes, etc. In actual scenarios, the first information operation indicator comparison result may refer to achieving a Pareto improvement.

[0094] The second information operation indicator comparison result refers to the comparison result between the indicator data of the information operation indicator obtained based on the operation data of the second test account on the first information and the indicator data of the same information operation indicator obtained based on the operation data of the second control account on the second information, specifically referring to the degree of improvement of the information operation indicator, such as a significant improvement in video viewing time, a significant improvement in the number of video exposures, a significant improvement in the number of video likes, etc. In actual scenarios, the second information operation indicator comparison result may refer to a significant consumption advantage.

[0095] It should be noted that the indicator data of the information operation indicator refers to the specific value of the information operation indicator, such as the specific video viewing time, the specific video exposure times, the specific video like times, etc.

[0096] Specifically, the server extracts the operation data of the first test account on the first information from the operation behavior log of the first test account, and analyzes the operation data of the first test account on the first information to obtain the indicator data of the corresponding first information operation indicator; extracts the operation data of the first control account on the second information from the operation behavior log of the first control account, and analyzes the operation data of the first control account on the second information to obtain the indicator data of the corresponding first information operation indicator; compares the above two indicator data to obtain the comparison result of the first information operation indicator.

[0097] Similarly, the server extracts the operation data of the second test account on the first information from the operation behavior log of the second test account, and analyzes the operation data of the second test account on the first information to obtain the indicator data of the corresponding second information operation indicator; extracts the operation data of the second control account on the second information from the operation behavior log of the second control account, and analyzes the operation data of the second control account on the second information to obtain the indicator data of the corresponding second information operation indicator; compares the above two indicator data to obtain the second information operation indicator comparison result.

[0098] In step S240, when the first information operation indicator comparison result satisfies the corresponding first preset indicator comparison result and the second information operation indicator comparison result satisfies the corresponding second preset indicator comparison result, the first information distribution model is determined to be the target information distribution model.

[0099] The first preset indicator comparison result refers to the degree of improvement of the preset information operation indicator, such as the degree of improvement of video viewing time, the degree of improvement of video exposure times, the degree of improvement of video likes, etc. In actual scenarios, the first preset indicator comparison result may refer to a 30% increase in video viewing time, a 40% increase in video exposure times, a 35% increase in video likes, etc.

[0100] Among them, the first information operation indicator comparison result satisfies the corresponding first preset indicator comparison result, which means that the degree of improvement of the information operation indicator represented by the first information operation indicator comparison result is greater than the preset degree of improvement of the information operation indicator; for example, the degree of improvement of the information operation indicator represented by the first information operation indicator comparison result is an increase of 30% in video viewing time, while the preset degree of improvement of the information operation indicator is an increase of 20% in video viewing time, indicating that the first information operation indicator comparison result satisfies the corresponding first preset indicator comparison result.

[0101] The second preset indicator comparison result also refers to the degree of improvement of the preset information operation indicator, such as the degree of improvement of video viewing time, the degree of improvement of video exposure times, the degree of improvement of video likes, etc. In actual scenarios, the second preset indicator comparison result may refer to a 30% increase in video viewing time, a 40% increase in video exposure times, a 35% increase in video likes, etc.

[0102] Among them, the second information operation indicator comparison result satisfies the corresponding second preset indicator comparison result, which means that the information operation indicator improvement degree represented by the second information operation indicator comparison result is greater than the preset information operation indicator improvement degree; for example, the information operation indicator improvement degree represented by the second information operation indicator comparison result is a 30% increase in the number of video exposures, while the preset information operation indicator improvement degree is a 20% increase in the number of video exposures, indicating that the second information operation indicator comparison result satisfies the corresponding second preset indicator comparison result. It should be noted that when the first information operation indicator comparison result satisfies the corresponding first preset indicator comparison result, and the second information operation indicator comparison result satisfies the corresponding second preset indicator comparison result, it means that the information distribution effect for the first test account and the second test account is better than the corresponding first control account and the second control account, and further indicates that the information distribution effect obtained based on the first information distribution model is better than the second information distribution model.

[0103] Specifically, the server obtains from the local database a first preset indicator comparison result corresponding to the first information operation indicator comparison result and a second preset indicator comparison result corresponding to the second information operation indicator comparison result; then compares the first information operation indicator comparison result with the corresponding first preset indicator comparison result, and compares the second information operation indicator comparison result with the corresponding second preset indicator comparison result; when the first information operation indicator comparison result satisfies the corresponding first preset indicator comparison result, and the second information operation indicator comparison result satisfies the corresponding second preset indicator comparison result, it indicates that the information distribution effect obtained based on the first information distribution model is better than that of the second information distribution model, and the first information distribution model is determined as the target information distribution model.

[0104] For example, when the comparison result of the first information operation indicator is a Pareto improvement, and the comparison result of the second information operation indicator is a significant consumption advantage, it means that the information distribution effect based on the first information distribution model is better than that of the second information distribution model, and the first information distribution model is determined as the target information distribution model.

[0105] In determining the above-mentioned information distribution model, sample data is obtained; the sample data at least includes first sample data, second sample data, third sample data and fourth sample data; the first sample data and the second sample data are respectively operation data of first distribution information including initial distribution information received by a first test account and a second test account; the third sample data and the fourth sample data are respectively operation data of second distribution information not including initial distribution information received by a first control account corresponding to the first test account and a second control account corresponding to the second test account; then, the first information distribution model is trained based on the sample data, and the second information distribution model is trained based on the data in the sample data except the first sample data and the second sample data. The first information distribution model is used to determine the first information, which is information sent to the first test account and the second test account; the second information distribution model is used to determine the second information, which is information sent to the first control account and the second control account; finally, the first information operation index comparison result is generated according to the operation data of the first test account on the first information and the operation data of the first control account on the second information, and the second information operation index comparison result is generated according to the operation data of the second test account on the first information and the operation data of the second control account on the second information; and when both the first information operation index comparison result and the second information operation index comparison result meet the corresponding threshold, the first information distribution model is determined to be the target information distribution model. In this way, when training the information distribution model, the first sample data and the second sample data that play an effect in the sample data are isolated, and then the second information distribution model to be trained is trained, so as to avoid the first sample data and the second sample data used for training the first information distribution model from being learned by the second information distribution model to be trained, so that the trained second information distribution model also has the experimental effect of the trained first information distribution model, so that the experimental effect of the trained first information distribution model cannot be truly judged, resulting in the information distribution model being difficult to be accurately determined in the iterative process, resulting in a low accuracy rate of determining the information distribution model.At the same time, after the first information distribution model is trained according to the sample data, and the second information distribution model is trained according to the data other than the first sample data and the second sample data in the sample data, the comparison results of the first information operation indicators generated according to the operation data of the first test account on the first information and the operation data of the first control account on the second information, as well as the comparison results of the second information operation indicators generated according to the operation data of the second test account on the first information and the operation data of the second control account on the second information are compared with the corresponding thresholds, so that the experimental effect of the first information distribution model and the experimental effect of the second information distribution model can be accurately judged, thereby achieving the purpose of accurately determining the information distribution model when isolating the sample data, so that the information distribution model can be accurately determined in the iterative process, and the accuracy of determining the information distribution model is further improved.

[0106] In an exemplary embodiment, if Figure 3 As shown, the above step 230 generates a comparison result of the first information operation index according to the operation data of the first test account on the first information and the operation data of the first control account on the second information, and specifically includes the following steps:

[0107] In step S310, first indicator data of the first information operation indicator is determined based on the operation data of the first test account on the first information, and second indicator data of the first information operation indicator is determined based on the operation data of the first control account on the second information.

[0108] In step S320, the first indicator data of the first information operation indicator and the second indicator data of the first information operation indicator are compared to obtain a first information operation indicator comparison result.

[0109] Among them, the first information operation indicator refers to the indicator used to measure the effectiveness of information distribution, such as video viewing time, video exposure times, video likes, etc.

[0110] Among them, the first indicator data refers to the information operation indicator data obtained by analyzing the operation data of the first test account on the first information, such as the viewing time of a specific video, the number of exposures of a specific video, the number of likes of a specific video, etc.

[0111] Among them, the second indicator data refers to the information operation indicator data obtained by analyzing the operation data of the first control account on the second information, such as the viewing time of a specific video, the number of exposures of a specific video, the number of likes of a specific video, etc.

[0112] Specifically, when there is only one first information operation indicator, the server obtains the operation data of the first test account on the first information, and the operation data of the first control account on the second information, and then takes the first information operation indicator as the statistical target, performs statistical analysis on the operation data of the first test account on the first information, and obtains the first indicator data of the first information operation indicator; takes the first information operation indicator as the statistical target, performs statistical analysis on the operation data of the first control account on the second information, and obtains the second indicator data of the first information operation indicator; finally, compares the first indicator data of the first information operation indicator with the second indicator data of the first information operation indicator to obtain the corresponding comparison result as the comparison result of the first information operation indicator.

[0113] For example, assuming that the first indicator data of the first information operation indicator is the video viewing time T1, and the second indicator data of the first information operation indicator is the video viewing time T2, then the comparison result of the first information operation indicator is the video viewing time improvement degree t=[(T1-T2)×100%] / T2. Assuming that the first indicator data of the first information operation indicator is the video exposure number N1, and the second indicator data of the first information operation indicator is the video exposure number N2, then the comparison result of the first information operation indicator is the video exposure number improvement degree n=[(N1-N2)×100%] / N2.

[0114] Furthermore, in the case where there are at least two first information operation indicators, the server obtains the operation data of the first test account on the first information, and the operation data of the first control account on the second information, and then takes each first information operation indicator as a statistical target, performs statistical analysis on the operation data of the first test account on the first information, and obtains the first indicator data of each first information operation indicator; takes each first information operation indicator as a statistical target, performs statistical analysis on the operation data of the first control account on the second information, and obtains the second indicator data of each first information operation indicator; compares the first indicator data of each first information operation indicator with the second indicator data of each first information operation indicator, and obtains the comparison results of each first information operation indicator; performs weighted sum processing on the comparison results of each first information operation indicator, and obtains the comparison result of the first information operation indicator.

[0115] The technical solution provided by the embodiment of the present disclosure can judge the information distribution effects of different distribution information and further judge the information distribution effects of different information distribution models by sending different distribution information to the first test account and the first control account, and then comparing the first indicator data of the first information operation indicator obtained based on the analysis of the operation data of the first test account on the first information with the second indicator data of the first information operation indicator obtained based on the analysis of the operation data of the first control account on the second information. This is conducive to accurately judging the model effects of different information distribution models and further improving the accuracy of determining the information distribution model.

[0116] In an exemplary embodiment, if Figure 4 As shown, the above step 230 generates a comparison result of the second information operation index according to the operation data of the second test account on the first information and the operation data of the second control account on the second information, and specifically includes the following steps:

[0117] In step S410, first indicator data of the second information operation indicator is determined based on the operation data of the second test account on the first information, and second indicator data of the second information operation indicator is determined based on the operation data of the second control account on the second information.

[0118] In step S420, the first indicator data of the second information operation indicator and the second indicator data of the second information operation indicator are compared to obtain a second information operation indicator comparison result.

[0119] Among them, the second information operation indicator refers to the indicator used to measure the effectiveness of information distribution, such as video viewing time, video exposure times, video likes times, etc.

[0120] Among them, the first indicator data of the second information operation indicator refers to the information operation indicator data obtained by analyzing the operation data of the second test account on the first information, such as the specific video viewing time, the specific video exposure times, the specific video like times, etc.

[0121] Among them, the second indicator data of the second information operation indicator refers to the information operation indicator data obtained by analyzing the operation data of the second control account on the second information, such as the viewing time of a specific video, the number of exposures of a specific video, the number of likes of a specific video, etc.

[0122] Specifically, when there is only one second information operation indicator, the server obtains the operation data of the second test account on the first information, and the operation data of the second control account on the second information, and then takes the second information operation indicator as the statistical target, performs statistical analysis on the operation data of the second test account on the first information, and obtains the first indicator data of the second information operation indicator; takes the second information operation indicator as the statistical target, performs statistical analysis on the operation data of the second control account on the second information, and obtains the second indicator data of the second information operation indicator; finally, compares the first indicator data of the second information operation indicator with the second indicator data of the second information operation indicator to obtain the corresponding comparison result as the comparison result of the second information operation indicator.

[0123] For example, assuming that the first indicator data of the second information operation indicator is the video viewing time T1, and the second indicator data of the second information operation indicator is the video viewing time T2, then the comparison result of the second information operation indicator is the video viewing time improvement degree t=[(T1-T2)×100%] / T2. Assuming that the first indicator data of the second information operation indicator is the video exposure number N1, and the second indicator data of the second information operation indicator is the video exposure number N2, then the comparison result of the second information operation indicator is the video exposure number improvement degree n=[(N1-N2)×100%] / N2.

[0124] Furthermore, in the case where there are at least two second information operation indicators, the server obtains the operation data of the second test account on the first information, and the operation data of the second control account on the second information, and then takes each second information operation indicator as a statistical target, performs statistical analysis on the operation data of the second test account on the first information, and obtains the first indicator data of each second information operation indicator; takes each second information operation indicator as a statistical target, performs statistical analysis on the operation data of the second control account on the second information, and obtains the second indicator data of each second information operation indicator; compares the first indicator data of each second information operation indicator with the second indicator data of each second information operation indicator, and obtains the comparison results of each second information operation indicator; performs weighted sum processing on the comparison results of each second information operation indicator, and obtains the comparison result of the second information operation indicator.

[0125] The technical solution provided by the embodiment of the present disclosure can judge the information distribution effects of different distribution information and further judge the information distribution effects of different information distribution models by sending different distribution information to the second test account and the second control account, and then comparing the first indicator data of the second information operation indicator obtained by analyzing the operation data of the second test account on the first information with the second indicator data of the second information operation indicator obtained by analyzing the operation data of the second control account on the second information. This is conducive to accurately judging the model effects of different information distribution models and further improving the accuracy of determining the information distribution model.

[0126] In an exemplary embodiment, the above-mentioned step S220, before training the first information distribution model based on the sample data and training the second information distribution model based on the data other than the first sample data and the second sample data in the sample data, also includes the following content: generating a first operation indicator comparison result based on the operation data of the first test account on the first distribution information and the operation data of the first control account on the second distribution information, and generating a second operation indicator comparison result based on the operation data of the second test account on the first distribution information and the operation data of the second control account on the second distribution information.

[0127] Among them, operation indicators refer to indicators used to measure the effectiveness of information distribution, specifically first operation indicators, second operation indicators, etc., such as video viewing time, video exposure times, video likes, etc.

[0128] Among them, the comparison result of the first operation indicator refers to the comparison result between the indicator data of the operation indicator obtained by analyzing the operation data of the first distribution information based on the first test account, and the indicator data of the same operation indicator obtained by analyzing the operation data of the second distribution information based on the first control account, specifically refers to the degree of improvement of the operation indicator, such as a significant increase in video viewing time, a significant increase in video exposure times, a significant increase in video likes, etc.

[0129] Among them, the second operation indicator comparison result refers to the comparison result between the indicator data of the operation indicator obtained by analyzing the operation data of the first distribution information based on the second test account, and the indicator data of the same operation indicator obtained by analyzing the operation data of the second distribution information based on the second control account, specifically refers to the degree of improvement of the operation indicator, such as a significant increase in video viewing time, a significant increase in video exposure times, a significant increase in video likes, etc.

[0130] Specifically, the server obtains the operation data of the first test account on the first distribution information, and analyzes the operation data of the first test account on the first distribution information to obtain the first indicator data of the corresponding first operation indicator; obtains the operation data of the first control account on the second distribution information, and analyzes the operation data of the first control account on the second distribution information to obtain the second indicator data of the corresponding first operation indicator; compares the above two indicator data to obtain the first operation indicator comparison result. For example, assuming that the first indicator data of the first operation indicator is the video viewing time T1, and the second indicator data of the first operation indicator is the video viewing time T2, then the first operation indicator comparison result is the video viewing time improvement degree t = [(T1-T2)×100%] / T2.

[0131] Similarly, the server obtains the operation data of the second test account on the first distribution information, and analyzes the operation data of the second test account on the first distribution information to obtain the first indicator data of the corresponding second operation indicator; obtains the operation data of the second control account on the second distribution information, and analyzes the operation data of the second control account on the second distribution information to obtain the second indicator data of the corresponding second operation indicator; compares the above two indicator data to obtain the second operation indicator comparison result. For example, assuming that the first indicator data of the second operation indicator is the number of video exposures N1, and the second indicator data of the second operation indicator is the number of video exposures N2, then the second operation indicator comparison result is the degree of improvement of the number of video exposures n = [(N1-N2)×100%] / N2.

[0132] Furthermore, the above-mentioned step S220, based on the sample data, trains a first information distribution model, and based on the data other than the first sample data and the second sample data in the sample data, trains a second information distribution model, specifically including: when the first operation indicator comparison result satisfies the corresponding first preset comparison result, and the second operation indicator comparison result satisfies the corresponding second preset comparison result, training the first information distribution model based on the sample data, and training the second information distribution model based on the data other than the first sample data and the second sample data in the sample data.

[0133] The first preset comparison result refers to the degree of improvement of the preset operation index, such as the degree of improvement of video viewing time, the degree of improvement of video exposure times, the degree of improvement of video likes, etc. In actual scenarios, the first preset comparison result may refer to a 30% increase in video viewing time, a 40% increase in video exposure times, a 35% increase in video likes, etc.

[0134] Among them, the first operation indicator comparison result satisfies the corresponding first preset comparison result, which means that the degree of improvement of the operation indicator represented by the first operation indicator comparison result is greater than the pre-set degree of improvement of the operation indicator; for example, the degree of improvement of the operation indicator represented by the first operation indicator comparison result is an increase of 30% in video viewing time, while the pre-set degree of improvement of the operation indicator is an increase of 20% in video viewing time, indicating that the first operation indicator comparison result satisfies the corresponding first indicator comparison result.

[0135] The second preset comparison result also refers to the degree of improvement of the preset operation index, such as the degree of improvement of video viewing time, the degree of improvement of video exposure times, the degree of improvement of video likes, etc. In actual scenarios, the second preset comparison result may refer to a 30% increase in video viewing time, a 40% increase in video exposure times, a 35% increase in video likes, etc.

[0136] Among them, the second operation indicator comparison result satisfies the corresponding second preset comparison result, which means that the degree of improvement of the operation indicator represented by the second operation indicator comparison result is greater than the preset degree of improvement of the operation indicator; for example, the degree of improvement of the operation indicator represented by the second operation indicator comparison result is an increase of 30% in the number of video exposures, while the preset degree of improvement of the operation indicator is an increase of 20% in the number of video exposures, indicating that the second operation indicator comparison result satisfies the corresponding second preset comparison result.

[0137] It should be noted that, when the comparison result of the first operation indicator satisfies the corresponding first preset comparison result, and the comparison result of the second operation indicator satisfies the corresponding second preset comparison result, it means that the information distribution effects for the first test account and the second test account are better than the corresponding first control account and the second control account, and further means that the operation data of the first test account on the first distribution information and the operation data of the second test account on the first distribution information are better than the operation data of the first control account on the second distribution information and the operation data of the second control account on the second distribution information, that is, the first sample data is better than the third sample data, and the second sample data is better than the fourth sample data, which means that the first sample data and the second sample data are both valid data, which is convenient for the subsequent isolation of the first sample data and the second sample data.

[0138] Specifically, the server obtains the first preset comparison result corresponding to the first operation indicator comparison result and the second preset comparison result corresponding to the second operation indicator comparison result from the local database; then compares the first operation indicator comparison result with the corresponding first preset comparison result, and compares the second operation indicator comparison result with the corresponding second preset comparison result; when the first operation indicator comparison result satisfies the corresponding first preset comparison result, and the second operation indicator comparison result satisfies the corresponding second preset comparison result, it means that the first sample data is better than the third sample data, and the second sample data is better than the fourth sample data, then the first sample data and the second sample data are confirmed to be valid data. Then, when both the first sample data and the second sample data are valid data, the server trains the neural network model to be trained according to the sample data to obtain the first information distribution model, and trains the neural network model to be trained according to the data in the sample data other than the first sample data and the second sample data to obtain the second information distribution model.

[0139] The technical solution provided by the embodiment of the present disclosure is to train the first information distribution model based on the sample data only when the first operation indicator comparison result satisfies the corresponding first preset comparison result and the second operation indicator comparison result satisfies the corresponding second preset comparison result, and to train the second information distribution model based on the data other than the first sample data and the second sample data in the sample data, thereby achieving the purpose of model training while isolating the valid sample data, avoiding the first sample data and the second sample data used for training the first information distribution model from being learned by the second information distribution model to be trained, so that the trained second information distribution model also has the same experimental effect as the trained first information distribution model, and thus it is impossible to truly judge the experimental effect of the trained first information distribution model, resulting in the information distribution model being difficult to be accurately determined during the iteration process, causing the defect of low determination accuracy of the information distribution model, and further improving the determination accuracy of the information distribution model.

[0140] In an exemplary embodiment, the above step S240, after determining that the first information distribution model is the target information distribution model, further includes the following content: inputting the account information of the first test account into the target information distribution model, and obtaining the distribution information for the first test account determined by the target information distribution model as new initial distribution information.

[0141] Specifically, after being trained, the target information distribution model can determine different distribution information based on different account information; that is, if the input is the account information of the first test account, then the output is the distribution information for the first test account, and the distribution information for the first test account is used as the new initial distribution information, thereby realizing the application of the target information distribution model.

[0142] The technical solution provided by the embodiment of the present disclosure realizes the acquisition of new initial distribution information, that is, it is possible to obtain new target distribution information containing all the distribution information of the initial distribution information through the new initial distribution information, thereby realizing the update of the target distribution information and improving the updating efficiency of the distribution information.

[0143] In an exemplary embodiment, the above-mentioned step S220, based on the sample data, trains the first information distribution model, and also includes: randomly sorting the sample data to obtain randomly sorted sample data; dividing the randomly sorted sample data into multiple sample data sets, and inputting each sample data set into the first information distribution model to be trained in turn for training to obtain the first information distribution model.

[0144] Specifically, the sample data contains multiple data. Scrambling the order of the data in the sample data can improve the random arrangement of the data. After inputting into the first information distribution model to be trained, the recognition ability and recognition accuracy of the first information distribution model to be trained for the sample data can be improved.

[0145] The technical solution provided by the embodiment of the present disclosure improves the training performance and learning ability of the first information distribution model, and ultimately improves the accuracy of determining the information distribution model.

[0146] In an exemplary embodiment, the above step S210, before obtaining sample data, also includes: in response to a test request carrying a test instruction identifier, obtaining a test instruction corresponding to the test instruction identifier; and according to the test instruction, filtering out information that meets the test instruction from the information to be distributed as initial distribution information.

[0147] Specifically, the test instruction identifier can indicate different test instructions. A test instruction corresponds to a group of initial distribution information of the same type or with the same characteristics in advance, which is equivalent to a preset information distribution strategy. According to the test instruction, the distribution information of the type or characteristics corresponding to the test instruction can be screened out from all the information to be distributed as the initial distribution information.

[0148] The technical solution provided by the embodiment of the present disclosure can determine the initial distribution information through a test request carrying a test instruction identifier, thereby determining the learning direction of the information distribution model to be obtained, that is, the training direction of the information distribution model is determined in advance through a test request carrying a test instruction identifier, thereby improving the training efficiency of the information distribution model and the accuracy of determining the information distribution model.

[0149] In an exemplary embodiment, the test request also carries the account information to be tested; the above step S210, before obtaining the sample data, also includes: according to the account information to be tested, obtaining a first preset number of account information from all account information as the first test account, and obtaining a second preset number of account information as the second test account.

[0150] Specifically, the account information to be tested is used to select account information used as a test account from all account information, and the account information to be tested is carried in the test request and can be directly transmitted to the server. The account information to be tested can be the type of account, the characteristics of the account, the preset range identified in the account information, and the preset number of account information. The first preset number is used to obtain a certain number of account information as the first test account, and the second preset number is used to obtain a certain number of account information as the second test account.

[0151] The technical solution provided by the embodiment of the present disclosure improves the efficiency of obtaining test accounts by setting the account information to be tested so that account information that meets the conditions of the account information to be tested can be obtained from all account information as test accounts.

[0152] In an exemplary embodiment, after obtaining a first preset number of account information as a first test account and a second preset number of account information as a second test account from all account information according to the account information to be tested, the method further includes: obtaining a first preset number of account information other than the first test account and the second test account from all account information as a first control account, and obtaining a second preset number of account information other than the first test account and the second test account from all account information as a second control account.

[0153] The first control account and the second control account are different control accounts.

[0154] Specifically, the first control account does not overlap with the first test account and the second test account, that is, the control account is selected from accounts other than the first test account and the second test account; similarly, the second control account does not overlap with the first test account and the second test account; the second control account can also be set not to overlap with the first control account. That is, the first test account, the second test account, the first control account and the second control account are independent of each other.

[0155] The technical solution provided by the embodiment of the present disclosure obtains a first test account, a second test account, a first control account, and a second control account that are independent of each other, so that the test of a certain test account will not affect other accounts, thereby ensuring that the test processes of various test accounts are independent and isolated from each other and are not affected.

[0156] Figure 5 is a flowchart of another information distribution model determination method according to an exemplary embodiment. Figure 5 As shown, the information distribution model determination method is used in the server 120, and includes the following steps:

[0157] In step S510, in response to a test request carrying a test instruction identifier, a test instruction corresponding to the test instruction identifier is obtained; according to the test instruction, information that meets the test instruction is screened from the information to be distributed as initial distribution information.

[0158] In step S520, the test request also carries the account information to be tested; according to the account information to be tested, a first preset number of account information is obtained from all the account information as the first test account, and a second preset number of account information is obtained as the second test account.

[0159] In step S530, a first preset amount of account information excluding the first test account and the second test account is obtained from all account information as a first control account, and a second preset amount of account information excluding the first test account and the second test account is obtained from all account information as a second control account.

[0160] In step S540, sample data is obtained; the sample data includes at least first sample data, second sample data, third sample data and fourth sample data; the first sample data and the second sample data are respectively operation data of first distribution information including initial distribution information received by the first test account and the second test account; the third sample data and the fourth sample data are respectively operation data of second distribution information not including initial distribution information received by the first control account corresponding to the first test account and the second control account corresponding to the second test account.

[0161] In step S550, a first operation indicator comparison result is generated based on the operation data of the first test account on the first distribution information and the operation data of the first control account on the second distribution information, and a second operation indicator comparison result is generated based on the operation data of the second test account on the first distribution information and the operation data of the second control account on the second distribution information.

[0162] In step S560, when the first operation indicator comparison result satisfies the corresponding first preset comparison result, and the second operation indicator comparison result satisfies the corresponding second preset comparison result, a first information distribution model is trained based on the sample data, and a second information distribution model is trained based on the data other than the first sample data and the second sample data in the sample data; the first information distribution model is used to determine the first information, which is the information sent to the first test account and the second test account; the second information distribution model is used to determine the second information, which is the information sent to the first control account and the second control account.

[0163] In step S570, the first indicator data of the first information operation indicator is determined based on the operation data of the first test account on the first information, and the second indicator data of the first information operation indicator is determined based on the operation data of the first control account on the second information; the first indicator data of the first information operation indicator and the second indicator data of the first information operation indicator are compared to obtain the comparison result of the first information operation indicator.

[0164] In step S580, the first indicator data of the second information operation indicator is determined based on the operation data of the second test account on the first information, and the second indicator data of the second information operation indicator is determined based on the operation data of the second control account on the second information; the first indicator data of the second information operation indicator and the second indicator data of the second information operation indicator are compared to obtain the comparison result of the second information operation indicator.

[0165] In step S590, when the first information operation indicator comparison result satisfies the corresponding first preset indicator comparison result and the second information operation indicator comparison result satisfies the corresponding second preset indicator comparison result, the first information distribution model is determined to be the target information distribution model.

[0166] In the above-mentioned information distribution model determination method, when training the information distribution model, the first sample data and the second sample data that play an effect in the sample data are isolated, and then the second information distribution model to be trained is trained to avoid the first sample data and the second sample data used to train the first information distribution model from being learned by the second information distribution model to be trained, so that the trained second information distribution model also has the same experimental effect as the trained first information distribution model, and thus it is impossible to truly judge the experimental effect of the trained first information distribution model, which makes it difficult to accurately determine the information distribution model during the iteration process, resulting in a low accuracy rate in determining the information distribution model. At the same time, after the first information distribution model is trained according to the sample data, and the second information distribution model is trained according to the data other than the first sample data and the second sample data in the sample data, the comparison results of the first information operation indicators generated according to the operation data of the first test account on the first information and the operation data of the first control account on the second information, as well as the comparison results of the second information operation indicators generated according to the operation data of the second test account on the first information and the operation data of the second control account on the second information are compared with the corresponding thresholds, so that the experimental effect of the first information distribution model and the experimental effect of the second information distribution model can be accurately judged, thereby achieving the purpose of accurately determining the information distribution model when isolating the sample data, so that the information distribution model can be accurately determined in the iterative process, and the accuracy of determining the information distribution model is further improved.

[0167] In one embodiment, another information distribution model determination method is provided, which is illustrated by taking a ramp-up experiment as a test instruction, a consumption observation group as a control account, and a ramp-up experiment group as a test account. The specific steps are as follows:

[0168] Step 1: Create an experiment. First, create a world for the ramp-up experiment. Figure 6 As shown, it is divided into a consumer observation group and a climbing group. The strategies related to the climbing itself are launched and observed in the climbing group, that is, P0 vs P1. Among them, P0 is the baseline group corresponding to the climbing group, and P1 is the experimental group corresponding to the climbing group. During the experiment, P1 is constrained to ensure that the consumption indicators do not drop too much, and aims to improve the content growth indicators and achieve Pareto improvement. In order to effectively observe the impact of P1 on the consumer-side indicators, the present invention also establishes a consumer observation experimental group in the same world to observe the value changes on the consumer side affected by the climbing, that is, the comparison of the consumer side, M0 vs M1. Among them, M0 is the consumer observation baseline group, and M1 is the consumer observation experimental group affected by the climbing experiment. M0 vs M1 mainly compares the indicators corresponding to the consumer side, such as duration, exposure, retention, DAU (Daily Active User, number of daily active users), etc.

[0169] Step 2: Train a full set of sorting models. Based on step 1, the present invention trains a corresponding full set of sorting models based on different traffic logs. First, remove the logs corresponding to the traffic P1 and M1 affected by the ramping strategy, recorded as log_all-log_P1-log_M1; then, train a set of online baseline sorting models model_0 including all sorting stages such as coarse sorting, fine sorting and re-sorting based on log_all-log_P1-log_M1. The original baseline model group constructed based on the traffic logs affected by the ramping strategy remains unchanged and is recorded as model_1; finally, in order to save costs, the present invention does not adopt the model cold start method in this process. Because the current baseline models of each sorting stage are based on deep neural networks, the optimization algorithms are SGD (Stochastic gradient descent) and variants of SGD, and there is basically no related batch algorithm, so the present invention adopts an incremental training strategy. Figure 7 As shown, when conducting online learning, the present invention first reshuffles the logs (log_all-log_M1-log_P1) that are not affected by the climbing strategy to train model model_0, and then reshuffles the climbing logs (log_M1+log_P1) to train model_1 in an incremental manner. Using this type of training method, the present invention can even support iterations of multiple groups of consumer observation experiments.

[0170] Step 3: Model launch. P1 and M1 launch the original baseline model model_1, and replace the baseline model model_1 of all other traffic (including other worlds) with model_0.

[0171] Step 4: Experiment iteration and full promotion. When P1 vs P0 achieves Pareto improvement through algorithm iteration of P1, and M1 vs M0 obtains obvious consumption advantage, that is, the positive benefit corresponding to M1 vs M0 is greater than the negative loss of P1 vs P0, full promotion can be achieved. P1 is fully promoted to the ramp-up baseline group (30%), and the consumption observation baseline group will fully promote model_1 to replace model_0, and then the main traffic will get the full promotion benefit of the ramp-up experiment and enter the next ramp-up experiment iteration cycle.

[0172] In one embodiment, in order to enable the master station to fully implement the experimental mechanism while effectively controlling the cost, an approximate solution is further designed to implement the experimental mechanism, that is, without retraining the coarse-fine ranking model, it is first verified on the LTR (Learning to Ranking) model, such as Figure 8As shown, compared with the complete solution that filters the traffic log according to the corresponding exptag (experimental group tag), this invention uses h10000-h10009 as htag (log tag) to filter the log; wherein, h10000 and h10009 both represent the log number, which can be adjusted according to the actual situation; still taking the ramp as an example, the specific steps are as follows:

[0173] Step 1: Create an experiment. The present invention still establishes a world of ramping experiments, which is divided into a consumer observation group and a ramping group. The strategy of ramping itself is launched and observed in the ramping group, that is, P0 vs P1. P1 should ensure that the consumption index does not drop too much as a constraint, improve the content growth index, and achieve Pareto improvement. In the same world, a consumer observation experimental group is established to observe the comparison of the consumer side affected by the ramping, that is, M0 vs M1. Among them, M0 is the consumer observation baseline group, and M1 is the consumer observation experimental group affected by the ramping experiment. M0 vs M1 mainly compares consumption indicators, such as duration, exposure, retention, DAU, etc.

[0174] Step 2: Training a full set of sorting models. Based on step 1, the present invention trains a corresponding full set of sorting models based on different traffic flows.

[0175] h10000-h10009 is determined as htag for climbing recall, recorded as R1; M0, M1, P0, P1 need to add climbing recall R1, and the original sorting baseline model model_0 in the coarse, fine and LTR stages that is not affected by climbing recall R1 is trained using log_all-log_R1, that is, the recall log R1 is removed for training. The LTR model model_1 is trained using all logs log_all, and P1 adds boost (a recall strategy) to the climbing recall.

[0176] Step 3: Model launch. P1 and M1 launch the baseline model model_1 affected by the ramp recall R1, and all other traffic (including other worlds) use the original baseline model model_0. The baseline model model_1 corresponding to LTR can be trained only when doing exploratory experiments, and model_0 is used for all traffic when not doing experiments.

[0177] Step 4: Experiment iteration and full promotion. When P1 vs P0 achieves Pareto improvement through algorithm iteration of P1, and M1 vs M0 obtains obvious consumption advantage, that is, the positive benefit corresponding to M1 vs M0 is greater than the negative loss of P1 vs P0, full promotion can be achieved. P1 is fully promoted to the ramp-up baseline group, and the consumption observation baseline group will fully promote model_1 to replace model_0, and then the main traffic will get the full promotion benefit of the ramp-up experiment and enter the next ramp-up experiment iteration cycle.

[0178] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0179] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0180] Based on the same inventive concept, an embodiment of the present disclosure also provides an information distribution model determination device for implementing the above-mentioned information distribution model determination method.

[0181] Fig. 9 is a block diagram of a device for determining an information distribution model according to an exemplary embodiment. Fig. 9 The device includes a data acquisition unit 902, a model training unit 904, a result generation unit 906 and a model determination unit 908.

[0182] The data acquisition unit 902 is configured to acquire sample data; the sample data includes at least first sample data, second sample data, third sample data and fourth sample data; the first sample data and the second sample data are respectively operation data of first distribution information including initial distribution information received by a first test account and a second test account; the third sample data and the fourth sample data are respectively operation data of second distribution information not including initial distribution information received by a first control account corresponding to the first test account and a second control account corresponding to the second test account;

[0183] The model training unit 904 is configured to execute training to obtain a first information distribution model based on the sample data, and to train to obtain a second information distribution model based on data other than the first sample data and the second sample data in the sample data; the first information distribution model is used to determine the first information, and the first information is the information sent to the first test account and the second test account; the second information distribution model is used to determine the second information, and the second information is the information sent to the first control account and the second control account.

[0184] The result generating unit 906 is configured to generate a first information operation indicator comparison result based on the operation data of the first test account on the first information and the operation data of the first control account on the second information, and to generate a second information operation indicator comparison result based on the operation data of the second test account on the first information and the operation data of the second control account on the second information.

[0185] The model determination unit 908 is configured to determine that the first information distribution model is the target information distribution model when the first information operation indicator comparison result meets the corresponding first preset indicator comparison result and the second information operation indicator comparison result meets the corresponding second preset indicator comparison result.

[0186] In an exemplary embodiment, the result generation unit 906 is further configured to determine the first indicator data of the first information operation indicator based on the operation data of the first test account on the first information, and to determine the second indicator data of the first information operation indicator based on the operation data of the first control account on the second information; and to compare the first indicator data of the first information operation indicator with the second indicator data of the first information operation indicator to obtain a comparison result of the first information operation indicator.

[0187] In an exemplary embodiment, the result generation unit 906 is further configured to determine the first indicator data of the second information operation indicator based on the operation data of the second test account on the first information, and to determine the second indicator data of the second information operation indicator based on the operation data of the second control account on the second information; and to compare the first indicator data of the second information operation indicator with the second indicator data of the second information operation indicator to obtain a comparison result of the second information operation indicator.

[0188] In an exemplary embodiment, the information distribution model determination device further includes a result determination module configured to generate a first operation indicator comparison result based on the operation data of the first test account on the first distribution information and the operation data of the first control account on the second distribution information, and generate a second operation indicator comparison result based on the operation data of the second test account on the first distribution information and the operation data of the second control account on the second distribution information;

[0189] The model training unit 904 is also configured to execute training to obtain a first information distribution model based on the sample data when the first operation indicator comparison result satisfies the corresponding first preset comparison result and the second operation indicator comparison result satisfies the corresponding second preset comparison result, and to train to obtain a second information distribution model based on data in the sample data other than the first sample data and the second sample data.

[0190] In an exemplary embodiment, the information distribution model determination device further includes an information updating unit configured to input the account information of the first test account into the target information distribution model to obtain distribution information for the first test account determined by the target information distribution model as new initial distribution information.

[0191] In an exemplary embodiment, the model training unit 904 is also configured to perform random sorting of the sample data to obtain randomly sorted sample data; divide the randomly sorted sample data into multiple sample data sets, and input each sample data set into the first information distribution model to be trained in turn for training to obtain the first information distribution model.

[0192] In an exemplary embodiment, the information distribution model determination device also includes an information acquisition unit, which is configured to execute a test request in response to a test instruction identifier, and obtain a test instruction corresponding to the test instruction identifier; according to the test instruction, filter out information that meets the test instruction from the information to be distributed as initial distribution information.

[0193] In an exemplary embodiment, the test request also carries information of the account to be tested;

[0194] The information acquisition unit is further configured to acquire a first preset amount of account information from all account information as first test accounts according to the account information to be tested, and acquire a second preset amount of account information as second test accounts.

[0195] In an exemplary embodiment, the information acquisition unit is further configured to acquire a first preset amount of account information excluding the first test account and the second test account from all the account information as a first control account, and to acquire a second preset amount of account information excluding the first test account and the second test account from all the account information as a second control account.

[0196] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0197] Each module in the above-mentioned information distribution model determination device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0198] Fig.10 1 is a block diagram of a server 1000 for implementing a method for determining an information distribution model according to an exemplary embodiment. Fig.10The server 1000 includes a processing component 1020, which further includes one or more processors, and a memory resource represented by a memory 1022 for storing instructions, such as applications, that can be executed by the processing component 1020. The application stored in the memory 1022 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1020 is configured to execute instructions to perform the above method.

[0199] The server 1000 may also include a power supply component 1024 configured to perform power management of the server 1000, a wired or wireless network interface 1026 configured to connect the server 1000 to a network, and an input / output (I / O) interface 1028. The server 1000 may operate based on an operating system stored in the memory 1022, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0200] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1022 including instructions, and the above instructions can be executed by a processor of the server 1000 to complete the above method. The storage medium can be a computer-readable storage medium, for example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0201] In an exemplary embodiment, a computer program product is also provided. The computer program product includes instructions, and the instructions can be executed by a processor of the server 1000 to complete the above method.

[0202] It should be noted that the above-mentioned devices, servers, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments and will not be described one by one here.

[0203] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0204] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for determining an information distribution model, characterized in that: include: Acquire sample data; the sample data at least includes first sample data, second sample data, third sample data and fourth sample data; The first sample data and the second sample data are respectively operation data of the first test account and the second test account with respect to the received first distribution information including the initial distribution information; The third sample data and the fourth sample data are respectively operation data of the first control account corresponding to the first test account and the second control account corresponding to the second test account for receiving the second distribution information that does not include the initial distribution information; A first information distribution model is trained based on the sample data, and a second information distribution model is trained based on data other than the first sample data and the second sample data in the sample data; the first information distribution model is used to determine first information, and the first information is information sent to the first test account and the second test account; The second information distribution model is used to determine second information, where the second information is information sent to the first control account and the second control account; generating a first information operation index comparison result based on the operation data of the first test account on the first information and the operation data of the first control account on the second information, and generating a second information operation index comparison result based on the operation data of the second test account on the first information and the operation data of the second control account on the second information; When the first information operation indicator comparison result satisfies the corresponding first preset indicator comparison result, and the second information operation indicator comparison result satisfies the corresponding second preset indicator comparison result, the first information distribution model is determined to be the target information distribution model.

2. The method according to claim 1, characterized in that The step of generating a comparison result of the first information operation index according to the operation data of the first test account on the first information and the operation data of the first control account on the second information includes: Determine first indicator data of the first information operation indicator according to the operation data of the first test account on the first information, and determine second indicator data of the first information operation indicator according to the operation data of the first control account on the second information; The first indicator data of the first information operation indicator and the second indicator data of the first information operation indicator are compared to obtain a comparison result of the first information operation indicator.

3. The method according to claim 1, characterized in that The step of generating a comparison result of the second information operation index according to the operation data of the second test account on the first information and the operation data of the second control account on the second information comprises: Determine first indicator data of the second information operation indicator according to the operation data of the second test account on the first information, and determine second indicator data of the second information operation indicator according to the operation data of the second control account on the second information; The first indicator data of the second information operation indicator and the second indicator data of the second information operation indicator are compared to obtain a comparison result of the second information operation indicator.

4. The method according to claim 1, characterized in that: Before training a first information distribution model according to the sample data and training a second information distribution model according to data other than the first sample data and the second sample data in the sample data, the method further includes: generating a first operation indicator comparison result according to the operation data of the first test account on the first distribution information and the operation data of the first control account on the second distribution information, and generating a second operation indicator comparison result according to the operation data of the second test account on the first distribution information and the operation data of the second control account on the second distribution information; The training of obtaining a first information distribution model based on the sample data, and the training of obtaining a second information distribution model based on data other than the first sample data and the second sample data in the sample data, include: When the first operation indicator comparison result satisfies the corresponding first preset comparison result, and the second operation indicator comparison result satisfies the corresponding second preset comparison result, a first information distribution model is trained based on the sample data, and a second information distribution model is trained based on the data in the sample data other than the first sample data and the second sample data.

5. The method according to any one of claims 1 to 4, characterized in that: After determining that the first information distribution model is a target information distribution model, the method further includes: The account information of the first test account is input into the target information distribution model to obtain the distribution information for the first test account determined by the target information distribution model as new initial distribution information.

6. The method according to any one of claims 1 to 4, characterized in that: The training of obtaining a first information distribution model according to the sample data further includes: Randomly sorting the sample data to obtain randomly sorted sample data; The randomly sorted sample data is divided into a plurality of sample data sets, and each sample data set is input into the first information distribution model to be trained in turn for training, so as to obtain the first information distribution model.

7. The method according to claim 1, characterized in that Before getting the sample data, also include: In response to a test request carrying a test instruction identifier, obtaining a test instruction corresponding to the test instruction identifier; According to the test instruction, information that meets the test instruction is screened out from the information to be distributed as the initial distribution information.

8. The method according to claim 7, characterized in that The test request also carries the account information to be tested; Before getting the sample data, also include: According to the account information to be tested, a first preset number of account information is obtained from all the account information as the first test accounts, and a second preset number of account information is obtained as the second test accounts.

9. The method according to claim 8, characterized in that After obtaining a first preset number of account information from all account information as the first test accounts according to the account information to be tested, and obtaining a second preset number of account information as the second test accounts, the method further includes: The first preset amount of account information excluding the first test account and the second test account is obtained from the entire account information as the first control account, and the second preset amount of account information excluding the first test account and the second test account is obtained from the entire account information as the second control account.

10. An information distribution model determination device, characterized in that: include: A data acquisition unit is configured to acquire sample data; the sample data includes at least first sample data, second sample data, third sample data and fourth sample data; the first sample data and the second sample data are respectively operation data of the first test account and the second test account on the received first distribution information including initial distribution information; The third sample data and the fourth sample data are respectively operation data of the first control account corresponding to the first test account and the second control account corresponding to the second test account for receiving the second distribution information that does not include the initial distribution information; a model training unit configured to execute training to obtain a first information distribution model based on the sample data, and to train to obtain a second information distribution model based on data in the sample data other than the first sample data and the second sample data; the first information distribution model is used to determine first information, the first information being information sent to the first test account and the second test account; The second information distribution model is used to determine second information, where the second information is information sent to the first control account and the second control account; A result generating unit is configured to generate a first information operation indicator comparison result according to the operation data of the first test account on the first information and the operation data of the first control account on the second information, and to generate a second information operation indicator comparison result according to the operation data of the second test account on the first information and the operation data of the second control account on the second information; The model determination unit is configured to determine that the first information distribution model is the target information distribution model when the first information operation indicator comparison result meets the corresponding first preset indicator comparison result and the second information operation indicator comparison result meets the corresponding second preset indicator comparison result.

11. The device according to claim 10, characterized in that The result generating unit is further configured to determine first indicator data of the first information operation indicator according to the operation data of the first test account on the first information, and to determine second indicator data of the first information operation indicator according to the operation data of the first control account on the second information; and to compare the first indicator data of the first information operation indicator with the second indicator data of the first information operation indicator to obtain a comparison result of the first information operation indicator.

12. The device according to claim 10, characterized in that The result generating unit is further configured to determine first indicator data of the second information operation indicator according to the operation data of the second test account on the first information, and determine second indicator data of the second information operation indicator according to the operation data of the second control account on the second information; The first indicator data of the second information operation indicator and the second indicator data of the second information operation indicator are compared to obtain a comparison result of the second information operation indicator.

13. The device according to claim 10, characterized in that The apparatus further includes a result determination module configured to generate a first operation indicator comparison result based on the operation data of the first test account on the first distribution information and the operation data of the first control account on the second distribution information, and to generate a second operation indicator comparison result based on the operation data of the second test account on the first distribution information and the operation data of the second control account on the second distribution information; The model training unit is further configured to execute, when the first operation indicator comparison result satisfies the corresponding first preset comparison result and the second operation indicator comparison result satisfies the corresponding second preset comparison result, training a first information distribution model according to the sample data and training a second information distribution model according to data in the sample data other than the first sample data and the second sample data.

14. The device according to any one of claims 10 to 13, characterized in that The device further includes an information updating unit configured to input the account information of the first test account into the target information distribution model, and obtain distribution information for the first test account determined by the target information distribution model as new initial distribution information.

15. The device according to any one of claims 10 to 13, characterized in that The model training unit is also configured to perform random sorting of the sample data to obtain randomly sorted sample data; divide the randomly sorted sample data into multiple sample data sets, and input each sample data set into the first information distribution model to be trained in turn for training to obtain the first information distribution model.

16. The device according to claim 10, characterized in that The device also includes an information acquisition unit, which is configured to execute a test request in response to a test instruction identifier and obtain a test instruction corresponding to the test instruction identifier; according to the test instruction, filter out information that meets the test instruction from the information to be distributed as the initial distribution information.

17. The device according to claim 16, characterized in that The test request also carries the account information to be tested; The information acquisition unit is further configured to acquire a first preset amount of account information from all account information as the first test accounts, and acquire a second preset amount of account information as the second test accounts according to the account information to be tested.

18. The device according to claim 17, characterized in that The information acquisition unit is further configured to acquire the first preset amount of account information excluding the first test account and the second test account from the entire account information as the first control account, and acquire the second preset amount of account information excluding the first test account and the second test account from the entire account information as the second control account.

19. A server, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the information distribution model determination method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute the information distribution model determination method according to any one of claims 1 to 9.

21. A computer program product, comprising instructions, characterized in that: When the instructions are executed by a processor of a server, the server is enabled to execute the information distribution model determination method according to any one of claims 1 to 9.

Images