Training and ranking methods for multi-agent models

Abstract

The present disclosure provides a training method, sorting method, apparatus, device, storage medium and computer program product for a multi-agent model, which relates to the field of artificial intelligence technology, specifically the field of deep learning and intelligent recommendation technology, and can be applied to scenarios such as data sorting. The specific implementation scheme is: obtaining a training sample set; performing the following training steps: selecting multiple training samples from the training sample set and inputting them into the multi-agent model for scoring, obtaining scoring results corresponding to each of the multiple agents; splicing the scoring results to obtain a joint score; inputting the joint score into a score evaluation environment to obtain reward values ​​corresponding to each of the multiple agents; training the multi-agent model based on the scoring results and reward values; and outputting the trained multi-agent model in response to the number of training times being equal to a preset number threshold. This improves the accuracy of sorting.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically the field of deep learning and intelligent recommendation technology, and can be applied to scenarios such as data sorting. In particular, it relates to a training method, sorting method, apparatus, device, storage medium, and computer program product for a multi-agent model. Background Art

[0002] When training a multi-agent model, a parameter sharing method is usually used, which is equivalent to first learning the common strategy among multiple agents in the multi-agent model, and then learning the specific strategy of each agent. However, this training method has high requirements on the correlation between agents, and training cannot be performed when the agents are unrelated. Summary of the Invention

[0003] The present disclosure provides a training method, sorting method, apparatus, device, storage medium and computer program product for a multi-agent model, which improves the accuracy of sorting.

[0004] According to one aspect of the present disclosure, a training method for a multi-agent model is provided, comprising: obtaining a training sample set; performing the following training steps: selecting multiple training samples from the training sample set and inputting them into the multi-agent model for scoring to obtain scoring results corresponding to each of the multiple agents; concatenating the scoring results to obtain a joint score; inputting the joint score into a score evaluation environment to obtain reward values ​​corresponding to each of the multiple agents; training the multi-agent model based on the scoring results and the reward values; and outputting the trained multi-agent model in response to the number of training times being equal to a preset number threshold.

[0005] According to another aspect of the present disclosure, a sorting method is provided, comprising: obtaining a plurality of data to be processed; inputting the plurality of data to be processed into a multi-agent model for scoring, and obtaining scoring results corresponding to each of the plurality of agents; and sorting the plurality of data to be processed based on the scoring results.

[0006] According to another aspect of the present disclosure, a training device for a multi-agent model is provided, including: a first acquisition module, configured to acquire a training sample set; a training module, configured to perform the following training steps: selecting multiple training samples from the training sample set and inputting them into the multi-agent model for scoring, to obtain scoring results corresponding to each of the multiple agents; splicing the scoring results to obtain a joint score; inputting the joint score into a score evaluation environment to obtain a reward value corresponding to each of the multiple agents; training the multi-agent model based on the scoring results and the reward value; and outputting the trained multi-agent model in response to the number of training times being equal to a preset number threshold.

[0007] According to another aspect of the present disclosure, a sorting device is provided, including: a second acquisition module, configured to acquire a plurality of data to be processed; a scoring module, configured to input the plurality of data to be processed into a multi-agent model for scoring, and obtain scoring results corresponding to each of the plurality of agents; and a sorting module, configured to sort the plurality of data to be processed based on the scoring results.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method and sorting method of the multi-agent model.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the training method and ranking method of the multi-agent model.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the training method and sorting method of the multi-agent model when executed by a processor.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0013] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0014] Figure 2 is a flowchart of an embodiment of a method for training a multi-agent model according to the present disclosure;

[0015] Figure 3 is a flowchart of another embodiment of a method for training a multi-agent model according to the present disclosure;

[0016] Figure 4 is a schematic diagram of a training method for a multi-agent model according to the present disclosure;

[0017] Figure 5 is a flow chart of an embodiment of a sorting method according to the present disclosure;

[0018] Figure 6is a structural diagram of an embodiment of a training device for a multi-agent model according to the present disclosure;

[0019] Figure 7 is a structural diagram of an embodiment of a sorting device according to the present disclosure;

[0020] Figure 8 It is a block diagram of an electronic device used to implement the training method or ranking method of the multi-agent model of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the multi-agent model training method or ranking method or multi-agent model training device or ranking device of the present disclosure can be applied.

[0023] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0024] Users can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to obtain multi-agent models or data sorting, etc. Various client applications, such as text data processing applications, can be installed on terminal devices 101, 102, 103.

[0025] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules, or as a single software program or software module. This is not specifically limited here.

[0026] The server 105 can provide various services based on determining a multi-agent model or data ranking. For example, the server 105 can analyze and process the recommended data and user data obtained from the terminal devices 101, 102, and 103, and generate processing results (such as determining a multi-agent model or determining the ranking of the recommended data, etc.).

[0027] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitations are given here.

[0028] It should be noted that the training method or sorting method of the multi-agent model provided in the embodiments of the present disclosure is generally executed by the server 105. Accordingly, the training device or sorting device of the multi-agent model is generally set in the server 105.

[0029] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0030] Continue to refer Figure 2 , which shows a process 200 of an embodiment of a multi-agent model training method according to the present disclosure. The multi-agent model training method includes the following steps:

[0031] Step 201: Obtain a training sample set.

[0032] In this embodiment, the execution subject of the training method of the multi-agent model (e.g. Figure 1 The server 105 shown in FIG. 105 can obtain the training sample set. The execution subject can obtain the existing sample set stored in the public database, or can obtain the training sample set through a terminal device (such as Figure 1 The terminal devices 101, 102, and 103 shown in the figure are used to collect samples. In this way, the execution entity can receive the samples collected by the terminal devices and store these samples locally, thereby generating a training sample set.

[0033] The training sample set may include at least one sample. The sample may include a data sample to be recommended. The data sample to be recommended may be a text sample, an image sample, a video sample, or a combination of text and image samples, which is not limited in this disclosure. Specifically, the data sample to be recommended may be sample data including any content to be recommended. For example, the data sample to be recommended may be advertising data in the form of a combination of text and images.

[0034] Step 202: Select multiple training samples from the training sample set and input them into the multi-agent model for scoring, to obtain scoring results corresponding to each of the multiple agents.

[0035] In this embodiment, after obtaining a training sample set, the aforementioned execution entity may select multiple training samples from the training sample set and input them into a multi-agent model for scoring. Specifically, the multi-agent model includes multiple agents, each of which can independently perform scoring. Multiple training samples may be randomly selected from the training sample set as input data, may be selected from the training sample set based on fixed intervals, or may be extracted from the training sample set based on a sample extraction algorithm, all of which are not limited in this disclosure. The number of selected samples can be flexibly set, and this disclosure does not impose any restrictions on this.

[0036] After obtaining multiple training samples, the multiple training samples can be used as input data and input into each agent of the multi-agent model for scoring. From the output end of each agent, multiple scoring results corresponding to the multiple training samples are output, thereby obtaining the scoring results corresponding to each of the multiple agents.

[0037] Step 203: Concatenate the scoring results to obtain a combined score.

[0038] In this embodiment, after obtaining the scoring results, the above-mentioned execution entity can splice the scoring results to obtain a joint score. Specifically, the multiple scoring results corresponding to each intelligent agent can be spliced ​​first to obtain a joint score corresponding to each intelligent agent, wherein the multiple scoring results of an intelligent agent can be spliced ​​based on any order, or the multiple scoring results of an intelligent agent can be spliced ​​based on the order of generation of the scoring results, or the multiple scoring results of an intelligent agent can be spliced ​​in descending order of the scoring results, and the present disclosure does not limit this. After obtaining a joint score corresponding to each intelligent agent, the multiple joint scores corresponding to multiple intelligent agents can be spliced ​​to obtain a joint score corresponding to the multi-agent model, wherein the multiple joint scores corresponding to multiple intelligent agents can be spliced ​​based on any order, or the multiple joint scores corresponding to multiple intelligent agents can be spliced ​​in descending order of the joint score corresponding to each intelligent agent, and the present disclosure does not limit this.

[0039] In some optional implementations of this embodiment, all scoring results corresponding to multiple agents can be sorted from large to small, and then all scoring results can be spliced ​​according to the sorting to obtain a joint score corresponding to the multi-agent model.

[0040] Step 204: Input the joint score into the score evaluation environment to obtain the reward values ​​corresponding to the multiple agents.

[0041] In this embodiment, after obtaining the joint score, the execution entity can input the joint score into the score evaluation environment to obtain reward values ​​corresponding to each of the multiple agents. The score evaluation environment is independent of the multi-agent model and can interact with the multi-agent model. Specifically, the joint score obtained based on the multi-agent model can be input as input data into the score evaluation environment, and multiple reward values ​​are output from the output terminal of the score evaluation environment. Each reward value corresponds to an agent in the multi-agent model and represents feedback on the scoring result of the agent. The reward value can be a positive number, representing positive feedback, or a negative number, representing negative feedback.

[0042] Step 205: Train the multi-agent model based on the scoring results and reward values.

[0043] In this embodiment, after obtaining the scoring results and reward values ​​corresponding to each of the multiple agents, the aforementioned execution entity can train the multi-agent model based on the scoring results and reward values. Specifically, each agent can be trained individually based on its corresponding scoring results and reward values. For example, a reward threshold can be pre-set. For each agent, the reward value corresponding to the agent is compared with the reward threshold. If the reward value corresponding to the agent does not meet the reward threshold, the parameters of the agent are adjusted to bring the reward value of the scoring result output for the adjusted agent closer to the reward threshold. The adjusted agents constitute the multi-agent model after a single training session.

[0044] Step 206: In response to the number of training times being equal to a preset number threshold, output the trained multi-agent model.

[0045] In this embodiment, after completing a training run for the multi-agent model, the execution entity may compare the number of training runs with a preset number threshold. Specifically, after completing a training run for the multi-agent model, the number of training runs is accumulated once, and the updated number of training runs is compared with the preset number threshold. For example, the preset number threshold may be set to 50,000. If the updated number of training runs equals the preset number threshold, the multi-agent model obtained from the last update is determined to be the trained multi-agent model, and the trained multi-agent model is output.

[0046] Step 207: In response to the number of training times being less than the preset number threshold, continue training.

[0047] In this embodiment, after the above-mentioned execution entity completes one training of the multi-agent model, it accumulates the number of training times once and compares the updated number of training times with the preset number threshold. If the updated number of training times is less than the preset number threshold, it is determined that the multi-agent model has not been trained, and steps 202-205 can be repeated to continue training until the updated number of training times is equal to the preset number threshold.

[0048] The training method for a multi-agent model provided by an embodiment of the present disclosure first obtains a training sample set, and then performs the following training steps: selecting multiple training samples from the training sample set and inputting them into the multi-agent model for scoring to obtain scoring results corresponding to each of the multiple agents; concatenating the scoring results to obtain a joint score; inputting the joint score into a score evaluation environment to obtain reward values ​​corresponding to each of the multiple agents; training the multi-agent model based on the scoring results and reward values; and outputting the trained multi-agent model in response to the number of training times being equal to a preset number threshold. The multi-agent model obtained based on the above training method can obtain a scoring result that integrates multiple agents, so that the scoring result can integrate multiple factors, thereby making the ranking obtained based on the scoring result more accurate.

[0049] Further reference Figure 3 , which shows a process 300 of another embodiment of a multi-agent model training method according to the present disclosure. The multi-agent model training method includes the following steps:

[0050] Step 301: Obtain a training sample set.

[0051] In this embodiment, the specific operation of step 301 has been Figure 2 Step 201 is described in detail in the illustrated embodiment and will not be repeated here.

[0052] It should be noted that the training sample set may include at least one training sample, wherein one training sample may include a piece of data to be recommended and corresponding user data, and the data to be recommended includes identification data and text data.

[0053] Specifically, the data to be recommended can be a combination of any identification data and text data. For example, the data to be recommended is an advertising content including identification data and text data, and the user data can include data such as user industry, user access frequency, the highest bid under the advertising unit, and the lowest bid under the advertising unit. Each piece of data to be recommended corresponds to a group of user data, and a piece of data to be recommended and the corresponding group of user data constitute a training sample.

[0054] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0055] Step 302: extract features from the training samples in the training sample set to obtain identification features, text features, and user features.

[0056] In this embodiment, after obtaining the training sample set, the execution entity may perform feature extraction on the training samples in the training sample set to obtain identification features, text features, and user features. Specifically, each training sample in the training sample set may be used as input data and input into a feature extraction model. The feature extraction model then outputs the sample features corresponding to the training sample. A sample feature includes an identification feature, a text feature, and at least one user feature.

[0057] In some optional implementations of this embodiment, the identification data in the training sample set can be hashed to obtain identification features; the text data in the training sample set can be processed based on a pre-trained language model to obtain text features; and the user data in the training sample set can be processed based on predetermined operation rules to obtain user features.

[0058] Specifically, the identification data of each training sample in the training sample set can be hashed to obtain an identification feature set; the text data of each training sample in the training sample set is input into a pre-trained language model, and the text features corresponding to the text data are output from the output end of the language model to obtain a text feature set, wherein the pre-trained language model is a model that can extract text features of text data; user data may include user industry, user access frequency, the highest bid under an advertising unit, the lowest bid under an advertising unit and other data, therefore, different operation rules can be pre-set for each type of user data, and the different user data of each training sample in the training sample set can be calculated based on the corresponding operation rules, and the calculation results are determined as the user features of the user data to be calculated, and the user features of all user data constitute a user feature set. Exemplarily, the operation rules can be averaging, normalization, maximum value and minimum value.

[0059] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0060] Step 303: Concatenate the identification features, text features, and user features corresponding to each training sample into a feature matrix, and determine the obtained multiple feature matrices as a sample feature set.

[0061] In this embodiment, after obtaining the identification feature set, text feature set, and user feature set, the execution entity may concatenate the identification features, text features, and user features corresponding to each training sample into a feature matrix, and determine the resulting multiple feature matrices as the sample feature set. Specifically, for each training sample, a corresponding identification feature may be found from the identification feature set, a corresponding text feature may be found from the text feature set, and at least one corresponding user feature may be found from the user feature set. The identification feature, the text feature, and the at least one user feature may be concatenated to form a feature matrix. The multiple feature matrices formed by all training samples constitute the sample feature set.

[0062] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0063] In some optional implementations of this embodiment, after obtaining the sample feature set, the number of training times and the number of scoring times can be initialized. The number of training times and the number of scoring times can be initialized to 0 or to any positive integer, which is not limited in this disclosure.

[0064] In some optional implementations of this embodiment, the multi-agent model may include a preference agent and a benefit agent, and the preference agent and the benefit agent each include a scoring network.

[0065] Specifically, after obtaining the sample feature set, an initialized multi-agent model can be created based on the ddpg (Deep Deterministic Policy Gradient) method, wherein the initialized multi-agent model includes an initialized preference agent and an initialized benefit agent, and the initialized preference agent and the initialized benefit agent each include a scoring network.

[0066] Step 304: Select multiple sample features from the sample feature set as the initial state.

[0067] In this embodiment, after obtaining the sample feature set, the execution entity may select multiple sample features from the sample feature set as the initial state. Specifically, multiple sample features may be randomly selected from the sample feature set; the sample features in the sample feature set may be sorted by user access frequency, and multiple sample features with the highest rankings may be selected; the sample features in the sample feature set may be classified based on user attributes, and sample features belonging to the same user may be grouped into one category, and at least one sample feature may be randomly selected from each category; or the sample features in each category may be sorted by user access frequency, and at least one sample feature with the highest rankings may be selected from each category. This disclosure does not limit this.

[0068] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0069] The selected multiple sample features are used as initial states for further input into the scoring network of the preference agent and the benefit agent for scoring.

[0070] Step 305: Input the initial state into the scoring network of the preference agent for scoring to obtain multiple preference scores.

[0071] In this embodiment, after obtaining the initial state, the execution entity can input the initial state into the preference agent's scoring network for scoring, thereby obtaining multiple preference scores. Specifically, the initial state can be used as input data and fed into the preference agent's scoring network. The preference agent's scoring network reads each sample feature in the initial state, performs a score based on each sample feature, and outputs multiple preference scores corresponding to the multiple sample features in the initial state from the output terminal of the preference agent's scoring network. The preference scores represent the user's level of interest in the recommended data; higher preference scores increase the probability that the user will select the recommended data.

[0072] In some optional implementations of this embodiment, the scoring network of the preference agent may add scoring action noise when scoring. By introducing noise to add randomness, the model is prevented from converging to a local optimal strategy.

[0073] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0074] Step 306: Input the initial state into the scoring network of the benefit agent for scoring to obtain multiple benefit scores.

[0075] In this embodiment, after obtaining the initial state, the execution entity can input the initial state into the scoring network of the benefit agent for scoring, thereby obtaining multiple benefit scores. Specifically, the initial state can be used as input data and input into the scoring network of the benefit agent. The scoring network of the benefit agent reads each sample feature in the initial state, performs a score based on each sample feature, and outputs multiple benefit scores corresponding to the multiple sample features in the initial state from the output terminal of the scoring network of the benefit agent. The benefit score represents the level of benefit that the recommended data can bring, with higher benefit scores indicating higher benefits for the recommended data.

[0076] In some optional implementations of this embodiment, the scoring network of the income agent can add scoring action noise when scoring. By introducing noise to add randomness, the model can be prevented from converging to a local optimal strategy.

[0077] Step 307: Concatenate the scoring results to obtain a combined score.

[0078] In this embodiment, the specific operation of step 307 has been Figure 2 Step 203 is described in detail in the illustrated embodiment and will not be repeated here.

[0079] Step 308: Input the joint score into the score evaluation environment, and obtain the preference reward value corresponding to the preference agent and the profit reward value corresponding to the profit agent based on the offline simulation process.

[0080] In this embodiment, after obtaining the joint score, the execution entity can input the joint score into the score evaluation environment, which, based on offline simulation processing, can obtain the preference reward value corresponding to the preference agent and the profit reward value corresponding to the profit agent. Specifically, the joint score can be input into the score evaluation environment, which calculates the joint score based on the preference parameters and the profit parameters to obtain the preference reward value corresponding to the preference agent and the profit reward value corresponding to the profit agent.

[0081] Step 309: The initial state, the plurality of preference scores, the plurality of benefit scores, the preference reward value, and the benefit reward value are stored as a piece of storage data in the training database.

[0082] In this embodiment, after obtaining the preference reward value and the profit reward value, the aforementioned execution entity can store the data. Specifically, the initial state, multiple preference scores, multiple profit scores, preference reward value, and profit reward value generated during a scoring process can be stored as a single piece of stored data in a training database. The training database is used to store the data generated during the scoring process, and the data in the training database can be used as training data to train the multi-agent model.

[0083] Step 310: Determine whether the number of scores of the multi-agent model meets a preset scoring threshold.

[0084] In this embodiment, after storing the data generated during a scoring process in a training database, the execution entity may determine whether the number of scores of the multi-agent model meets a preset scoring threshold. Specifically, after storing the data generated during a scoring process in the training database, the number of scores may be accumulated once, and the updated accumulated number of scores may be compared with the preset scoring threshold to determine whether the number of scores of the multi-agent model meets the preset scoring threshold. The preset scoring threshold is a pre-set value, and illustratively, the preset scoring threshold is 5 times.

[0085] Step 311: In response to the number of scoring times being less than the preset scoring times threshold, multiple sample features are reselected from the sample feature set as the initial state, and are respectively input into the scoring network of the preference agent and the scoring network of the benefit agent for scoring, that is, returning to step 304 and scoring again.

[0086] In this embodiment, when the execution subject determines that the number of scoring times is less than the preset scoring number threshold, it can reselect multiple sample features from the sample feature set as the initial state and execute the scoring process again. Specifically, multiple sample features can be randomly selected from the sample features that have not been selected in the sample feature set as the initial state, or the sample feature with the largest sum of the preference reward value and the benefit reward value among the selected sample features in the sample feature set can be first obtained as the target sample feature, and then the similarity between the sample features that have not been selected in the sample feature set and the target sample feature is calculated. In order to select multiple sample features from the sample features that have not been selected in the sample feature set in descending order of similarity, the initial state is then input into the scoring network of the preference agent and the scoring network of the benefit agent for scoring, which is equivalent to executing steps 304-309 again.

[0087] Step 312: In response to the number of scoring times being equal to the preset scoring times threshold, a plurality of stored data are selected from the training database.

[0088] In this embodiment, when the execution entity determines that the number of scoring passes equals a preset scoring threshold, multiple pieces of stored data may be selected from the training database. Specifically, if the updated cumulative number of scoring passes equals the preset scoring threshold, the multi-agent model is trained once based on the data generated by multiple scoring processes in the current training step, and steps 312-316 may be executed. In step 312, multiple pieces of stored data may be randomly selected from the training database.

[0089] In some optional implementations of this embodiment, the preference agent and the benefit agent each include an evaluation network.

[0090] The selected multiple stored data are used to further train the scoring network and evaluation network of the preference agent and the benefit agent.

[0091] Step 313: Based on the multiple preference reward values ​​and the multiple preference scores in the multiple stored data, the evaluation network of the preference agent is trained.

[0092] In this embodiment, after selecting multiple pieces of stored data, the execution subject can train the evaluation network of the preference agent based on the multiple preference reward values ​​and multiple preference scores in the multiple pieces of stored data. The evaluation network of the preference agent can output a preference evaluation value based on at least one input preference score. The preference evaluation value represents a feedback on the input preference score. The preference evaluation value can be a positive number, representing positive feedback, or a negative number, representing negative feedback. Specifically, for each piece of stored data selected, multiple preference scores and a preference reward value in the stored data can be obtained, and the obtained multiple preference scores are input into the evaluation network of the preference agent to obtain a preference evaluation value. The loss value is calculated based on the obtained preference evaluation value and the preference reward value. In response to the loss value being greater than the preset loss threshold, the parameters of the evaluation network of the preference agent are adjusted. The above training process is repeated based on each piece of stored data selected.

[0093] Step 314: Based on the multiple profit reward values ​​and multiple profit scores in the multiple stored data, the evaluation network of the profit agent is trained.

[0094] In this embodiment, after selecting multiple pieces of stored data, the execution entity can train the evaluation network of the profit agent based on the multiple profit reward values ​​and multiple profit scores in the multiple pieces of stored data. The evaluation network of the profit agent can output a profit evaluation value based on at least one input profit score. The profit evaluation value represents a feedback on the input profit score. The profit evaluation value can be a positive number, representing positive feedback, or a negative number, representing negative feedback. Specifically, for each piece of stored data selected, multiple profit scores and a profit reward value in the piece of stored data can be obtained, and the obtained multiple profit scores are input into the evaluation network of the profit agent to obtain a profit evaluation value. The loss value is calculated based on the obtained profit evaluation value and the profit reward value. In response to the loss value being greater than the preset loss threshold, the parameters of the evaluation network of the profit agent are adjusted. The above training process is repeated based on each piece of stored data selected.

[0095] Step 315: Based on the multiple initial states in the multiple stored data and the trained evaluation network of the preference agent, the scoring network of the preference agent is trained.

[0096] In this embodiment, after obtaining the evaluation network of the trained preference agent, the above-mentioned execution subject can train the scoring network of the preference agent based on multiple initial states in multiple stored data and the evaluation network of the trained preference agent. Among them, the scoring network of the preference agent can output a preference score based on an input sample feature. Specifically, for each selected stored data, an initial state in the stored data can be obtained, and the obtained initial state can be input into the scoring network of the preference agent to obtain multiple preference scores corresponding to the multiple sample features in the initial state. The obtained multiple preference scores are input into the evaluation network of the trained preference agent to obtain a preference evaluation value, and the parameters of the scoring network of the preference agent are adjusted so that the preference score output by the adjusted scoring network of the preference agent can obtain a higher preference evaluation value.

[0097] Step 316: Based on the multiple initial states in the multiple stored data and the evaluation network of the trained profit agent, the scoring network of the profit agent is trained.

[0098] In this embodiment, after obtaining the evaluation network of the trained profit agent, the execution subject can train the scoring network of the profit agent based on multiple initial states in multiple stored data and the evaluation network of the trained profit agent. The scoring network of the profit agent can output a profit score based on a sample feature of the input. Specifically, for each selected stored data, an initial state in the stored data can be obtained, and the obtained initial state can be input into the scoring network of the profit agent to obtain multiple profit scores corresponding to the multiple sample features in the initial state. The obtained multiple profit scores are input into the evaluation network of the trained profit agent to obtain a profit evaluation value. The parameters of the scoring network of the profit agent are adjusted so that the profit score output by the adjusted scoring network of the profit agent can obtain a higher profit evaluation value.

[0099] Step 317: In response to the number of training times being equal to a preset threshold, the trained multi-agent model is output.

[0100] In this embodiment, the specific operation of step 317 has been Figure 2 Step 206 is described in detail in the illustrated embodiment and will not be repeated here.

[0101] Step 318: In response to the number of training times being less than the preset threshold, the data in the training database is cleared, the scoring times are reset, and the training step is performed again.

[0102] In this embodiment, after determining that the number of training times is less than a preset threshold, the execution subject may re-execute the training steps. Specifically, the data in the training database may be cleared, the scoring times may be reset, and the training steps 304-316 may be re-executed based on the last updated multi-agent model. In this way, the scoring times may be recalculated in a new training step, and training may be resumed based on the scoring data generated by the multi-agent model in that training step.

[0103] It should be noted that, in the three pairs of steps 305 and 306, steps 313 and 314, and steps 315 and 316, the order of execution is not distinguished. In a pair of steps, any one step can be executed first, or both steps can be executed at the same time.

[0104] from Figure 3 It can be seen that Figure 2 Compared with the corresponding embodiments, the training method of the multi-agent model in this embodiment, the multi-agent model includes a preference agent and a benefit agent, a joint score is obtained based on the preference agent and the benefit agent, the joint score is input into the score evaluation environment, the preference reward value and the benefit reward value are obtained based on offline simulation processing, based on the initial state, preference reward value, and preference score, the evaluation network of the preference agent is first trained, and then the scoring network of the preference agent is trained to obtain a trained preference agent, based on the initial state, benefit reward value, and benefit score, the evaluation network of the benefit agent is first trained, and then the scoring network of the benefit agent is trained to obtain a trained benefit agent, which can be optimized simultaneously when the preference agent and the benefit agent are unrelated, and the trained multi-agent model can comprehensively consider user preferences and the benefits of the data to be recommended, and make a comprehensive score for the recommended data, so that the sorting result of the data to be recommended based on the comprehensive score is more accurate and reasonable.

[0105] Further reference Figure 4 , which shows a schematic diagram 400 of the training method of the multi-agent model according to the present disclosure, from Figure 4As can be seen from the figure, the multi-agent model can include a preference agent and a benefit agent. The preference agent and the benefit agent respectively include a scoring network and an evaluation network. When training the multi-agent model, multiple sample features can be selected from the sample feature set as the initial state. The initial state is input into the scoring network of the preference agent and the scoring network of the benefit agent respectively to obtain multiple preference scores and multiple benefit scores. The multiple preference scores and multiple benefit scores are spliced ​​to obtain a joint score. The joint score is input into the score evaluation environment to obtain a preference reward value and a benefit reward value. Based on the preference reward value and the preference score, the evaluation network of the preference agent is first trained. Then, based on the initial state and the trained evaluation network of the preference agent, the scoring network of the preference agent is trained to obtain the trained preference agent. Based on the benefit reward value and the benefit score, the evaluation network of the benefit agent is first trained. Then, based on the initial state and the trained evaluation network of the benefit agent, the scoring network of the benefit agent is trained to obtain the trained benefit agent. The trained preference agent and the trained benefit agent form the trained multi-agent model. Make the ranking of recommended data obtained based on the trained multi-agent model more accurate and reasonable.

[0106] Further reference Figure 5 , which shows a process 500 of an embodiment of a sorting method according to the present disclosure. The sorting method includes the following steps:

[0107] Step 501: Acquire multiple data to be processed.

[0108] In this embodiment, the execution subject can obtain multiple data to be processed. The execution subject can obtain multiple existing data to be processed stored in the public database, or can obtain the data through a terminal device (such as Figure 1 The terminal devices 101, 102, 103 shown in the figure are used to collect a plurality of data to be processed.

[0109] A piece of data to be processed may include a piece of data to be recommended. The data to be recommended may be text, an image, a video, or a combination of text and images, which is not limited in this disclosure. Specifically, the data to be recommended may include any content to be recommended. For example, the data to be recommended may be advertising data in the form of a combination of text and images.

[0110] In some optional implementations of this embodiment, one piece of data to be processed may include a piece of data to be recommended and corresponding user data, and the data to be recommended includes identification data and text data. Specifically, the data to be recommended may be any combination of identification data and text data. For example, the data to be recommended is an advertisement content including identification data and text data, and the user data may include data such as the user's industry, user access frequency, the highest bid under the advertising unit, and the lowest bid under the advertising unit. Each piece of data to be recommended corresponds to a group of user data, and a piece of data to be recommended and the corresponding group of user data constitute one piece of data to be processed.

[0111] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0112] Step 502: Input the plurality of data to be processed into the multi-agent model for scoring, and obtain the scoring results corresponding to the plurality of agents.

[0113] In this embodiment, after obtaining the plurality of data to be processed, the execution entity may input the plurality of data to be processed into a multi-agent model for scoring, thereby obtaining scoring results corresponding to each of the plurality of agents. Specifically, the multi-agent model includes a plurality of agents, each of which can independently perform scoring. The plurality of data to be processed may be input as input data to each agent in the multi-agent model for scoring, and the output terminal of each agent outputs a plurality of scoring results corresponding to the plurality of data to be processed, thereby obtaining scoring results corresponding to each of the plurality of agents.

[0114] In some optional implementations of this embodiment, the multi-agent model may include a preference agent and a benefit agent, and the preference agent and the benefit agent each include a scoring network.

[0115] In some optional implementations of this embodiment, multiple data to be processed can be input into a multi-agent model for scoring to obtain scoring results corresponding to multiple agents in the following manner: feature extraction is performed on multiple data to be processed to obtain identification features, text features and user features; the identification features, text features and user features corresponding to each data to be processed are spliced ​​into a feature matrix to obtain multiple feature matrices; the multiple feature matrices are input into a scoring network of a preference agent for scoring to obtain multiple preference scores; the multiple feature matrices are input into a scoring network of a benefit agent for scoring to obtain multiple benefit scores.

[0116] Step 503: Sort the plurality of data to be processed based on the scoring results.

[0117] In this embodiment, after obtaining the scoring results, the execution entity can sort the multiple data to be processed based on the scoring results. Specifically, each data to be processed corresponds to multiple scoring results output by multiple agents. The multiple scoring results corresponding to each data to be processed can be added together to obtain a comprehensive score, and the multiple data to be processed can be sorted based on the multiple comprehensive scores.

[0118] In some optional implementations of this embodiment, multiple pieces of to-be-recommended data among the multiple pieces of to-be-processed data may be sorted based on multiple preference scores and multiple benefit scores.

[0119] Specifically, each data to be processed corresponds to a preference score and a benefit score. The preference scores and benefit scores corresponding to multiple data to be processed can be weighted and summed respectively to obtain multiple comprehensive scores corresponding to the multiple data to be processed, and the multiple data to be recommended in the multiple data to be processed can be sorted based on the multiple comprehensive scores.

[0120] from Figure 5 It can be seen that the sorting method in this embodiment can sort multiple pieces of data to be recommended based on preference scores and revenue scores, taking into account both user preferences and the revenue of the data to be recommended, making the sorting results more reasonable, improving users' willingness to use the recommended data, and increasing the revenue of the data to be recommended.

[0121] Further references Figure 6 As an implementation of the above-mentioned multi-agent model training method, the present disclosure provides an embodiment of a multi-agent model training device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0122] like Figure 6 As shown, the multi-agent model training apparatus 600 of this embodiment may include a first acquisition module 601 and a training module 602. The first acquisition module 601 is configured to acquire a training sample set; the training module 602 is configured to perform the following training steps: selecting multiple training samples from the training sample set and inputting them into the multi-agent model for scoring to obtain scoring results corresponding to each of the multiple agents; concatenating the scoring results to obtain a joint score; inputting the joint score into a score evaluation environment to obtain reward values ​​corresponding to each of the multiple agents; training the multi-agent model based on the scoring results and reward values; and outputting the trained multi-agent model in response to the number of training times being equal to a preset number threshold.

[0123] In this embodiment, the training device 600 of the multi-agent model: the specific processing of the first acquisition module 601 and the training module 602 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201-207 in the corresponding embodiment are not repeated here.

[0124] In some optional implementations of this embodiment, a training sample in the training sample set includes a piece of data to be recommended and corresponding user data, and the data to be recommended includes identification data and text data; before training the multi-agent model, the training device 600 of the multi-agent model also includes: a feature extraction module, which is configured to extract features from the training samples in the training sample set to obtain identification features, text features and user features; a feature splicing module, which is configured to splice the identification features, text features and user features corresponding to each training sample into a feature matrix, and determine the obtained multiple feature matrices as a sample feature set.

[0125] In some optional implementations of this embodiment, the feature extraction module includes: a first extraction submodule, configured to perform hash processing on the identification data in the training sample set to obtain identification features; a second extraction submodule, configured to process the text data in the training sample set based on a pre-trained language model to obtain text features; and a third extraction submodule, configured to process the user data in the training sample set based on predetermined operation rules to obtain user features.

[0126] In some optional implementations of this embodiment, the multi-agent model includes a preference agent and a benefit agent, and the preference agent and the benefit agent each include a scoring network; the training module 602 includes: a first selection submodule, configured to select multiple sample features from the sample feature set as the initial state; a first scoring submodule, configured to input the initial state into the scoring network of the preference agent for scoring, and obtain multiple preference scores; a second scoring submodule, configured to input the initial state into the scoring network of the benefit agent for scoring, and obtain multiple benefit scores.

[0127] In some optional implementations of this embodiment, the training module 602 also includes: a calculation submodule, which is configured to input the joint score into the score evaluation environment, and obtain the preference reward value corresponding to the preference agent and the profit reward value corresponding to the profit agent based on offline simulation processing.

[0128] In some optional implementations of this embodiment, the training module 602 also includes: a storage submodule, configured to store the initial state, multiple preference scores, multiple profit scores, preference reward values ​​and profit reward values ​​as a storage data in the training database; a first judgment submodule, configured to judge whether the number of scores of the multi-agent model meets the preset scoring number threshold; a second judgment submodule, configured to execute training of the multi-agent model based on the scoring results and reward values ​​in response to the number of scores being equal to the preset scoring number threshold; a third judgment submodule, configured to re-select multiple sample features from the sample feature set as the initial state in response to the number of scores being less than the preset scoring number threshold, and input them into the scoring network of the preference agent and the scoring network of the profit agent for scoring respectively.

[0129] In some optional implementations of this embodiment, the preference agent and the profit agent each include an evaluation network; the training module 602 also includes: a second selection submodule, configured to select multiple stored data from the training database; a first training submodule, configured to train the evaluation network of the preference agent based on multiple preference reward values ​​and multiple preference scores in the multiple stored data; a second training submodule, configured to train the evaluation network of the profit agent based on multiple profit reward values ​​and multiple profit scores in the multiple stored data; a third training submodule, configured to train the scoring network of the preference agent based on multiple initial states in the multiple stored data and the trained evaluation network of the preference agent; a fourth training submodule, configured to train the scoring network of the profit agent based on multiple initial states in the multiple stored data and the trained evaluation network of the profit agent.

[0130] In some optional implementations of this embodiment, the multi-agent model training device 600 also includes: a repeated training module, which is configured to clear the data in the training database, reset the scoring times, and execute the training steps again in response to the number of training times being less than a preset threshold.

[0131] Further references Figure 7 As an implementation of the above-mentioned sorting method, the present disclosure provides an embodiment of a sorting device. Figure 5 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0132] like Figure 7As shown, the sorting device 700 of this embodiment may include a second acquisition module 701, a scoring module 702, and a sorting module 703. The second acquisition module 701 is configured to acquire a plurality of data to be processed; the scoring module 702 is configured to input the plurality of data to be processed into a multi-agent model for scoring, and obtain scoring results corresponding to each of the plurality of agents; and the sorting module 703 is configured to sort the plurality of data to be processed based on the scoring results.

[0133] In this embodiment, the specific processing of the sorting device 700: the second acquisition module 701, the scoring module 702, and the sorting module 703 and the technical effects thereof can be referred to respectively. Figure 5 The relevant descriptions of steps 501-503 in the corresponding embodiment are not repeated here.

[0134] In some optional implementations of this embodiment, one piece of data to be processed includes one piece of data to be recommended and corresponding user data, the data to be recommended includes identification data and text data, the multi-agent model includes a preference agent and a benefit agent, and the preference agent and the benefit agent each include a scoring network; the scoring module 702 includes: a fourth extraction submodule, configured to perform feature extraction on multiple data to be processed to obtain identification features, text features and user features; a splicing submodule, configured to splice the identification features, text features and user features corresponding to each piece of data to be processed into a feature matrix to obtain multiple feature matrices; a third scoring submodule, configured to input multiple feature matrices into the scoring network of the preference agent for scoring to obtain multiple preference scores; a fourth scoring submodule, configured to input multiple feature matrices into the scoring network of the benefit agent for scoring to obtain multiple benefit scores.

[0135] In some optional implementations of this embodiment, the sorting module 703 includes: a sorting submodule configured to sort the multiple pieces of to-be-recommended data among the multiple pieces of to-be-processed data based on the multiple preference scores and the multiple benefit scores.

[0136] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0137] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0138] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0139] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0140] The computing unit 801 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the training method or ranking generation method of the multi-agent model. For example, in some embodiments, the training method or ranking generation method of the multi-agent model can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the training method or ranking generation method of the multi-agent model described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the training method or ranking generation method of the multi-agent model in any other appropriate manner (e.g., by means of firmware).

[0141] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0143] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0145] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0146] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a server in a distributed system or a server integrated with blockchain. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server may be a server in a distributed system or a server integrated with blockchain. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0147] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0148] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for training a multi-agent model, comprising: Obtain a training sample set; Perform the following training steps: Selecting multiple training samples from the training sample set and inputting them into the scoring network of each agent in the multi-agent model for scoring, thereby obtaining scoring results corresponding to each of the multiple agents; The scoring results are combined to obtain a combined score; Inputting the joint score into a score evaluation environment to obtain reward values ​​corresponding to each of the multiple agents; The multi-agent model is trained based on the scoring results and the reward values, and each agent is trained individually based on the scoring results and reward values ​​corresponding to each agent, including: training the evaluation network of the corresponding agent based on the scoring results and the reward values ​​of each agent; and training the scoring network of the corresponding agent based on the multiple training samples and the trained evaluation network. In response to the number of training times being equal to a preset number threshold, outputting the trained multi-agent model; Wherein, one training sample in the training sample set includes a piece of data to be recommended and corresponding user data, and the data to be recommended includes identification data and text data; Before training the multi-agent model, the method further comprises: Extracting features from the training samples in the training sample set to obtain identification features, text features, and user features; The identification features, text features and user features corresponding to each training sample are spliced ​​into a feature matrix, and the obtained multiple feature matrices are determined as the sample feature set.

2. The method according to claim 1, wherein The feature extraction of the training samples in the training sample set to obtain identification features, text features and user features includes: Performing hash processing on the identification data in the training sample set to obtain the identification feature; Processing the text data in the training sample set based on a pre-trained language model to obtain the text features; The user data in the training sample set is processed based on a predetermined operation rule to obtain the user features.

3. The method according to claim 1 or 2, wherein: The multi-agent model includes a preference agent and a benefit agent; The step of selecting multiple training samples from the training sample set and inputting them into the multi-agent model for scoring, and obtaining the scoring results corresponding to the multiple agents includes: Selecting multiple sample features from the sample feature set as an initial state; Inputting the initial state into the scoring network of the preference agent for scoring to obtain a plurality of preference scores; The initial state is input into the scoring network of the profit agent for scoring to obtain multiple profit scores.

4. The method according to claim 3, wherein: Inputting the joint score into the score evaluation environment to obtain the reward values ​​corresponding to each of the multiple agents includes: The joint score is input into the score evaluation environment, and the preference reward value corresponding to the preference agent and the profit reward value corresponding to the profit agent are obtained based on offline simulation processing.

5. The method according to claim 4, wherein Inputting the joint score into a score evaluation environment to obtain reward values ​​corresponding to each of the multiple agents further comprises: storing the initial state, the plurality of preference scores, the plurality of benefit scores, the preference reward value, and the benefit reward value as a piece of storage data in a training database; Determine whether the number of scores of the multi-agent model meets a preset scoring number threshold; In response to the number of scoring being equal to the preset scoring threshold, executing the training of the multi-agent model based on the scoring result and the reward value; In response to the number of scoring times being less than the preset scoring times threshold, a plurality of sample features are reselected from the sample feature set as initial states and are respectively input into the scoring network of the preference agent and the scoring network of the benefit agent for scoring.

6. The method according to claim 5, wherein: The training of the multi-agent model based on the scoring result and the reward value includes: Selecting a plurality of stored data from the training database; Training an evaluation network of the preference agent based on a plurality of preference reward values ​​and a plurality of preference scores in the plurality of stored data; Training an evaluation network of the profit agent based on a plurality of profit reward values ​​and a plurality of profit scores in the plurality of stored data; Training a scoring network of the preference agent based on a plurality of initial states in the plurality of stored data and a trained evaluation network of the preference agent; Based on the multiple initial states in the multiple stored data and the evaluation network of the trained profit agent, the scoring network of the profit agent is trained.

7. The method according to claim 6, further comprising: In response to the number of training times being less than the preset number threshold, the data in the training database is cleared, the scoring times are reset, and the training step is performed again.

8. A sorting method comprising: Get multiple data to be processed; Inputting the plurality of data to be processed into a multi-agent model for scoring to obtain scoring results corresponding to the plurality of agents, wherein the multi-agent model is trained based on any one of claims 1 to 7; Based on the scoring result, the plurality of data to be processed are sorted.

9. The method according to claim 8, wherein A piece of data to be processed includes a piece of data to be recommended and corresponding user data, the data to be recommended includes identification data and text data, the multi-agent model includes a preference agent and a benefit agent, and the preference agent and benefit agent each include a scoring network; Inputting the plurality of data to be processed into the multi-agent model for scoring, and obtaining the scoring results corresponding to the plurality of agents includes: Extracting features from the plurality of data to be processed to obtain identification features, text features, and user features; The identification features, text features, and user features corresponding to each data to be processed are concatenated into a feature matrix to obtain multiple feature matrices; Inputting the plurality of feature matrices into the scoring network of the preference agent for scoring to obtain a plurality of preference scores; The multiple feature matrices are input into the scoring network of the profit agent for scoring to obtain multiple profit scores.

10. The method according to claim 9, wherein: Sorting the plurality of to-be-processed data based on the scoring results includes: Based on the plurality of preference scores and the plurality of revenue scores, a plurality of pieces of to-be-recommended data in the plurality of to-be-processed data are sorted.

11. A multi-agent model training device, comprising: A first acquisition module is configured to acquire a training sample set; The training module is configured to perform the following training steps: selecting a plurality of training samples from the training sample set and inputting them into the scoring network of each agent in the multi-agent model for scoring, thereby obtaining scoring results corresponding to each of the plurality of agents; The scoring results are combined to obtain a combined score; the combined score is input into a score evaluation environment to obtain reward values ​​corresponding to each of the multiple agents; The multi-agent model is trained based on the scoring result and the reward value, and each agent is trained individually based on the scoring result and the reward value corresponding to each agent, including: training the evaluation network of the corresponding agent based on the scoring result and the reward value of each agent; training the scoring network of the corresponding agent based on the multiple training samples and the trained evaluation network; and outputting the trained multi-agent model in response to the number of training times being equal to a preset number threshold; Wherein, one training sample in the training sample set includes a piece of data to be recommended and corresponding user data, and the data to be recommended includes identification data and text data; Before training the multi-agent model, the apparatus further comprises: A feature extraction module is configured to extract features from the training samples in the training sample set to obtain identification features, text features and user features; The feature concatenation module is configured to concatenate the identification features, text features, and user features corresponding to each training sample into a feature matrix, and determine the obtained multiple feature matrices as a sample feature set.

12. The device according to claim 11, wherein The feature extraction module includes: A first extraction submodule is configured to perform hash processing on the identification data in the training sample set to obtain the identification feature; A second extraction submodule is configured to process the text data in the training sample set based on a pre-trained language model to obtain the text features; The third extraction submodule is configured to process the user data in the training sample set based on a predetermined operation rule to obtain the user features.

13. The device according to claim 11 or 12, wherein: The multi-agent model includes a preference agent and a benefit agent; The training module includes: A first selection submodule is configured to select a plurality of sample features from the sample feature set as an initial state; A first scoring submodule is configured to input the initial state into the scoring network of the preference agent for scoring to obtain a plurality of preference scores; The second scoring submodule is configured to input the initial state into the scoring network of the profit agent for scoring to obtain multiple profit scores.

14. The device according to claim 13, wherein The training module also includes: The calculation submodule is configured to input the joint score into the score evaluation environment, and obtain the preference reward value corresponding to the preference agent and the profit reward value corresponding to the profit agent based on offline simulation processing.

15. The device according to claim 14, wherein The training module also includes: A storage submodule is configured to store the initial state, the plurality of preference scores, the plurality of profit scores, the preference reward value, and the profit reward value as a piece of storage data into a training database; A first judgment submodule is configured to judge whether the number of scores of the multi-agent model meets a preset scoring number threshold; A second judgment submodule is configured to, in response to the number of scoring being equal to the preset scoring threshold, execute the training of the multi-agent model based on the scoring result and the reward value; The third judgment submodule is configured to reselect multiple sample features from the sample feature set as the initial state in response to the number of scoring times being less than the preset scoring number threshold, and input them into the scoring network of the preference agent and the scoring network of the benefit agent for scoring respectively.

16. The device according to claim 15, wherein The training module also includes: A second selection submodule is configured to select a plurality of stored data from the training database; A first training submodule is configured to train an evaluation network of the preference agent based on a plurality of preference reward values ​​and a plurality of preference scores in the plurality of stored data; a second training submodule configured to train an evaluation network of the profit agent based on a plurality of profit reward values ​​and a plurality of profit scores in the plurality of stored data; A third training submodule is configured to train a scoring network of the preference agent based on the multiple initial states in the multiple stored data and the trained evaluation network of the preference agent; The fourth training submodule is configured to train the scoring network of the profit agent based on the multiple initial states in the multiple stored data and the evaluation network of the trained profit agent.

17. The device according to claim 16, wherein The device further comprises: The repeated training module is configured to clear the data in the training database, reset the scoring times, and execute the training step again in response to the number of training times being less than the preset number threshold.

18. A sorting device, comprising: A second acquisition module is configured to acquire a plurality of data to be processed; a scoring module configured to input the plurality of to-be-processed data into a multi-agent model for scoring, and obtain scoring results corresponding to the plurality of agents, wherein the multi-agent model is trained based on any one of claims 11-17; The sorting module is configured to sort the plurality of data to be processed based on the scoring result.

19. The device according to claim 18, wherein A piece of data to be processed includes a piece of data to be recommended and corresponding user data, the data to be recommended includes identification data and text data, the multi-agent model includes a preference agent and a benefit agent, and the preference agent and benefit agent each include a scoring network; The scoring module includes: a fourth extraction submodule, configured to perform feature extraction on the plurality of data to be processed to obtain identification features, text features, and user features; The splicing submodule is configured to splice the identification features, text features and user features corresponding to each piece of data to be processed into a feature matrix to obtain multiple feature matrices; A third scoring submodule is configured to input the multiple feature matrices into the scoring network of the preference agent for scoring to obtain multiple preference scores; The fourth scoring submodule is configured to input the multiple feature matrices into the scoring network of the profit agent for scoring to obtain multiple profit scores.

20. The device according to claim 19, wherein The sorting module includes: The sorting submodule is configured to sort the plurality of to-be-recommended data in the plurality of to-be-processed data based on the plurality of preference scores and the plurality of revenue scores.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images