Model training method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a model training method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: inputting a plurality of state action pairs of a training sample sequence in a training data set into a decision model, and carrying out the decision processing; obtaining a target loss value corresponding to the decision model and a loss value corresponding to each training sample sequence; if the target loss value does not meet the preset convergence condition, updating model parameters of the decision model according to the target loss value; determining quality evaluation corresponding to the training sample sequence according to the loss value corresponding to the training sample sequence, and screening out the training sample sequence corresponding to the quality evaluation meeting a preset quality condition from the training data set to obtain a target training sample sequence; continuing to train the updated decision model according to the target training sample sequence; and if the target loss value meets a preset convergence condition, taking the decision model as a trained target decision model. According to the embodiment of the invention, the simulation learning effect of the decision model can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a model training method, device, electronic device and computer-readable storage medium. Background Art

[0002] Imitation learning refers to a method of letting the model learn behavioral strategies from expert data. Expert data is usually a state-action pair (s, a), which represents the expert's decision a for state s. The effect of imitation learning depends largely on the quality of expert data. If the quality of expert data is low, the effect of imitation learning is poor. In order to ensure the effect of imitation learning, expert data is usually cleaned.

[0003] At present, the methods for cleaning expert data mainly include manual cleaning and trained evaluation model cleaning. Manual cleaning is inefficient, and the cleaning of trained evaluation models depends on the quality of the trained evaluation models. When imitation learning has high requirements for the quality of expert data, the expert data cleaned by the trained evaluation model cannot meet the requirements, which ultimately leads to poor imitation learning results. Summary of the invention

[0004] The embodiments of the present application provide a model training method, device, electronic device and computer-readable storage medium, which can make the updated training data compatible with the decision model and improve the decision effect of the target decision model obtained by training the decision model according to the updated training data.

[0005] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:

[0006] Obtaining a training data set corresponding to the decision model, the training data set including a plurality of training sample sequences, the training sample sequences including a plurality of state-action pairs, the state-action pairs including the state of the expert and the decision action made by the expert in the state;

[0007] Inputting multiple state-action pairs of the training sample sequence in the training data set into the decision model for decision processing, and obtaining a target loss value corresponding to the decision model and a loss value corresponding to each training sample sequence;

[0008] If the target loss value does not meet the preset convergence condition, the model parameters of the decision model are updated according to the target loss value to obtain an updated decision model, and the updated decision model is used as the decision model;

[0009] Determine the quality evaluation corresponding to the training sample sequence according to the loss value corresponding to the training sample sequence, and select the training sample sequence corresponding to the quality evaluation that meets the preset quality condition from the training data set to obtain the target training sample sequence;

[0010] According to the target training sample sequence, an updated training data set is generated, and the updated data set is used as the training data set, and the step of inputting a plurality of state-action pairs in the training sample sequence in the training data set into the decision model for decision processing is returned to obtain the target loss value corresponding to the decision model and the loss value of each training sample sequence;

[0011] If the above target loss value meets the preset convergence condition, the above decision model is regarded as the trained decision model.

[0012] In a second aspect, an embodiment of the present application further provides a model training device, the device comprising:

[0013] An acquisition module is used to acquire a training data set corresponding to the decision model, wherein the training data set includes a plurality of training sample sequences, and the training sample sequences include a plurality of state-action pairs, wherein the state-action pairs include the state of the expert and the decision action made by the expert in the state;

[0014] An input module, used to input multiple state-action pairs of the training sample sequence in the training data set into the decision model for decision processing, and obtain a target loss value corresponding to the decision model and a loss value corresponding to each training sample sequence;

[0015] An updating module, configured to update the model parameters of the decision model according to the target loss value if the target loss value does not meet the preset convergence condition, obtain an updated decision model, and use the updated decision model as the decision model;

[0016] A screening module is used to determine the quality evaluation corresponding to the training sample sequence according to the loss value corresponding to the training sample sequence, and screen out the training sample sequence corresponding to the quality evaluation that meets the preset quality condition from the training data set to obtain the target training sample sequence;

[0017] A return module is used to generate an updated training data set according to the target training sample sequence, and use the updated data set as the training data set, and return to execute the step of inputting multiple state-action pairs in the training sample sequence in the training data set into the decision model for decision processing, and obtaining the target loss value corresponding to the decision model and the loss value of each training sample sequence;

[0018] As a module, if the above target loss value meets the preset convergence condition, the above decision model will be used as the trained target decision model.

[0019] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory storing a plurality of instructions; the processor loads instructions from the memory to execute any one of the model training methods provided in the embodiment of the present application.

[0020] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute any model training method provided in the embodiment of the present application.

[0021] In an embodiment of the present application, a training data set corresponding to a decision model is obtained, the training data set includes multiple training sample sequences, the training sample sequences include multiple state-action pairs, the state-action pairs include the state of the expert and the decision action made by the expert under the state; the multiple state-action pairs of the training sample sequences in the training data set are input into the decision model for decision processing, and a target loss value corresponding to the decision model and a loss value corresponding to each training sample sequence are obtained; if the target loss value does not meet the preset convergence condition, the model parameters of the decision model are updated according to the target loss value to obtain an updated decision model, and the updated decision model is used as the decision model; according to the loss value corresponding to the training sample sequence, the quality evaluation corresponding to the training sample sequence is determined, and the training data set is screened. A training sample sequence corresponding to a quality evaluation that meets a preset quality condition is selected to obtain a target training sample sequence; an updated training data set is generated according to the target training sample sequence, and the updated data set is used as the training data set, and the step of inputting multiple state-action pairs in the training sample sequence in the training data set into the decision model for decision processing is returned to obtain a target loss value corresponding to the decision model and a loss value of each training sample sequence is obtained; if the target loss value meets a preset convergence condition, the decision model is used as a trained decision model, and the training data set is screened and updated according to the decision model, so that the updated training data set is compatible with the decision model, thereby improving the decision effect of the target decision model obtained by training the decision model according to the updated training data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1is a schematic diagram of a model training system provided in an embodiment of the present application;

[0024] Figure 2 This is a flow chart of an embodiment of the model training method provided in the embodiment of the present application;

[0025] Figure 3 It is a scene diagram of another embodiment of the flow chart of the model training method provided in the embodiment of the present application;

[0026] Figure 4 It is a process schematic diagram of the model training method provided in the embodiment of the present application;

[0027] Figure 5 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0028] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0030] Before explaining the embodiments of the present application in detail, some terms involved in the embodiments of the present application are first explained.

[0031] Among them, in the description of the embodiment of the present application, the terms "first", "second", etc. can be used in this article to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. In addition, the term "plurality" refers to two or more. The terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] The embodiments of the present application provide a model training method, device, electronic device and computer-readable storage medium. Specifically, the model training method of the embodiments of the present application can be executed by an electronic device, wherein the electronic device can be a terminal or a server.

[0033] The terminal can be a terminal device such as a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC), a personal digital assistant (PDA), etc. The terminal can also include a client, which can be a game application client, a browser client carrying a game program, or an instant messaging client, etc.

[0034] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0035] For example, Figure 1 As shown, the electronic device is explained using the server 20 as an example. The terminal 10 can collect multiple training sample sequences through the client. The training sample sequences include multiple state-action pairs. The state-action pairs include the state of the expert and the decision-making actions made by the expert under the state, and send the multiple training sample sequences to the server 20.

[0036] The server 20 determines the training data set corresponding to the decision model according to multiple training sample sequences. Input multiple state-action pairs of the training sample sequence in the training data set into the decision model for decision processing, and obtain the target loss value corresponding to the decision model and the loss value corresponding to each training sample sequence; if the target loss value does not meet the preset convergence condition, update the model parameters of the decision model according to the target loss value to obtain an updated decision model, and use the updated decision model as the decision model; determine the quality evaluation corresponding to the training sample sequence according to the loss value corresponding to the training sample sequence, and select the training sample sequence corresponding to the quality evaluation that meets the preset quality condition from the training data set to obtain the target training sample sequence; generate an updated training data set according to the target training sample sequence, and use the updated data set as the training data set, and return to execute the step of inputting multiple state-action pairs in the training sample sequence in the training data set into the decision model for decision processing, and obtain the target loss value corresponding to the decision model and the loss value of each training sample sequence; if the target loss value meets the preset convergence condition, the decision model is used as the target decision model after training. The server 20 returns the target decision model to the terminal 10.

[0037] The terminal 10 obtains the pending decision state, and inputs the pending decision state into the target decision model for decision processing, obtains the target decision action in the pending decision state, and executes the target decision action.

[0038] The following is a detailed description of the embodiments in conjunction with the accompanying drawings. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0039] In this embodiment, a terminal is taken as an example for explanation. This embodiment provides a model training method, such as Figure 2 As shown, the specific process of the model training method can be as follows:

[0040] 201. Obtain a training data set corresponding to the decision model, where the training data set includes multiple training sample sequences, the training sample sequences include multiple state-action pairs, and the state-action pairs include the state of the expert and the decision action made by the expert in the state.

[0041] The decision model refers to a network system which is formed by connecting various network layers and can realize the decision-making function. The network layer may include at least one neuron, and the neuron refers to a computing unit.

[0042] The type of decision model can be selected according to actual conditions. For example, the decision model can be a convolutional neural network model (CNN) or a recurrent neural network (RNN), which is not limited in the embodiments of the present application.

[0043] It should be understood that the decision model can be applied to various decision scenarios. For example, the decision model can be applied to a game scenario, in which case the decision model can be called a game assistant or a game robot. Another example is that the decision model can be applied to an autonomous driving scenario, in which case the decision model can be called a driving assistant or a driving robot.

[0044] Optionally, the decision model may or may not have a corresponding visualization object in the decision scene. For example, the visualization object corresponding to the decision model in the game scene may be a virtual game character. For another example, the decision model may not have a corresponding visualization object in the autonomous driving scene.

[0045] Since the quality of state-action pairs of the same expert at different times may be inconsistent, the multiple training sample sequences included in the training data set may be multiple training sample sequences of one expert.

[0046] Since the qualities of state-action pairs of different experts may be inconsistent, the multiple training sample sequences included in the training data set may be training sample sequences of multiple experts.

[0047] The multiple state-action pairs included in a training sample sequence may be state-action pairs of an expert in a task. For example, when the decision-making scenario is a game scenario, a task may be a game match. In this case, the state-action pairs of a player in a game match may be combined into a training sample sequence. For another example, when the decision-making scenario is an autonomous driving scenario, a task may be to reach a certain destination. In this case, the state-action pairs of the user in the process of driving from the beginning to the destination may be combined into a training sample sequence. Among them, the multiple state-action pairs may be stored in the training sample sequence in chronological order.

[0048] The state in the state-action pair refers to the situation that the expert is in, and the action in the state-action pair refers to the decision-making action made by the expert in that state. For example, when the decision-making scenario is an autonomous driving scenario, the state can refer to the driving state that the expert is in, and the action can refer to the driving behavior made by the expert in that driving state.

[0049] 202. Input multiple state-action pairs of the training sample sequence in the training data set into the decision model for decision processing, and obtain the target loss value corresponding to the decision model and the loss value corresponding to each training sample sequence.

[0050] Among them, the process of inputting multiple state-action pairs of the training sample sequence in the training data set into the decision model for decision processing, and obtaining the loss value corresponding to each training sample sequence can be:

[0051] Input multiple state-action pairs of a training sample sequence in the training data set into the decision model for decision processing, and obtain the sub-loss value corresponding to each state-action pair in the training sample sequence;

[0052] According to the sub-loss value corresponding to each state-action pair in the training sample sequence, the loss value corresponding to the training sample sequence is determined.

[0053] The method for calculating the target loss value can be selected according to actual conditions, and the embodiment of the present application does not limit it here.

[0054] For example, multiple state-action pairs of the training sample sequence in the training data set are input into the decision model for decision processing, and the process of obtaining the target loss value corresponding to the decision model can be:

[0055] The loss value corresponding to each training sample sequence is weighted to obtain the target loss value corresponding to the decision model.

[0056] For another example, multiple state-action pairs of the training sample sequence in the training data set are input into the decision model for decision processing, and the process of obtaining the target loss value corresponding to the decision model can also be:

[0057] According to the sub-loss value corresponding to each state-action pair, the target loss value corresponding to the decision model is determined.

[0058] Optionally, the loss function for calculating the loss value corresponding to the training sample sequence and the loss function for calculating the target loss value may be the same or different, and this embodiment of the present application is not limited thereto.

[0059] The loss function for calculating the loss value corresponding to the training sample sequence and the loss function for calculating the target loss value can be selected according to actual conditions. For example, the loss function for calculating the loss value corresponding to the training sample sequence can be a mean squared error loss function (MSE) or an average loss function, and the loss function for calculating the target loss value can be a cross entropy function or a mean squared error loss function, which is not limited in the embodiments of the present application.

[0060] Optionally, the terminal may be input into a decision model multiple state-action pairs of training sample sequences with a preset number threshold in the training data set for decision processing to obtain a target loss value corresponding to the decision model, and the preset number threshold is less than or equal to the number of training sample sequences included in the training data set.

[0061] Among them, the terminal may obtain the target loss value corresponding to the decision model after inputting multiple state-action pairs of a training sample sequence with a preset number threshold in the training data set into the decision model for decision processing, that is, determining the target loss value corresponding to the decision model based on multiple state-action pairs of a training sample sequence with a preset number threshold in the training data set.

[0062] Alternatively, the terminal may also determine the target loss value corresponding to the decision model only according to the state-action pairs of the first target number of training sample sequences in the training data set. In this case, multiple state-action pairs of the training sample sequences in the training data set are input into the decision model for decision processing. The process of obtaining the target loss value corresponding to the decision model and the loss value corresponding to each training sample sequence may be:

[0063] Filter out a first target number of training sample sequences from the training data set;

[0064] Multiple state-action pairs of the first target number of training sample sequences are input into the decision model for decision processing to obtain a target loss value corresponding to the decision model and a loss value corresponding to the first target number of training sample sequences.

[0065] The first target quantity is less than the preset quantity threshold and may be a factor of the preset quantity threshold. That is, at this time, the target loss value of the decision model is determined according to a plurality of state-action pairs of the training sample sequence of the first target quantity.

[0066] The method for filtering out the first target number of training sample sequences from the training data set can be selected according to the actual situation. For example, the first target number of training sample sequences can be filtered out from the training data set according to a random rule. For another example, the first target number of training sample sequences can be filtered out from the training data set according to the corresponding time sequence of state-action pairs in the training sample sequence. The embodiments of the present application are not limited here.

[0067] 203. If the target loss value does not meet the preset convergence condition, the model parameters of the decision model are updated according to the target loss value to obtain an updated decision model, and the updated decision model is used as the decision model.

[0068] After obtaining the target loss value, the terminal can determine whether the decision model converges according to the target loss value. If the decision model converges, it is determined that the target loss value meets the preset convergence condition. If the decision model does not converge, it is determined that the target loss value does not meet the preset convergence condition. The convergence of the decision model means that the target loss value of the decision model gradually approaches a certain value.

[0069] 204. Determine the quality evaluation corresponding to the training sample sequence according to the loss value corresponding to the training sample sequence, and select the training sample sequence corresponding to the quality evaluation that meets the preset quality condition from the training data set to obtain the target training sample sequence.

[0070] The terminal may directly use the loss value corresponding to the training sample sequence as the quality evaluation corresponding to the training sample sequence, or the terminal may use the opposite of the loss value corresponding to the training sample sequence as the quality evaluation corresponding to the training sample sequence.

[0071] After obtaining the quality evaluation corresponding to the training sample sequence, the terminal may first determine the quality evaluation that meets the preset quality condition, and then select the training sample sequence corresponding to the quality evaluation that meets the preset quality condition from the training data set to obtain the target training sample sequence.

[0072] The method for determining the quality evaluation that meets the preset quality conditions can be selected according to the actual situation. For example, the quality evaluation that meets the preset quality conditions can be determined based on the quality average value and clustering method corresponding to the quality evaluation. The embodiment of the present application is not limited here.

[0073] When the quality evaluation that meets the preset quality condition is determined according to the quality average value corresponding to the quality evaluation, the process of selecting the training sample sequence corresponding to the quality evaluation that meets the preset quality condition from the training data set to obtain the target training sample sequence can be:

[0074] Determine the quality average value corresponding to the quality evaluation;

[0075] Determine a quality evaluation greater than the quality average as a quality evaluation that meets the preset quality conditions;

[0076] The training sample sequences corresponding to the quality evaluations that meet the preset quality conditions are selected from the training data set to obtain the target training sample sequences.

[0077] When the quality evaluation that meets the preset quality condition is determined according to the clustering method, the training sample sequence corresponding to the quality evaluation that meets the preset quality condition is screened out from the training data set, and the process of obtaining the target training sample sequence can be:

[0078] Perform clustering on the quality evaluations corresponding to the training sample sequences to obtain multiple quality categories;

[0079] According to the quality evaluation in the quality category, determine the quality level corresponding to the quality category;

[0080] According to the quality level, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition is selected from the training data set to obtain a target training sample sequence.

[0081] Among them, the quality evaluations corresponding to the training sample sequences are clustered to obtain multiple quality categories, which can be understood as clustering relatively close quality evaluations into one quality category. The method for clustering the quality evaluations corresponding to the training sample sequences can be selected according to the actual situation. For example, the quality evaluations corresponding to the training sample sequences can be clustered by the K-means algorithm, or the quality evaluations corresponding to the training sample sequences can be clustered by the density-based clustering method. The embodiments of the present application are not limited here.

[0082] The method for determining the quality level corresponding to a quality category based on the quality evaluation in the quality category can be set according to actual conditions. For example, the quality evaluation corresponding to the class center of the quality category can be used as the quality level corresponding to the quality category, or the average value of the quality evaluation in the quality category can be used as the quality level corresponding to the quality category, or the sum of the quality evaluations in the quality category can be used as the quality level corresponding to the quality category. The embodiments of the present application are not limited here.

[0083] After obtaining the quality level corresponding to the quality category, the terminal selects the training sample sequence corresponding to the quality evaluation that meets the preset quality condition from the training data set according to the quality level to obtain the target training sample sequence.

[0084] Optionally, according to the quality level, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition is screened out from the training data set, and the process of obtaining a target training sample sequence may be:

[0085] Determine the level mean corresponding to the quality level;

[0086] The quality evaluation in the quality category where the quality level greater than the level average is located is determined as the quality evaluation that meets the preset quality conditions;

[0087] The training sample sequences corresponding to the quality evaluations that meet the preset quality conditions are selected from the training data set to obtain the target training sample sequences.

[0088] Alternatively, according to the quality level, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition is selected from the training data set, and the process of obtaining a target training sample sequence may also be:

[0089] According to the quality level, determining the screening ratio of the training sample sequences in the quality category corresponding to the quality level;

[0090] According to the screening ratio, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition is screened out from the training sample sequence in the quality category corresponding to the quality level to obtain a target training sample sequence.

[0091] Among them, the quality levels can be sorted in order from large to small or from small to large to obtain the quality levels sorted in order of size, and then the screening ratios corresponding to the size order of the quality levels are obtained, and the screening ratios corresponding to the size order of the quality levels are used as the screening ratios of the training sample sequences in the quality categories corresponding to the quality levels.

[0092] For example, the quality levels include quality level L1, quality level L2 and quality level L3, quality level L1 is the quality level of quality category C1, quality level L2 is the quality level of quality category C2, quality level L3 is the quality level of quality category C3, quality level L1 is greater than quality level L2, and quality level L2 is greater than quality level L3, then the size order of quality level L1 is the largest order, and the screening ratio corresponding to the largest order is 100%, and 100% of the training sample sequences are screened out from quality category C1 as the training sample sequences corresponding to the quality evaluation that meets the preset quality conditions, the size order of quality level L2 is the second largest order, and the screening ratio corresponding to the second largest order is 50%, and 50% of the training sample sequences are screened out from quality category C2 as the training sample sequences corresponding to the quality evaluation that meets the preset quality conditions, the size order of quality level L3 is the third largest order, and the screening ratio corresponding to the third largest order is 20%, and 20% of the training sample sequences are screened out from quality category C3 as the training sample sequences corresponding to the quality evaluation that meets the preset quality conditions.

[0093] Since the quality evaluation corresponding to the training sample sequence obtained by the decision model may be wrong, that is, the quality evaluation corresponding to the training sample sequence with better quality may be misjudged as a lower quality evaluation, and the quality evaluation corresponding to the training sample sequence with poorer quality may be misjudged as a higher quality evaluation, therefore, in an embodiment of the present application, the quality evaluations corresponding to the training sample sequences are clustered to obtain multiple quality categories; according to the quality level, the screening ratio of the training sample sequences in the quality category corresponding to the quality level is determined; according to the screening ratio, the training sample sequences corresponding to the quality evaluations that meet the preset quality conditions are screened out from the training sample sequences in the quality category corresponding to the quality level to obtain the target training sample sequence, so that the target training sample sequence is screened out in each quality category, so that the quality evaluation of the target training sample sequence may be higher or lower, thereby reducing the phenomenon of misjudged training sample sequences being screened out, and further improving the decision effect of the target decision model obtained by training the decision model according to the target training sample sequence.

[0094] Optionally, the screening ratio corresponding to the order of quality levels may be fixed or dynamically changed. When the screening ratio corresponding to the order of quality levels is dynamically changed, the process of determining the screening ratio of the training sample sequence in the quality category corresponding to the quality level according to the quality level may be:

[0095] According to the quality level, determining the initial screening ratio of the training sample sequences in the quality category corresponding to the quality level;

[0096] Obtain the target number of training sample sequences in the quality category corresponding to the quality level;

[0097] According to the target quantity, the initial screening ratio is adjusted to obtain the screening ratio of the training sample sequence in the quality category corresponding to the quality level.

[0098] The training sample sequence in the quality category corresponding to the quality level refers to the training sample sequence corresponding to the quality evaluation in the quality category corresponding to the quality level. If the target number is greater than the first preset training number threshold, the initial screening ratio is reduced; if the target number is less than or equal to the first preset training number threshold and greater than the second preset training number threshold, the initial screening ratio is used as the screening ratio; if the target number is less than or equal to the second preset training number threshold, the initial screening ratio is increased.

[0099] In an embodiment of the present application, the initial screening ratio is adjusted according to the target number of training sample sequences in the quality category corresponding to the quality level to obtain the screening ratio of the training sample sequences in the quality category corresponding to the quality level, so as to realize dynamic adjustment of the screening ratio according to the target number, thereby further reducing the phenomenon of misjudged training sample sequences with better quality being screened out, and further improving the decision effect of the target decision model obtained by training the decision model according to the target training sample sequence.

[0100] When the screening ratios corresponding to the order of quality levels change dynamically, the process of determining the screening ratios of training sample sequences in the quality categories corresponding to the quality levels according to the quality levels may also be:

[0101] Determine the level difference between each quality level;

[0102] According to the level difference, the screening ratio of the training sample sequences in the quality category corresponding to the quality level is determined.

[0103] Among them, the larger the level difference is, the larger the difference in screening ratios between training sample sequences in quality categories corresponding to the quality levels corresponding to the level difference is, and the smaller the level difference is, the smaller the difference in screening ratios between training sample sequences in quality categories corresponding to the quality levels corresponding to the level difference is.

[0104] For example, the quality levels include quality level L1, quality level L2 and quality level L3, quality level L1 is the quality level of quality category C1, quality level L2 is the quality level of quality category C2, quality level L3 is the quality level of quality category C3, quality level L1 is greater than quality level L2, quality level L2 is greater than quality level L3, then the size order of quality level L1 is the largest order, the size order of quality level L2 is the second largest order, the size order of quality level L3 is the third largest order, the level difference between quality level L1 and quality level L2 is a1, and the level difference between quality level L2 and quality level L3 is a2, then find the first screening ratio corresponding to a1 in the largest order, the second screening ratio corresponding to a1 in the second largest order, the third screening ratio corresponding to a2 in the second largest order, and the fourth screening ratio corresponding to a2 in the third largest order.

[0105] The first screening ratio is used as the screening ratio of the training sample sequence in the quality category corresponding to the quality level L1. If the second screening ratio and the third screening ratio are the same, the second screening ratio or the third screening ratio is used as the screening ratio of the training sample sequence in the quality category corresponding to the quality level L2, and the fourth screening ratio is used as the screening ratio of the training sample sequence in the quality category corresponding to the quality level L3.

[0106] If the second screening ratio and the third screening ratio are different, the second screening ratio is used as the screening ratio of the training sample sequence in the quality category corresponding to the quality level L2, and then the ratio difference between the third screening ratio and the fourth screening ratio is determined, and the ratio difference is added to the second screening ratio to obtain the fifth screening ratio, and the fifth screening ratio is used as the screening ratio of the training sample sequence in the quality category corresponding to the quality level L3.

[0107] In an embodiment of the present application, the level difference between each quality level is determined; based on the level difference, the screening ratio of the training sample sequence in the quality category corresponding to the quality level is determined, and the screening ratio is dynamically adjusted according to the level difference, so that the phenomenon of misjudged training sample sequences with better quality being screened out can be further reduced, and the decision effect of the target decision model obtained by training the decision model according to the target training sample sequence can be further improved.

[0108] Optionally, according to the screening ratio, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition is screened out from the training sample sequence in the quality category corresponding to the quality level. The method for obtaining the target training sample sequence can be set according to actual conditions, and the embodiment of the present application is not limited here.

[0109] Exemplarily, according to the screening ratio, the process of screening out the training sample sequences corresponding to the quality evaluation that meet the preset quality conditions from the training sample sequences in the quality category corresponding to the quality level, and obtaining the target training sample sequence may be:

[0110] According to the rule of from large to small, the quality evaluation of the screening ratio is screened out from the training sample sequence in the quality category corresponding to the quality level to obtain the first target quality evaluation;

[0111] The training sample sequence corresponding to the first target quality evaluation is used as the training sample sequence corresponding to the quality evaluation of the preset quality condition to obtain a target training sample sequence.

[0112] For example, the quality level includes quality level L1, quality level L1 includes quality evaluations p1, p2, p3, p4 and p5, p1>p2>p3>p4>p5, and the screening ratio is 60%. Then, according to the rule from large to small, 60% of the quality evaluations are screened out from the training sample sequence in the quality category corresponding to the quality level, which are p1, p2 and p3 respectively. That is, the first target quality evaluation is p1, p2 and p3.

[0113] As another example, according to the screening ratio, the process of screening out the training sample sequence corresponding to the quality evaluation that meets the preset quality condition from the training sample sequence in the quality category corresponding to the quality level, and obtaining the target training sample sequence may also be:

[0114] According to the random rule, a quality evaluation of the screening ratio is screened out from the training sample sequence in the quality category corresponding to the quality level to obtain a second target quality evaluation;

[0115] The training sample sequence corresponding to the second target quality evaluation is used as the training sample sequence corresponding to the quality evaluation of the preset quality condition to obtain a target training sample sequence.

[0116] For example, the quality level includes quality level L1, quality level L1 includes quality evaluations p1, p2, p3, p4 and p5, p1>p2>p3>p4>p5, and the screening ratio is 60%. Then, according to the random rule, 60% of the quality evaluations are screened out from the training sample sequence in the quality category corresponding to the quality level, namely, the first target quality evaluation is p1, p2 and p5.

[0117] As another example, according to the screening ratio, the process of screening out the training sample sequence corresponding to the quality evaluation that meets the preset quality condition from the training sample sequence in the quality category corresponding to the quality level, and obtaining the target training sample sequence may also be:

[0118] Taking the class center in the quality category corresponding to the quality level as a benchmark, a quality evaluation of a screening ratio is screened out from the training sample sequence in the quality category corresponding to the quality level to obtain a third target quality evaluation;

[0119] The training sample sequence corresponding to the third target quality evaluation is used as the training sample sequence corresponding to the quality evaluation of the preset quality condition to obtain a target training sample sequence.

[0120] For example, the quality level includes quality level L1, quality level L1 includes quality evaluations p1, p2, p3, p4 and p5, p1>p2>p3>p4>p5, the class center is p3, the screening ratio is 60%, and based on the class center, 60% of the quality evaluations are screened out from the training sample sequence in the quality category corresponding to the quality level, namely, p2, p3 and p4. That is, the third target quality evaluation is p2, p3 and p4.

[0121] In some embodiments, when a plurality of state-action pairs of a first target number of training sample sequences in a training data set are input into a decision model for decision processing, and a target loss value corresponding to the decision model and a loss value of the first target number of training sample sequences are obtained, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition is screened out from the training data set to obtain a target training sample sequence, including:

[0122] determining a second target number of training sample sequences for which quality evaluation exists;

[0123] If the second target quantity meets the preset quantity threshold, a training sample sequence corresponding to a quality evaluation that meets the preset quality condition is screened out from the training data set to obtain a target training sample sequence.

[0124] When multiple state-action pairs of the first target number of training sample sequences in the training data set are input into the decision model for decision processing, and the target loss value corresponding to the decision model and the loss value of the first target number of training sample sequences are obtained, since the first target number is less than the preset number threshold, the second target number is less than or equal to the preset number threshold. If the target loss value does not meet the preset convergence condition and the second target number is less than the preset number threshold, the model parameters of the decision model are updated according to the target loss value to obtain an updated decision model, and the updated decision model is used as the decision model to return to the step of screening out the first target number of training sample sequences from the training data set. If the target loss value does not meet the preset convergence condition and the first target number is equal to the preset number threshold, the training sample sequence corresponding to the quality evaluation that meets the preset quality condition is screened from the training data set to obtain the target training sample sequence, so that the loss value of the training sample sequence can be obtained through the updated decision model, the accuracy of the loss value of the training sample sequence is improved, that is, the accuracy of the quality evaluation of the training sample sequence is improved, thereby improving the decision effect of the target model obtained by training the decision model according to the training sample sequence, that is, improving the effect of imitation learning of the decision model.

[0125] 205. Generate an updated training data set according to the target training sample sequence, and use the updated data set as the training data set, return to execute the step of inputting multiple state-action pairs in the training sample sequence in the training data set into the decision model for decision processing, and obtain the target loss value corresponding to the decision model and the loss value of each training sample sequence.

[0126] After obtaining the target training sample sequence, the terminal can form the target training sample sequence into an updated training data set, use the updated data set as the training data set, return to execute the step of inputting multiple state-action pairs in the training sample sequence in the training data set into the decision model for decision processing, and obtain the target loss value corresponding to the decision model and the loss value of each training sample sequence, so as to continue to train the decision model according to the updated data set, and realize the screening and updating of the training data set according to the decision model, so that the updated training data set is compatible with the decision model, improve the decision effect of the target decision model obtained by training the decision model according to the updated training data set, and have high mobility and implementation value.

[0127] 206. If the target loss value meets the preset convergence condition, the decision model is used as the target decision model after training.

[0128] If the target loss value meets the preset convergence condition, the decision model can be used as the target decision model for training completion.

[0129] In some embodiments, if the target loss value satisfies the preset convergence condition, after the decision model is used as the trained target decision model, the method further includes:

[0130] Get the pending decision status;

[0131] Input the decision-to-be-decided state into the target decision model for decision processing, and obtain the target decision action for the decision-to-be-decided state;

[0132] Execute the target decision action.

[0133] From the above content, it can be seen that in the embodiment of the present application, a training data set corresponding to the decision model is obtained, the training data set includes multiple training sample sequences, the training sample sequence includes multiple state-action pairs, the state-action pair includes the state of the expert and the decision action made by the expert in the state; the multiple state-action pairs of the training sample sequence in the training data set are input into the decision model for decision processing, and the target loss value corresponding to the decision model and the loss value corresponding to each training sample sequence are obtained; if the target loss value does not meet the preset convergence condition, the model parameters of the decision model are updated according to the target loss value to obtain an updated decision model, and the updated decision model is used as the decision model; according to the loss value corresponding to the training sample sequence, the quality evaluation corresponding to the training sample sequence is determined, and the training sample sequence is obtained from the loss value. A training sample sequence corresponding to a quality evaluation that meets a preset quality condition is screened out from the data set to obtain a target training sample sequence; an updated training data set is generated according to the target training sample sequence, and the updated data set is used as the training data set, and the step of inputting a plurality of state-action pairs in the training sample sequence in the training data set into the decision model for decision processing is returned to obtain a target loss value corresponding to the decision model and a loss value of each training sample sequence is obtained; if the target loss value meets a preset convergence condition, the decision model is used as a trained decision model, and the training data set is screened and updated according to the decision model, so that the updated training data set is compatible with the decision model, thereby improving the decision effect of the target decision model obtained by training the decision model according to the updated training data set.

[0134] According to the following Figure 3 , the model training method provided in this application is further explained.

[0135] 301. The terminal obtains a training data set corresponding to the decision model, where the training data set includes multiple training sample sequences, the training sample sequences include multiple state-action pairs, and the state-action pairs include the state of the expert and the decision action made by the expert in the state.

[0136] 302. The terminal selects a first target number of training sample sequences from the training data set according to a random rule, and inputs multiple state-action pairs of the first target number of training sample sequences into the decision model for decision processing, to obtain a target loss value corresponding to the decision model and a loss value corresponding to the first target number of training sample sequences.

[0137] 303. If the target loss value does not meet the preset convergence condition, the terminal updates the model parameters of the decision model according to the target loss value, obtains an updated decision model, and uses the updated decision model as the decision model.

[0138] 304. The terminal determines a quality evaluation corresponding to the training sample sequence according to the loss value corresponding to the training sample sequence, and determines a second target number of training sample sequences having the quality evaluation.

[0139] 305. If the second target quantity meets the preset quantity threshold, the terminal performs clustering processing on the quality evaluations corresponding to the training sample sequence to obtain multiple quality categories.

[0140] 306. The terminal determines a quality level corresponding to the quality category according to the quality evaluation in the quality category, and determines a screening ratio of the training sample sequence in the quality category corresponding to the quality level according to the quality level.

[0141] 307. The terminal selects, according to the screening ratio, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition from the training sample sequences in the quality category corresponding to the quality level to obtain a target training sample sequence.

[0142] According to the screening ratio, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition is screened out from the training sample sequence in the quality category corresponding to the quality level. The process of obtaining the target training sample sequence can be specifically referred to the above-mentioned model training method embodiment, which will not be repeated in this example.

[0143] 308. The terminal generates an updated training data set according to the target training sample sequence, and uses the updated data set as the training data set, and then returns to execute 302.

[0144] 309. If the second target quantity does not meet the preset quantity threshold, the terminal returns to execute 302.

[0145] 3010. If the target loss value meets the preset convergence condition, the terminal uses the decision model as the trained target decision model.

[0146] In an embodiment of the present application, in the process of training the decision model according to the target loss value, the loss values ​​of different training sample sequences are recorded, and then the quality evaluation of the training sample sequence is determined according to the loss value of the training sample sequence. The smaller the loss value, the higher the quality evaluation. Then, the quality evaluation is clustered to obtain a quality category. From the quality levels corresponding to the quality categories, the training sample sequences corresponding to the quality evaluations that meet the preset quality conditions are screened out from the quality categories to obtain the target training sample sequence, so as to screen out the training sample sequences with poor quality. Then, the decision model is continued to be trained with the target training sample sequence, such as Figure 4 shown.

[0147] The specific implementation method and corresponding beneficial effects of this embodiment can be specifically referred to the above-mentioned model training method embodiment, and this example will not be repeated here.

[0148] In order to better implement the above method, an embodiment of the present application also provides a model training device, which can be specifically integrated into an electronic device, such as a computer device, which can be a terminal, a server, or other device.

[0149] Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop computer, a personal computer and other devices; the server can be a single server or a server cluster composed of multiple servers.

[0150] For example, in this embodiment, the method of the embodiment of the present application is described in detail by taking the model training device specifically integrated in the terminal as an example. This embodiment provides a model training device, such as Figure 5 As shown, the model training device may include:

[0151] The acquisition module 501 is used to acquire a training data set corresponding to the decision model. The training data set includes multiple training sample sequences. The training sample sequences include multiple state-action pairs. The state-action pairs include the state of the expert and the decision action made by the expert in the state.

[0152] The input module 502 is used to input multiple state-action pairs of the training sample sequence in the training data set into the decision model for decision processing, and obtain the target loss value corresponding to the decision model and the loss value corresponding to each training sample sequence.

[0153] The updating module 503 is used to update the model parameters of the decision model according to the target loss value if the target loss value does not meet the preset convergence condition, obtain the updated decision model, and use the updated decision model as the decision model.

[0154] The screening module 504 is used to determine the quality evaluation corresponding to the training sample sequence according to the loss value corresponding to the training sample sequence, and screen out the training sample sequence corresponding to the quality evaluation that meets the preset quality condition from the training data set to obtain the target training sample sequence.

[0155] Return module 505, used to generate an updated training data set according to the target training sample sequence, and use the updated data set as the training data set, and return to execute the step of inputting multiple state-action pairs in the training sample sequence in the training data set into the decision model for decision processing, and obtaining the target loss value corresponding to the decision model and the loss value of each training sample sequence;

[0156] As module 506, if the target loss value meets the preset convergence condition, the decision model is used as the trained target decision model.

[0157] In some embodiments, the screening module 504 is specifically configured to:

[0158] Perform clustering on the quality evaluations corresponding to the training sample sequences to obtain multiple quality categories;

[0159] According to the quality evaluation in the quality category, determine the quality level corresponding to the quality category;

[0160] According to the quality level, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition is selected from the training data set to obtain a target training sample sequence.

[0161] In some embodiments, the screening module 504 is specifically configured to:

[0162] According to the quality level, determining the screening ratio of the training sample sequences in the quality category corresponding to the quality level;

[0163] According to the screening ratio, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition is screened out from the training sample sequence in the quality category corresponding to the quality level to obtain a target training sample sequence.

[0164] In some embodiments, the screening module 504 is specifically configured to:

[0165] According to the quality level, determining the initial screening ratio of the training sample sequences in the quality category corresponding to the quality level;

[0166] Obtain the target number of training sample sequences in the quality category corresponding to the quality level;

[0167] According to the target quantity, the initial screening ratio is adjusted to obtain the screening ratio of the training sample sequence in the quality category corresponding to the quality level.

[0168] In some embodiments, the screening module 504 is specifically configured to:

[0169] Determine the level difference between each quality level;

[0170] According to the level difference, the screening ratio of the training sample sequences in the quality category corresponding to the quality level is determined.

[0171] In some embodiments, the input module 502 is specifically used for:

[0172] Filter out a first target number of training sample sequences from the training data set;

[0173] Multiple state-action pairs of the first target number of training sample sequences are input into the decision model for decision processing to obtain a target loss value corresponding to the decision model and a loss value corresponding to the first target number of training sample sequences.

[0174] The screening module 504 is specifically used for:

[0175] determining a second target number of training sample sequences for which quality evaluation exists;

[0176] If the second target quantity meets the preset quantity threshold, a training sample sequence corresponding to a quality evaluation that meets the preset quality condition is screened out from the training data set to obtain a target training sample sequence.

[0177] In some embodiments, the model training device further includes a return execution module, and the return execution module is specifically used to:

[0178] If the second target quantity does not meet the preset quantity threshold, the process returns to the step of selecting the first target quantity of training sample sequences from the training data set.

[0179] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation methods and corresponding beneficial effects of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0180] Accordingly, an embodiment of the present application further provides an electronic device, which may be a terminal, and the terminal may be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC), a personal digital assistant (PDA), or other terminal devices. Figure 6 As shown, Figure 6A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 600 includes a processor 601 having one or more processing cores, a memory 602 having one or more computer-readable storage media, and a computer program stored in the memory 602 and executable on the processor. The processor 601 is electrically connected to the memory 602. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0181] The processor 601 is the control center of the electronic device 600. It uses various interfaces and lines to connect various parts of the entire electronic device 600, executes various functions of the electronic device 600 and processes data by running or loading software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, thereby monitoring the electronic device 600 as a whole.

[0182] In the embodiment of the present application, the processor 601 in the electronic device 600 will load instructions corresponding to the processes of one or more application programs into the memory 602 according to the following steps, and the processor 601 will run the application programs stored in the memory 602 to implement various functions, such as:

[0183] Obtaining a training data set corresponding to the decision model, the training data set includes multiple training sample sequences, the training sample sequences include multiple state-action pairs, the state-action pairs include the state of the expert and the decision action made by the expert in the state;

[0184] Input multiple state-action pairs of the training sample sequence in the training data set into the decision model for decision processing, and obtain the target loss value corresponding to the decision model and the loss value corresponding to each training sample sequence;

[0185] If the target loss value does not meet the preset convergence condition, the model parameters of the decision model are updated according to the target loss value to obtain an updated decision model, and the updated decision model is used as the decision model;

[0186] According to the loss value corresponding to the training sample sequence, the quality evaluation corresponding to the training sample sequence is determined, and the training sample sequence corresponding to the quality evaluation that meets the preset quality condition is screened out from the training data set to obtain the target training sample sequence;

[0187] Generate an updated training data set according to the target training sample sequence, use the updated data set as the training data set, and return to execute the step of inputting multiple state-action pairs in the training sample sequence in the training data set into the decision model for decision processing, and obtain the target loss value corresponding to the decision model and the loss value of each training sample sequence;

[0188] If the target loss value meets the preset convergence condition, the decision model is taken as the target decision model after training.

[0189] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the detailed description of the model training method above, which will not be elaborated here.

[0190] Optional, such as Figure 6 As shown, the electronic device 600 further includes: a touch screen 603, a radio frequency circuit 604, an audio circuit 605, an input unit 606, and a power supply 607. The processor 601 is electrically connected to the touch screen 603, the radio frequency circuit 604, the audio circuit 605, the input unit 606, and the power supply 607, respectively. Those skilled in the art can understand that Figure 6 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0191] The touch display screen 603 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 603 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device, and these graphical user interfaces may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 601, and can receive the command sent by the processor 601 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 601 to determine the type of touch event, and then the processor 601 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 603 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 603 can also be used as a part of the input unit 606 to realize the input function.

[0192] The radio frequency circuit 604 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other electronic devices through wireless communication, and to send and receive signals between the network device or other electronic devices.

[0193] The audio circuit 605 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 605 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 605 and converted into audio data, and then the audio data is output to the processor 601 for processing, and then sent to another electronic device through the radio frequency circuit 604, or the audio data is output to the memory 602 for further processing. The audio circuit 605 may also include an earplug jack to provide communication between an external headset and an electronic device.

[0194] The input unit 606 may be used to receive input numbers, character information or user feature information (such as fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0195] The power supply 607 is used to supply power to various components of the electronic device 600. Optionally, the power supply 607 can be logically connected to the processor 601 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 607 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0196] although Figure 6 Not shown, the electronic device 600 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0197] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0198] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0199] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored, and the computer program can be loaded by a processor to execute any model training method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0200] Obtaining a training data set corresponding to the decision model, the training data set includes multiple training sample sequences, the training sample sequences include multiple state-action pairs, the state-action pairs include the state of the expert and the decision action made by the expert in the state;

[0201] Input multiple state-action pairs of the training sample sequence in the training data set into the decision model for decision processing, and obtain the target loss value corresponding to the decision model and the loss value corresponding to each training sample sequence;

[0202] If the target loss value does not meet the preset convergence condition, the model parameters of the decision model are updated according to the target loss value to obtain an updated decision model, and the updated decision model is used as the decision model;

[0203] According to the loss value corresponding to the training sample sequence, the quality evaluation corresponding to the training sample sequence is determined, and the training sample sequence corresponding to the quality evaluation that meets the preset quality condition is screened out from the training data set to obtain the target training sample sequence;

[0204] Generate an updated training data set according to the target training sample sequence, use the updated data set as the training data set, and return to execute the step of inputting multiple state-action pairs in the training sample sequence in the training data set into the decision model for decision processing, and obtain the target loss value corresponding to the decision model and the loss value of each training sample sequence;

[0205] If the target loss value meets the preset convergence condition, the decision model is taken as the target decision model after training.

[0206] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the detailed description of the model training method above, which will not be elaborated here.

[0207] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0208] Since the computer program stored in the computer-readable storage medium can execute any one of the model training methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the model training methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0209] The above is a detailed introduction to a model training method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a training data set corresponding to the decision model, the training data set comprising a plurality of training sample sequences, the training sample sequences comprising a plurality of state-action pairs, the state-action pairs comprising a state of an expert and a decision action made by the expert under the state; Inputting a plurality of state-action pairs of the training sample sequence in the training data set into the decision model for decision processing, and obtaining a target loss value corresponding to the decision model and a loss value corresponding to each of the training sample sequences; If the target loss value does not meet the preset convergence condition, the model parameters of the decision model are updated according to the target loss value to obtain an updated decision model, and the updated decision model is used as the decision model; Determine the quality evaluation corresponding to the training sample sequence according to the loss value corresponding to the training sample sequence, and select the training sample sequence corresponding to the quality evaluation that meets the preset quality condition from the training data set to obtain the target training sample sequence; Generate an updated training data set according to the target training sample sequence, use the updated data set as the training data set, and return to execute the step of inputting multiple state-action pairs in the training sample sequence in the training data set into the decision model for decision processing, and obtain the target loss value corresponding to the decision model and the loss value of each training sample sequence; If the target loss value meets the preset convergence condition, the decision model is used as the trained target decision model.

2. The model training method according to claim 1, characterized in that: The step of selecting a training sample sequence corresponding to a quality evaluation that meets a preset quality condition from the training data set to obtain a target training sample sequence includes: Performing clustering processing on the quality evaluations corresponding to the training sample sequences to obtain multiple quality categories; Determining the quality level corresponding to the quality category according to the quality evaluation in the quality category; According to the quality level, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition is screened out from the training data set to obtain a target training sample sequence.

3. The model training method according to claim 2, characterized in that: The step of selecting, according to the quality level, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition from the training data set to obtain a target training sample sequence includes: According to the quality level, determining a screening ratio of training sample sequences in a quality category corresponding to the quality level; According to the screening ratio, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition is screened out from the training sample sequences in the quality category corresponding to the quality level to obtain a target training sample sequence.

4. The model training method according to claim 3, characterized in that: The step of determining, according to the quality level, a screening ratio of training sample sequences in a quality category corresponding to the quality level includes: According to the quality level, determining an initial screening ratio of training sample sequences in a quality category corresponding to the quality level; Obtaining a target number of training sample sequences in a quality category corresponding to the quality level; The initial screening ratio is adjusted according to the target quantity to obtain the screening ratio of the training sample sequences in the quality category corresponding to the quality level.

5. The model training method according to claim 3, characterized in that: The step of determining, according to the quality level, a screening ratio of training sample sequences in a quality category corresponding to the quality level includes: determining level differences between each of said quality levels; According to the level difference, a screening ratio of training sample sequences in the quality category corresponding to the quality level is determined.

6. The model training method according to claim 1, characterized in that: The process of inputting a plurality of state-action pairs of the training sample sequence in the training data set into the decision model for decision processing to obtain the target loss value corresponding to the decision model and the loss value corresponding to each of the training sample sequences may be: Filtering a first target number of training sample sequences from the training data set; Inputting a plurality of state-action pairs of the first target number of training sample sequences into the decision model for decision processing, and obtaining a target loss value corresponding to the decision model and a loss value corresponding to the first target number of training sample sequences; The step of selecting a training sample sequence corresponding to a quality evaluation that meets a preset quality condition from the training data set to obtain a target training sample sequence includes: determining a second target number of training sample sequences for which quality evaluation exists; If the second target quantity meets a preset quantity threshold, a training sample sequence corresponding to a quality evaluation that meets a preset quality condition is screened out from the training data set to obtain a target training sample sequence.

7. The model training method according to claim 6, characterized in that: After determining the second target number of training sample sequences with quality evaluation, the method further includes: If the second target quantity does not meet the preset quantity threshold, the process returns to the step of selecting the first target quantity of training sample sequences from the training data set.

8. A model training device, characterized in that: The device comprises: An acquisition module is used to acquire a training data set corresponding to the decision model, wherein the training data set includes a plurality of training sample sequences, wherein the training sample sequences include a plurality of state-action pairs, wherein the state-action pairs include the state of the expert and the decision action made by the expert under the state; An input module, used to input multiple state-action pairs of the training sample sequence in the training data set into the decision model for decision processing, and obtain a target loss value corresponding to the decision model and a loss value corresponding to each of the training sample sequences; An updating module, configured to update the model parameters of the decision model according to the target loss value if the target loss value does not meet the preset convergence condition, obtain an updated decision model, and use the updated decision model as the decision model; A screening module, used to determine the quality evaluation corresponding to the training sample sequence according to the loss value corresponding to the training sample sequence, and screen out the training sample sequence corresponding to the quality evaluation that meets the preset quality condition from the training data set to obtain the target training sample sequence; A return module, used to generate an updated training data set according to the target training sample sequence, and use the updated data set as the training data set, and return to execute the step of inputting multiple state-action pairs in the training sample sequence in the training data set into the decision model for decision processing, and obtaining the target loss value corresponding to the decision model and the loss value of each training sample sequence; As a module, if the target loss value meets the preset convergence condition, the decision model is used as the trained target decision model.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the model training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute the model training method described in any one of claims 1 to 7.

Images