Method, device, equipment and storage medium for training model

Abstract

The embodiments of the present disclosure disclose a method and device for training a model. A specific implementation of the method includes: obtaining a training sample set; performing the following training steps: using the state information of the object at each time point included in the training sample as the model input, using the object's action information as the model output, and solving the candidate parameters of the model based on the loss function and the objective function. The loss function is constructed based on the adversarial learning algorithm, and the candidate parameters are a set of optimal parameters of the adversarial parties obtained using the objective function; when the candidate parameters meet the preset conditions, the candidate parameters are determined as the final parameters of the model, and the final model after training is output. The preset conditions are used to characterize the setting of the set of optimal parameters of the adversarial parties based on the extreme value principle of both parties; when the candidate parameters do not meet the preset conditions, the above training steps are repeated. This scheme implements a new model training method and device.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, specifically to the field of artificial intelligence technology, and in particular to a method and device for training a model. Background Art

[0002] In recent years, adversarial reinforcement learning has achieved great success in fields such as autonomous driving and competitive games. The introduction of adversarial neural networks enables the model to better handle problems with adversarial scenarios and improves the robustness of the model and the efficiency of obtaining training data. However, the existing adversarial reinforcement learning model lacks theory and is insufficient to support the design of new algorithms. We hope to model adversarial reinforcement learning problems through optimal control and provide a mathematical theoretical basis for designing new algorithms, so as to use adversarial reinforcement learning to solve problems in autonomous driving. Summary of the invention

[0003] Embodiments of the present disclosure provide a method, apparatus, device, and storage medium for training a model and a method, apparatus, device, and storage medium for generating information.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for training a model, the method comprising: obtaining a training sample set, wherein the training samples in the training sample set include state information of the object at each time point and action information of the object; performing the following training steps: using the state information of the object at each time point included in the training sample as model input, and using the action information of the object corresponding to the input state information of the object at each time point as model output, and solving the candidate parameters of the model based on the loss function and the objective function, wherein the loss function is constructed based on an adversarial learning algorithm, and the candidate parameters are a set of optimal parameters of the adversarial parties obtained using the objective function; in response to the candidate parameters satisfying preset conditions, determining the candidate parameters as the final parameters of the model, and outputting the final model after training, wherein the preset conditions are used to characterize the setting of the set of optimal parameters of the adversarial parties based on the extreme value principle of both parties; in response to the candidate parameters not satisfying the preset conditions, repeating the above training steps.

[0005] In some embodiments, each time point is a time point in a continuous state of the object, and the model is used to solve the optimal control problem of the object in a continuous state; and the loss function is constructed based on a combination of an adversarial learning algorithm and a reinforcement learning algorithm.

[0006] In some embodiments, in response to candidate parameters satisfying preset conditions, the candidate parameters are determined as final parameters of the model, and the final model after training is output, including: in response to candidate parameters satisfying preset conditions, the candidate parameters are determined as final parameters of the model, and the final model after training is output, wherein the preset conditions are: the loss value corresponding to the set of optimal parameters of the adversarial parties is equal to the unique solution of the first equation, and the solution of the first equation is used to characterize the solution to the problem existing in the model training.

[0007] In some embodiments, the preset condition includes a first sub-condition and a second sub-condition; in response to the candidate parameters satisfying the preset condition, the candidate parameters are determined as the final parameters of the model, and the final model after training is output, including: in response to the candidate parameters satisfying the first sub-condition and the second sub-condition at the same time, the candidate parameters are determined as the final parameters of the model, and the final model after training is output, wherein the first sub-condition is used to characterize the setting of the set of optimal parameters of the adversarial parties based on the principle of extreme values ​​of both parties, and the second sub-condition is: the loss value corresponding to the set of optimal parameters of the adversarial parties is equal to the unique solution of the first equation.

[0008] In some embodiments, the problem is a saddle point problem.

[0009] In some embodiments, the first equation is constructed based on nonlinear control theory; the first equation is the Hamilton-Jacobi-Isaac equation or the Hamilton-Jacobi-Bellman equation.

[0010] In a second aspect, an embodiment of the present disclosure provides a method for generating information, comprising: obtaining state information of an object at each time point, each time point being each time point when the object is in a continuous state; inputting the state information of the object at each time point into a pre-trained information generation model to generate action information of the object, wherein the information generation model is trained by a method such as any embodiment of the above-mentioned method for training a model.

[0011] In a third aspect, an embodiment of the present disclosure provides a device for training a model, comprising: an acquisition unit, configured to acquire a training sample set, wherein the training samples in the training sample set include state information of the object at each time point and action information of the object; a training unit, configured to perform the following training steps: using the state information of the object at each time point included in the training sample as model input, and using the action information of the object corresponding to the input state information of the object at each time point as model output, and solving the candidate parameters of the model based on the loss function and the objective function, wherein the loss function is constructed based on an adversarial learning algorithm, and the candidate parameters are a set of optimal parameters of the adversarial parties obtained using the objective function; in response to the candidate parameters satisfying preset conditions, determining the candidate parameters as final parameters of the model, and outputting the final model after training, wherein the preset conditions are used to characterize the setting of the set of optimal parameters of the adversarial parties based on the extreme value principle of both parties; a repetition unit, configured to repeat the above training steps in response to the candidate parameters not satisfying the preset conditions.

[0012] In some embodiments, each time point is a time point in a continuous state of the object, and the model is used to solve the optimal control problem of the object in a continuous state; and the loss function is constructed based on a combination of an adversarial learning algorithm and a reinforcement learning algorithm.

[0013] In some embodiments, the training model is further configured to determine the candidate parameters as the final parameters of the model in response to the candidate parameters satisfying preset conditions, and output the final model after training, wherein the preset conditions are: the loss value corresponding to the set of optimal parameters of the adversarial parties is equal to the unique solution of the first equation, and the solution of the first equation is used to characterize the solution to the problem existing in the model training.

[0014] In some embodiments, the preset condition includes a first sub-condition and a second sub-condition; the training model is further configured to determine the candidate parameter as the final parameter of the model in response to the candidate parameter satisfying the first sub-condition and the second sub-condition at the same time, and output the final model after training, wherein the first sub-condition is used to characterize the setting of the set of optimal parameters of the adversarial parties based on the principle of extreme values ​​of both parties, and the second sub-condition is: the loss value corresponding to the set of optimal parameters of the adversarial parties is equal to the unique solution of the first equation.

[0015] In some embodiments, the problem in the training unit is a saddle point problem.

[0016] In some embodiments, the first equation in the training unit is constructed based on nonlinear control theory; the first equation is the Hamilton-Jacobi-Isaac equation or the Hamilton-Jacobi-Bellman equation.

[0017] In a fourth aspect, the disclosed embodiment provides a device for generating information, comprising: an information acquisition unit, configured to acquire state information of an object at each time point, each time point being each time point when the object is in a continuous state; an information generation unit, configured to input the state information of the object at each time point into a pre-trained information generation model to generate action information of the object, wherein the information generation model is trained by a method such as any embodiment of the method for training a model described above.

[0018] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner in the first aspect or the second aspect.

[0019] In a sixth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute a method described in any implementation of the first aspect or the second aspect.

[0020] The method and device for training a model provided by the embodiments of the present disclosure adopt the method of obtaining a training sample set, wherein the training samples in the training sample set include state information of the object at each time point and action information of the object, and perform the following training steps: using the state information of the object at each time point included in the training sample as the model input, and using the action information of the object corresponding to the state information of the input object at each time point as the model output, and solving the candidate parameters of the model based on the loss function and the objective function, wherein the loss function is constructed based on the adversarial learning algorithm, and the candidate parameters are the set of optimal parameters of the adversarial parties obtained by using the objective function. In response to the candidate parameters satisfying preset conditions, the candidate parameters are determined as the final parameters of the model, and the final model after training is output, wherein the preset conditions are used to characterize the setting of the set of optimal parameters of the adversarial parties based on the principle of extreme values ​​of both parties. In response to the candidate parameters not satisfying the preset conditions, the above training steps are repeatedly performed, and by judging the necessary and sufficient conditions for the optimality of the model, the reinforcement learning training is guided in turn, and then the principle of extreme values ​​of both parties in the optimal control theory is generalized to the model training, so as to realize a new model training method and device.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a schematic diagram of a first embodiment of a method for training a model according to the present disclosure;

[0024] Figure 2 is a scene graph that can implement the method for training a model in an embodiment of the present disclosure;

[0025] Figure 3 is a schematic diagram of a second embodiment of a method for training a model according to the present disclosure;

[0026] Figure 4 is a schematic diagram of a first embodiment of a method for generating information according to the present disclosure;

[0027] Figure 5 is a structural schematic diagram of an embodiment of an apparatus for training a model according to the present disclosure;

[0028] Figure 6 is a structural schematic diagram of an embodiment of a device for generating information according to the present disclosure;

[0029] Figure 7 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0032] Figure 1 A schematic diagram 100 of a first embodiment of a method for training a model according to the present disclosure is shown. The method for training a model comprises the following steps:

[0033] Step 101: Obtain a training sample set.

[0034] In this embodiment, the executor of the method for training the model can obtain a training sample set from other electronic devices or locally through a wired connection or a wireless connection, wherein the training samples in the training sample set include state information of the object at each time point and action information of the object, and the state of the object can be the motion state of a vehicle in autonomous driving, the motion state of a game character's limbs, and various possible states of the object.

[0035] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G, 4G, 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other wireless connection methods currently known or to be developed in the future.

[0036] Step 102, perform the following training steps.

[0037] Step 1021, using the state information of the object at each time point included in the training sample as the model input, using the action information of the object corresponding to the input state information of the object at each time point as the model output, and solving the candidate parameters of the model based on the loss function and the objective function.

[0038] In this embodiment, the execution entity can use a deep learning algorithm, take the state information of the object at each time point included in the training samples in the training sample set obtained in step 101 as input, and take the action information of the object corresponding to the state information of the input object at each time point as output, and solve the candidate parameters of the model based on the loss function and the objective function, wherein the loss function is constructed based on the adversarial learning algorithm, and the candidate parameters are a set of optimal parameters of the adversarial parties obtained by using the objective function.

[0039] Step 1022, in response to the candidate parameters satisfying the preset conditions, the candidate parameters are determined as the final parameters of the model, and the trained final model is output.

[0040] In this embodiment, the execution subject can judge the candidate parameters of the model according to the preset conditions. When the candidate parameters of the model meet the preset conditions, the candidate parameters are determined as the final parameters of the model, and the final model after training is output, thereby completing the training of the model. Among them, the preset conditions are used to characterize the setting of the set of optimal parameters of the two opposing parties based on the extreme value principle of both parties. The setting of the conditions can be based on a threshold or an analysis, comparison or calculation with the previous optimal parameters to set the set of optimal parameters of the two opposing parties, or it can be based on various existing technologies or theories for solving problems in the model to set the set of optimal parameters of the two opposing parties. The condition can be a limiting condition for limiting the set of optimal parameters, or it can be a combination of multiple conditions for limiting the set of optimal parameters. The preset conditions can classify and limit the set of optimal parameters according to different business needs.

[0041] In some optional implementations of this embodiment, in response to the candidate parameters satisfying the preset conditions, the candidate parameters are determined as the final parameters of the model, and the final model after training is output, including: in response to the candidate parameters satisfying the preset conditions, the candidate parameters are determined as the final parameters of the model, and the final model after training is output, wherein the preset conditions are: the loss value corresponding to the set of optimal parameters of the adversarial parties is equal to the unique solution of the first equation, and the solution of the first equation is used to characterize the problem existing in the model training, the problem can be a saddle point problem, the first equation can be constructed based on nonlinear control theory, and the first equation is the Hamilton-Jacobi-Isaac equation (i.e., the HJI equation) or the Hamilton-Jacobi-Bellman equation (i.e., the HJB equation). It solves the situation that various problems (such as saddle point problems) caused by adversarial neural networks cannot be explained in the existing model, and the problems in the existing model are optimized by setting the necessary conditions for the optimality of the model to achieve the optimality of the model.

[0042] Step 103: In response to the candidate parameters not satisfying the preset conditions, the above training steps are repeated.

[0043] In this embodiment, when the execution subject determines that the candidate parameters do not meet the preset conditions, it jumps to step 102 and repeats the above step 102.

[0044] It should be noted that the training process of the above model is a well-known technology that is currently widely studied and applied, and will not be elaborated here.

[0045] Continue to see Figure 2, the method 200 for training a model in this embodiment runs in a server 201. The server 201 first obtains a training sample set 202, wherein the training samples in the training sample set include state information of the object at each time point and action information of the object, and then the server 201 performs the following training steps: the state information of the object at each time point included in the training sample is used as the model input, and the action information of the object corresponding to the state information of the input object at each time point is used as the model output, and the candidate parameters of the model are obtained based on the loss function and the objective function, wherein the loss function is constructed based on the adversarial learning algorithm, and the candidate parameters are the set of optimal parameters of the adversarial parties obtained by using the objective function, and in response to the candidate parameters satisfying the preset conditions, the candidate parameters are determined as the final parameters of the model, and the final model 203 after training is output, wherein the preset conditions are used to characterize the setting of the set of optimal parameters of the adversarial parties based on the extreme value principle of both parties, and in response to the candidate parameters not satisfying the preset conditions, the above training steps are repeated.

[0046] The method for training a model provided by the above-mentioned embodiment of the present disclosure adopts the method of obtaining a training sample set, wherein the training samples in the training sample set include state information of the object at each time point and action information of the object, and performing the following training steps: using the state information of the object at each time point included in the training sample as the model input, and using the action information of the object corresponding to the state information of the input object at each time point as the model output, and solving the candidate parameters of the model based on the loss function and the objective function, wherein the loss function is constructed based on the adversarial learning algorithm, and the candidate parameters are a set of optimal parameters of the adversarial parties obtained by using the objective function. In response to the candidate parameters satisfying the preset conditions, the candidate parameters are determined as the final parameters of the model, and the final model after training is output, wherein the preset conditions are used to characterize the setting of the set of optimal parameters of the adversarial parties based on the principle of the extreme value of both parties. In response to the candidate parameters not satisfying the preset conditions, the above-mentioned training steps are repeatedly performed, and by judging the necessary and sufficient conditions for the optimality of the model, the reinforcement learning training is guided in turn, and then the principle of the extreme value of both parties in the optimal control theory is generalized to the model training, so as to realize a new model training method.

[0047] Further references Figure 3 , which shows a schematic diagram 300 of a second embodiment of a method for training a model. The process of the method includes the following steps:

[0048] Step 301: Obtain a training sample set.

[0049] In this embodiment, the executor of the method for training the model can obtain a training sample set from other electronic devices or locally through a wired connection or a wireless connection, wherein the training samples in the training sample set include state information of the object at each time point and action information of the object, each time point is a time point in a continuous state of the object, and the continuous state of the object can be the continuous motion state of a vehicle in autonomous driving, the continuous motion state of a game character's limbs, and various possible continuous states of the object.

[0050] Step 302, perform the following training steps.

[0051] Step 3021, using the state information of the object at each time point included in the training sample as the model input, using the action information of the object corresponding to the input state information of the object at each time point as the model output, and solving the candidate parameters of the model based on the loss function and the objective function.

[0052] In this embodiment, the execution subject can use a deep learning algorithm, take the state information of the object at each time point included in the training sample set in step 301 as input, take the action information of the object corresponding to the state information of the input object at each time point as output, and solve the candidate parameters of the model based on the loss function and the objective function. Among them, the model is used to solve the optimal control problem of the object in a continuous state, the loss function is constructed based on the combination of the adversarial learning algorithm and the reinforcement learning algorithm, the model parameters of the adversarial parties in the loss function are the parameter functions of the object in the continuous state about time, and the training goal of the model is to find the set of optimal parameters of the adversarial parties, that is, the candidate parameters are the set of optimal parameters of the adversarial parties obtained by using the objective function.

[0053] To further illustrate, the trained model is an adversarial reinforcement learning model that combines reinforcement learning and adversarial learning. There are an original model Z and an adversarial model D. The loss function is composed of the states and parameters x in the two models. z ,θ d ,θ z ,θ d The model parameters of the adversarial parties are the parameter functions θ of the objects in the continuous state with respect to time t. z (t),θ d (t). Adversarial reinforcement learning is a game problem between two dynamic systems, namely quantitative differential game, and the loss function of the model is: in L represents the penalty for each state and action in training, Φ represents the termination penalty for the last state, and their joint distribution follows (x0, y0) ~ μ. The training goal is to find the optimal Make

[0054] Step 3022, in response to the candidate parameters satisfying the preset conditions, the candidate parameters are determined as the final parameters of the model, and the trained final model is output.

[0055] In this embodiment, the preset condition may include a first sub-condition and a second sub-condition. The execution subject may judge the candidate parameters of the model according to the first sub-condition and the second sub-condition. When the candidate parameters of the model satisfy the first sub-condition and the second sub-condition at the same time, the candidate parameters are determined as the final parameters of the model, and the final model after training is output, thereby completing the training of the model. Among them, the first sub-condition is used to characterize the setting of the set of optimal parameters of the two adversaries based on the principle of the extreme value of both parties, and the second sub-condition is: the loss value corresponding to the set of optimal parameters of the two adversaries is equal to the unique solution of the first equation. The solution of the first equation is used to characterize the solution of the saddle point problem existing in the model training, and the first equation is the HJI equation. Both the first sub-condition and the second sub-condition can be a limiting condition for limiting the set of optimal parameters, or a combination of multiple conditions for limiting the set of optimal parameters.

[0056] The method of optimal control of neural networks generally uses mean field optimal control to give the necessary and sufficient conditions for the convergence of neural networks and analyzes the mean field optimal control method of generalization error and the conditions based on the convergence of neural networks, and the successive approximation method that solves partial differential equations to make different parameters meet the convergence conditions one by one.

[0057] Based on the above optimal control neural network scenario, further examples are given to illustrate the setting of the first condition in the preset conditions. First, define the Hamiltonian H: = -L + ψ T f, based on the generalization of the traditional two-way extreme value principle in the mean field sense, for the model in the above example, the preset conditions include: assuming that f is bounded, f and L are bounded about θ z and θ d Continuous, f, L and Φ are continuously differentiable with respect to x, μ has compact support, let is the optimal parameter of the model, x * (t) is the corresponding optimal state, then there exists ψ * and ξ such that,

[0058]

[0059]

[0060]

[0061] and,

[0062]

[0063] where H(x(t),θz (t),θ d (t),ψ(t)):=-L(x(t),θ z (t),θ d (t))+ψ T (t)f(x(t),θ z (t),θ d (t)).

[0064] Therefore, any subsequent sub-strategy in the optimal strategy is also optimal, that is, the strategy starting from a certain middle moment to the end moment corresponds to the control problem from this middle moment to the end moment. This is the principle of dynamic programming.

[0065] Based on this principle, for the setting of the second condition in the preset conditions, consider starting from any intermediate time s to the end time t f The control problem and its corresponding optimal loss function v(s,μ) are assumed to be bounded, f, L and Φ are Liebschitz continuous with respect to x, and the Liebschitz constants of f and L are related to θ z and θ d No relation. is the optimal parameter of the model, then its corresponding loss function and v is the unique solution of the mean field HJI equation,

[0066] The second condition represents the generalization of the traditional HJI equation in the mean field sense.

[0067] In the above example, the model can give a corresponding optimal solution starting from any initial value, so the quantitative differential game problem in the sense of mean field is studied, which is different from the traditional quantitative differential game problem that only studies the optimality conditions of the dynamic system for determining the initial value.

[0068] It should be noted that the above-mentioned preset conditions, first conditions and second conditions are not limited to those listed above.

[0069] Step 303: In response to the candidate parameters not satisfying the preset conditions, the above training steps are repeated.

[0070] In this embodiment, when the execution subject determines that the candidate parameters do not satisfy the preset conditions (ie, the candidate parameters do not satisfy the first sub-condition and the second sub-condition at the same time), the process jumps to step 302 and repeats the above step 302.

[0071] In some optional implementations of this embodiment, the first equation may be the Hamilton-Jacobi-Bellman equation, ie, the HJB equation. Another training method for an adversarial reinforcement learning model based on optimal control theory is implemented.

[0072] from Figure 3 It can be seen that Figure 1 Compared with the corresponding embodiment, the schematic diagram 300 of the method for training the model in this embodiment converts the model parameters in the prior art from the parameters in each time step into a function about time in a continuous sense. The optimal control problem is to find the set of optimal parameters of the adversarial parties with the minimum loss function value, that is, to replace the optimization problem of discrete time steps with the mean field optimal control problem of the continuous power system, which solves the problem that the prior art can only use a single power system for optimal control reinforcement learning, and cannot explain the saddle point problem caused by the introduction of adversarial neural networks. By satisfying the two-party extreme value principle and HJI equation in the mean field sense during modeling, the optimality of the model has been achieved, and a new training method for adversarial reinforcement learning models based on optimal control theory has been realized. At the same time, it avoids the problem that each time step in the prior art contains a deep neural network and a large number of parameters need to be learned, simplifies the training process of the model, and improves the training efficiency of the model. The performance of existing adversarial reinforcement learning technology in specific application scenarios (such as autonomous driving, etc.). Optimizing the model based on preset conditions can further optimize a series of problems in scenarios such as autonomous driving.

[0073] Further references Figure 4 , which shows a schematic diagram 400 of a first embodiment of a method for generating information according to the present disclosure. The method for generating information comprises the following steps:

[0074] Step 401, obtaining state information of the object at each time point.

[0075] In this embodiment, the execution entity (such as a server or terminal device) can obtain the state information of the object at each time point from other electronic devices or locally through a wired connection or a wireless connection, and each time point is a time point when the object is in a continuous state.

[0076] Step 402: input the state information of the object at each time point into a pre-trained information generation model to generate the object's motion information.

[0077] In this embodiment, the execution subject can input the state information of the object at each time point obtained in step 401 into a pre-trained information generation model, and generate the object's action information through model prediction. The information generation model is trained by the method of any embodiment of the above-mentioned method for training a model.

[0078] from Figure 4 It can be seen that Figure 1Compared with the corresponding embodiment, the process 400 of the method for generating information in this embodiment highlights the step of using the trained information generation model to generate the action information of the object. Therefore, the solution described in this embodiment can achieve more efficient and accurate information prediction.

[0079] Further references Figure 5 , as a response to the above Figures 1 to 3 The present disclosure provides an embodiment of a device for training a model, and the device embodiment is Figure 1 Corresponding to the method embodiment shown in FIG. 1 , in addition to the features described below, the device embodiment may also include Figure 1 The same or corresponding features of the method embodiment shown, and the generation of Figure 1 The method embodiment shown has the same or corresponding effects, and the device can be specifically applied to various electronic devices.

[0080] like Figure 5 As shown, the device 500 for training a model in this embodiment includes: an acquisition unit 501, a training unit 502 and a repetition unit 503, wherein the acquisition unit is configured to acquire a training sample set, wherein the training samples in the training sample set include state information of the object at each time point and action information of the object; the training unit is configured to perform the following training steps: using the state information of the object at each time point included in the training sample as the model input, using the action information of the object corresponding to the input state information of the object at each time point as the model output, and solving the candidate parameters of the model based on the loss function and the objective function, wherein the loss function is constructed based on the adversarial learning algorithm, and the candidate parameters are a set of optimal parameters of the adversarial parties obtained by using the objective function; in response to the candidate parameters satisfying the preset conditions, determining the candidate parameters as the final parameters of the model, and outputting the final model after training, wherein the preset conditions are used to characterize the setting of the set of optimal parameters of the adversarial parties based on the extreme value principle of both parties; the repetition unit is configured to repeat the above training steps in response to the candidate parameters not satisfying the preset conditions.

[0081] In this embodiment, the specific processing of the acquisition unit 501, the training unit 502 and the repeating unit 503 of the device 500 for training the model and the technical effects thereof can be referred to respectively. Figure 1 The relevant descriptions of steps 101 to 103 in the corresponding embodiment are not repeated here.

[0082] In some optional implementations of this embodiment, each time point is a time point in a continuous state of the object, and the model is used to solve the optimal control problem of the object in a continuous state; and the loss function is constructed based on a combination of an adversarial learning algorithm and a reinforcement learning algorithm.

[0083] In some optional implementations of this embodiment, the training model is further configured to determine the candidate parameters as the final parameters of the model in response to the candidate parameters satisfying preset conditions, and output the final model after training, wherein the preset conditions are: the loss value corresponding to the set of optimal parameters of the adversarial parties is equal to the unique solution of the first equation, and the solution of the first equation is used to characterize the solution to the problem existing in the model training.

[0084] In some optional implementations of this embodiment, the preset condition includes a first sub-condition and a second sub-condition; the training model is further configured to determine the candidate parameter as the final parameter of the model in response to the candidate parameter satisfying the first sub-condition and the second sub-condition at the same time, and output the final model after training, wherein the first sub-condition is used to characterize the setting of the set of optimal parameters of the adversarial parties based on the principle of extreme values ​​of both parties, and the second sub-condition is: the loss value corresponding to the set of optimal parameters of the adversarial parties is equal to the unique solution of the first equation.

[0085] In some optional implementations of this embodiment, the problem in the training unit is a saddle point problem.

[0086] In some optional implementations of this embodiment, the first equation in the training unit is constructed based on nonlinear control theory; the first equation is the Hamilton-Jacobi-Isaac equation or the Hamilton-Jacobi-Bellman equation.

[0087] Continue to refer Figure 6 , as a response to the above Figure 4 The present disclosure provides an embodiment of a device for generating information. Figure 4 Corresponding to the method embodiment shown in FIG. 1 , in addition to the features described below, the device embodiment may also include Figure 4 The same or corresponding features of the method embodiment shown, and the generation of Figure 4 The method embodiment shown has the same or corresponding effects, and the device can be specifically applied to various electronic devices.

[0088] like Figure 6 As shown, the device 600 for generating information in this embodiment includes: an information acquisition unit 601 and an information generation unit 602, wherein the information acquisition unit is configured to acquire state information of the object at each time point, and each time point is each time point when the object is in a continuous state; the information generation unit is configured to input the state information of the object at each time point into a pre-trained information generation model to generate action information of the object, wherein the information generation model is trained by a method such as any embodiment of the method for training a model described above.

[0089] In this embodiment, the specific processing of the information acquisition unit 601 and the information generation unit 602 of the device for generating information 600 and the technical effects thereof can be referred to in Figure 4 The relevant descriptions of step 401 to step 402 in the corresponding embodiment are not repeated here.

[0090] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0091] like Figure 7 , is a block diagram of an electronic device for a method for training a model according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0092] like Figure 7 As shown, the electronic device includes: one or more processors 701, a memory 702, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 701 is taken as an example.

[0093] The memory 702 is a non-transitory computer-readable storage medium provided by the present disclosure. The memory stores instructions executable by at least one processor to enable at least one processor to perform the method for training a model provided by the present disclosure. The non-transitory computer-readable storage medium of the present disclosure stores computer instructions, which are used to enable a computer to perform the method for training a model provided by the present disclosure.

[0094] The memory 702 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the method for training a model in the embodiment of the present disclosure (for example, the attached Figure 5 The processor 701 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 702, that is, implements the method for training the model in the above method embodiment.

[0095] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device for training the model, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the electronic device for training the model via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0096] The electronic device for the method of training a model may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0097] The input device 703 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device for training the model, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 704 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0098] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0100] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0101] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0102] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0103] According to the technical solution of the embodiment of the present disclosure, a training sample set is obtained, wherein the training samples in the training sample set include state information of the object at each time point and action information of the object, and the following training steps are performed: the state information of the object at each time point included in the training sample is used as the model input, and the action information of the object corresponding to the state information of the input object at each time point is used as the model output, and the candidate parameters of the model are solved based on the loss function and the objective function, wherein the loss function is constructed based on the adversarial learning algorithm, and the candidate parameters are the set of optimal parameters of the adversarial parties obtained by using the objective function. In response to the candidate parameters satisfying the preset conditions, the candidate parameters are determined as the final parameters of the model, and the final model after training is output, wherein the preset conditions are used to characterize the setting of the set of optimal parameters of the adversarial parties based on the principle of the extreme value of both parties. In response to the candidate parameters not satisfying the preset conditions, the above training steps are repeated, and by judging the necessary and sufficient conditions for the optimality of the model, the reinforcement learning training is guided in turn, and then the principle of the extreme value of both parties in the optimal control theory is generalized to the model training, so as to realize a new model training method and device.

[0104] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0105] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a model, comprising: Acquire a training sample set, wherein the training samples in the training sample set include state information of an object at each time point and action information of the object, wherein each time point is a time point in a continuous state of the object, and the continuous state of the object is a continuous motion state of a vehicle in an automatic driving, or a continuous motion state of a limb of a game character; The following training steps are performed: the state information of the object at each time point included in the training sample is used as the model input, the action information of the object corresponding to the input state information of the object at each time point is used as the model output, and the candidate parameters of the model are obtained based on the loss function and the objective function, wherein the loss function is constructed based on the combination of the adversarial learning algorithm and the reinforcement learning algorithm, and the candidate parameters are the set of optimal parameters of the adversarial parties obtained by using the objective function; in response to the candidate parameters satisfying the preset conditions, the candidate parameters are determined as the final parameters of the model, and the final model after training is output, wherein the preset conditions are used to characterize the setting of the set of optimal parameters of the adversarial parties based on the extreme value principle of both parties; In response to the candidate parameters not satisfying the preset condition, the above training steps are repeated.

2. The method according to claim 1, wherein: The model is used to solve the optimal control problem of the object in a continuous state.

3. The method according to any one of claims 1 or 2, wherein: In response to the candidate parameters satisfying a preset condition, determining the candidate parameters as final parameters of the model, and outputting a final model after training, comprising: In response to the candidate parameters satisfying preset conditions, the candidate parameters are determined as final parameters of the model, and the trained final model is output, wherein the preset conditions are: the loss value corresponding to the set of optimal parameters of the adversarial parties is equal to the unique solution of the first equation, and the solution of the first equation is used to characterize the solution to the problem existing in the model training.

4. The method according to claim 3, wherein: The preset condition includes a first sub-condition and a second sub-condition; in response to the candidate parameter satisfying the preset condition, determining the candidate parameter as the final parameter of the model, and outputting the final model after training, including: In response to the candidate parameters satisfying both the first sub-condition and the second sub-condition, the candidate parameters are determined as the final parameters of the model, and the trained final model is output, wherein the first sub-condition is used to characterize the setting of the set of optimal parameters of the two adversarial parties based on the principle of extreme values ​​of both parties, and the second sub-condition is: the loss value corresponding to the set of optimal parameters of the two adversarial parties is equal to the unique solution of the first equation.

5. The method according to claim 3, wherein: The problem described is a saddle point problem.

6. The method according to any one of claims 3 or 4, wherein: The first equation is constructed based on nonlinear control theory; the first equation is the Hamilton-Jacobi-Isaac equation or the Hamilton-Jacobi-Bellman equation.

7. A method for generating information, comprising: Acquire state information of an object at each time point, where each time point is each time point when the object is in a continuous state; The state information of the object at each time point is input into a pre-trained information generation model to generate the action information of the object, wherein the information generation model is trained by the method according to any one of claims 1-6.

8. A device for training a model, comprising: an acquisition unit configured to acquire a training sample set, wherein the training samples in the training sample set include state information of an object at each time point and action information of the object, wherein each time point is each time point in a continuous state of the object, and the continuous state of the object is a continuous motion state of a vehicle in an automatic driving, or a continuous motion state of a limb of a game character; The training unit is configured to perform the following training steps: using the state information of the object at each time point included in the training sample as the model input, using the action information of the object corresponding to the input state information of the object at each time point as the model output, and solving the candidate parameters of the model based on the loss function and the objective function, wherein the loss function is constructed based on the combination of the adversarial learning algorithm and the reinforcement learning algorithm, and the candidate parameters are the set of optimal parameters of the adversarial parties obtained by using the objective function; in response to the candidate parameters satisfying the preset conditions, determining the candidate parameters as the final parameters of the model, and outputting the final model after training, wherein the preset conditions are used to characterize the setting of the set of optimal parameters of the adversarial parties based on the extreme value principle of both parties; The repeating unit is configured to repeatedly execute the above training steps in response to the candidate parameters not satisfying the preset conditions.

9. The device according to claim 8, wherein: The training model is further configured to determine the candidate parameters as the final parameters of the model in response to the candidate parameters satisfying preset conditions, and output the final model after training, wherein the preset conditions are: the loss value corresponding to the set of optimal parameters of the adversarial parties is equal to the unique solution of the first equation, and the solution of the first equation is used to characterize the solution to the problem existing in the model training.

10. A device for generating information, comprising: An information acquisition unit is configured to acquire state information of the object at each time point, wherein each time point is each time point when the object is in a continuous state; The information generation unit is configured to input the state information of the object at each time point into a pre-trained information generation model to generate action information of the object, wherein the information generation model is trained by the method according to one of claims 1-6.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6 or the method of claim 7.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method of any one of claims 1 to 6 or the method of claim 7.

Images