Reinforcement learning method, device and system based on distributed system and storage medium

Abstract

The application discloses a reinforcement learning method, device and system based on a distributed system and a storage medium. The distributed system comprises a training end, an evaluation end, a database and multiple learning ends. The method comprises the following steps: based on the multiple learning ends, multiple training samples are processed respectively to generate multiple sample data which is broadcast to the database respectively; after the training end receives the multiple sample data broadcast by the database, the training end performs reinforcement learning on the multiple sample data to generate training parameters which are broadcast to the database; after the evaluation end receives the training parameters broadcast by the database, the evaluation end evaluates the training parameters, and the reinforcement learning of the training end is completed after the evaluation of the training parameters is passed. The distributed system is used for reinforcement learning, parallel scheduling and processing of various resources are realized by the database, a large amount of sample data can be quickly generated for learning and training, the result obtained through learning and training is accurate, and the efficiency of reinforcement learning is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, in particular to a reinforcement learning method, device and system based on a distributed system and a storage medium. BACKGROUND

[0002] Reinforcement learning is an important machine learning method for solving the problem of how to make the best decision, and is widely used in the fields of robot control, game theory and automatic driving.

[0003] Current reinforcement learning usually generates sample data for training through a single computer or other terminal device, but is limited by the memory resources and processing speed of the single terminal device, and the number of generated sample data is limited and the rate is slow. The amount of sample data is related to the accuracy of the reinforcement learning result, and the less the sample data, the less the sample data for training, and the lower the accuracy of the result. The speed is related to the efficiency of reinforcement learning, and the slower the rate, the slower the convergence speed and the lower the learning efficiency. Therefore, the low accuracy and low training efficiency of current reinforcement learning are technical problems to be solved. SUMMARY

[0004] The main purpose of the present application is to provide a reinforcement learning method, device and system based on a distributed system and a storage medium, which aims to solve the technical problems of low accuracy and low training efficiency of existing reinforcement learning.

[0005] To achieve the above purpose, the present application provides a reinforcement learning method based on a distributed system, which comprises a training end, an evaluation end, a database and a plurality of learning ends. The reinforcement learning method based on the distributed system comprises the following steps:

[0006] Based on the processing of the training sample by the plurality of learning ends, a plurality of sample data are generated, and the plurality of sample data are broadcast to the database respectively;

[0007] After the training end receives the plurality of sample data broadcast by the database, the training end performs reinforcement learning on the plurality of sample data, generates training parameters and broadcasts them to the database;

[0008] After the evaluation end receives the training parameters broadcast by the database, the evaluation end evaluates the training parameters, and completes the reinforcement learning of the training end after the evaluation of the training parameters is passed.

[0009] Optionally, the step of evaluating the training parameters by the evaluation end comprises:

[0010] The evaluation end calculates the training parameters based on a preset evaluation strategy, generates a return value, and determines whether the return value is greater than a preset threshold.

[0011] If the return value is greater than the preset threshold, it is determined that the evaluation of the training parameters is passed.

[0012] Optionally, the step of determining whether the return value is greater than the preset threshold comprises:

[0013] If the return value is less than or equal to the preset threshold, the plurality of learning ends perform the steps of processing the training samples respectively based on the training parameters to generate a plurality of sample data.

[0014] Optionally, the number of training ends is a plurality, and the step of the evaluation end calculating the training parameters based on a preset evaluation strategy to generate a return value comprises:

[0015] The evaluation end calculates the training parameters corresponding to the plurality of training ends based on a preset evaluation strategy to generate intermediate return values corresponding to the plurality of training ends.

[0016] Compare the plurality of intermediate return values to determine the intermediate return value with the largest value among the intermediate return values, and take the intermediate return value with the largest value as the return value.

[0017] Optionally, the step of generating training parameters and broadcasting them to the database comprises:

[0018] The database broadcasts the training parameters to the evaluation end and the plurality of learning ends based on a message queue mode.

[0019] Optionally, the step of the database broadcasting the training parameters to the evaluation end and the plurality of learning ends based on a message queue mode comprises:

[0020] The database determines whether the registration state of the evaluation end is valid based on the registration state of the evaluation end.

[0021] If the registration state of the evaluation end is valid, the training parameters are broadcast to the evaluation end based on a message queue mode.

[0022] The database determines the target learning end among the plurality of learning ends based on the registration state of the plurality of learning ends, and broadcasts the training parameters to each of the target learning ends based on a message queue mode.

[0023] Optionally, the step of generating training parameters and broadcasting them to the database comprises:

[0024] The training end broadcasts the training parameters to the database based on a message queue mode after generating the training parameters.

[0025] Optionally, the step of processing the training samples by the plurality of learning ends respectively comprises:

[0026] The plurality of learning ends respectively register to the distributed system based on the learning registration threads.

[0027] The evaluation end registers to the distributed system based on the evaluation registration thread.

[0028] Optionally, the step of broadcasting the plurality of sample data to the database respectively comprises:

[0029] The plurality of learning ends broadcast the plurality of sample data to the database based on a message subscription mode.

[0030] Optionally, the step of broadcasting the plurality of sample data to the database respectively comprises:

[0031] The database broadcasts the plurality of sample data to the training end based on a message subscription mode.

[0032] Further, to achieve the above-mentioned purpose, the application further provides a reinforcement learning device based on a distributed system, which comprises:

[0033] A learning module is configured to process training samples by a plurality of learning ends respectively, generate a plurality of sample data, and broadcast the plurality of sample data to the database respectively.

[0034] A training module is configured to perform reinforcement learning on the plurality of sample data after the training end receives the plurality of sample data broadcast by the database, and generate training parameters and broadcast the training parameters to the database.

[0035] An evaluation module is configured to evaluate the training parameters after the evaluation end receives the training parameters broadcast by the database, and complete the reinforcement learning of the training end after the evaluation of the training parameters is passed.

[0036] Optionally, the evaluation module further comprises:

[0037] An evaluation unit is configured to calculate the training parameters based on a preset evaluation strategy, generate a reward value, and determine whether the reward value is greater than a preset threshold.

[0038] A determination module is configured to determine that the evaluation of the training parameter is passed if the return value is greater than a preset threshold.

[0039] Optionally, the evaluation module further comprises:

[0040] An execution unit is configured to execute a step of processing training samples respectively based on the training parameter to generate multiple pieces of sample data if the return value is less than or equal to a preset threshold.

[0041] Optionally, the number of the training ends is multiple, and the evaluation unit is further configured to:

[0042] calculate, based on a preset evaluation strategy, training parameters respectively corresponding to the multiple training ends to generate intermediate return values respectively corresponding to the multiple training ends;

[0043] compare the multiple intermediate return values to determine an intermediate return value with the largest value among the multiple intermediate return values, and take the intermediate return value with the largest value as the return value.

[0044] Optionally, the reinforcement learning device based on a distributed system further comprises:

[0045] A broadcast module is configured to broadcast the training parameter to the evaluation end and the multiple learning ends respectively based on a message queue mode.

[0046] Optionally, the broadcast module further comprises:

[0047] A judgment unit is configured to judge whether a registration state of the evaluation end is valid based on the registration state of the evaluation end.

[0048] A broadcast unit is configured to broadcast the training parameter to the evaluation end based on a message queue mode if the registration state of the evaluation end is valid.

[0049] A determination unit is configured to determine a target learning end in the multiple learning ends based on registration states of the multiple learning ends, and broadcast the training parameter to each of the target learning ends based on a message queue mode.

[0050] Optionally, the training module further comprises:

[0051] After the training parameter is generated, the training parameter is broadcast to the database based on a message queue mode.

[0052] Optionally, the reinforcement learning device based on a distributed system further comprises:

[0053] A starting module is configured to start a learning registration thread for each of the multiple learning ends, and register each of the learning ends to the distributed system based on each of the learning registration threads.

[0054] The registration module is configured to start an evaluation registration thread based on the evaluation end, and register the evaluation end to the distributed system based on each evaluation registration thread.

[0055] Further, to achieve the above object, the present application also provides a distributed system, which comprises a training end, an evaluation end, a database and a plurality of learning ends, and further comprises a memory, a processor and a distributed system-based reinforcement learning program stored in the memory and executable on the processor, wherein the distributed system-based reinforcement learning program implements the steps of the distributed system-based reinforcement learning method when executed by the processor.

[0056] Further, to achieve the above object, the present application also provides a storage medium, which stores a distributed system-based reinforcement learning program, wherein the distributed system-based reinforcement learning program implements the steps of the distributed system-based reinforcement learning method when executed by a processor.

[0057] The distributed system-based reinforcement learning method, device, system and storage medium of the present application are based on a distributed system comprising a training end, an evaluation end, a database and a plurality of learning ends. Each learning end processes a training sample to generate a plurality of sample data, which is broadcast to the database for storage and then broadcast to the training end by the database. After receiving the plurality of sample data, the training end performs reinforcement learning on the plurality of sample data to generate training parameters, which is broadcast to the database for storage and then broadcast to the evaluation end by the database. After receiving the training parameters, the evaluation end evaluates the training parameters to determine the training effect of the training end. If the evaluation of the training parameters is passed, it indicates that the training has achieved good results, and the reinforcement learning of the training end is completed. The present application performs reinforcement learning through a distributed system, and the database in the system realizes parallel scheduling and processing of various resources, which can quickly generate a large amount of sample data for learning and training. The result obtained through learning and training is accurate, and the convergence speed is accelerated, thereby improving the efficiency of reinforcement learning. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The structure diagram of the hardware running environment involved in the distributed system embodiment of the present application;

[0059] Figure 2 The flowchart of the first embodiment of the distributed system-based reinforcement learning method of the present application.

[0060] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0061] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0062] This invention provides a distributed system, with reference to Figure 1 , Figure 1 This is a schematic diagram of the hardware operating environment involved in the distributed system embodiment of the present invention.

[0063] like Figure 1 As shown, the distributed system may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or stable non-volatile memory, such as disk storage. Optionally, the memory 1005 may also be a distributed storage system independent of the aforementioned processor 1001.

[0064] Those skilled in the art will understand that Figure 1 The hardware structure of the distributed system shown does not constitute a limitation on the distributed system. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0065] like Figure 1 As shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and a reinforcement learning program based on a distributed system. The operating system is a program that manages and controls the hardware and software resources of the distributed system, supporting the operation of the network communication module, the user interface module, the reinforcement learning program based on the distributed system, and other programs or software. The network communication module manages and controls the network interface 1004; the user interface module manages and controls the user interface 1003.

[0066] exist Figure 1 In the distributed system hardware architecture shown, network interface 1004 is mainly used to connect to the backend server and communicate data with it; user interface 1003 is mainly used to connect to the client (user end) and communicate data with it; processor 1001 can call the reinforcement learning program based on the distributed system stored in memory 1005 and perform the following operations:

[0067] The multiple learning ends respectively process the training samples to generate multiple sample data, and broadcast the multiple sample data to the database;

[0068] After receiving the multiple sample data broadcast by the database, the training end performs reinforcement learning on the multiple sample data to generate training parameters and broadcast the training parameters to the database;

[0069] After receiving the training parameters broadcast by the database, the evaluation end evaluates the training parameters, and completes the reinforcement learning of the training end after the evaluation of the training parameters is passed.

[0070] Further, the step of evaluating the training parameters by the evaluation end comprises:

[0071] The evaluation end calculates the training parameters based on a preset evaluation strategy to generate a return value, and determines whether the return value is greater than a preset threshold;

[0072] If the return value is greater than the preset threshold, it is determined that the evaluation of the training parameters is passed.

[0073] Further, after the step of determining whether the return value is greater than the preset threshold, the processor 1001 can call the reinforcement learning program based on the distributed system stored in the memory 1005, and perform the following operations:

[0074] If the return value is less than or equal to the preset threshold, the multiple learning ends perform the step of respectively processing the training samples to generate multiple sample data based on the training parameters.

[0075] Further, the number of training ends is multiple, and the step of calculating the return value by the evaluation end based on the preset evaluation strategy comprises:

[0076] The evaluation end calculates the training parameters corresponding to the multiple training ends based on the preset evaluation strategy to generate intermediate return values corresponding to the multiple training ends;

[0077] Compare the multiple intermediate return values to determine the intermediate return value with the maximum value, and take the intermediate return value with the maximum value as the return value.

[0078] Further, after the step of generating training parameters and broadcasting them to the database, the processor 1001 can call the reinforcement learning program based on the distributed system stored in the memory 1005, and perform the following operations:

[0079] The database broadcasts the training parameters to the evaluation end and the plurality of learning ends respectively based on a message queue mode.

[0080] Further, the step of broadcasting the training parameters to the evaluation end and the plurality of learning ends respectively based on a message queue mode by the database comprises:

[0081] The database determines whether the registration state of the evaluation end is valid based on the registration state of the evaluation end.

[0082] If the registration state of the evaluation end is valid, the database broadcasts the training parameters to the evaluation end based on a message queue mode.

[0083] The database determines a target learning end from the plurality of learning ends based on the registration state of the plurality of learning ends, and broadcasts the training parameters to each of the target learning ends based on a message queue mode.

[0084] Further, the step of generating the training parameters and broadcasting the training parameters to the database comprises:

[0085] The training end broadcasts the training parameters to the database based on a message queue mode after generating the training parameters.

[0086] Further, before the step of processing the training samples by the plurality of learning ends respectively, the processor 1001 can call the reinforcement learning program based on a distributed system stored in the memory 1005, and perform the following operations:

[0087] Start a learning registration thread based on each of the plurality of learning ends, and register each of the plurality of learning ends to the distributed system based on each of the learning registration threads.

[0088] Start an evaluation registration thread based on the evaluation end, and register the evaluation end to the distributed system based on each of the evaluation registration threads.

[0089] Further, the step of broadcasting the plurality of sample data to the database respectively comprises:

[0090] The plurality of learning ends broadcast the plurality of sample data to the database respectively based on a message subscription mode.

[0091] Further, after the step of broadcasting the plurality of sample data to the database respectively, the processor 1001 can call the reinforcement learning program based on a distributed system stored in the memory 1005, and perform the following operations:

[0092] The database broadcasts the plurality of sample data to the training end based on a message subscription mode.

[0093] The specific implementation of the distributed system of the present application is basically the same as each embodiment of the reinforcement learning method based on the distributed system described below, and will not be repeated here.

[0094] The present application also provides a reinforcement learning method based on a distributed system.

[0095] Reference Figure 2 , Figure 2 The flowchart of the first embodiment of the reinforcement learning method based on the distributed system of the present application.

[0096] The embodiments of the reinforcement learning method based on the distributed system provided by the present application need to be explained. Although the logical order is shown in the flowchart, in some cases, the steps shown or described here can be performed in a different order. Specifically, in the reinforcement learning method based on the distributed system of the present application, the distributed system includes a training end, an evaluation end, a database, and multiple learning ends. The reinforcement learning method based on the distributed system includes:

[0097] Step S10, based on multiple learning ends respectively processing training samples, generating multiple sample data, and broadcasting the multiple sample data to the database respectively;

[0098] The reinforcement learning method based on the distributed system in the present embodiment is applied to a distributed system, which includes a training end, an evaluation end, a database, and multiple learning ends. The training end is used for reinforcement learning training. The evaluation end is used to evaluate whether the training result of reinforcement learning reaches the optimal. The database is used to store and broadcast various data generated during the learning process. The multiple learning ends are used to generate sample data for training the training end. The database is preferably a database supporting a message queue, such as a redis database, to quickly synchronize data through a network queue. Moreover, the training end, the evaluation end, and the multiple learning ends can correspond to different computer devices, respectively, each computer device realizing the functions of each end to achieve the purpose of reinforcement learning; or they can correspond to the same computer device, with different processes in the computer device processing in parallel to realize the functions of each end to achieve the purpose of reinforcement learning to get the optimal decision.

[0099] Understandably, after the training end trains the training parameters, on the one hand, the training parameters are broadcast to the evaluation end for evaluation, and on the other hand, the training parameters are broadcast to each learning end. If the evaluation end determines that the training effect does not reach the optimal, each learning end continues to generate new sample data based on the received training parameters for the training end to train. In order to ensure that the training end accurately synchronizes the training parameters to the evaluation end and the multiple learning ends, it is necessary to register the started evaluation end and learning end first. Specifically, the step of based on multiple learning ends respectively processing training samples includes:

[0100] Step a, based on a plurality of said learning end respectively start learning registration thread, and based on each said learning registration thread, each said learning end is registered to the distributed system respectively;

[0101] Step b, based on the evaluation end start evaluation registration thread, and based on each said evaluation registration thread, the evaluation end is registered to the distributed system.

[0102] Further, a plurality of learning end respectively start each learning registration thread, to register each learning end to the database in the distributed system through each learning registration thread, facilitate to realize the data synchronization with the training end. At the same time, the evaluation end starts its evaluation registration thread, to register the evaluation end to the database in the distributed system through the evaluation registration thread, realize the data synchronization between the training end. It needs to be explained that the fault expiration time is set for each registered learning end or evaluation end, if any one of the fault equipment is detected, such as heart loss, the data of the fault equipment recorded in the database will be cleared within the fault expiration time, to avoid the data synchronization to it, ensure the accuracy and safety of data synchronization.

[0103] Further, in the process of reinforcement learning, the scene to be learned is set first, such as the game of go or maze. The initial scene data of the learning scene is set for each learning end, such as the number of road strips supported in the initial maze, and the initial scene data is used as the training sample of each learning end. Each learning end processes its own training sample to generate sample data, which represents the result of processing the training sample, such as the path walked in the maze, the number of steps walked out of the maze, the time spent, etc. Each learning end transmits the sample data generated by itself to the database cache, so as to synchronize the transmission to the training end for training.

[0104] Further, said step of broadcasting a plurality of said sample data to said database comprises:

[0105] Step c, a plurality of said learning end based on message subscription mode respectively broadcast a plurality of said sample data to said database.

[0106] Further, in order to improve the efficiency of real-time data transmission between each learning end and the database, each learning end broadcasts its own sample data to the database through message subscription. Message subscription is a message communication mode in the redis database, in which the sender (pub) sends messages and the subscriber (sub) receives messages. In the embodiment, the learning end is the sender and the training end is the subscriber. The learning end, as a message producer, publishes messages (i.e. sample data) to the database, which are consumed by the training end as a message consumer, i.e. the training end trains the obtained sample data.

[0107] Further, after the multiple learning ends broadcast their own sample data to the database through message subscription, the training end, as a message consumer in message subscription, consumes the sample data as messages. That is, the database broadcasts multiple sample data to the training end based on message subscription, so that the training end learns and trains the obtained multiple sample data to obtain the optimal training parameters in the required learning scenario. For example, for the above-mentioned maze, the fastest path out of the maze is the optimal training parameter.

[0108] Step S20, after the training end receives the multiple sample data broadcast by the database, the training end learns and trains the multiple sample data to generate training parameters and broadcast them to the database.

[0109] Further, after receiving the multiple sample data broadcast by the database based on message subscription, the training end learns and trains the multiple sample data based on a preset algorithm. The preset algorithm is a machine learning algorithm set in advance according to requirements, such as the gradient descent algorithm. The training end learns and trains the multiple sample data to obtain training parameters representing the optimal decision performance. The parameters representing the optimal decision performance are different according to different learning scenarios. For example, for the above-mentioned maze, the parameters representing the optimal decision performance are the decision path out of the maze. The training end synchronizes the obtained training parameters to the database through broadcasting, so that the training parameters are quickly synchronized to the evaluation end based on the network queue of the database for evaluation of the decision performance. Specifically, the step of generating training parameters and broadcasting them to the database includes:

[0110] Step d, after generating the training parameters, the training end broadcasts the training parameters to the database based on message queue.

[0111] Likewise, in consideration of the efficiency of data synchronization, after the training end generates the training parameters through reinforcement learning, the training parameters are broadcast to the database in a message queue-based manner, so as to quickly synchronize the training parameters to the database by means of the network queue, and then to the evaluation end by the database. For the message queue, a message producer produces a message and sends it to the queue, and a message consumer takes out and consumes the message from the queue. In the embodiment, the training end is the message producer, the database is the message consumer, and the broadcast training parameters are the generated messages.

[0112] Further, the step of broadcasting the generated training parameters to the database comprises:

[0113] Step e: The database broadcasts the training parameters to the evaluation end and the plurality of learning ends in a message queue-based manner.

[0114] Further, after receiving the broadcast training parameters from the training end in a message queue-based manner, the database also broadcasts the training parameters to the evaluation end and the plurality of learning ends in a message queue-based manner, so as to evaluate the training parameters at the evaluation end. If the evaluation at the evaluation end determines that the training parameters have not yet reached the optimal decision performance, the plurality of learning ends generate new sample data based on the received training parameters for the training end to continue training, until the training parameters obtained by the training end reach the optimal decision performance.

[0115] Step S30: After receiving the broadcast training parameters from the database, the evaluation end evaluates the training parameters, and completes the reinforcement learning of the training end after the evaluation of the training parameters is passed.

[0116] Further, a preset evaluation strategy is preset in the evaluation end, and after receiving the broadcast training parameters from the database, the training parameters are evaluated by the preset evaluation strategy to determine whether the training parameters reach the optimal performance set in the preset evaluation strategy. If the optimal performance is reached, it is determined that the evaluation of the training parameters is passed, and the reinforcement learning of the training end is completed. If the optimal performance is not reached, new sample data is generated by the plurality of learning ends and synchronized to the training end for continued training, and the optimal decision performance of the training parameters is ensured through multiple iterative training of a large amount of sample data.

[0117] The reinforcement learning method based on the distributed system of the present application comprises a training end, an evaluation end, a database and a plurality of learning ends. Each learning end processes training samples to generate a plurality of sample data, which is broadcast to the database for storage and then broadcast to the training end by the database. After receiving the plurality of sample data, the training end performs reinforcement learning on the plurality of sample data to generate training parameters, which is broadcast to the database for storage and then broadcast to the evaluation end by the database. After receiving the training parameters, the evaluation end evaluates the training parameters to determine the training effect of the training end. If the evaluation of the training parameters is passed, it indicates that the training has achieved good results, and the reinforcement learning of the training end is completed. The present application performs reinforcement learning through a distributed system, and the database in the system realizes parallel scheduling and processing of various resources. A large amount of sample data can be quickly generated for learning and training. The results obtained by learning and training are accurate, the convergence speed is accelerated, and the efficiency of reinforcement learning is improved.

[0118] Further, based on the first embodiment of the reinforcement learning method based on the distributed system of the present application, a second embodiment of the reinforcement learning method based on the distributed system of the present application is proposed.

[0119] The second embodiment of the reinforcement learning method based on the distributed system is different from the first embodiment of the reinforcement learning method based on the distributed system in that the step of evaluating the training parameters by the evaluation end comprises:

[0120] In step S31, the evaluation end calculates the training parameters based on a preset evaluation strategy to generate a reward value, and determines whether the reward value is greater than a preset threshold.

[0121] In step S32, if the reward value is greater than the preset threshold, it is determined that the evaluation of the training parameters is passed.

[0122] In step S33, if the reward value is less than or equal to the preset threshold, a plurality of learning ends perform the step of processing training samples to generate a plurality of sample data based on the training parameters.

[0123] The preset evaluation strategy of the evaluation end is different according to different learning scenarios. For example, for the learning scenario of playing Go, the preset evaluation strategy can be to win the opponent with the least number of steps. For the learning scenario of a maze, the preset evaluation strategy can be to walk out of the maze with the shortest path. When evaluating the training parameters, the evaluation end first calls the preset evaluation strategy set according to the learning scenario, calculates the training parameters through the preset evaluation strategy, and generates a return value. The higher the return value, the better the decision effect. The preset evaluation strategy can set different scores for each item of data in the training parameters, and the final score represents the return value. For example, for the maze scenario, the score is reduced by 1 for each step in the path. The more steps taken, the less the score is reduced, and the lower the corresponding return value, the worse the decision effect. For the case where the maze is not walked out, the score value is set to 0, indicating a failed decision.

[0124] Further, in order to represent the good or bad of the decision performance, a preset threshold is set in advance, and the generated return value is compared with the preset threshold to determine whether the return value is greater than the preset threshold. If it is determined that the return value is greater than the preset threshold, it means that the return value is high and the decision effect is good, so it is determined that the evaluation of the training parameters is passed. The training parameters passed by the evaluation represent that the reinforcement learning has good decision performance, which is used as a condition for the training end to end the reinforcement learning.

[0125] Further, if it is determined through comparison that the return value is not greater than the preset threshold, it means that the return value is low and the decision effect is not good, and reinforcement learning training needs to be continued. At this time, the training sample is processed by each learning end based on the broadcasted training parameters to generate new sample data, which is broadcast to the database. The database is synchronized to the training end for continuous training to obtain new training parameters, which are evaluated by the evaluation end until the return value obtained by the evaluation end is greater than the preset threshold, and it is determined that the evaluation of the training parameters is passed.

[0126] It should be noted that, in order to avoid the situation that the return value is high due to accidental factors, i.e. the decision effect is good by chance, the embodiment sets a continuous discrimination mechanism. That is, after the return value is greater than the preset threshold for the first time, representing a good decision effect, the training end continues to generate respective sample data from multiple learning ends, which are synchronized to the training end to obtain training parameters, which are evaluated by the evaluation end. If the return values generated by the evaluation for a continuous number of times are all greater than the preset threshold, it means that the training parameters obtained by the reinforcement learning are stable and can be used to make good decisions continuously, so it is determined that the evaluation of the training parameters is passed. If the return value is not greater than the preset threshold at some time during the continuous training, it means that the training parameters obtained by the reinforcement learning are not stable and cannot be used to make good decisions continuously, so it is determined that the evaluation of the training parameters has not passed and reinforcement learning training needs to be continued.

[0127] The embodiment generates a reward value by evaluating the end needle for the training parameter, evaluates the training parameter by the reward value, and ensures that the decision formed by the reinforcement learning of the training end has better performance. A continuous evaluation mechanism is set for multiple times to eliminate the influence of accidental factors and ensure the accuracy and stability of the decision performance.

[0128] Further, based on the first or second embodiment of the distributed system-based reinforcement learning method of the application, a third embodiment of the distributed system-based reinforcement learning method of the application is proposed.

[0129] The third embodiment of the distributed system-based reinforcement learning method differs from the first or second embodiment of the distributed system-based reinforcement learning method in that the number of training ends is multiple, the evaluation end calculates the training parameters based on a preset evaluation strategy to generate a reward value, and the steps of generating a reward value include:

[0130] In step S311, the evaluation end calculates the training parameters corresponding to multiple training ends based on a preset evaluation strategy to generate intermediate reward values corresponding to multiple training ends.

[0131] In step S312, multiple intermediate reward values are compared to determine the intermediate reward value with the largest value, and the intermediate reward value with the largest value is taken as the reward value.

[0132] In order to ensure the optimal decision obtained by reinforcement learning, multiple training ends are provided in the distributed system, and each training end is provided with a different machine learning algorithm. Multiple copies of sample data generated by each learning end are synchronized to multiple training ends, and multiple training ends respectively train multiple copies of sample data by using their own machine learning algorithms to obtain their own training parameters, which are synchronized to the evaluation end. After reading each training parameter, the evaluation end calculates each training parameter based on a preset evaluation strategy to obtain an intermediate reward value representing the training effect of each training end. Then, the intermediate reward values are compared to determine the intermediate reward value with the largest value, which represents the training end with the best training effect. The intermediate reward value with the largest value is taken as the reward value for evaluation to determine whether the optimal performance end condition is reached, i.e., whether the reward value is greater than a preset threshold within a preset number of times. If the end condition is reached, the training of each training end is ended, the target training parameter generating the reward value is determined from each training parameter, and the strategy corresponding to the training parameter is taken as the optimal strategy obtained by reinforcement learning. Otherwise, if the optimal performance end condition is not reached, the training of each training end is continued until the optimal performance end condition is reached.

[0133] The embodiment sets multiple training ends in the distributed system, respectively trains sample data, obtains multiple training parameters for evaluation, improves evaluation accuracy, and is beneficial to quickly determine an optimal decision obtained through reinforcement learning, and improves the efficiency of reinforcement learning.

[0134] Further, based on the first, second or third embodiment of the reinforcement learning method based on the distributed system, the fourth embodiment of the reinforcement learning method based on the distributed system is provided.

[0135] The fourth embodiment of the reinforcement learning method based on the distributed system is different from the first, second or third embodiment of the reinforcement learning method based on the distributed system in that the step of broadcasting the training parameters to the evaluation end and the multiple learning ends by the database based on the message queue mode comprises:

[0136] Step e1, the database judges whether the registration state of the evaluation end is valid based on the registration state of the evaluation end;

[0137] Step e2, if the registration state of the evaluation end is valid, the training parameters are broadcast to the evaluation end based on the message queue mode;

[0138] Step e3, the database determines a target learning end in the multiple learning ends based on the registration state of the multiple learning ends, and broadcasts the training parameters to each target learning end based on the message queue mode.

[0139] In order to ensure the accuracy and effectiveness of the training parameter synchronization, the registration state validity detection mechanism is provided in the embodiment. Specifically, after each learning end is successfully registered based on the learning registration thread of each learning end, the registration state of each learning end is in a valid state. Similarly, after the evaluation end is successfully registered based on the evaluation registration thread, the registration state of the evaluation end is in a valid state. Before the training parameters are synchronized to the evaluation end through the database, it is judged whether the registration state of the evaluation end is valid, i.e. whether it is in a valid state; if it is in a valid state, the training parameters are broadcast to the evaluation end based on the message queue mode, and the training parameters are synchronized to the evaluation end for evaluation.

[0140] Further, for the multiple learning ends, the evaluation ends with the registration state in a valid state are determined, and the evaluation ends are determined as target learning ends. Further, the training parameters are broadcast to each target learning end based on the message queue mode, the training parameters are synchronized to the target learning end currently online, so that when the optimal performance is not reached through evaluation, the target learning end currently online generates new sample data for training based on the training parameters.

[0141] The embodiment determines the effective states of the learning end and the evaluation end before training parameter synchronization is performed, so as to accurately synchronize the training parameters to the effective evaluation end and learning end, and ensure the effectiveness of evaluation and the effectiveness of continuous training.

[0142] The application also provides a reinforcement learning device based on a distributed system.

[0143] The learning module is configured to process training samples respectively based on multiple learning ends, generate multiple pieces of sample data, and broadcast the multiple pieces of sample data to the database respectively;

[0144] The training module is configured to perform reinforcement learning on the multiple pieces of sample data after the training end receives the multiple pieces of sample data broadcast by the database, and broadcast training parameters to the database;

[0145] The evaluation module is configured to evaluate the training parameters after the evaluation end receives the training parameters broadcast by the database, and complete the reinforcement learning of the training end after the evaluation of the training parameters is passed.

[0146] Further, the evaluation module further comprises:

[0147] The evaluation unit is configured to calculate the training parameters based on a preset evaluation strategy, generate a return value, and determine whether the return value is greater than a preset threshold;

[0148] The determination module is configured to determine that the evaluation of the training parameters is passed if the return value is greater than the preset threshold.

[0149] Further, the evaluation module further comprises:

[0150] The execution unit is configured to perform the steps of processing training samples respectively and generating multiple pieces of sample data based on the training parameters if the return value is less than or equal to the preset threshold.

[0151] Further, the number of training ends is multiple, and the evaluation unit is further configured to:

[0152] Calculate training parameters corresponding to multiple training ends respectively based on a preset evaluation strategy, to generate intermediate return values corresponding to multiple training ends respectively;

[0153] Compare the multiple intermediate return values, determine an intermediate return value with the largest value among the intermediate return values, and take the intermediate return value with the largest value as the return value.

[0154] Further, the reinforcement learning device based on the distributed system further comprises:

[0155] a broadcast module configured to broadcast the training parameter to the evaluation end and the plurality of learning ends respectively based on a message queue manner.

[0156] Further, the broadcast module further comprises:

[0157] a judging unit configured to judge whether the registration state of the evaluation end is valid based on the registration state of the evaluation end;

[0158] a broadcast unit configured to broadcast the training parameter to the evaluation end based on a message queue manner if the registration state of the evaluation end is valid;

[0159] a determining unit configured to determine a target learning end in the plurality of learning ends based on the registration state of the plurality of learning ends, and broadcast the training parameter to each of the target learning end based on a message queue manner.

[0160] Further, the training module further comprises:

[0161] after the training parameter is generated, the training parameter is broadcast to the database based on a message queue manner.

[0162] Further, the reinforcement learning device based on a distributed system further comprises:

[0163] a starting module configured to start a learning registration thread for each of the plurality of learning ends, and register each of the learning ends to the distributed system based on each of the learning registration thread;

[0164] a registration module configured to start an evaluation registration thread based on the evaluation end, and register the evaluation end to the distributed system based on each of the evaluation registration thread.

[0165] Further, the learning module is further configured to:

[0166] the plurality of learning ends broadcast a plurality of the sample data to the database respectively based on a message subscription manner.

[0167] Further, the learning module is further configured to:

[0168] the database broadcasts a plurality of the sample data to the training end based on a message subscription manner.

[0169] The embodiment of the reinforcement learning device based on a distributed system is basically the same as the above-mentioned reinforcement learning method based on a distributed system, and will not be repeated here.

[0170] In addition, the embodiment of the present application also proposes a storage medium.

[0171] The storage medium stores a distributed system-based reinforcement learning program, which, when executed by the processor, implements the steps of the distributed system-based reinforcement learning method as described above.

[0172] The storage medium of the present application can be a computer-readable storage medium, and the detailed embodiments thereof are basically the same as those of the distributed system-based reinforcement learning method described above, and thus will not be described herein again.

[0173] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the specific embodiments described above, which are merely illustrative rather than restrictive. Those of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims. Any equivalent structure or equivalent flow conversion made by using the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, all belong to the protection of the present application.

Claims

1. A reinforcement learning method based on distributed systems, characterized in that, The distributed system includes a training terminal, an evaluation terminal, a database, and multiple learning terminals. The reinforcement learning method based on the distributed system includes the following steps: The training samples are processed by multiple learning endpoints to generate multiple sets of sample data, and the multiple sets of sample data are broadcast to the database. The database is used to store and broadcast the data generated during the learning process, and the database supports message queues. After receiving multiple sets of sample data broadcast from the database, the training terminal performs reinforcement learning on the multiple sets of sample data, generates training parameters, and broadcasts them to the database. After receiving the training parameters broadcast by the database, the evaluation end calculates the training parameters based on a preset evaluation strategy, generates a reward value, and determines whether the reward value is greater than a preset threshold. If the reward value is greater than the preset threshold, the evaluation of the training parameters is deemed to have passed, and the reinforcement learning of the training end is completed after the evaluation of the training parameters has passed.

2. The reinforcement learning method based on distributed systems as described in claim 1, characterized in that, The step of determining whether the return value is greater than a preset threshold includes: If the reward value is less than or equal to a preset threshold, then the multiple learning terminals will perform the steps of processing the training samples respectively and generating multiple sets of sample data based on the training parameters.

3. The reinforcement learning method based on distributed systems as described in claim 1, characterized in that, The number of training terminals is multiple, and the evaluation terminal calculates the training parameters and generates a reward value based on a preset evaluation strategy, including the following steps: The evaluation end calculates the training parameters corresponding to the multiple training ends based on a preset evaluation strategy, and generates intermediate reward values ​​corresponding to the multiple training ends respectively. The multiple intermediate return values ​​are compared to determine the intermediate return value with the largest value, and the intermediate return value with the largest value is taken as the return value.

4. The reinforcement learning method based on distributed systems as described in claim 1, characterized in that, The step of generating training parameters and broadcasting them to the database is followed by: The database broadcasts the training parameters to the evaluation end and multiple learning ends respectively using a message queue method.

5. The reinforcement learning method based on distributed systems as described in claim 4, characterized in that, The step of broadcasting the training parameters to the evaluation end and multiple learning ends based on a message queue method includes: The database determines whether the registration status of the evaluation terminal is valid based on the registration status of the evaluation terminal. If the registration status of the evaluation terminal is valid, the training parameters will be broadcast to the evaluation terminal using a message queue method. The database determines the target learning end among the multiple learning ends based on their registration status, and broadcasts the training parameters to each target learning end using a message queue.

6. The reinforcement learning method based on distributed systems as described in any one of claims 1-5, characterized in that, The step of generating training parameters and broadcasting them to the database includes: After generating the training parameters, the training terminal broadcasts the training parameters to the database using a message queue.

7. The reinforcement learning method based on distributed systems as described in claim 1, characterized in that, Before the step of processing the training samples separately based on multiple learning terminals, the following are included: Learning registration threads are started based on multiple learning terminals, and each learning terminal is registered to the distributed system based on each learning registration thread. An evaluation registration thread is started based on the evaluation terminal, and the evaluation terminal is registered to the distributed system based on each evaluation registration thread.

8. The reinforcement learning method based on distributed systems as described in claim 1, characterized in that, The step of broadcasting multiple sets of the sample data to the database includes: Multiple learning endpoints broadcast multiple sets of sample data to the database based on a message subscription method.

9. The reinforcement learning method based on distributed systems as described in claim 1, characterized in that, The step of broadcasting multiple sets of the sample data to the database includes: The database broadcasts multiple sets of sample data to the training terminal based on a message subscription method.

10. A reinforcement learning device based on a distributed system, characterized in that, The reinforcement learning device based on a distributed system includes: The learning module is used to process training samples based on multiple learning endpoints, generate multiple sets of sample data, and broadcast the multiple sets of sample data to a database. The database is used to store and broadcast data generated during the learning process, and the database supports message queues. The training module is used to perform reinforcement learning on the multiple sets of sample data broadcast by the database after receiving them at the training end, and to generate training parameters which are then broadcast to the database. The evaluation module is used to calculate the training parameters based on a preset evaluation strategy after receiving the training parameters broadcast by the database at the evaluation end, generate a reward value, and determine whether the reward value is greater than a preset threshold. If the reward value is greater than the preset threshold, the evaluation of the training parameters is determined to be passed, and the reinforcement learning at the training end is completed after the evaluation of the training parameters is passed.

11. The reinforcement learning device based on a distributed system as described in claim 10, characterized in that, The evaluation module also includes: An execution unit is configured to, if the reward value is less than or equal to a preset threshold, perform the steps of processing the training samples respectively and generating multiple sets of sample data based on the training parameters.

12. The reinforcement learning device based on a distributed system as described in claim 10, characterized in that, The number of training terminals is multiple, and the evaluation module is further used for: Based on a preset evaluation strategy, the training parameters corresponding to the multiple training ends are calculated to generate intermediate reward values ​​corresponding to the multiple training ends. The multiple intermediate return values ​​are compared to determine the intermediate return value with the largest value, and the intermediate return value with the largest value is taken as the return value.

13. The reinforcement learning device based on a distributed system as described in claim 10, characterized in that, The reinforcement learning device based on a distributed system further includes: The broadcast module is used to broadcast the training parameters to the evaluation end and multiple learning ends respectively based on a message queue.

14. The reinforcement learning device based on a distributed system as described in claim 13, characterized in that, The broadcast module also includes: The judgment unit is used to determine whether the registration status of the evaluation terminal is valid based on the registration status of the evaluation terminal. A broadcast unit is used to broadcast the training parameters to the evaluation terminal based on a message queue if the registration status of the evaluation terminal is valid. The determining unit is configured to determine the target learning end among the multiple learning ends based on the registration status of the multiple learning ends, and broadcast the training parameters to each of the target learning ends in a message queue manner.

15. The reinforcement learning device based on a distributed system as described in any one of claims 11-14, characterized in that, The training module also includes: After generating the training parameters, the training parameters are broadcast to the database using a message queue.

16. The reinforcement learning device based on a distributed system as described in claim 10, characterized in that, The reinforcement learning device based on a distributed system further includes: The startup module is used to start learning registration threads based on multiple learning terminals respectively, and register each learning terminal to the distributed system based on each learning registration thread; The registration module is used to start an evaluation registration thread based on the evaluation terminal, and register the evaluation terminal to the distributed system based on each evaluation registration thread.

17. A distributed system, characterized in that, The distributed system includes a training end, an evaluation end, a database, and multiple learning ends. It also includes a memory, a processor, and a distributed system-based reinforcement learning program stored in the memory and executable on the processor. When the distributed system-based reinforcement learning program is executed by the processor, it implements the steps of the distributed system-based reinforcement learning method as described in any one of claims 1-9.

18. A storage medium, characterized in that, The storage medium stores a reinforcement learning program based on a distributed system, which, when executed by a processor, implements the steps of the reinforcement learning method based on a distributed system as described in any one of claims 1-9.

Images