Model decision supervision method, system and equipment based on reinforcement learning and medium

Abstract

The invention discloses a model decision supervision method and system based on reinforcement learning, equipment and a medium, and relates to the technical field of model application, and the method comprises the steps: transmitting environment information and equipment information of to-be-controlled equipment to a decision model, and obtaining a decision strategy, the decision-making model is a model obtained by performing intensified training on the basis of actual equipment information, actual environment information and a corresponding actual decision-making strategy in advance; inputting the decision strategy into a preset supervision model to obtain a strategy evaluation result, the supervision model being obtained by training with an actual decision strategy as model input data and an actual strategy evaluation result corresponding to the actual decision strategy as a model training label, and the actual strategy evaluation result being determined with a forward value condition as a standard; and if it is detected that the strategy evaluation result meets the forward value condition, controlling the to-be-controlled devices according to the decision strategy. The technical scheme of the invention aims to solve the technical problem of how to prevent data output by a model from generating negative effects.

Description

Technical Field

[0001] The present application relates to the field of model application technology, and in particular to a model decision supervision method, system, device and medium based on reinforcement learning. Background Art

[0002] In order to make the model predictive, it is usually necessary to use a large amount of data to train the model. Therefore, current model training is usually designed based on data sets involved in public channels such as the Internet, books, news articles, and social media.

[0003] However, the data sets in public channels may contain social and cultural biases, historical biases, etc., which may cause the model to learn and amplify the biases in the training data, and then cause the model to produce harmful outputs, such as misleading information, inaccurate predictions, or inappropriate recommendations.

[0004] Therefore, how to avoid negative impacts on the data output by the model is a technical problem that has yet to be solved by those skilled in the art. Summary of the invention

[0005] The main purpose of this application is to provide a model decision supervision method, system, device and medium based on reinforcement learning, aiming to solve the technical problem of how to avoid negative impacts on the data output by the model.

[0006] To achieve the above objectives, the present application proposes a model decision supervision method based on reinforcement learning, the method comprising:

[0007] The environmental information and the device information of each device to be controlled are transmitted to the decision model to obtain a decision strategy, wherein the decision model is a model obtained by intensive training based on the actual device information and the actual environmental information and the corresponding actual decision strategy in advance;

[0008] Input the decision strategy into a preset supervision model to obtain a strategy evaluation result, wherein the supervision model is trained using the actual decision strategy as model input data and the actual strategy evaluation result corresponding to the actual decision strategy as a model training label, wherein the actual strategy evaluation result is determined based on a preset positive value condition;

[0009] If it is detected that the strategy evaluation result satisfies the preset positive value condition, the operation of each device to be controlled is controlled accordingly according to the decision strategy.

[0010] In one embodiment, before the step of transmitting the environmental information and the device information of each device to be controlled to the decision model to obtain the decision strategy, the method further includes:

[0011] Obtaining preset model training parameters and obtaining an actual decision strategy corresponding to both actual device information and actual environment information, wherein the model training parameters include a strategy network and a value network;

[0012] The actual device information and the actual environment information are each converted into a tensor, and based on the model training parameters, the tensors and the actual decision-making strategy, the preset machine learning model is trained, and the trained machine learning model is used as a decision model.

[0013] In one embodiment, the decision strategy includes a collaborative relationship and a behavior pattern, and the step of controlling each of the devices to be controlled according to the decision strategy includes:

[0014] Determining a target device specified by the cooperation relationship among the devices to be controlled;

[0015] In the behavior pattern, searching for a corresponding target behavior pattern of the target device, wherein the behavior pattern includes various operating parameters of the device;

[0016] The target device is controlled to operate according to the target behavior mode.

[0017] In one embodiment, the strategy evaluation result includes: a strategy score, and the method further includes:

[0018] Detecting whether the strategy evaluation result meets the preset positive value condition;

[0019] The step of detecting whether the strategy evaluation result meets the preset positive value condition includes:

[0020] Obtaining a preset strategy score threshold, and comparing the strategy score with the strategy score threshold;

[0021] If it is detected that the strategy score is greater than or equal to the strategy score threshold, it is determined that the strategy evaluation result meets the preset positive value condition;

[0022] Otherwise, it is determined that the strategy evaluation result does not meet the preset positive value condition.

[0023] In one embodiment, after the step of detecting whether the strategy evaluation result satisfies a preset positive value condition, the method further includes:

[0024] Calculate a first number of times that the strategy evaluation result output by the supervisory model satisfies a preset positive value condition within a preset time period, and calculate a second number of times that the supervisory model outputs the strategy evaluation result;

[0025] If it is detected that the ratio between the first number of times and the second number of times is greater than a preset threshold, then when it is detected that the strategy evaluation result output by the supervision model does not meet the preset positive value condition, the decision strategy corresponding to the strategy evaluation result is output, and the decision strategy after manual correction is received;

[0026] Each of the devices to be controlled is controlled based on the manually corrected decision strategy.

[0027] In one embodiment, the method further comprises:

[0028] If it is detected that the policy evaluation result does not satisfy the preset positive value condition, the step of passing the environmental information and the device information of each device to be controlled to the decision model to obtain the decision strategy is returned to execution until it is detected that the policy evaluation result satisfies the preset positive value condition.

[0029] In one embodiment, after the step of inputting the decision strategy into a preset supervision model to obtain a strategy evaluation result, the method further includes:

[0030] The strategy evaluation result is transmitted to the decision model, and the decision model is optimized based on the strategy evaluation result.

[0031] In addition, to achieve the above purpose, the present application also proposes a model decision supervision system based on reinforcement learning, and the model decision supervision system based on reinforcement learning includes:

[0032] In the strategy generation module, the user transmits the environment information and the device information of each device to be controlled to the decision model to obtain the decision strategy, wherein the decision model is a model obtained by intensive training based on the actual device information and the actual environment information and the corresponding actual decision strategy in advance;

[0033] A strategy evaluation module, used to input the decision strategy into a preset supervision model to obtain a strategy evaluation result, wherein the supervision model is obtained by training with the actual decision strategy as model input data and the actual strategy evaluation result corresponding to the actual decision strategy as a model training label, wherein the actual strategy evaluation result is determined based on a preset positive value condition;

[0034] The device control module is used to control the operation of each device to be controlled accordingly according to the decision strategy if it is detected that the strategy evaluation result meets the preset positive value condition.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a model decision supervision device based on reinforcement learning, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the model decision supervision method based on reinforcement learning as described above.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the model decision supervision method based on reinforcement learning as described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the model decision supervision method based on reinforcement learning as described above.

[0038] In an embodiment of the present application, the present application obtains a decision strategy by transmitting environmental information and device information of each device to be controlled to a decision model, wherein the decision model is a model obtained by intensive training based on actual device information and actual environmental information and the corresponding actual decision strategy in advance, and a decision strategy matching the environmental information and device information can be obtained based on the decision model; then the decision strategy is input into a preset supervision model to obtain a strategy evaluation result, and the supervision model is trained with the actual decision strategy as model input data and the actual strategy evaluation result corresponding to the actual decision strategy as the model training label, wherein the actual strategy evaluation result is determined based on a preset positive value condition, and whether the decision strategy output by the decision model will have a negative impact can be judged based on the supervision model; then if it is detected that the strategy evaluation result meets the preset positive value condition, each device to be controlled is controlled according to the decision strategy, so that each device to be controlled can be controlled based on the decision strategy on the premise of avoiding the negative impact of the decision strategy output by the decision model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1A flowchart of the first embodiment of the model decision supervision method based on reinforcement learning provided in this application;

[0042] Figure 2 A data flow diagram of an embodiment of a model decision supervision method based on reinforcement learning of this application;

[0043] Figure 3 This is a flow chart of another embodiment of the model decision supervision method based on reinforcement learning of the present application;

[0044] Figure 4 This is a schematic diagram of the module structure of the model decision supervision system based on reinforcement learning in an embodiment of the present application;

[0045] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the model decision supervision method based on reinforcement learning in an embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] The main solution of the embodiment of the present application is: passing environmental information and device information of each device to be controlled to a decision model to obtain a decision strategy, wherein the decision model is a model obtained by intensive training based on actual device information, actual environmental information and corresponding actual decision strategies in advance; inputting the decision strategy into a preset supervision model to obtain a strategy evaluation result, wherein the supervision model is trained using the actual decision strategy as model input data and the actual strategy evaluation result corresponding to the actual decision strategy as a model training label, wherein the actual strategy evaluation result is determined based on a preset positive value condition; if it is detected that the strategy evaluation result meets the preset positive value condition, the operation of each device to be controlled is controlled accordingly according to the decision strategy.

[0050] It is understandable that in order for a model to have predictive capabilities, it is usually necessary to use a large amount of data to train the model. Therefore, current model training is usually designed based on data sets involved in public channels such as the Internet, books, news articles, and social media.

[0051] However, there may be social and cultural biases, historical biases and other data in the data sets of public channels, which may cause the model to learn and amplify the biases in the training data, and then cause the model to produce harmful outputs, such as misleading information, inaccurate predictions, decisions that are not in compliance with laws and regulations, or inappropriate recommendations.

[0052] Therefore, how to avoid negative impacts on the data output by the model is a technical problem that has yet to be solved by those skilled in the art.

[0053] To solve this technical problem, the present application obtains a decision strategy by transmitting environmental information and device information of each device to be controlled to a decision model, wherein the decision model is a model obtained by intensive training based on actual device information and actual environmental information and the corresponding actual decision strategy in advance, and a decision strategy matching the environmental information and device information can be obtained based on the decision model; then the decision strategy is input into a preset supervision model to obtain a strategy evaluation result, and the supervision model is trained with the actual decision strategy as model input data and the actual strategy evaluation result corresponding to the actual decision strategy as the model training label, wherein the actual strategy evaluation result is determined based on a preset positive value condition, and whether the decision strategy output by the decision model will have a negative impact can be judged based on the supervision model; then if it is detected that the strategy evaluation result meets the preset positive value condition, each device to be controlled is controlled according to the decision strategy, so that each device to be controlled can be controlled based on the decision strategy under the premise of avoiding the negative impact of the decision strategy output by the decision model.

[0054] It should be noted that the execution subject of the reinforcement learning-based model decision supervision method of the present application, the reinforcement learning-based model decision supervision device, can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a server, etc. The following takes the server as an example to illustrate various embodiments of the reinforcement learning-based model decision supervision method of the present application.

[0055] Based on this, this application provides a model decision supervision method based on reinforcement learning, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the model decision supervision method based on reinforcement learning in this application.

[0056] In this embodiment, the model decision supervision method based on reinforcement learning includes steps S10 to S30:

[0057] Step S10, transmitting the environmental information and the device information of each device to be controlled to the decision model to obtain a decision strategy, wherein the decision model is a model obtained by intensive training based on actual device information, actual environmental information and corresponding actual decision strategies in advance;

[0058] It can be understood that in this embodiment, the decision model is a model for making a decision response to a specific environment and the device information of each device to be controlled in the specific environment. Therefore, in this embodiment, the environmental information refers to the information in the environment where each device to be controlled is located. For example, the environmental information can be information such as temperature and humidity. The device to be controlled refers to the device to be controlled, for example, the device to be controlled can be each sensor terminal. The device information refers to the working parameters of the device.

[0059] In addition, it should be noted that before making a decision on the environmental information and device information through the decision model, the decision model needs to be trained. Among them, the actual device information refers to the device information dedicated to training the decision model, the actual environmental information refers to the environmental information dedicated to training the decision model, and the actual decision strategy can be a strategy manually input based on the actual device information and the actual environmental information. In addition, the above decision strategy refers to the control principle output by the decision model.

[0060] In this embodiment, under the condition that a response needs to be made through a decision model, for a decision model that has been trained, the environmental information of each device to be controlled and the working parameters of each device to be controlled can be input into the trained decision model to obtain the control principles corresponding to the environmental information and the information of each device.

[0061] For example, in a computer room with multiple servers, multiple servers need to be deployed to avoid the problem of a single server being overloaded. Since the temperature in the computer room will also affect the performance of the server, the working parameter values ​​of each server and the ambient temperature can be input into the decision model, so as to obtain the server control strategy of the computer room based on the decision model.

[0062] Step S20, inputting the decision strategy into a preset supervision model to obtain a strategy evaluation result, wherein the supervision model is obtained by training with the actual decision strategy as model input data and the actual strategy evaluation result corresponding to the actual decision strategy as a model training label, wherein the actual strategy evaluation result is determined based on a preset positive value condition;

[0063] It should be noted that, in this embodiment, the supervision model is used to determine whether the decision strategy output by the decision model meets the preset positive value conditions. It is understandable that in order for the model to have the ability to judge, the model needs to be trained in advance. In this embodiment, the actual strategy evaluation result is the evaluation result corresponding to the actual decision strategy. The actual strategy evaluation result can be automatically evaluated by the model or manually evaluated, and this application does not limit this.

[0064] It can be understood that when training a supervision model, a large number of strategies output by the decision model can be obtained in advance, or a large number of manually input strategies can be obtained. Then, each strategy can be automatically evaluated by the model to obtain the strategy evaluation results corresponding to each strategy, or the manually input strategy evaluation results corresponding to each strategy can be received, so that the supervision model can be trained with the strategy evaluation results as labels.

[0065] It should be noted that the strategy evaluation result can be a score or "whether it meets the preset positive value conditions", and this application does not impose any restrictions on this.

[0066] Please refer to Figure 2 , Figure 2 This is a data flow diagram of an embodiment of a model decision supervision method based on reinforcement learning in this application. The data of the devices 1 to 3 to be controlled and the data in the environment will be transmitted to the decision model, and then the output of the decision model will be transmitted to the supervision model, so as to achieve the purpose of judging whether the decision strategy will have a negative impact through the supervision model.

[0067] Step S30: If it is detected that the strategy evaluation result satisfies the preset positive value condition, the operation of each device to be controlled is controlled accordingly according to the decision strategy.

[0068] It can be understood that when it is detected that the strategy evaluation result meets the preset positive value condition, it indicates that the decision strategy will not cause negative impact, and thus, the operation of each device to be controlled can be controlled according to the decision strategy.

[0069] In a feasible implementation, the decision strategy includes a collaborative relationship and a behavior pattern, wherein the collaborative relationship refers to the cooperative relationship between a device to be controlled and another device to be controlled, and the behavior pattern refers to the control content of the device to be controlled. For example, the behavior pattern corresponding to a device to be controlled may be to adjust the interval of the collected signal to 10ms.

[0070] Based on this, the above step S30 includes:

[0071] Step S301, determining a target device specified by the cooperation relationship in each of the devices to be controlled;

[0072] It is understandable that when the cooperation relationship and the devices to be controlled are not in one-to-one correspondence, it is necessary to determine the device to be controlled corresponding to the cooperation relationship among the devices to be controlled according to the cooperation relationship, that is, to obtain the target device.

[0073] Exemplarily, the number or ID of the device to be controlled may be identified in the collaboration relationship, so that the target device may be determined among multiple devices to be controlled based on the number or ID.

[0074] Step S302, searching the behavior pattern for a corresponding target behavior pattern of the target device, wherein the behavior pattern includes various operating parameters of the device;

[0075] It is understandable that after determining the target device, it is also necessary to determine the control content corresponding to the target device in the behavior mode, that is, to determine the target behavior mode.

[0076] Step S303: Control the target device to operate according to the target behavior mode.

[0077] Therefore, for a decision strategy, one or more target devices can be determined among the devices to be controlled, and then the target devices can be controlled based on the control content corresponding to the target devices, thereby achieving the control purpose.

[0078] In addition, in a feasible implementation, the model decision supervision method based on reinforcement learning also includes:

[0079] Step S40, if it is detected that the policy evaluation result does not meet the preset positive value condition, then return to execute the step of transmitting the environmental information and the device information of each device to be controlled to the decision model to obtain the decision strategy, until it is detected that the policy evaluation result meets the preset positive value condition.

[0080] It can be understood that in this embodiment, in order to achieve full automation of model decision supervision based on reinforcement learning, when the strategy evaluation result does not meet the preset positive value condition, the decision model can be prompted to output a decision strategy whose strategy evaluation result meets the preset positive value condition in a loop.

[0081] In a feasible implementation, the model decision supervision method based on reinforcement learning also includes:

[0082] Step S50: transferring the policy evaluation result to the decision model, and optimizing the decision model based on the policy evaluation result.

[0083] It can be understood that in this embodiment, the decision model can be further optimized by passing the strategy evaluation results to the decision model, so that the decision model can learn how to generate a decision strategy that meets the preset positive value conditions, thereby improving the accuracy of the decision model in generating decision strategies that meet the preset positive value conditions.

[0084] In a possible embodiment, please refer to Figure 3 Before the above step S10, the method further includes:

[0085] Step S100, obtaining preset model training parameters, and obtaining an actual decision strategy corresponding to both actual device information and actual environment information, wherein the model training parameters include a strategy network and a value network;

[0086] It should be noted that the policy network is a function used to determine the action decision in a given state. Simply put, the policy network is a mapping from state to action. The value network is a function used to evaluate the value of a state or state-action pair. It should be noted that the policy network and the value network are both model components of reinforcement learning, and this application will not go into details about the policy network and the value network. In addition, it should be noted that in addition to the policy network and the value network, the model training parameters may also include learning rate, epochs, total number of iterations, number of states, number of hidden layers, etc., and this application does not limit this.

[0087] In step S200, the actual device information and the actual environment information are converted into tensors respectively, and based on the model training parameters, the tensors and the actual decision-making strategy, the preset machine learning model is trained, and the trained machine learning model is used as the decision model.

[0088] It should be noted that in deep learning and machine learning, a tensor is a multidimensional array and one of the basic structures for processing and representing data. It is understandable that tensors have efficient computing capabilities and are easy to operate. Therefore, this application improves data processing efficiency by converting device information and environmental information into tensors.

[0089] Exemplarily, the steps of training the decision model through reinforcement learning, fine-tuning, and proximal policy optimization algorithm are as follows:

[0090] 1. Import the Pytorch library and the RL_brain library, and then define the linear layer policy network and value network through the li near method. Specifically, the main function of the policy network is to generate the probability distribution of actions in a given state, which can be achieved through the policy gradient method to ensure that the direction of the policy update can increase the total return. The value network is used to estimate the expected return of a given state, that is, the expected value of the total return that can be obtained by following the current policy starting from this state. The output of the value network can be used to calculate the advantage function to help the agent optimize the policy. Among them, the advantage function is a key component in policy optimization and is used to guide the parameter update of the policy network.

[0091] 2. Define training parameters and optimizers, such as Adam, and configure training parameters such as learning rate and epochs.

[0092] 3. Extract the content of the training data set (actual device information and actual environment information), convert the data into tensors through the tensor() method, so that the training data set can be used for model training, and set the training parameters based on the PPO function provided by the RL_brain library.

[0093] 4. Use the PPO function in the RL_brain library to build a model (build a decision model that has not been trained yet), configure the total number of iterations, the number of states, the number of hidden layers, the number of actions, the learning rate of the policy network and the value network, and the relevant parameters of the advantage function (configure training parameters). Then use the l earn() function to train the model and save the return of each iteration. When the training meets the total number of iterations, the model training ends.

[0094] In an embodiment of the present application, the present application obtains a decision strategy by transmitting environmental information and device information corresponding to each device to be controlled to a decision model, wherein the decision model is a model obtained by intensive training based on actual device information and actual environmental information and the corresponding actual decision strategy in advance, and a decision strategy matching the environmental information and device information can be obtained based on the decision model; then the decision strategy is input into a preset supervision model to obtain a strategy evaluation result, and the supervision model is trained with the actual decision strategy as model input data and the actual strategy evaluation result corresponding to the actual decision strategy as the model training label, wherein the actual strategy evaluation result is determined based on a preset positive value condition, and whether the decision strategy output by the decision model will have a negative impact can be judged based on the supervision model; then if it is detected that the strategy evaluation result meets the preset positive value condition, each device to be controlled is controlled according to the decision strategy, so that each device to be controlled can be controlled based on the decision strategy on the premise of avoiding the negative impact of the decision strategy output by the decision model.

[0095] Furthermore, based on the above-mentioned first embodiment of the model decision supervision method based on reinforcement learning of the present application, a second embodiment of the model decision supervision method based on reinforcement learning of the present application is proposed.

[0096] In this embodiment, the strategy evaluation result includes: strategy score, and the above-mentioned model decision supervision method based on reinforcement learning also includes:

[0097] Step S60, detecting whether the strategy evaluation result meets a preset positive value condition;

[0098] In a feasible implementation manner, the above step S60 includes:

[0099] Step S601, obtaining a preset strategy score threshold, and comparing the strategy score with the strategy score threshold;

[0100] It should be noted that the strategy score refers to the score output by the supervisory model after evaluating the decision strategy. The strategy score threshold is a pre-set score used to represent the satisfaction of the preset positive value condition. For example, the strategy score threshold can be 90, under the premise that the higher the score, the more the decision strategy meets the preset positive value condition.

[0101] Step S602: if it is detected that the strategy score is greater than or equal to the strategy score threshold, it is determined that the strategy evaluation result meets a preset positive value condition;

[0102] Step S603: otherwise, it is determined that the strategy evaluation result does not meet the preset positive value condition.

[0103] In this embodiment, the present application uses a score to determine whether the strategy evaluation result meets the preset positive value condition, which not only simplifies the judgment process, but also can intuitively evaluate the gap between the decision strategy and the preset positive value condition, thereby reducing the difficulty of measuring the decision-making ability of the decision model.

[0104] In a feasible implementation manner, after the above step S60, the method further includes:

[0105] Step S70, calculating a first number of times that the strategy evaluation result output by the supervisory model satisfies a preset positive value condition within a preset time period, and calculating a second number of times that the supervisory model outputs the strategy evaluation result;

[0106] For example, assuming that within a preset time period, the decision model outputs 100 decision strategies, and accordingly, the supervision model performs 100 evaluations. If the number of times the strategy evaluation results meet the preset positive value conditions is 80, the first number is 80 and the second number is 100.

[0107] Step S80, if it is detected that the ratio between the first number of times and the second number of times is greater than a preset threshold, then when it is detected that the strategy evaluation result output by the supervision model does not meet the preset positive value condition, the decision strategy corresponding to the strategy evaluation result is output, and the decision strategy after manual correction is received;

[0108] It should be noted that the preset threshold is a pre-set ratio used to evaluate the decision-making ability of the decision model.

[0109] Step S90: controlling each of the devices to be controlled based on the manually corrected decision strategy.

[0110] In this embodiment, when it is detected that the ratio of the first number to the second number is greater than the preset threshold, it indicates that the decision model has a strong ability to output decision strategies that meet the preset positive value conditions. However, at this time, a small number of decision strategies output by the decision model still do not meet the preset positive value conditions. Therefore, in order to avoid the negative impact of a small number of decision strategies that do not meet the preset positive value conditions output by the decision model, the present application can output decision strategies that do not meet the preset positive value conditions, and then correct the output decision strategies that do not meet the preset positive value conditions through manual correction to avoid the negative impact of decision strategies that do not meet the preset positive value conditions.

[0111] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the model decision supervision method based on reinforcement learning of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0112] This application also provides a model decision supervision system based on reinforcement learning, please refer to Figure 4 , the model decision supervision system based on reinforcement learning includes:

[0113] Strategy generation module 10, the user transmits the environment information and the device information of each device to be controlled to the decision model to obtain the decision strategy, wherein the decision model is a model obtained by intensive training based on the actual device information and the actual environment information and the corresponding actual decision strategy in advance;

[0114] A strategy evaluation module 20 is used to input the decision strategy into a preset supervision model to obtain a strategy evaluation result, wherein the supervision model is obtained by training with the actual decision strategy as model input data and the actual strategy evaluation result corresponding to the actual decision strategy as a model training label, wherein the actual strategy evaluation result is determined based on a preset positive value condition;

[0115] The device control module 30 is used to control the operation of each device to be controlled accordingly according to the decision strategy if it is detected that the strategy evaluation result meets the preset positive value condition.

[0116] In one embodiment, the model decision supervision system based on reinforcement learning includes:

[0117] A parameter acquisition module, used to acquire preset model training parameters and to acquire an actual decision strategy corresponding to both actual device information and actual environmental information, wherein the model training parameters include a strategy network and a value network;

[0118] The model training module is used to convert the actual device information and the actual environmental information into tensors respectively, and train the preset machine learning model based on the model training parameters, the tensors and the actual decision-making strategy, and use the trained machine learning model as the decision model.

[0119] In one embodiment, the decision strategy includes a collaborative relationship and a behavior pattern, and the device control module 30 is further used to:

[0120] Determining a target device specified by the cooperation relationship among the devices to be controlled;

[0121] In the behavior pattern, searching for a corresponding target behavior pattern of the target device, wherein the behavior pattern includes various operating parameters of the device;

[0122] The target device is controlled to operate according to the target behavior mode.

[0123] In one embodiment, the strategy evaluation result includes: a strategy score, and the model decision supervision system based on reinforcement learning also includes:

[0124] A value detection module, used to detect whether the strategy evaluation result meets the preset positive value condition;

[0125] The values ​​detection module is also used for:

[0126] Obtaining a preset strategy score threshold, and comparing the strategy score with the strategy score threshold;

[0127] If it is detected that the strategy score is greater than or equal to the strategy score threshold, it is determined that the strategy evaluation result meets the preset positive value condition;

[0128] Otherwise, it is determined that the strategy evaluation result does not meet the preset positive value condition.

[0129] In one embodiment, the model decision supervision system based on reinforcement learning further includes:

[0130] A frequency statistics module, used to calculate the first number of times that the strategy evaluation result output by the supervision model meets the preset positive value condition within a preset time period, and calculate the second number of times that the supervision model outputs the strategy evaluation result;

[0131] A manual correction module, configured to output a decision strategy corresponding to the strategy evaluation result if it is detected that the ratio between the first number of times and the second number of times is greater than a preset threshold, and receive a decision strategy after manual correction when it is detected that the strategy evaluation result output by the supervision model does not meet a preset positive value condition;

[0132] The correction control module is used to control each of the devices to be controlled based on the decision strategy after manual correction.

[0133] In one embodiment, the model decision supervision system based on reinforcement learning further includes:

[0134] A loop module is used to return to the step of transmitting the environmental information and the device information of each device to be controlled to the decision model to obtain the decision strategy if it is detected that the strategy evaluation result does not meet the preset positive value condition, until it is detected that the strategy evaluation result meets the preset positive value condition.

[0135] In one embodiment, the model decision supervision system based on reinforcement learning further includes:

[0136] The optimization module is used to transfer the strategy evaluation result to the decision model and optimize the decision model based on the strategy evaluation result.

[0137] The model decision supervision system based on reinforcement learning provided by the present application adopts the model decision supervision method based on reinforcement learning in the above-mentioned embodiment, which can solve the technical problem of how to avoid the negative impact of the data output by the model. Compared with the prior art, the beneficial effects of the model decision supervision system based on reinforcement learning provided by the present application are the same as the beneficial effects of the model decision supervision method based on reinforcement learning provided by the above-mentioned embodiment, and the other technical features of the model decision supervision system based on reinforcement learning are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0138] The present application provides a model decision supervision device based on reinforcement learning, and the model decision supervision device based on reinforcement learning includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the model decision supervision method based on reinforcement learning in the above-mentioned embodiment one.

[0139] Reference below Figure 5 , which shows a structural diagram of a reinforcement learning-based model decision supervision device suitable for implementing an embodiment of the present application. Figure 5 The reinforcement learning-based model decision supervision device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0140] like Figure 5As shown, the model decision supervision device based on reinforcement learning may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read On ly Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the model decision supervision device based on reinforcement learning are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the reinforcement learning-based model decision supervision device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a reinforcement learning-based model decision supervision device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0141] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0142] The model decision supervision device based on reinforcement learning provided by the present application adopts the model decision supervision method based on reinforcement learning in the above-mentioned embodiment, which can solve the technical problem of how to avoid the negative impact of the data output by the model. Compared with the prior art, the beneficial effects of the model decision supervision device based on reinforcement learning provided by the present application are the same as the beneficial effects of the model decision supervision method based on reinforcement learning provided by the above-mentioned embodiment, and the other technical features in the model decision supervision device based on reinforcement learning are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0143] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0144] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0145] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the model decision supervision method based on reinforcement learning in the above-mentioned embodiment.

[0146] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read On ly Memory), an erasable programmable read-only memory (EPROM: Erasable Programmab le Read On ly Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read On ly Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0147] The above-mentioned computer-readable storage medium may be included in the model decision supervision device based on reinforcement learning; or it may exist independently without being assembled into the model decision supervision device based on reinforcement learning.

[0148] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the model decision supervision device based on reinforcement learning, the model decision supervision device based on reinforcement learning: transmits environmental information and device information corresponding to each device to be controlled to the decision model to obtain a decision strategy, wherein the decision model is a model obtained by reinforcement training based on actual device information and actual environmental information and the corresponding actual decision strategy; inputs the decision strategy into a preset supervision model to obtain a strategy evaluation result, wherein the supervision model is trained with the actual decision strategy as model input data and the actual strategy evaluation result corresponding to the actual decision strategy as the model training label, wherein the actual strategy evaluation result is determined based on a preset positive value condition; if it is detected that the strategy evaluation result meets the preset positive value condition, the operation of each device to be controlled is controlled accordingly according to the decision strategy.

[0149] The computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0150] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0151] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0152] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned model decision supervision method based on reinforcement learning, and can solve the technical problem of how to avoid the negative impact of the data output by the model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the model decision supervision method based on reinforcement learning provided in the above-mentioned embodiment, and will not be repeated here.

[0153] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned reinforcement learning-based model decision supervision method.

[0154] The computer program product provided by this application can solve the technical problem of how to avoid the negative impact of the data output by the model. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the model decision supervision method based on reinforcement learning provided by the above embodiment, which will not be repeated here.

[0155] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A model decision supervision method based on reinforcement learning, It is characterized in that The model decision supervision method based on reinforcement learning includes: The environmental information and the device information of each device to be controlled are transmitted to the decision model to obtain a decision strategy, wherein the decision model is a model obtained by intensive training based on the actual device information and the actual environmental information and the corresponding actual decision strategy in advance; Input the decision strategy into a preset supervision model to obtain a strategy evaluation result, wherein the supervision model is trained using the actual decision strategy as model input data and the actual strategy evaluation result corresponding to the actual decision strategy as a model training label, wherein the actual strategy evaluation result is determined based on a preset positive value condition; If it is detected that the strategy evaluation result satisfies the preset positive value condition, the operation of each device to be controlled is controlled accordingly according to the decision strategy.

2. The model decision supervision method based on reinforcement learning as claimed in claim 1, It is characterized in that Before the step of transmitting the environmental information and the device information of each device to be controlled to the decision model to obtain the decision strategy, the method further includes: Obtaining preset model training parameters and obtaining an actual decision strategy corresponding to both actual device information and actual environment information, wherein the model training parameters include a strategy network and a value network; The actual device information and the actual environment information are each converted into a tensor, and based on the model training parameters, the tensors and the actual decision-making strategy, the preset machine learning model is trained, and the trained machine learning model is used as a decision model.

3. The model decision supervision method based on reinforcement learning as claimed in claim 1, It is characterized in that The decision strategy includes a collaborative relationship and a behavior pattern, and the step of controlling each of the devices to be controlled according to the decision strategy includes: Determining a target device specified by the cooperation relationship among the devices to be controlled; In the behavior pattern, searching for a corresponding target behavior pattern of the target device, wherein the behavior pattern includes various operating parameters of the device; The target device is controlled to operate according to the target behavior mode.

4. The model decision supervision method based on reinforcement learning as claimed in claim 1, It is characterized in that The strategy evaluation result includes: a strategy score, and the method further includes: Detecting whether the strategy evaluation result meets the preset positive value condition; The step of detecting whether the strategy evaluation result meets the preset positive value condition includes: Obtaining a preset strategy score threshold, and comparing the strategy score with the strategy score threshold; If it is detected that the strategy score is greater than or equal to the strategy score threshold, it is determined that the strategy evaluation result meets the preset positive value condition; Otherwise, it is determined that the strategy evaluation result does not meet the preset positive value condition.

5. The model decision supervision method based on reinforcement learning as claimed in claim 4, It is characterized in that After the step of detecting whether the strategy evaluation result meets the preset positive value condition, the method further includes: Calculate a first number of times that the strategy evaluation result output by the supervisory model satisfies a preset positive value condition within a preset time period, and calculate a second number of times that the supervisory model outputs the strategy evaluation result; If it is detected that the ratio between the first number of times and the second number of times is greater than a preset threshold, then when it is detected that the strategy evaluation result output by the supervision model does not meet the preset positive value condition, the decision strategy corresponding to the strategy evaluation result is output, and the decision strategy after manual correction is received; Each of the devices to be controlled is controlled based on the manually corrected decision strategy.

6. The model decision supervision method based on reinforcement learning as claimed in claim 1, It is characterized in that The method further comprises: If it is detected that the policy evaluation result does not satisfy the preset positive value condition, the step of passing the environmental information and the device information of each device to be controlled to the decision model to obtain the decision strategy is returned to execution until it is detected that the policy evaluation result satisfies the preset positive value condition.

7. The model decision supervision method based on reinforcement learning as claimed in claim 1, It is characterized in that After the step of inputting the decision strategy into a preset supervision model to obtain a strategy evaluation result, the method further includes: The strategy evaluation result is transmitted to the decision model, and the decision model is optimized based on the strategy evaluation result.

8. A model decision supervision system based on reinforcement learning, It is characterized in that The model decision supervision system based on reinforcement learning includes: In the strategy generation module, the user transmits the environment information and the device information of each device to be controlled to the decision model to obtain the decision strategy, wherein the decision model is a model obtained by intensive training based on the actual device information and the actual environment information and the corresponding actual decision strategy in advance; A strategy evaluation module, used to input the decision strategy into a preset supervision model to obtain a strategy evaluation result, wherein the supervision model is obtained by training with the actual decision strategy as model input data and the actual strategy evaluation result corresponding to the actual decision strategy as a model training label, wherein the actual strategy evaluation result is determined based on a preset positive value condition; The device control module is used to control the operation of each device to be controlled accordingly according to the decision strategy if it is detected that the strategy evaluation result meets the preset positive value condition.

9. A model decision supervision device based on reinforcement learning, It is characterized in that The reinforcement learning-based model decision supervision device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the reinforcement learning-based model decision supervision method as described in any one of claims 1 to 7.

10. A storage medium, It is characterized in that The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model decision supervision method based on reinforcement learning are implemented as described in any one of claims 1 to 7.

Images