Environmental state novelty evaluation method and device and storage medium
By using multiple sets of random noise and environmental state transition functions in high-dimensional complex environments to evaluate environmental state novelty, the problem of poor evaluation accuracy in the prior art is solved, and higher evaluation accuracy and lower computing resource consumption is achieved.
Patent Information
- Application Number
- CN202410744135.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-11
- Publication Date
- 2025-03-04
AI Technical Summary
The accuracy of environmental state novelty evaluation in the prior art is difficult to guarantee, especially in high-dimensional complex environments.
By using the environmental state transition function to obtain the low-dimensional environmental characteristic prediction value of each set of random noise and the next moment corresponding to each set of random noise, the average error is calculated as the environmental state novelty.
The accuracy of environmental state novelty evaluation is improved, and the chance of a single model and the high energy consumption of multiple parallel models are avoided.
Smart Images
Figure CN119557580A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and storage medium for evaluating the novelty of an environmental state. Background Art
[0002] Reinforcement learning is a learning method based on reward signals, which enables intelligent agents to learn effective behavior strategies in complex environments. However, reinforcement learning also faces some challenges, among which sparse rewards are one of the main difficulties.
[0003] The most effective way to overcome the sparse reward problem is to use intrinsic rewards to drive the agent. The core of designing intrinsic rewards is how to reasonably evaluate the novelty of a state. Novelty can be simply understood as the agent's familiarity with the state. If the agent frequently visits the same state, the novelty of the state will gradually decrease, while the novelty of the state that the agent rarely visits or has never visited will be higher.
[0004] In a countable state environment, the novelty of a state can be measured simply by counting the states. However, in a high-dimensional state complex environment, counting the states is not feasible and the state novelty needs to be evaluated. However, the accuracy of the current environmental state novelty evaluation is difficult to guarantee. Summary of the invention
[0005] The present invention provides an environment state novelty assessment method, device and storage medium, which are used to solve the technical problem of poor accuracy of environment state novelty assessment in the prior art.
[0006] In a first aspect, the present invention provides a method for evaluating the novelty of an environmental state, comprising: Based on multiple groups of random noises and the current environmental state, and using an environmental state transfer function, a low-dimensional environmental feature prediction value at the next moment corresponding to each group of random noises is obtained; the environmental state transfer function is used to predict the low-dimensional environmental feature at the next moment according to the current environmental state; The average error is calculated according to the true value of the low-dimensional environmental feature at the next moment and the predicted value of the low-dimensional environmental feature at the next moment corresponding to each group of random noise, and the average error is used as the novelty of the environmental state.
[0007] In some embodiments, the method further comprises: Inputting the sample environmental state data into a feature extraction network to obtain a first low-dimensional environmental feature corresponding to a first moment and a second low-dimensional environmental feature corresponding to a second moment; the feature extraction network is used to extract features from the environmental state to obtain low-dimensional environmental features; the first moment and the second moment are two adjacent moments; the sample environmental state data includes multiple sets of environmental state pairs at adjacent moments; Based on the first low-dimensional environmental features, the environmental features corresponding to the second moment are predicted using a generative network, and an environmental state transfer function is fitted based on the prediction results and the second low-dimensional environmental features; the generative network is used to predict the low-dimensional environmental features of the input environmental state at the next moment.
[0008] In some embodiments, after obtaining the first low-dimensional environment feature corresponding to the first moment and the second low-dimensional environment feature corresponding to the second moment, the method further includes: Inputting the first low-dimensional environmental feature and the second low-dimensional environmental feature into an agent strategy prediction network to obtain an agent decision prediction value; the agent strategy prediction network is used to predict the agent decision based on the environmental state; Calculate a first loss based on the agent decision true value and the agent decision predicted value; The agent policy prediction network and the feature extraction network are trained based on the first loss.
[0009] In some embodiments, the predicting the environmental features corresponding to the second moment based on the first low-dimensional environmental features and using a generative network includes: An adversarial network is generated according to the generation conditions of the generation network and the discriminant network; the discriminant network is used to discriminate whether the input is the same as the second low-dimensional environment feature, and the discriminant result is true when the input is the same as the second low-dimensional environment feature, and the discriminant result is false when the input is not the same as the second low-dimensional environment feature; The network parameters of the generation network are fixed, and the first low-dimensional environmental feature, the true value of the agent's decision and random noise are input into the generation network to obtain the predicted value of the second low-dimensional environmental feature.
[0010] In some embodiments, the step of fitting the environmental state transfer function based on the prediction result and the second low-dimensional environmental feature includes: Marking the second low-dimensional environmental feature prediction value as false, and inputting the second low-dimensional environmental feature prediction value into a discrimination module to obtain a first discrimination result; Marking the second low-dimensional environmental feature as true, and inputting the second low-dimensional environmental feature into a discrimination module to obtain a second discrimination result; Calculating a second loss based on the first discrimination result and the second discrimination result; The discriminant network is trained using the second loss and different random noises until the first discriminant result is true.
[0011] In some embodiments, the predicting the environmental features corresponding to the second moment based on the first low-dimensional environmental features and using a generative network further includes: The network parameters of the discriminant network are fixed, and the first low-dimensional environmental feature, the true value of the agent's decision and random noise are input into the generative network to obtain the predicted value of the second low-dimensional environmental feature.
[0012] In some embodiments, the step of fitting the environmental state transfer function based on the prediction result and the second low-dimensional environmental feature further includes: Inputting the predicted value of the second low-dimensional environmental feature into a discrimination module to obtain a third discrimination result; Calculate a third loss based on the third discrimination result, the second low-dimensional environmental feature and the second low-dimensional environmental feature prediction value; Training the generating network using the third loss and different random noises until the third discrimination result is true; The trained generative network is used as a model of the environment state transfer function.
[0013] In a second aspect, the present invention provides an environment state novelty assessment device, comprising a prediction module and a calculation module.
[0014] A prediction module, for obtaining a prediction value of a low-dimensional environmental feature at the next moment corresponding to each group of random noises based on multiple groups of random noises and the environmental state at the current moment and using an environmental state transfer function; the environmental state transfer function is used to predict the low-dimensional environmental feature at the next moment according to the environmental state at the current moment; The calculation module is used to calculate the average error according to the real value of the low-dimensional environmental feature at the next moment and the predicted value of the low-dimensional environmental feature at the next moment corresponding to each group of random noise, and use the average error as the novelty of the environmental state.
[0015] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for evaluating the novelty of an environmental state as described in the first aspect above is implemented.
[0016] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for evaluating the novelty of an environmental state as described in the first aspect above.
[0017] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for evaluating the novelty of an environmental state as described in the first aspect above.
[0018] The environmental state novelty evaluation method, device and storage medium provided by the present invention are based on multiple groups of random noises and the environmental state at the current moment and use the environmental state transfer function to obtain the predicted value of the low-dimensional environmental feature at the next moment corresponding to each group of random noises; the environmental state transfer function is used to predict the low-dimensional environmental feature at the next moment according to the environmental state at the current moment; the average error is calculated according to the real value of the low-dimensional environmental feature at the next moment and the predicted value of the low-dimensional environmental feature at the next moment corresponding to each group of random noises, and the average error is used as the environmental state novelty. Using multiple groups of random noise inputs, the average error of the environmental state characteristics is obtained as the environmental state novelty, avoiding the contingency of a single model and the high energy consumption of multiple parallel models, and improving the accuracy of the environmental state novelty measurement. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0020] Figure 1 It is one of the flow charts of the environmental state novelty assessment method provided by the present invention.
[0021] Figure 2 This is the second flow chart of the environmental state novelty assessment method provided by the present invention.
[0022] Figure 3 It is the intrinsic reward generation framework of the environmental state novelty evaluation method provided by the present invention
[0023] Figure 4 It is a structural schematic diagram of the environment state novelty assessment device provided by the present invention.
[0024] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0025] At present, the powerful representation ability of deep learning models is generally used to complete the evaluation of state novelty. A common practice is to use deep learning models to model the state transition dynamics in the environment. There are two main methods to obtain novelty through deep learning. One is to train a forward model and use the prediction error of the model to measure the novelty of the state. This method often has a large degree of randomness because the model can only output a single environmental state. Another method measures the novelty of the state by the uncertainty of the predicted output of multiple forward models, that is, the variance of the predicted output of multiple forward models is used as the intrinsic reward signal, and the novelty of the state is measured by the uncertainty of the predicted output of multiple forward models. This approach can more accurately measure the novelty of the state and is not easily affected by noise in the environment, but it requires training multiple forward models, which will increase the difficulty of model training and consume more computing resources and time.
[0026] Based on the above technical problems, the present invention proposes a method for evaluating the novelty of an environmental state, inputting multiple groups of random noises and the environmental state at the current moment into an environmental state transfer function to obtain a low-dimensional environmental feature prediction value at the next moment corresponding to each group of random noises, wherein the environmental state transfer function is used to predict the low-dimensional environmental features at the next moment based on the environmental state at the current moment, and then calculating the average error based on the true value of the low-dimensional environmental features at the next moment and the predicted value of the low-dimensional environmental features at the next moment corresponding to each group of random noises, and using the average error as the novelty of the environmental state, which is not easily affected by the noise in the environment, thereby improving the accuracy of the evaluation of the novelty of the environmental state.
[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] Figure 1 is one of the flow charts of the environmental state novelty evaluation method provided by the present invention, such as Figure 1 As shown, the present invention provides a method for evaluating the novelty of an environment state. The method includes the following steps 101 and 102.
[0029] Step 101: Based on multiple groups of random noises and the current environmental state, an environmental state transfer function is used to obtain a predicted value of a low-dimensional environmental feature at the next moment corresponding to each group of random noises; the environmental state transfer function is used to predict the low-dimensional environmental feature at the next moment according to the current environmental state.
[0030] Specifically, the environment state of the reinforcement learning scenario can be countable or uncountable.
[0031] First, the current environmental state is obtained, and then based on multiple groups of random noises and the current environmental state, the environmental state transfer function is used to obtain the low-dimensional environmental feature prediction value of the next moment corresponding to each group of random noises.
[0032] Optionally, the random noise includes but is not limited to any form of heat map, feature map, list and / or tuple, and the expression includes but is not limited to any form of tensor, vector and / or numerical value.
[0033] The environment state transfer function is pre-trained and is used to predict the low-dimensional environment features at the next moment based on the environment state at the current moment.
[0034] For example, get the current environment state S at time t t , multiple groups of random noise include Z1, Z2, …, Z n (n is a positive integer), get the current environmental state S t Low-dimensional environment characteristics Will And multiple groups of random noises are input into the environmental state transfer function to obtain multiple groups of low-dimensional environmental feature prediction values at the next moment, i.e., moment t+1, including the low-dimensional environmental feature prediction value at moment t+1 corresponding to Z1 and the low-dimensional environmental feature prediction value at moment t+1 corresponding to z2.
[0035] Step 102: Calculate the average error based on the real value of the low-dimensional environmental feature at the next moment and the predicted value of the low-dimensional environmental feature at the next moment corresponding to each group of random noise, and use the average error as the novelty of the environmental state.
[0036] Specifically, after the intelligent agent makes a decision and performs the corresponding action, it can obtain the true value of the low-dimensional environmental features at the next moment, calculate the average error based on the true value of the low-dimensional environmental features at the next moment and the predicted value of the low-dimensional environmental features at the next moment corresponding to each group of random noise, and use the average error as the novelty of the environmental state.
[0037] In some embodiments, environmental novelty may be defined as:
[0038] in, refers to the novelty of the environment in which the agent is located at time t; n refers to the total number of random noise samples input; error i It refers to the error between the output of the i-th noise sample and the real value representation of the low-dimensional environmental feature at the next moment. The environmental novelty can be expressed as Other calculation methods are also possible.
[0039] The environmental state novelty evaluation method provided in the embodiment of the present application uses multiple sets of random noise inputs to obtain the output average error, which can be used as a method for evaluating environmental novelty. The input of multiple sets of random noise can avoid the contingency of a single model and the high energy consumption of multiple parallel models, thereby comprehensively measuring the environmental state novelty and improving the accuracy of environmental state novelty evaluation.
[0040] In some embodiments, the method further comprises: Inputting the sample environmental state data into a feature extraction network to obtain a first low-dimensional environmental feature corresponding to a first moment and a second low-dimensional environmental feature corresponding to a second moment; the feature extraction network is used to extract features from the environmental state to obtain low-dimensional environmental features; the first moment and the second moment are two adjacent moments; the sample environmental state data includes multiple sets of environmental state pairs at adjacent moments; Based on the first low-dimensional environmental features, the environmental features corresponding to the second moment are predicted using a generative network, and an environmental state transfer function is fitted based on the prediction results and the second low-dimensional environmental features; the generative network is used to predict the low-dimensional environmental features of the input environmental state at the next moment.
[0041] Specifically, before evaluating the novelty of the environmental state, the environmental state transfer function is first fitted based on the sample environmental state data.
[0042] Optionally, the sample environment state data includes multiple sets of environment state pairs at adjacent moments, for example, one set of environment state pairs is the sample environment state s at moment t. t and the environmental state s at time t+1 t+1 .
[0043] The feature extraction network is used to extract features of the environment state to obtain low-dimensional environment state features. The input is the reinforcement learning environment state at adjacent moments, and the output is a low-dimensional reinforcement learning environment state feature pair.
[0044] Figure 2 FIG. 2 is a flow chart of the environmental state novelty evaluation method provided by the present invention. Figure 2 As shown, paired sample environmental state data such as the environmental state at the first moment (moment t) and the second moment (moment t+1) are input into the feature extraction network for feature extraction to obtain the first low-dimensional environmental feature corresponding to the first moment and the second low-dimensional environmental feature corresponding to the second moment.
[0045] Optionally, the feature extraction network can be a deep learning model, or a traditional machine learning algorithm model, such as support vector machines (SVM), K-nearest neighbor or logistic regression, or a hybrid algorithm.
[0046] Optionally, the environmental state characteristics include but are not limited to any form of heat maps, feature maps, lists and / or tuples, and the expressions include but are not limited to any form of tensors, vectors and / or numbers.
[0047] like Figure 2 As shown, after obtaining the first low-dimensional environmental feature, the first low-dimensional environmental feature is input into the generation network, the environmental feature prediction value corresponding to the second moment is output, and the environmental state transfer function is fitted based on the second environmental feature prediction value and the second low-dimensional environmental feature.
[0048] The generation network is used to predict the low-dimensional environmental features of the input environmental state at the next moment. In some embodiments, the generation network can be a deep learning network or any other type of machine learning model.
[0049] In some embodiments, after obtaining the first low-dimensional environment feature corresponding to the first moment and the second low-dimensional environment feature corresponding to the second moment, the method further includes: Inputting the first low-dimensional environmental feature and the second low-dimensional environmental feature into an agent strategy prediction network to obtain an agent decision prediction value; the agent strategy prediction network is used to predict the agent decision based on the environmental state; Calculate a first loss based on the agent decision true value and the agent decision predicted value; The agent policy prediction network and the feature extraction network are trained based on the first loss.
[0050] Specifically, the agent strategy prediction network is used to predict the agent strategy. The input is the first and second low-dimensional environmental features at adjacent moments, and the output is the agent decision prediction value.
[0051] like Figure 2 As shown, the first low-dimensional environmental feature and the second low-dimensional environmental feature are input into the intelligent agent strategy prediction network to obtain the intelligent agent decision prediction value.
[0052] Optionally, the agent strategy prediction network can be an inverse model, or other deep learning networks, or traditional machine learning models including but not limited to decision trees, SVMs, K nearest neighbors, or logistic regression.
[0053] While obtaining the sample environment state data, the agent decision that causes the transition between each pair of environment states is also obtained, that is, the true value of the agent decision.
[0054] After obtaining the agent decision prediction value by using the agent strategy prediction network, the first loss is obtained by using the agent decision true value and the agent decision prediction value. The first loss is used to train the agent strategy prediction network and the feature extraction network.
[0055] Optionally, the expression of the first loss is as follows:
[0056] Among them, L1 refers to the first loss, CrossEntropyLoss refers to the cross entropy error function, and a t Refers to the true value of the agent's decision, Refers to predicting the agent's decision prediction value. The agent strategy prediction module and feature extraction module are trained using L1 loss.
[0057] In some embodiments, the predicting the environmental features corresponding to the second moment based on the first low-dimensional environmental features and using a generative network includes: An adversarial network is generated according to the generation conditions of the generation network and the discriminant network; the discriminant network is used to discriminate whether the input is the same as the second low-dimensional environment feature, and the discriminant result is true when the input is the same as the second low-dimensional environment feature, and the discriminant result is false when the input is not the same as the second low-dimensional environment feature; The network parameters of the generation network are fixed, and the first low-dimensional environmental feature, the true value of the agent's decision and random noise are input into the generation network to obtain the predicted value of the second low-dimensional environmental feature.
[0058] Specifically, the discriminant network is used to discriminate whether the input is identical to the second low-dimensional environment feature, and the discriminant result is true when the input is identical to the second low-dimensional environment feature, and the discriminant result is false when the input is different from the second low-dimensional environment feature.
[0059] For example, the environmental features corresponding to the second moment predicted by the generative network will be Input to the discriminant network, if The environmental features corresponding to the actual second moment (i.e., the second low-dimensional environmental features) If they agree, then the judgment is true; if The second low-dimensional environment features If they are inconsistent, the result is false.
[0060] Optionally, the discriminant network may be a deep learning network or any other type of machine learning model.
[0061] Specifically, a conditional generative adversarial network is composed of a generative network and a discriminative network to fit the environment state transfer function. Figure 2 As shown, under the premise of fixing the network parameters of the generating network, the step of predicting the environmental characteristics at the second moment is performed, including: inputting the first low-dimensional environmental characteristics, the true value of the agent's decision and the random noise into the generating network to obtain the predicted value of the second low-dimensional environmental characteristics.
[0062] In some embodiments, the step of fitting the environmental state transfer function based on the prediction result and the second low-dimensional environmental feature includes: Marking the second low-dimensional environmental feature prediction value as false, and inputting the second low-dimensional environmental feature prediction value into a discrimination module to obtain a first discrimination result; Marking the second low-dimensional environmental feature as true, and inputting the second low-dimensional environmental feature into a discrimination module to obtain a second discrimination result; Calculating a second loss based on the first discrimination result and the second discrimination result; The discriminant network is trained using the second loss and different random noises until the first discriminant result is true.
[0063] Specifically, the network parameters of the generated network are fixed, the predicted value of the second low-dimensional environmental feature is obtained, the predicted value of the second low-dimensional environmental feature is marked as false, and input into the discrimination module to obtain the first discrimination result; the second low-dimensional environmental feature is marked as true, and input into the discrimination module to obtain the second discrimination result. Then, the second loss is calculated based on the first discrimination result and the second discrimination result, and the discrimination network is trained using the second loss and different random noises until the discrimination network outputs the correct result (that is, the first discrimination result is true).
[0064] Optionally, the expression of the second loss function for training the discriminant network is as follows:
[0065] Among them, L2 refers to the second loss function, D refers to the discriminant function of the discriminant network, G refers to the generating function of the generating network, z refers to random noise, P refers to the environment state transfer function, s t refers to the environmental state at time t, a t Refers to the true value of the agent's decision, Refers to the feature extraction function of the feature extraction network.
[0066] The second loss function may be L2, or other calculation methods.
[0067] In some embodiments, the predicting the environmental features corresponding to the second moment based on the first low-dimensional environmental features and using a generative network further includes: The network parameters of the discriminant network are fixed, and the first low-dimensional environmental feature, the true value of the agent's decision and random noise are input into the generative network to obtain a prediction of the second low-dimensional environmental feature.
[0068] Specifically, a conditional generative adversarial network is composed of a generative network and a discriminative network. Under the premise of fixing the network parameters of the discriminative network, a step of predicting the environmental characteristics at the second moment is performed, including: inputting the first low-dimensional environmental characteristics, the true value of the agent's decision, and random noise into the generative network to obtain the predicted value of the second low-dimensional environmental characteristics.
[0069] In some embodiments, the step of fitting the environmental state transfer function based on the prediction result and the second low-dimensional environmental feature further includes: Inputting the predicted value of the second low-dimensional environmental feature into a discrimination module to obtain a third discrimination result; Calculate a third loss based on the third discrimination result, the second low-dimensional environmental feature and the second low-dimensional environmental feature prediction value; Training the generating network using the third loss and different random noises until the third discrimination result is true; The trained generative network is used as a model of the environment state transfer function.
[0070] Specifically, the network parameters of the discriminant network are fixed, the predicted value of the second low-dimensional environmental feature is obtained, the predicted value of the second low-dimensional environmental feature is input into the discriminant module, and the third discriminant result is obtained, which is beneficial to calculate the third loss based on the third discriminant result, the second low-dimensional environmental feature and the predicted value of the second low-dimensional environmental feature, and the network is generated by using the third loss and different random noise training until the discriminant network outputs the result, that is, the third discriminant result is true.
[0071] Optionally, the expression of the third loss function used to train the generator network is as follows:
[0072] Among them, L3 refers to the third loss function, D refers to the discriminant function of the discriminant network, G refers to the generating function of the generating network, z refers to random noise, P refers to the environment state transfer function, s t refers to the environmental state at time t, a t Refers to the true value of the agent's decision, Refers to the feature extraction function of the feature extraction network, and ω is the coefficient used to adjust the two losses.
[0073] The third loss function may be L3, or other calculation methods.
[0074] Specifically, the training process of the discriminant network and the generator network is executed cyclically until the second loss function and the third loss function converge, thereby completing the optimization of the discriminant network and the generator network.
[0075] Obtain as much and comprehensive reinforcement learning environment state data as possible, and use different sample environment state data to perform the following steps in a loop: feature extraction, use the first loss to train the agent strategy prediction network and feature extraction network, and optimize the discriminant network and the generative network until the first loss, the second loss, and the third loss converge. At this point, the function G fitted by the generative network is the environment state transfer function.
[0076] Figure 3 is the intrinsic reward generation framework of the environmental state novelty evaluation method provided by the present invention, such as Figure 3 As shown, multiple groups of random noises (including random noise 1 to random noise n) and the current environmental state are input into the environmental state transfer function to obtain the low-dimensional environmental feature prediction value of the next moment corresponding to each group of random noise (including predicted environmental state 1 to predicted environmental state n); the average prediction error is calculated according to the true value of the low-dimensional environmental feature at the next moment and the predicted value of the low-dimensional environmental feature, and the average prediction error is used as the novelty of the environmental state.
[0077] After evaluating the novelty of the environment state, an agent intrinsic reward is also generated based on the environment state and the novelty of the environment state.
[0078] In some embodiments, the agent intrinsically rewards Can be evaluated as:
[0079] Among them, d is the vector dimension corresponding to the low-dimensional environment state feature, Refers to the true value of the low-dimensional environment feature at the next moment, Refers to the predicted value of the low-dimensional environmental features at the next moment, error i It refers to the error between the predicted value of the low-dimensional environmental feature corresponding to the i-th random noise and the true value representation, and n refers to the total number of random noise samples input.
[0080] The environmental state novelty assessment method provided in the embodiment of the present application constructs a conditional generative adversarial network to cross-train the discriminant network and the generative network, so that the fitted environmental state transfer function is more accurate and effective, thereby being able to more accurately assess the environmental state novelty in practical applications.
[0081] The information transmission methods provided in the above embodiments are further described below through specific examples: Example 1: Environmental state transfer function fitting.
[0082] Step 1: Obtain sample environment state data and agent decision data for reinforcement learning scenarios.
[0083] Step 2: Input the paired environmental state data obtained in step 1 into the feature extraction module (i.e., feature extraction network) to obtain low-dimensional representation of environmental state features, wherein the state at the previous moment generates the first low-dimensional environmental feature and the state at the next moment generates the second low-dimensional environmental feature.
[0084] Step 3: Input the first and second low-dimensional environmental features generated in step 2 into the agent strategy prediction module (i.e., the agent strategy prediction network) to obtain the agent decision prediction value.
[0085] Step 4: Use the actual value of the agent decision obtained in step 1 and the predicted value of the agent decision obtained in step 3 to obtain the first loss, which is used to train the agent strategy prediction module and feature extraction module.
[0086] Step 5: Input the first low-dimensional environmental feature, the true value of the agent's decision, and the random noise into the generation module (i.e., the generation network) to generate a second low-dimensional environmental feature generation value (i.e., the second low-dimensional environmental feature prediction value). The second low-dimensional environmental feature generation value is the second low-dimensional environmental feature predicted by the generation module.
[0087] Step 6: Input the first low-dimensional environment feature, the true value of the agent's decision, the second low-dimensional environment feature, and the generated value of the second low-dimensional environment feature into the discrimination module (i.e., the discrimination network) to determine whether the generated value of the second low-dimensional environment feature is consistent with the second low-dimensional environment feature. If they are consistent, it is judged to be true, otherwise it is false.
[0088] Step 7: The conditional generative adversarial network is composed of the generation module and the discrimination module. The parameters of the generation module are fixed, and step 5 is executed to mark the output as false; mark the second low-dimensional environmental feature result as true. Input them into the discrimination module respectively and calculate the second loss. Use the second loss to train the discrimination module until the output result of the discrimination module is the correct result.
[0089] Step 8: The generator module and the discriminant module form a conditional generative adversarial network, fix the discriminant module parameters, execute step 5, obtain the second low-dimensional environment feature generation value, input it to the discriminant module, and calculate the third loss. Use the third loss to train the generator module until the discriminant module outputs a true result.
[0090] Step 9: Loop through steps 5 to 8, using different random noises to fit the environment transfer function until the second loss and the third loss converge.
[0091] Step 10, using different paired data to loop through steps 2 to 9 until the first loss, the second loss, and the third loss converge. At this point, the generation module is the environmental state transfer function fitting.
[0092] Example 2: Environmental state novelty assessment.
[0093] Step 1: Input the current environment state (the environment state at time t) into the feature extraction module to obtain the first low-dimensional environment feature.
[0094] Step 2: The agent makes corresponding decisions and performs actions.
[0095] Step 3: Input the current environment state (the environment state at time t+1 in this case) into the feature extraction module to obtain the second low-dimensional environment feature.
[0096] Step 4: Input multiple groups of random noise together with the first low-dimensional environmental characteristics and the true value of the agent's decision into the environmental state transfer function, obtain multiple groups of second low-dimensional environmental characteristics, compare them with the second low-dimensional environmental characteristics, and calculate the average difference of the low-dimensional environmental characteristics as the environmental novelty.
[0097] Step 5: Combine environmental rewards and environmental novelty to generate agent intrinsic rewards.
[0098] Figure 4 is a schematic diagram of the structure of the environment state novelty assessment device provided by the present invention, such as Figure 4 As shown, the present invention provides an environment state novelty assessment device, including a prediction module 401 and a calculation module 402.
[0099] The prediction module 401 is used to obtain the low-dimensional environmental feature prediction value of the next moment corresponding to each group of random noise based on multiple groups of random noises and the environmental state at the current moment and using the environmental state transfer function; the environmental state transfer function is used to predict the low-dimensional environmental features at the next moment according to the environmental state at the current moment.
[0100] The calculation module 402 is used to calculate the average error according to the real value of the low-dimensional environmental feature at the next moment and the predicted value of the low-dimensional environmental feature at the next moment corresponding to each group of random noise, and use the average error as the novelty of the environmental state.
[0101] In some embodiments, it also includes: A first acquisition module is used to input the sample environmental state data into a feature extraction network to obtain a first low-dimensional environmental feature corresponding to a first moment and a second low-dimensional environmental feature corresponding to a second moment; the feature extraction network is used to extract features from the environmental state to obtain low-dimensional environmental features; the first moment and the second moment are two adjacent moments; the sample environmental state data includes multiple sets of environmental state pairs at adjacent moments; The second acquisition module is used to predict the environmental characteristics corresponding to the second moment based on the first low-dimensional environmental characteristics and using the generation network, and fit the environmental state transfer function based on the prediction results and the second low-dimensional environmental characteristics; the generation network is used to predict the low-dimensional environmental characteristics of the input environmental state at the next moment.
[0102] In some embodiments, it also includes: A third acquisition module is used to input the first low-dimensional environmental feature and the second low-dimensional environmental feature into an intelligent agent strategy prediction network to obtain an intelligent agent decision prediction value; the intelligent agent strategy prediction network is used to predict the intelligent agent decision based on the environmental state; A first loss acquisition module is used to calculate a first loss based on the true value of the agent decision and the predicted value of the agent decision; a training module is used to train the agent strategy prediction network and the feature extraction network based on the first loss.
[0103] In some embodiments, the second acquisition module includes: A first generating unit is used to generate an adversarial network according to the generating network and the generating network generating conditions; the discriminating network is used to discriminate whether the input is the same as the second low-dimensional environment feature, and the discriminating result is true when the input is the same as the second low-dimensional environment feature, and the discriminating result is false when the input is not the same as the second low-dimensional environment feature; The first acquisition unit is used to fix the network parameters of the generation network, and input the first low-dimensional environmental feature, the true value of the agent decision and the random noise into the generation network to obtain the predicted value of the second low-dimensional environmental feature.
[0104] In some embodiments, the second acquisition module further includes: A first discrimination unit is used to mark the second low-dimensional environmental feature prediction value as false, and input the second low-dimensional environmental feature prediction value into a discrimination module to obtain a first discrimination result; A second discrimination unit, used for marking the second low-dimensional environmental feature as true, and inputting the second low-dimensional environmental feature into a discrimination module to obtain a second discrimination result; A second loss obtaining unit, configured to calculate a second loss based on the first discrimination result and the second discrimination result; The first training unit is used to train the discriminant network by using the second loss and different random noises until a first discriminant result is true.
[0105] In some embodiments, the second acquisition module further includes: The second acquisition unit is used to fix the network parameters of the discriminant network, and input the first low-dimensional environmental feature, the true value of the agent's decision and the random noise into the generation network to obtain the predicted value of the second low-dimensional environmental feature.
[0106] In some embodiments, the second acquisition module further includes: a third discrimination unit, used for inputting the predicted value of the second low-dimensional environmental feature into a discrimination module to obtain a third discrimination result; a third loss acquisition unit, used for calculating a third loss based on the third discrimination result, the second low-dimensional environmental feature and the predicted value of the second low-dimensional environmental feature; A second training unit, configured to train the generating network using the third loss and different random noises until the third discrimination result is true; A determination unit is used to model the trained generative network as a transition function of the environment state.
[0107] Specifically, the above-mentioned environmental state novelty assessment device provided by the present invention can implement all the method steps implemented by the above-mentioned environmental state novelty assessment method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.
[0108] It should be noted that the division of units / modules in the above-mentioned embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.
[0109] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory 503 communicate with each other via the communication bus 504. The processor 501 may call the logic instructions in the memory 503 to execute the environment state novelty evaluation method, which includes: Based on multiple groups of random noises and the current environmental state, and using an environmental state transfer function, a low-dimensional environmental feature prediction value at the next moment corresponding to each group of random noises is obtained; the environmental state transfer function is used to predict the low-dimensional environmental feature at the next moment according to the current environmental state; The average error is calculated according to the true value of the low-dimensional environmental feature at the next moment and the predicted value of the low-dimensional environmental feature at the next moment corresponding to each group of random noise, and the average error is used as the novelty of the environmental state.
[0110] Specifically, the processor 501 can be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD), and the processor can also adopt a multi-core architecture.
[0111] The logic instructions in the memory 503 can be implemented in the form of software functional units and can be stored in a processor-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.
[0112] In some embodiments, a computer program product is further provided, the computer program product comprising a computer program, the computer program may be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the environmental state novelty assessment method provided by each of the above method embodiments, the method comprising: Based on multiple groups of random noises and the current environmental state, and using an environmental state transfer function, a low-dimensional environmental feature prediction value at the next moment corresponding to each group of random noises is obtained; the environmental state transfer function is used to predict the low-dimensional environmental feature at the next moment according to the current environmental state; The average error is calculated according to the true value of the low-dimensional environmental feature at the next moment and the predicted value of the low-dimensional environmental feature at the next moment corresponding to each group of random noise, and the average error is used as the novelty of the environmental state.
[0113] Specifically, the above-mentioned computer program product provided in the embodiment of the present application can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0114] In some embodiments, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program is used to enable a computer to execute the environmental state novelty assessment method provided by each of the above method embodiments, the method comprising: Based on multiple groups of random noises and the current environmental state, and using an environmental state transfer function, a low-dimensional environmental feature prediction value at the next moment corresponding to each group of random noises is obtained; the environmental state transfer function is used to predict the low-dimensional environmental feature at the next moment according to the current environmental state; The average error is calculated according to the true value of the low-dimensional environmental feature at the next moment and the predicted value of the low-dimensional environmental feature at the next moment corresponding to each group of random noise, and the average error is used as the novelty of the environmental state.
[0115] Specifically, the above-mentioned computer-readable storage medium provided by the present invention can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effects. The parts and beneficial effects that are the same as the method embodiments in this embodiment will not be described in detail here.
[0116] It should be noted that the computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.
[0117] It should also be noted that the terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are usually of the same type, and the number of objects is not limited. For example, the first object can be one or more.
[0118] In the present invention, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.
[0119] In the present invention, the term "plurality" refers to two or more than two, and other quantifiers are similar to them.
[0120] In the present invention, "determine B based on A" means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "determine B based on A and C", "determine B based on A, C and E", "determine C based on A, and further determine B based on C", etc. In addition, it can also include taking A as a condition for determining B, for example, "when A meets the first condition, use the first method to determine B"; for another example, "when A meets the second condition, determine B", etc.; for another example, "when A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "when A meets the first condition, use the first method to determine C, and further determine B based on C", etc.
[0121] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.
[0122] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer executable instructions. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0123] These processor executable instructions may also be stored in a processor readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0124] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0125] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A method for evaluating the novelty of an environmental state, characterized in that: include: Based on multiple groups of random noises and the current environmental state, and using the environmental state transfer function, a low-dimensional environmental feature prediction value at the next moment corresponding to each group of random noises is obtained; The environmental state transfer function is used to predict the low-dimensional environmental characteristics at the next moment according to the environmental state at the current moment; The average error is calculated according to the true value of the low-dimensional environmental feature at the next moment and the predicted value of the low-dimensional environmental feature at the next moment corresponding to each group of random noise, and the average error is used as the novelty of the environmental state.
2. The method for evaluating novelty of environmental status according to claim 1, characterized in that: The method further comprises: Inputting the sample environmental state data into a feature extraction network to obtain a first low-dimensional environmental feature corresponding to a first moment and a second low-dimensional environmental feature corresponding to a second moment; the feature extraction network is used to extract features from the environmental state to obtain low-dimensional environmental features; the first moment and the second moment are two adjacent moments; the sample environmental state data includes multiple sets of environmental state pairs at adjacent moments; Based on the first low-dimensional environmental features, the environmental features corresponding to the second moment are predicted using a generative network, and an environmental state transfer function is fitted based on the prediction results and the second low-dimensional environmental features; the generative network is used to predict the low-dimensional environmental features of the input environmental state at the next moment.
3. The method for evaluating novelty of environmental status according to claim 2, characterized in that: After obtaining the first low-dimensional environment feature corresponding to the first moment and the second low-dimensional environment feature corresponding to the second moment, the method further includes: Inputting the first low-dimensional environmental feature and the second low-dimensional environmental feature into an agent strategy prediction network to obtain an agent decision prediction value; the agent strategy prediction network is used to predict the agent decision based on the environmental state; Calculate a first loss based on the agent decision true value and the agent decision predicted value; The agent policy prediction network and the feature extraction network are trained based on the first loss.
4. The method for evaluating novelty of environmental status according to claim 2, characterized in that: The predicting the environmental features corresponding to the second moment based on the first low-dimensional environmental features and using a generative network includes: An adversarial network is generated according to the generation conditions of the generation network and the discriminant network; the discriminant network is used to discriminate whether the input is the same as the second low-dimensional environment feature, and the discriminant result is true when the input is the same as the second low-dimensional environment feature, and the discriminant result is false when the input is not the same as the second low-dimensional environment feature; The network parameters of the generation network are fixed, and the first low-dimensional environmental feature, the true value of the agent's decision and random noise are input into the generation network to obtain the predicted value of the second low-dimensional environmental feature.
5. The method for evaluating novelty of environmental status according to claim 4, characterized in that: The step of fitting the environmental state transfer function based on the prediction result and the second low-dimensional environmental feature comprises: Marking the second low-dimensional environmental feature prediction value as false, and inputting the second low-dimensional environmental feature prediction value into a discrimination module to obtain a first discrimination result; Marking the second low-dimensional environmental feature as true, and inputting the second low-dimensional environmental feature into a discrimination module to obtain a second discrimination result; Calculating a second loss based on the first discrimination result and the second discrimination result; The discriminant network is trained using the second loss and different random noises until the first discriminant result is true.
6. The method for evaluating novelty of environmental status according to claim 4, characterized in that: The step of predicting the environmental features corresponding to the second moment based on the first low-dimensional environmental features and using a generative network also includes: The network parameters of the discriminant network are fixed, and the first low-dimensional environmental feature, the true value of the agent's decision and the random noise are input into the generative network to obtain the predicted value of the second low-dimensional environmental feature.
7. The method for evaluating novelty of environmental status according to claim 6, characterized in that: The step of fitting the environmental state transfer function based on the prediction result and the second low-dimensional environmental feature further includes: Inputting the second low-dimensional environmental feature prediction value into a discrimination module to obtain a third discrimination result; Calculate a third loss based on the third discrimination result, the second low-dimensional environmental feature and the second low-dimensional environmental feature prediction value; Training the generating network using the third loss and different random noises until the third discrimination result is true; The trained generative network is used as a model of the environment state transfer function.
8. An environmental state novelty assessment device, characterized in that: include: A prediction module is used to obtain a low-dimensional environmental feature prediction value at the next moment corresponding to each group of random noise based on multiple groups of random noises and the environmental state at the current moment and using an environmental state transfer function; The environmental state transfer function is used to predict the low-dimensional environmental characteristics at the next moment according to the environmental state at the current moment; The calculation module is used to calculate the average error according to the real value of the low-dimensional environmental feature at the next moment and the predicted value of the low-dimensional environmental feature at the next moment corresponding to each group of random noise, and use the average error as the novelty of the environmental state.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for evaluating the novelty of an environmental state as described in any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for evaluating the novelty of an environmental state as claimed in any one of claims 1 to 7 is implemented.