A case-based reasoning based skill learning and generalization method for robot agent

Abstract

The application relates to the field of robot agent skill learning, in particular to a robot agent skill learning and generalization method based on case reasoning, which comprises the following steps: constructing a case library, obtaining a target case most similar to each state vector of a robot agent in a current task scene in the case library, obtaining an initial action strategy in the current task scene, obtaining a target action strategy, and obtaining a final action strategy. The method overcomes the problem of low action strategy explainability caused by using a deep neural network to fit an action strategy, and improves the task efficiency of the robot agent.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot agent skill learning, in particular to a robot agent skill learning and generalization method based on case-based reasoning. BACKGROUND

[0002] The learning of robot agent skills is a very hot topic at present, and reinforcement learning and its derivative methods are a main means for learning robot operation skills.

[0003] However, reinforcement learning also has inherent defects, which are particularly evident in the learning process of robot agent skills, one of which is the difficulty in transferring between similar task scenarios.

[0004] Based on this, some skills to improve the adaptability of algorithms to tasks have emerged, such as domain randomization, domain adaptation, meta-learning, etc. These methods have achieved certain results, but they are all based on the actual data or scene model information of the scene for training, and a large amount of data and computing resources are required to achieve good results, which leads to low training efficiency. Since the training process often relies on the fitting of deep neural networks, there is a lack of credible explanation of whether the trained algorithm can adapt to new task scenarios, i.e. due to the difference in environmental state between old and new scenarios, it is impossible to determine whether the action strategy trained in the old scenario can complete the task in the new scenario under the condition that people require, resulting in a lack of explainability of the training results.

[0005] Therefore, it is necessary to provide a robot agent skill learning and generalization method based on case-based reasoning to solve the above problems. SUMMARY

[0006] The present application provides a robot agent skill learning and generalization method based on case-based reasoning, which then overcomes the problem of unequal state and action dimensions in new and old task scenarios through a mapping relationship. Then, the adaptation of the action strategy in the new task scenario is realized in combination with the action of the target case and the way in which the initial action strategy is provided by human experts, and the optimization of the action strategy is realized using the parameter exploration-based policy gradient (PGPE) method, to solve the problem of lack of explainability of the training results due to the difference in environmental state between old and new scenarios, i.e. it is impossible to determine whether the action strategy trained in the old scenario can complete the task in the new scenario under the condition that people require.

[0007] The robot agent skill learning and generalization method based on case-based reasoning of the present application adopts the following technical scheme: comprising:

[0008] In the action strategy of the robot agent in the known task scenario, the state-action pair of the robot agent at each time step is taken as a case, and a case library is constructed;

[0009] obtaining a correspondence between state vectors of a current task scenario and known task scenarios, and a correspondence between action vectors; obtaining, according to the correspondence between the state vectors, one target case in a case library that is most similar to each state vector of a robot agent in the current task scenario;

[0010] obtaining an initial action policy in the current task scenario according to each action vector and state vector in the current task scenario, and a correspondence between each state vector and a state vector in a corresponding target case;

[0011] optimizing the initial action policy by using a policy gradient algorithm of parameter exploration to obtain an optimized action policy, and extending a feature parameter in the optimized action policy by using a random exploration manner to obtain a target feature parameter;

[0012] obtaining a target action policy according to the target feature parameter and the initial action policy;

[0013] optimizing the target action policy by using the policy gradient algorithm of parameter exploration to obtain a final action policy.

[0014] Preferably, the state-action pair is an action vector and a time vector corresponding to each time step.

[0015] Preferably, the step of obtaining the correspondence between the state vectors of the current task scenario and the known task scenarios, and the correspondence between the action vectors comprises:

[0016] corresponding state vectors with the same physical properties in the current task scenario and the known task scenarios;

[0017] corresponding action vectors with the same effects in the current task scenario and the known task scenarios;

[0018] Thus, the correspondence between the state vectors and the correspondence between the action vectors between the current task scenario and the known task scenarios are obtained.

[0019] Preferably, the step of obtaining one target case in the case library of the known task scenario that is most similar to the state of the robot agent in the current task scenario comprises:

[0020] constructing a similarity function of the known task scenario and the current task scenario according to a distance norm between two state vectors of the known task scenario and the current task scenario;

[0021] obtaining one target case in the case library of the known task scenario that is most similar to the state of the robot agent in the current task scenario according to the similarity function of the known task scenario and the current task scenario.

[0022] Preferably, the step of obtaining the optimized action policy comprises:

[0023] taking multiple vector values for the characteristic parameter of the initial action policy using a Gaussian distribution;

[0024] updating the initial action policy according to the characteristic parameter at each vector value to obtain an updated action policy, and executing each updated action policy by the robot agent to obtain a trajectory corresponding to each updated action policy;

[0025] presetting a corresponding reward value according to the trajectory corresponding to each updated action policy, and obtaining a mean value of the characteristic parameter of each updated action policy according to each reward value;

[0026] until the mean value of the characteristic parameter converges, taking the mean value of the characteristic parameter at this time as the characteristic parameter of the optimized action policy, and obtaining the optimized action policy.

[0027] Preferably, the step of obtaining the target characteristic parameter comprises:

[0028] executing the optimized action policy by the robot agent;

[0029] until the state vector of the robot agent reaches a preset state vector change range for the first time, randomly selecting a state vector in the state vector change range as the state vector of the robot agent at this time;

[0030] using the robot agent to execute a task according to the state vector at this time, and obtaining a reward value of the task completion after each execution of the task;

[0031] taking a target state vector corresponding to the trajectory of the maximum reward value in all reward values;

[0032] taking the vector size of the target state vector as the target characteristic parameter.

[0033] Preferably, the expression of the target action policy is:

[0034]

[0035] In the formula, θ represents the characteristic parameter of the optimized action policy, and the characteristic parameter is the characteristic parameter of the action policy;

[0036] θ new represents the target characteristic parameter;

[0037] s current represents the state vector of the robot agent at the current time;

[0038] T a (a old) represents the correspondence between the action vector of the robot agent at the current time of the current task scenario and the action vector at the current time of the known task scenario.

[0039] Preferably, before the final action policy is obtained by optimizing the target action policy using the policy gradient algorithm with parameter exploration, the method further comprises:

[0040] classifying the cases in the case base using a supervised pattern recognition method in machine learning;

[0041] The classification result is expressed in the form of a rule language description: "if the state (satisfies the condition), then perform (an action)".

[0042] The target action policy is updated according to the rule language form to obtain an updated target action policy.

[0043] Preferably, the step of updating the target action policy according to the rule language form to obtain an updated target action policy comprises:

[0044] Let the expression of the rule language description form be a old = F(s old ), which represents the mapping of the state vector to the action vector in the known task scenario, then the expression of the updated target action policy is:

[0045]

[0046] In the formula, θ represents the characteristic parameter of the initial action policy;

[0047] θ new represents the target characteristic parameter;

[0048] s current represents the state vector of the robot agent at each previous time in the current task scenario;

[0049] F(s ref ) represents the action vector in the known task scenario described using a complete rule language;

[0050] T a (F(s ref )) represents the correspondence between the action vector of the robot agent at each time in the current task scenario and the action vector at each corresponding time in the known task scenario;

[0051] In the formula, the state vector s ref is obtained from the case base through the correspondence of the state vector s ref = T s (s current ).

[0052] Preferably, the step of classifying the cases in the case base using a supervised pattern recognition method in machine learning comprises:

[0053] Each case is taken as a sample, the state vector of the case is taken as the feature vector of the sample input, and the action strategy of the case is taken as the output category of the training sample, so that the category of each case is recognized through the supervised pattern recognition method.

[0054] The present application has the following beneficial effects:

[0055] 1. By constructing a similarity function based on the state of the task scene, the state of the robot agent in the current task scene is compared with the state of the cases in the case base, the most similar case is selected as the target case, the problem of inequality between the state and action dimensions in the new and old task scenes is overcome through the mapping relationship, the adaptation of the action strategy in the new task scene is realized in combination with the action of the target case and the way of providing the initial action strategy by the human expert, and the optimization of the action strategy is realized by using the policy gradient exploration based on parameters (PGPE), so that the problem of low action strategy interpretability caused by using the deep neural network to fit the action strategy is overcome.

[0056] 2. The present application uses a complete rule-based language to express the discrete action strategy in the case base through supervised pattern recognition, and replaces the case base, so that the problem of low task efficiency caused by the need to perform traversal search on the case base when using the case base to select the action is overcome. DETAILED DESCRIPTION

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0058] Figure 1 A flowchart of an embodiment of a robot agent skill learning and generalization method based on case-based reasoning of the present application;

[0059] Figure 2 A schematic diagram of a low-dimensional mountain-car simulation scene in the present embodiment;

[0060] Figure 3 A schematic diagram of a high-dimensional mountain-car simulation scene in the present embodiment;

[0061] Figure 4 Simulation results obtained by using the initial action strategy and the target action strategy for iteration;

[0062] Figure 5 continuing iteration of the updated target action policy not dependent on the policy library. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0064] An embodiment of a robot agent skill learning and generalization method based on case-based reasoning of the present application, as shown in FIG. 1, comprises the following steps. Figure 1

[0065] S1, constructing a case library;

[0066] Specifically, the state-action pair of the robot agent at each time step in the action policy of the robot agent in the known task scene is taken as a case, and a case library is constructed.

[0067] wherein the state-action pair is the action vector and the time vector corresponding to each time step, that is, a known action policy is used to execute several rounds in the known task scene, and the state-action pair of the robot agent at each time step in the execution task process is obtained, and the state-action pair is denoted as S i i i , that is, S i is a case, wherein s i represents the state vector corresponding to the i th time step, and a i represents the action vector corresponding to the i th time step; then cases with similar states are merged, that is, a threshold value δ is set, if the state s j at the j th time step and the state s i at the i th time step satisfy: ||s i -s j || < δ, s j is deleted, s i is retained, and finally M = 600-700 cases are retained to constitute the case library C case = {S i | i = 1, 2, …, M}, and M represents the total number of cases in the case library.

[0068] S2, obtaining a target case most similar to each state vector of the robot agent in the current task scene from the case library;

[0069] ​​​Specifically, a correspondence relationship between a current task scene and a state vector of a known task scene and a correspondence relationship between action vectors are obtained, and one target case in a case library that is most similar to each state vector of the robot agent in the current task scene is obtained according to the correspondence relationship between the state vectors.

[0070] Wherein, since people master the state space and action space of the robot agent in the new task scene and the known task scene, the correspondence relationship of the state vector and the correspondence relationship of the action vector of the robot agent in the new task scene and the known task scene are set in combination with human domain knowledge. The correspondence relationship of the state vector is T S and the correspondence relationship of the action vector T a The principle followed is that the state vectors with the same physical properties correspond to each other, and the action vectors with the same effects correspond to each other. Specifically, the task scene is determined by an expert in combination, and the result is often expressed in the form of a table or "if-then" rule language.

[0071] Wherein, the step of obtaining one target case in the case library of the known task scene that is most similar to the state of the robot agent in the current task scene comprises: constructing a similarity function of the known task scene and the current task scene according to the distance norm between the two state vectors of the known task scene and the current task scene; and obtaining one target case in the case library of the known task scene that is most similar to the state of the robot agent in the current task scene according to the similarity function of the known task scene and the current task scene, that is, the expression of the similarity function of the known task scene and the current task scene is:

[0072] f sim (s current ,s old )=||(s current -s old )|| n

[0073] In the formula, f sim (s current ,s old ) represents the similarity function of the current task scene and the known task scene;

[0074] s current represents the state vector in the current task scene;

[0075] s old represents the state vector in the known task scene;

[0076] ‖ ‖ represents the distance norm;

[0077] n represents the norm number, which is generally 2.

[0078] It should be noted that the expression of the similarity function of the known task scene and the current task scene is an algebraic equation, and the embodiment uses the distance norm between the states of different task scenes as a measurement standard.

[0079] The expression of the similarity function can obtain the state-action pair of each time step in the current task scene, the similarity of each case in the case library, and select the case S case in the case library C i that minimizes the similarity function value (i.e., maximizes the similarity) as our target case S ref . ref ref , where s ref represents the state vector corresponding to the case S ref , and a ref represents the action vector corresponding to the case S ref .

[0080] Thus, we have obtained a target case in the case library of the known task scene that is most similar to each state vector of the robot agent in the current task scene.

[0081] S3, obtaining an initial action policy under the current task scene;

[0082] Specifically, according to each action vector and state vector under the current task scene, and the corresponding relationship between each state vector and the state vector in the corresponding target case, an initial action policy under the current task scene is obtained.

[0083] In this embodiment, the initial action policy under the new task scene is specified in combination with the scene knowledge observed by the human operator under the current task scene, i.e., the expression of the initial action policy is:

[0084] a current = A θ (s current , T a (a old ))

[0085] In the formula, a current represents the action vector of the robot agent at each time in the current task scene;

[0086] A θ (s current , T a (a old )) represents the initial action policy of the robot agent at each time in the current task scene; ​

[0087] T a (a old ) represents the correspondence between the action vector of the robot agent at each time of the current task scene and the action vector of the robot agent at each corresponding time of the known task scene;

[0088] a old represents the action vector of the robot agent at each time of the known task scene;

[0089] s current represents the state vector of the robot agent at each time of the current task scene;

[0090] θ represents the characteristic parameter of the initial action policy;

[0091] It should be noted that, since in actual tasks, such correspondence is often not unique, human beings need to select the characteristic parameter from s current Define the action policy A θ (·);θ represents the characteristic parameter of the initial action policy, which is selected by human operators from s current according to the scene knowledge.

[0092] S4, obtaining a target action policy;

[0093] Specifically, the initial action policy is optimized by using a parameter exploration policy gradient algorithm (PGPE algorithm) to obtain an optimized action policy, the characteristic parameter in the optimized action policy is expanded by using a random exploration method to obtain a target characteristic parameter, and the target action policy is obtained according to the target characteristic parameter and the initial action policy.

[0094] Wherein, obtaining the optimized action policy comprises: taking a plurality of vector values for the characteristic parameter of the initial action policy by using a Gaussian distribution; obtaining an updated action policy according to the characteristic parameter under each vector value updating the initial action policy, the robot agent executing each updated action policy to obtain a trajectory corresponding to each updated action policy; according to the trajectory corresponding to each updated action policy, a preset reward value is obtained, and the characteristic parameter mean value of each updated action policy is obtained according to each reward value; until the characteristic parameter mean value converges, the characteristic parameter mean value at this time is taken as the characteristic parameter of the optimized action policy, and the optimized action policy is obtained, that is, N vector values are taken for the characteristic parameter θ by using the Gaussian distribution, and the N characteristic parameters θ are represented as: θ1, θ2, …, θ N ~ Φ(μ,σ 2 ), Φ(μ,σ 2 ) represents a Gaussian distribution with mean μ and standard deviation σ, then the updated action policy of the initial action policy is: According to the robot agent to perform each updated action policy A θ (s current , T a (a ref )) and generate N segments of trajectory ρ:<ρ1,ρ2,…,ρ N According to the results of the robot agent performing the task, that is, the generated trajectory, the human sets the reward value J(ρ i ) of the robot agent each time the updated action policy is executed according to the task completion, J(ρ i ) represents the reward value of the robot agent when the i-th updated action policy is executed, that is, the setting rule of the reward value J(ρ i ) is that the reward value of each time step is-1, and if the task is completed, the reward value is 0 and the next iteration is started; otherwise, the task is ended after 500 time steps of the maximum task time; according to the reward value, the mean value of the characteristic parameters of each updated action policy is obtained, that is, the gradient of the reward value to the variance of the characteristic parameters is calculated according to the gradient ascent algorithm, and finally the update of the parameters is realized. The expression of the mean value of the characteristic parameters of the updated action policy is:

[0095]

[0096] In the formula, Δμ represents the mean value of the characteristic parameters of the updated action policy;

[0097] N represents the total number of characteristic parameters;

[0098] θ i represents the characteristic parameters of the i-th updated action policy;

[0099] μ represents the mean value of the characteristic parameters of the initial action policy;

[0100] J(ρ i ) represents the reward value of the robot agent when the i-th updated action policy is executed;

[0101] σ is the standard deviation of the characteristic parameters of the initial action policy in the Gaussian distribution;

[0102] In addition, it should be noted that the policy gradient algorithm (PGPE algorithm) is a prior art algorithm, and the present embodiment will not be described in detail.

[0103] Until the mean value of the characteristic parameters converges, the mean value of the characteristic parameters at the time of convergence is taken as the characteristic parameters of the optimized action policy, and the optimized action policy is obtained.

[0104] In the embodiment, the step of obtaining the target feature parameter includes: the robot agent executes the optimized action strategy; until the state vector of the robot agent reaches the preset state vector variation range for the first time, a state vector in the state vector variation range is randomly selected as the state vector of the robot agent at this time; the robot agent executes the task according to the state vector at this time, and obtains a reward value of the task completion after each execution of the task; the target state vector corresponding to the trajectory of the maximum reward value in all reward values is obtained; and the vector size of the target state vector is taken as the target feature parameter.

[0105] In the embodiment, it needs to be noted that the feature parameter is expanded based on the method combining the knowledge in the field and random exploration, that is, if the obtained reward value J(ρ i ) is not satisfactory, the feature parameter θ in the optimized action strategy needs to be expanded, that is, a certain state vector variation range W is first selected according to the knowledge in the field, so that the robot agent executes the optimized action strategy, when the robot agent reaches the state vector variation range W for the first time at t time, a state vector s w in the state vector variation range W is randomly selected to replace the state vector s t of the robot agent at this time, and the optimized action strategy is continuously used to execute the task until the task is completed. Specifically, the expression of the action strategy when the feature parameter is explored is as follows:

[0106]

[0107]

[0108] In the formula, s t represents the state vector of the robot agent at t time;

[0109] i represents the number of times that the robot agent enters the selected state vector variation range W;

[0110] a t represents the action vector of the robot agent at t time;

[0111] s w represents the state vector of the robot agent after replacement at t time, that is, the state vector randomly selected in the state vector variation range W;

[0112] a ref represents the action vector of the robot agent at t time in the known task scene;

[0113] T a (a ref ) represents the corresponding relationship between the action vector of the robot agent at t time and the action vector of the robot agent at t time in the known task scene.

[0114] A θ represents the action policy of the robot agent at time t;

[0115] According to the expression of the action policy during the exploration of the characteristic parameter, the K tasks are executed, that is, K trajectories are obtained, and the reward value J(ρ j ), j∈[1, K] of completing the task each time is obtained, and the trajectory corresponding to the maximum reward value in the K reward values is selected as the target state vector s k , and the vector size of s k is taken as the target characteristic parameter, and the expression of the target characteristic parameter is:

[0116]

[0117] In the formula, θ new represents the target characteristic parameter;

[0118] J(ρ j ) represents the jth reward value in the K reward values;

[0119] s k represents the target state vector corresponding to the trajectory of the maximum reward value in the K reward values;

[0120] max represents the maximum value;

[0121] It should be noted that if the reward values J(ρ j ), j∈[1, K] obtained after the expansion of the action policy are all less than the reward value obtained by using the initial action policy, the state vector variation range W or the value of the task number K is increased until the satisfactory J(ρ j ) is selected.

[0122] Wherein, the target action policy is obtained according to the target characteristic parameter and the initial action policy, and the expression of the target action policy is:

[0123]

[0124] In the formula, θ represents the characteristic parameter of the optimized action policy, and the characteristic parameter is the characteristic parameter of the action policy;

[0125] θ new represents the target characteristic parameter;

[0126] s current represents the state vector of the robot agent at each time in the current task scene;

[0127] T a (a old) represents the correspondence between the action vector of the robot agent at each time in the current task scenario and the action vector at each time in the known task scenario.

[0128] It should be noted that the target action policy is: when the robot agent moves according to the initial action policy A θ (s current ,T a (a old )) and enters the set state vector change range W for the first time, the robot agent will move towards the vector direction of the target feature parameter θ new , until the target feature parameter θ new or the task boundary is reached, and then the robot agent continues to move according to the initial action policy A θ (s current ,T a (a old )) until the task is completed.

[0129] S5, obtaining a final action policy;

[0130] Specifically, the target action policy is optimized by a parameter exploration policy gradient algorithm to obtain the final action policy.

[0131] Since the use of the target action policy depends on the action vector a old in the known task scenario, the selection of the action vector a old in the known task scenario depends on the a ref in the target case, and the use of the a ref in the target case requires traversal search of the case library, which is not conducive to improving the task execution efficiency, therefore, it is necessary to extract the mathematical expression of the relationship between the state vector and the action vector of all cases in the case library, and to correspond the state vector change range and the action policy one by one.

[0132] In order to improve the efficiency of the algorithm, the embodiment uses a supervised pattern recognition method in machine learning to classify the cases in the case base; wherein the classification result is expressed in the form of a rule language description: "if the state (satisfies the condition), then execute (an action)"; the target action strategy is updated according to the form of the rule language to obtain an updated target action strategy; the updated target action strategy is used as the target action strategy for obtaining the final target action strategy. Specifically, the supervised pattern recognition method in machine learning is used to classify the cases in the case base: each case is a sample, the state vector of the case is the feature vector of the sample, and the action strategy of the case is the category of the sample. By training the supervised pattern recognition method, the optimal classification hyperplane of different categories can be found, and the classification result can always be expressed in the form of "if the state (satisfies the condition), then execute (an action)". Therefore, the role of case classification of the case base is to describe the action strategy under the original environmental condition using complete rule language. Let the mathematical expression of the action strategy under the original environmental condition be: old a old = F(s old ), which represents the mapping from the state vector to the action vector in the known task scenario, i.e. the corresponding action at a state. The expression a old = F(s new ) in the expression of the target action strategy represents the transformation from the action to the state, and the updated target action strategy is obtained. At this time, the action strategy under the current task scenario is the updated target action strategy, i.e. the expression is:

[0133]

[0134] wherein θ represents the characteristic parameter of the initial action strategy;

[0135] θ new represents the target characteristic parameter;

[0136] s current represents the state vector of the robot agent at each previous time in the current task scenario;

[0137] F(s ref ) represents the action vector under the known task scenario described using complete rule language;

[0138] T a (F(s ref )) represents the correspondence between the action vector of the robot agent at each time in the current task scenario and the action vector of the corresponding time in the known task scenario;

[0139] wherein the correspondence of the state vector is s ref = T s (scurrent )get;

[0140] Finally, the updated target action policy is optimized using a policy gradient algorithm based on parameter exploration to obtain the final action policy.

[0141] It also includes verification of the method of the present invention:

[0142] To demonstrate the effectiveness of the framework proposed in this patent, this embodiment uses a mountain-car task scenario as an example to verify that the method proposed in the patent can meet the needs of mobile robots for efficient skill transfer and learning from simple low-dimensional scenarios to complex high-dimensional task scenarios. In a low-dimensional mountain-car training scenario, such as... Figure 2 As shown, the robot agent starts from the lowest point and can only move horizontally left or right, that is, by applying leftward or rightward acceleration to itself. Its vertical position changes with the undulations of the terrain. Its goal is to reach the flagpole at the highest point within a specified time. Therefore, the state vector space s of the task environment we are interested in includes the horizontal position x and the horizontal velocity. Represented as In this environment, the state space range is: x∈[-1.5,0.7], The action vector includes three options: "left," "right," and "no action." However, in high-dimensional mountain-car training scenarios, such as... Figure 3 As shown, its characteristics are that the terrain cross-section viewed from the x or y direction is similar to that of the low-dimensional scene, and the motion target of the robot agent is also similar to that of the low-dimensional scene, both aiming to reach the highest point target within a specified time. However, since the task space now has an additional spatial degree of freedom, the state vector space s of the robot agent is expanded to the horizontal position x and y, and the horizontal velocity... Represented as Since the horizontal position determines the robot's vertical position, the position and velocity in the vertical z-direction are not included in the state vector space. Therefore, the range of the state space in this environment is: x, y ∈ [-1.5, 0.7]. The action vector space is expanded to include five action options: "North", "West", "East", "South", and "No Action". "North" and "South" apply acceleration along the positive and negative directions of the y-axis, respectively; "West" and "East" apply acceleration along the positive and negative directions of the x-axis, respectively; and "No Action" does not apply acceleration.

[0143] Step 1, in particular in combination with the method of the present application, the low-dimensional mountain-car scenario is a known task scenario, the high-dimensional mountain-car scenario is a current task scenario, first, the case library is generated, that is, the action strategy is trained in the low-dimensional mountain-car task scenario, the deep Q-learning method is used for training, and after 1600 rounds of skill training, the training is completed. Then use the trained action strategy to perform several tasks, and take the state-action pair S i = <s i , a i as a case at each time step. Finally, cases with similar states are merged, and M = 600-700 cases are retained to form a case library C case = {S i |i = 1, 2, …, M}.

[0144] Step 2, set the corresponding relationship between the state variables and the action variables between the high-dimensional mountain-car task scenario and the low-dimensional mountain-car task scenario, the principle followed is that the state variables with the same or similar physical properties correspond to each other, and the action variables with the same or similar effects correspond to each other. Therefore, we correspond the state vectors representing the position to each other, and correspond the action vectors representing the speed to each other; correspond the action vectors moving towards the target to each other, and correspond the action vectors moving away from the target to each other. The corresponding relationship of the state vector T S is as shown in Table 1 below:

[0145] Table 1

[0146]

[0147] The corresponding relationship of the action vector T a is as shown in Table 2 below:

[0148] Table 2

[0149] High-dimensional scenario Low-dimensional scenario No action No action North Right East Right South Left West Left

[0150] In Table 2, "north" and "south" respectively represent applying acceleration along the positive and negative directions of the y-axis in Figure 3 ; "west" and "east" represent applying acceleration along the positive and negative directions of the x-axis; "no action" means no acceleration is applied.

[0151] Get the most similar target case, that is, first construct a similarity function, in the high-dimensional mountain-car task scenario, the state variable that can be taken is: Use the distance norm between the corresponding state vectors to construct the similarity function between the new and old task scenarios:

[0152]

[0153] wherein, represents a state vector under a high-dimensional task scenario; represents a low-dimensional task scenario state vector, the low-dimensional task scenario state vector being obtained from a case library C case .

[0154] Then, a case with the highest similarity (i.e., the smallest similarity function) is selected as a target case under the high-dimensional task scenario, and the target case is represented as S ref = <s ref , a ref >, wherein represents a state vector corresponding to the case S ref , and a ref represents an action vector of the case S ref , and thus, the target case under the new task scenario can be obtained.

[0155] Step 3, based on the correspondence relationship between the action vectors of the high-dimensional mountain-car task scenario and the low-dimensional mountain-car task scenario, and in combination with the scenario knowledge obtained by the human operator observing the new task scenario, an initial action strategy A θ (·) under the new task scenario is defined as follows:

[0156]

[0157] wherein a current refers to the action of the current robot intelligent agent;

[0158] T a (a old ) represents the correspondence relationship of the action variable;

[0159] x current represents an x-coordinate position under the high-dimensional task scenario;

[0160] θ represents a characteristic parameter.

[0161] Step 4, the initial action strategy is optimized by using a parameter exploration-based policy gradient algorithm to obtain an optimized action strategy; the characteristic parameter in the optimized action strategy is expanded by using a random exploration method to obtain a target characteristic parameter, that is, according to the step S4, the setting rule of the reward value J(ρ i ) is that the reward value at each time step is -1, and if the task is completed, the reward value is 0 and the next iteration is started; otherwise, the task is ended after 500 time steps of the maximum task time, then the gradient of the reward value with respect to the characteristic parameter variance is calculated according to the gradient ascent algorithm, and finally the parameter is updated, in this embodiment, the mean value μ of the characteristic parameter converges to about 0.3, and the reward value J(ρ iOnce the value stabilizes at around -297, the feature parameters are taken to be equal to the mean of the feature parameters at convergence. At this point, the action strategy optimization is complete, and the optimized action strategy is obtained.

[0162] Then, the feature parameters in the optimized action strategy are expanded using a random exploration method to obtain the target feature parameters. This involves selecting a certain range of state vector changes, W, based on domain knowledge. In this example, the initial y-position of the robot is changed for exploration, and W ∈ [-0.7, -0.3]. This is because the initial action strategy expression only constrains the x-direction; changing the y-direction position might alter the robot's velocity and contribute to task completion. When the robot begins to move, the state parameter y in W is randomly selected. w Replace the current position of the robot agent in the y-direction. t That is, in the expression of the action strategy when exploring feature parameters at this time, i = 0.

[0163] Then, the target action policy is obtained based on the target feature parameters and the initial action policy. When unfolded, it becomes:

[0164]

[0165] In this example, the optimized feature parameter θ = 0.15, and the target feature parameter θ new =0.35, at this point the reward value J(ρ) i The value stabilizes at around -260. Specifically, the simulation graphs for the reward values ​​of the optimized initial and target action strategies are shown below. Figure 4 As shown, Figure 4 In the first 10 rounds, the results are optimized using the initial action strategy; in the middle 10 rounds, the reward value fluctuates greatly because the value is randomly selected within the range W of the state vector change; and in the last 20 rounds, the results are optimized based on the expanded target action strategy. It can be seen that the expansion of the action strategy can significantly improve the quality of the reward value.

[0166] Step 5: Optimize the target action policy using the policy gradient algorithm that utilizes parameter exploration to obtain the final action policy;

[0167] Specifically, the decision tree classification algorithm is used to classify the cases in the case library. This is a mature machine learning algorithm with high computational efficiency. Ultimately, multiple optimal classification hyperplanes for action policies in a low-dimensional task environment are obtained, which can be represented as: a old =F(s) old ),available:

[0168]

[0169] Based on this, we obtain the updated target action policy with a complete description of the final high-dimensional task environment, expressed as:

[0170]

[0171] Then, the target action policy is optimized using the policy gradient algorithm based on parameter exploration to obtain the final action policy, i.e., the optimized feature parameter θ = -0.12, and the target feature parameter θ new =0.25, such as Figure 5 As shown, the reward value J(ρ) at this time i It stabilized at around -196, such as Figure 5 As shown, the final action strategy for the high-dimensional scene has now been obtained; from Figure 5 It can be observed that starting from round 40, the updated target action strategy is used, and after about 3 more iterations, the reward value increases significantly again. This indicates that action strategies that do not rely on the strategy library are more efficient than methods that consult the strategy library.

[0172] Therefore, through verification, the following conclusions can be drawn: (1) The transfer of robot agent skills is essentially the expansion of the robot agent's knowledge of scene understanding, and the comparison of similarity is an effective means of understanding new task scenarios. On this basis, combined with certain prior knowledge of human domains, a basic understanding of new task scenarios can be formed; (2) Continuous and complete action strategies are superior to discrete action strategies that depend on case libraries, and it is easier to find better state sequence paths when using reinforcement learning algorithms for optimization.

[0173] In summary, the robot agent skill learning and generalization method based on case reasoning provided by this invention constructs a similarity function based on the task scenario state, then compares the robot agent's state in the current task scenario with the states of cases in the case library, selecting the most similar case as the target case. This overcomes the problem of unequal state and action dimensions in new and old task scenarios through mapping relationships. Furthermore, it adapts the action strategy to new task scenarios by combining the action of the target case with the initial action strategy provided by human experts. The method of Parameter Exploration-Based Policy Gradient (PGPE) is used to optimize the action strategy, overcoming the problem of low interpretability of action strategies caused by using deep neural networks to fit action strategies. Finally, this invention uses supervised pattern recognition on discrete action strategies in the case library to express them using a complete rule-based language, replacing the case library. This overcomes the limitation of low algorithm efficiency caused by traversing and searching the case library for action selection.

[0174] The above merely provides the preferred embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A case-based reasoning based skill learning and generalization method for robot agent, the method is applied to a mobile robot, and the method is characterized in that, The application comprises the following steps: a case library is constructed by taking the state-action pair of the robot agent at each time step in the action policy of the robot agent in a known task scenario as a case; a correspondence between the state vectors of the current task scenario and the known task scenario and a correspondence between the action vectors are obtained; one target case in the case library that is most similar to the state of the robot agent in the current task scenario is obtained according to the correspondence between the state vectors; an initial action policy in the current task scenario is obtained according to each action vector and state vector in the current task scenario and the correspondence between each state vector and the state vector in the corresponding target case; the initial action policy is optimized by using a parameter exploration policy gradient algorithm to obtain an optimized action policy; a target feature parameter is obtained by extending the feature parameter in the optimized action policy in a random exploration manner; the step of obtaining the target feature parameter comprises the following steps: the robot agent executes the optimized action policy; when the state vector of the robot agent reaches a preset state vector variation range for the first time, a state vector in the state vector variation range is randomly selected as the state vector of the robot agent at this time; the robot agent executes a task according to the state vector at this time and obtains a reward value of the task after each execution; a target state vector corresponding to the trajectory of the maximum reward value in all reward values is obtained; and the vector size of the target state vector is taken as the target feature parameter; a target action policy is obtained according to the target feature parameter and the initial action policy; and the target action policy is optimized by using the parameter exploration policy gradient algorithm to obtain a final action policy.

2. The case-based reasoning based skill learning and generalization method for a robot agent according to claim 1, characterized in that, In the application, the state-action pair is the action vector and the time vector corresponding to each time step. 3.The case-based reasoning based skill learning and generalization method for a robot agent according to claim 1, wherein, The step of obtaining the correspondence between the state vectors of the current task scenario and the known task scenario and the correspondence between the action vectors comprises the following steps: the state vectors with the same physical properties in the current task scenario and the known task scenario are corresponded; the action vectors with the same effects in the current task scenario and the known task scenario are corresponded; and the correspondence between the state vectors and the correspondence between the action vectors between the current task scenario and the known task scenario are obtained.

4. The case-based reasoning based skill learning and generalization method for a robot agent according to claim 1, wherein, The step of obtaining one target case in the case library of the known task scenario that is most similar to the state of the robot agent in the current task scenario comprises the following steps: a similarity function of the known task scenario and the current task scenario is constructed according to the distance norm between the two state vectors of the known task scenario and the current task scenario; one target case in the case library of the known task scenario that is most similar to the state of the robot agent in the current task scenario is obtained according to the similarity function of the known task scenario and the current task scenario.

5. The case-based reasoning based skill learning and generalization method for a robot agent according to claim 1, wherein, The step of obtaining the optimized action policy comprises the following steps: a plurality of vector values are taken for the feature parameter of the initial action policy by using a Gaussian distribution; an updated action policy is obtained by updating the initial action policy according to the feature parameter under each vector value; the robot agent executes each updated action policy to obtain a trajectory corresponding to each updated action policy; and The reward value corresponding to each updated action policy is preset according to the trajectory corresponding to the updated action policy, and a feature parameter mean value of each updated action policy is obtained according to each reward value; Until the feature parameter mean value converges, the feature parameter mean value at this time is taken as a feature parameter of an optimized action policy, and the optimized action policy is obtained.

6. The case-based reasoning based skill learning and generalization method for a robot agent according to claim 1, wherein, The expression of the target action policy is: In the formula, represents the target action policy; characteristic parameters representing an initial action policy; representing a target feature parameter; x(t) represents the state vector of the robot agent at the current time instant; represents a correspondence between an action vector of the robot agent at a current time instant of a current task scenario and an action vector of a known task scenario at the current time instant; represents an action vector of the robot agent at each time instant of the known task scenario.

7. The case-based reasoning based skill learning and generalization method for a robot agent according to claim 1, wherein, Before the target action policy is optimized by using the parameter exploration-based policy gradient algorithm to obtain a final action policy, the method further includes: using a supervised pattern recognition method in machine learning to classify the cases in the case library; The classification result is expressed in the form of a rule language description of "if the state (satisfies the condition), then execute (a certain) action"; The target action policy is updated according to the rule language form to obtain an updated target action policy; The updated target action policy is taken as the target action policy for obtaining the final target action policy. 8.The case-based reasoning based skill learning and generalization method of a robot agent according to claim 7, wherein, The step of updating the target action policy according to the rule language form to obtain the updated target action policy includes: Let the expression of the description form of the rule-based language be , which represents the mapping of the state vector to the action vector in the known task scenario. Then the expression of the updated target action policy is In the formula, represents the updated target action policy; characteristic parameters representing an initial action policy; representing a target feature parameter; represents the state vector of the robotic agent at each time instant in the current task scenario; represents action vectors in known task scenarios described using a complete rule-based language; a correspondence between an action vector of the robot agent at each time instant of the current task scenario and an action vector of each corresponding time instant of the known task scenario; where the state vector The corresponding relationship of the state vector is obtained by from the case library. 9.The case-based reasoning based skill learning and generalization method of a robot agent according to claim 7, wherein, The step of using the supervised pattern recognition method in machine learning to classify the cases in the case library includes: Each case is taken as a sample, a state vector of the case is taken as a feature vector of a sample input, an action policy of the case is taken as a training sample output category, and the category of each case is recognized by the supervised pattern recognition method.

Images