Robot obstacle avoidance method and device based on migration reinforcement learning

Abstract

The embodiment of the invention provides a robot obstacle avoidance method and device based on migration reinforcement learning. The method is applied to the technical field of robot obstacle avoidance control, and comprises the following steps: acquiring a target domain data set, preprocessing the data set, inputting the preprocessed data set into a trained Transform model, and outputting a new data sample through the Transform model; mixing the original data sample and the new data sample to obtain a target domain synthetic data set; performing target domain data distribution modeling according to the target domain synthetic data set by adopting a conditional variation auto-encoder, and determining a state transition probability of data on a target domain; determining the state transition probability of the data on the source domain according to the real-time interaction data of the state action of the robot by using a dual-binary classifier architecture; based on the state transition probability of the target domain and the state transition probability of the source domain, the robot obstacle avoidance strategy trained on the source domain is migrated, the cost loss in the training process is reduced, and the environmental adaptability of the obstacle avoidance strategy is improved.

Description

Technical Field

[0001] This application relates to the technical field of robot obstacle avoidance control, and particularly to a robot obstacle avoidance method and device based on transfer reinforcement learning. Background Art

[0002] With the vigorous development of artificial intelligence technology, robots have gradually integrated into people's production and life. For example, floor-sweeping robots, inspection robots, and robotic dogs have brought convenience to all aspects of people's lives. Among them, the obstacle avoidance function is a necessary basis for robots to move quickly.

[0003] Most current robot obstacle avoidance strategies are directly implemented using online reinforcement learning algorithms, such as the Proximal Policy Optimization (PPO) algorithm, the Deep Q-Network (DQN) algorithm, etc.

[0004] However, the training process of online reinforcement learning algorithms requires a large amount of interaction with the environment online to learn the potential features in the environment. This process will cause wear and tear on the robot device, resulting in high training costs for the algorithm. Moreover, once the environment changes slightly, the obstacle avoidance algorithm needs to be re-learned and adjusted, and its adaptability is also very limited. Summary of the Invention

[0005] This application provides a robot obstacle avoidance method and device based on transfer reinforcement learning to solve the problems of high training costs and poor environmental adaptability of existing robot obstacle avoidance algorithms.

[0006] In a first aspect, this application provides a robot obstacle avoidance method based on transfer reinforcement learning, and the method includes:

[0007] Obtain a target domain data set, where the target domain data set includes: trajectory data, and preprocess the trajectory data;

[0008] Use the preprocessed trajectory data as the original data sample and input it into the trained Transformer network model, and output a new data sample through the Transformer network model;

[0009] Mix the original data sample and the new data sample to obtain a target domain synthetic data set;

[0010] Use a conditional variational autoencoder to model the data distribution of the target domain according to the target domain synthetic data set, and determine the state transition probability of the data on the target domain;

[0011] Obtain the real-time interaction data of the robot's action state, and use the dual binary classifier architecture to determine the state transition probability of the data on the source domain according to the real-time interaction data;

[0012] Transfer the obstacle avoidance strategy of the robot trained on the source domain based on the state transition probability of the data on the target domain and the state transition probability of the data on the source domain.

[0013] Optionally, the step of using the conditional variational autoencoder to model the data distribution of the target domain according to the synthetic dataset of the target domain to obtain the state transition probability of the data on the target domain includes:

[0014] Decompose the trajectory data into the first state data at the next moment and the state-action data at the current moment;

[0015] Recombine the first state data and the state-action data at the current moment to obtain recombined state-action data;

[0016] Input the recombined state-action data into the encoder of the conditional variational autoencoder for analysis and processing to obtain the mean and variance of the recombined state-action data;

[0017] Recombine the state-action data at the current moment according to the data distribution to obtain a conditional state representation, and input the conditional state representation into the decoder of the conditional variational autoencoder for analysis and processing to obtain the second state data at the next moment;

[0018] Determine the loss function value of the conditional variational autoencoder according to the first state data and the second state data;

[0019] Determine the state transition probability of the data on the target domain according to the loss function value of the conditional variational autoencoder.

[0020] Optionally, the dual binary classifier architecture includes: a first classifier and a second classifier. The step of using the dual binary classifier architecture to determine the state transition probability of the data on the source domain according to the real-time interaction data includes:

[0021] Analyze and process the state data, action data and the state data at the next moment of the robot through the trained first classifier, and output a first conditional probability;

[0022] Analyze and process the state data and action data of the robot at the current moment through the trained second classifier, and output a second conditional probability;

[0023] Determine the state transition probability of the data on the source domain according to the first conditional probability and the second conditional probability.

[0024] Optionally, the state transition probability of the data on the source domain satisfies the following formula:

[0025] logp_source(s′|s,a) = logp(source|s,a,s′) - logp(source|s,a)

[0026] Among them, p_source(s'|s,a) is the state transition probability of the data in the source domain, s is the state at the current moment, a is the action at the current moment, s' is the state at the next moment, p(source|s,a,s') is the first conditional probability, and p(source|s,a) is the second conditional probability.

[0027] Optionally, migrating the robot obstacle avoidance strategy trained on the source domain based on the state transition probability of the data in the target domain and the state transition probability of the data in the source domain includes:

[0028] Judging whether the state transition probability of the data in the target domain is consistent with the state transition probability of the data in the source domain;

[0029] When the state transition probability of the data in the target domain is consistent with the state transition probability of the data in the source domain, migrating the robot obstacle avoidance strategy according to the original reward value in the source domain;

[0030] When the state transition probability of the data in the target domain is inconsistent with the state transition probability of the data in the source domain, correcting the original reward value in the source domain according to the difference between the state transition probability of the data in the target domain and the state transition probability of the data in the source domain, and migrating the robot obstacle avoidance strategy according to the corrected reward value.

[0031] Optionally, correcting the original reward value in the source domain according to the difference between the state transition probability of the data in the target domain and the state transition probability of the data in the source domain satisfies the following formula:

[0032] r adapted (s,a,s′) = r source (s,a) + β · (log p target (s′|s,a) - log p source (s′|s,a))

[0033] Among them, s is the state at the current moment, a is the action at the current moment, s' is the state at the next moment, r adapted (s,a,s') is the corrected reward value, r source (s,a) is the original reward value in the source domain, β is the balance parameter, p target (s'|s,a) represents the state transition probability of the data in the target domain, p source (s'|s,a) represents the state transition probability of the data in the source domain.

[0034] In a second aspect, the present application provides a robot obstacle avoidance device based on transfer reinforcement learning, and the device includes:

[0035] An acquisition module, configured to acquire a target domain data set, where the target domain data set includes: trajectory data;

[0036] A processing module, configured to preprocess the trajectory data;

[0037] The processing module is further configured to input the preprocessed trajectory data as an original data sample into a trained Transformer network model, and output a new data sample through the Transformer network model;

[0038] The processing module is further configured to mix the original data sample and the new data sample to obtain a target domain synthetic data set;

[0039] A determination module, configured to use a conditional variational autoencoder to model the data distribution of the target domain according to the target domain synthetic data set, and determine the state transition probability of the data on the target domain;

[0040] The acquisition module is further configured to acquire real-time interaction data of the robot action state;

[0041] The determination module is further configured to use a dual binary classifier architecture to determine the state transition probability of the data on the source domain according to the real-time interaction data;

[0042] The processing module is further configured to transfer the robot obstacle avoidance strategy trained on the source domain based on the state transition probability of the data on the target domain and the state transition probability of the data on the source domain.

[0043] In a third aspect, the present application provides a robot obstacle avoidance device based on transfer reinforcement learning, including:

[0044] A memory;

[0045] A processor;

[0046] Wherein, the memory stores computer execution instructions;

[0047] The processor executes the computer execution instructions stored in the memory to implement the robot obstacle avoidance method based on transfer reinforcement learning as described in the first aspect and various possible implementation manners of the first aspect.

[0048] In a fourth aspect, the present application provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the robot obstacle avoidance method based on transfer reinforcement learning as described in the first aspect and various possible implementation manners of the first aspect.

[0049] In a fifth aspect, the present application provides a computer program product, including a computer program, which when executed by a processor, implements the robot obstacle avoidance method based on transfer reinforcement learning as described in the above first aspect and various possible implementation manners of the first aspect.

[0050] The present application provides a robot obstacle avoidance method and device based on transfer reinforcement learning. The method obtains a target domain data set, preprocesses the data set and inputs it into a trained Transformer model, and outputs new data samples through the Transformer model; mixes the original data samples and the new data samples to obtain a target domain synthetic data set; uses a conditional variational autoencoder to model the data distribution of the target domain based on the target domain synthetic data set to determine the state transition probability of the data on the target domain; uses a double binary classifier architecture to determine the state transition probability of the data on the source domain based on the real-time interaction data of the robot state actions; based on the state transition probability of the target domain and the state transition probability of the source domain, migrates the robot obstacle avoidance strategy trained on the source domain, reduces the cost loss in the training process, and increases the environmental adaptability of the obstacle avoidance strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0052] Figure 1 It is a flowchart of the robot obstacle avoidance method based on transfer reinforcement learning provided by an embodiment of the present application Figure 1 ;

[0053] Figure 2 It is a schematic diagram of the CVAE processing flow provided by an embodiment of the present application;

[0054] Figure 3 It is a structural diagram of the double binary classifier model provided by an embodiment of the present application;

[0055] Figure 4 It is a schematic diagram of the structure of the robot obstacle avoidance device based on transfer reinforcement learning provided by an embodiment of the present application;

[0056] Figure 5 It is a schematic diagram of the structure of the robot obstacle avoidance device based on transfer reinforcement learning provided by an embodiment of the present application.

[0057] Through the above accompanying drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These accompanying drawings and the written description are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below with reference to the accompanying drawings in this application. Apparently, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the protection scope of this application.

[0059] In the specification and claims of this invention and the above accompanying drawings, terms such as "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this invention described here can be implemented in an order different from those illustrated or described here.

[0060] In the embodiments of this application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0061] In practical applications, the obstacle avoidance function of robots mainly relies on the Reinforce Learning (RL) algorithm to achieve. Through the interaction with the environment, the robot can learn the optimal strategy and adapt to complex and dynamic environments without explicit modeling. During the training process, robots with centimeter-level positioning need to interact with the environment a large number of times to learn an excellent strategy. Such interactions will undoubtedly bring high costs under real conditions. The robot may not be able to avoid obstacles in time during the training process and collide with obstacles, which may lead to the loss of hardware devices. For example, in the warehousing and logistics scenario, the robot needs to shuttle between narrow shelf aisles to transport goods. Frequent collisions will not only damage components such as the robot's own sensors and casings but may also damage the goods on the shelves, increasing the operating cost. Secondly, the time cost required for training is relatively high. To achieve an ideal obstacle avoidance effect, the robot needs to conduct a large number of tests and learn in various complex scenarios, which may take weeks or even months.

[0062] Implementing robot obstacle avoidance through the offline reinforcement learning algorithm can reduce the trial-and-error cost to a certain extent. However, during the process of training the obstacle avoidance strategy, it highly depends on the data quality and data scale. When the data quality is low or the scale is small, the trained strategy often cannot achieve normal obstacle avoidance. In addition, due to the fact that data in different environments cannot be reused, there is also a cost for obtaining the data itself.

[0063] In view of the above problems, the present application proposes a robot obstacle avoidance method based on transfer reinforcement learning. By using a Transformer-based trajectory data generation method, the temporal change relationship of target domain data is captured, and more state-action pair samples that conform to the dynamics of the target domain are generated, improving the sample efficiency and diversity of target domain data. By calculating the online interaction data of the robot source domain using a double binary classifier, the state-action transition probability of the source domain is obtained, and combined with a conditional variational autoencoder to model the offline data of the target domain, learning the latent feature representation of the target domain environment. According to the difference between the state transfer distributions of the source domain and the target domain, the reward function of the source domain training strategy is adjusted, and a robot obstacle avoidance strategy applicable to the target domain is trained on the source domain using reinforcement learning, greatly reducing the cost loss during the training process and also increasing the environmental adaptability of the obstacle avoidance strategy.

[0064] The following uses specific embodiments to elaborate in detail on the technical solution of the present application and how the technical solution of the present application solves the above technical problems. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0065] Figure 1 Flow diagram of the robot obstacle avoidance method based on transfer reinforcement learning provided by the embodiments of the present application Figure 1 As Figure 1 shown, the robot obstacle avoidance method based on transfer reinforcement learning provided in this embodiment includes:

[0066] S101: Obtain a target domain data set, where the target domain data set includes: trajectory data, and preprocess the trajectory data.

[0067] Among them, the target domain represents the actual operating environment of the robot, for example, there are obstacles that the robot needs to avoid. The trajectory data includes: the action data and state data of the robot.

[0068] It can be understood that this step is a pre-step for trajectory sample generation, aiming to enable the Transformer to more accurately model and generate the features of the target domain data, mainly including three steps: linear interpolation, adding a small amount of noise, and standardization and normalization.

[0069] It is understandable that in the actual environment, the target domain data often has local missing due to acquisition interruption or sensor failure. Directly using such data to train the Transformer will disrupt the continuity of the time series modeling, resulting in non-physical state jumps or incoherent action changes in the generated samples. Therefore, interpolation processing is required for the missing trajectory segments, which mainly involves the calculation of action and state data, and a linear interpolation method is specifically adopted to satisfy the following formula:

[0070]

[0071] Among them, s t is the state data generated by interpolation, a t is the action data generated by interpolation, s t-1 is the state data at the previous moment, a t-1 is the action data at the previous moment, s t+1 is the state data at the next moment, a t+1 is the action data at the next moment.

[0072] The interpolation process can repair the temporal coherence of the trajectory and provide a reliable template for the Transformer to generate complete and smooth sequences.

[0073] Secondly, a small amount of noise is added to the interpolated data. In this way, the robustness of the model can be improved, and the diversity of the generated samples can also be enhanced through data augmentation. For the subsequent Conditional Variational Autoencoder (CVAE), the modeled state transition probability is more exploratory, avoiding getting stuck in local optima during policy transfer. Specifically, a small amount of Gaussian noise is added to the action and state. Gaussian noise has the characteristics of a mean of zero and a controllable variance, and can retain key data features to a large extent. The injection method of Gaussian noise satisfies the following formula:

[0074] s noisy = s + ∈ s

[0075] a noisy = a + ∈ a

[0076] Among them, ∈ s is the small amount of noise added to the state, ∈ a is the small amount of noise added to the action. Both follow a Gaussian distribution with a mean of 0 and a controllable variance of σ, that is, ∈ s , ∈ a ~N(0,σ), s noisy is the state data after noise addition, a noisyThe action data after noise addition, s is the state data after linear interpolation, and a is the action data after linear interpolation.

[0077] Finally, standardize and normalize the data processed in the above two steps. Since the trajectory data of offline reinforcement learning usually contains multi-dimensional continuous state and action features, if directly input into the model, the dimension difference will cause unstable gradient updates, and the generated data is difficult to conform to objective laws. Therefore, standardization and normalization operations are usually performed before data input. It should be noted that the theoretical value range of state data is (-∞, +∞), and the value of the state is a statistical value, which is the result calculated according to the environment and actions. Therefore, it is more suitable to use standardization to process state data so that each dimension feature follows the standard normal distribution. Specifically, the standardization process satisfies the following formula:

[0078]

[0079] where s norm is the standardized state, s is the original state, μ s is the mean of the state, and σ s is the variance of the state.

[0080] For action data, there are significant differences in the value ranges and measurement units of these actions, but at the same time, they have a certain value range. Therefore, normalization operation is more suitable for dealing with this multi-scale feature, reducing the sensitivity of the model to outliers, and ensuring that all features are compared and analyzed on the same scale. Specifically, linearly scale the continuous action space to the (-1, 1) interval, which satisfies the following formula:

[0081]

[0082] where a min is the minimum value of the action on the finite data, and a max is the maximum value of the action on the finite data.

[0083] After the above data preprocessing, the sample efficiency can be significantly improved, so that the generated samples not only cover a wider state-action space, but also retain the core laws of the dynamic characteristics of the target domain, thus providing a high-fidelity training basis for subsequent policy transfer.

[0084] S102: Input the preprocessed trajectory data as the original data sample into the trained Transformer network model, and output new data samples through the Transformer network model.

[0085] It can be understood that before generating samples, it is first necessary to train the Transformer network model. By inputting action data and state data into the trained Transformer network model, action samples and state samples for the next moment can be generated.

[0086] In an optional embodiment, the training process of the model includes: converting the format of the preprocessed state and action data, that is, concatenating the state data and action data along the time dimension to form a state-action sequence and inputting it into the Transformer. After the Transformer extracts features from the data, it predicts and generates the state-action for the next time. Subsequently, the mean squared error (MSE) between the real next state-action and the predicted next state-action is calculated through a loss function, and the model parameters are updated by gradient according to the error to gradually minimize the error value. Finally, the goal of generating the ideal next moment data sample is achieved. At this time, the network parameters of the Transformer model are fixed to complete the algorithm training.

[0087] It can also be understood that for sample generation: at this time, the original action state data can be input into the trained Tranformer model to generate new data samples. Subsequently, the new sample dataset and the original dataset can be mixed as the input data for the subsequent module, so as to reduce the cost of data acquisition when it is difficult to obtain data in the robot target domain.

[0088] In order to make the Transformer better fit the reinforcement learning data, in this embodiment, the core multi-head self-attention of the Transformer is adjusted so that it can not only learn the temporal information in the trajectory data, but also rely on the reward value to judge the importance of the current action state to the overall dataset, and then calculate the influence of the current state-action on the overall distribution modeling.

[0089] Specifically, the adjusted attention mechanism adds reward guidance. In the multi-head self-attention mechanism, a part of the attention heads are allocated to specifically process the reward information, satisfying the following formula:

[0090]

[0091] Among them, Attention i(Q, K, V) is the calculation result, that is, the attention weight matrix obtained by operating on Q, K, and V. K, K, and V represent the query vector, key vector, and value vector respectively. These vectors are calculated by the Transformer based on the input data. The meaning of this attention formula is that based on the Q vector, by calculating the dot product between the Q vector and all K vectors, a similarity matrix is obtained. After normalization through the Softmax operation, it is converted into a probability weight matrix, and then multiplied by the V vector matrix and weighted summation is performed to obtain the new representation of the current element, d k represents the scaling factor, and its actual value is the dimension of the K vector. The modified attention adds value guidance, that is, β·R. β is the discount factor, which is manually specified and adjusted at any time during the training process. The reward matrix R is extended from the reward sequence in the form of a diagonal matrix. The reward sequence is similar to the action state sequence, and there is corresponding action state reward data at each time step. The previous data preprocessing only targets the action and state data. Although the reward data sequence does not directly participate in the two steps of noise addition and normalization, it still undergoes the same linear interpolation process. Therefore, the preprocessed action state can still correspond to the reward one-to-one in the time dimension. The reward matrix depends on the reward sequence for generation.

[0092] S103: Mix the original data sample and the new data sample to obtain a synthetic dataset for the target domain.

[0093] S104: Use a conditional variational autoencoder to model the data distribution of the target domain based on the synthetic dataset for the target domain, and determine the state transition probability of the data on the target domain.

[0094] It can be understood that the source domain represents a simulated simulation environment. For example, it can be assumed that there are no obstacles in this environment, that is, there is no cost loss caused by collisions. When migrating the robot obstacle avoidance strategy trained on the source domain to the target domain, it is necessary to eliminate the differences between the source domain environment and the target domain environment. The differences between the source domain environment and the target domain environment are fundamentally due to the imbalance of data such as action state rewards in the two environments. Specifically, it is mainly reflected in two dimensions: data distribution shift and state transition probability. Therefore, it is necessary to model and analyze the data of the target domain and the source domain to eliminate this difference and finally achieve the obstacle avoidance function of the robot on the target domain.

[0095] Based on the above analysis, in this embodiment, a Conditional Variational Autoencoder (CVAE) is used to model the offline data. When given appropriate input data and generation conditions, it can greatly ensure the timeliness of the offline data in the robot target domain, effectively capture the dynamic characteristics and potential structure of the target domain data, thereby accurately modeling the data distribution and state transition probability, and significantly enhancing the generalization ability and adaptability of the policy.

[0096] Specifically, the essence of the state transition probability in the target domain is the mapping relationship from the state-action pair (s, a) to the next state s', that is, it is necessary to model p(s'|s, a). In CVAE, by introducing the conditional variable y = (s, a), the process of generating the target s' is explicitly dependent on the current state-action pair, and its generation process satisfies the following formula:

[0097] p(s'|s, a) = ∫p(s'|z, s, a)p(z|s, a)dz

[0098] Among them, the latent variable z encodes the implicit dynamic characteristics of the state transition and is the process value obtained by the CVAE encoder. The meaning of the formula is to calculate the expectation of the target value (state transition probability) on the continuous distribution, which is decomposed into two parts by the latent variable z: p(s'|z, s, a) represents the probability distribution of the next state s' given the latent variable z, the current state s, and the action a; p(z|s, a) represents the probability distribution of the latent variable z under the current state s and the action a. Through the integral operation, the influence of the latent variable is taken into account to obtain the overall probability distribution of the next state under the current state and action, that is, p(s'|s, a). This value is both the state transition probability and the calculated value of the loss function of CVAE. Based on the above analysis, taking the action a, the state s, and the next state s' at the next moment as the input data of CVAE, the required state transition probability value can be obtained by using the loss function value of CVAE at this time.

[0099] The loss function of CVAE consists of two parts. The first part is the mean square error, and the second part is the KL divergence. The calculation formula is as follows:

[0100] CVAE loss = Recon loss + β * KL loss

[0101] Among them, Recon loss represents the mean square error, KL loss represents the KL divergence, and β represents the scaling factor of the KL divergence constraint, which can be manually adjusted during the training process.

[0102] Specifically, the calculation formula of the mean squared error is as follows:

[0103]

[0104] In the formula, N represents the number of samples, s' represents the actual state at the next moment after the input data state s, which is a value obtained from the data, and s' gen represents the state value generated by the CVAE.

[0105] The calculation method formula of the KL divergence is as follows:

[0106] KL loss = D KL (q(s′|z,s,a)||p(z|s,a))

[0107] Among them, q(s'|z,s,a) is the approximate posterior distribution, which represents the probability distribution of generating the next state s' given the latent variable z and the conditional variables (s,a). It is modeled by the CVAE and used to approximate the true posterior distribution q(z|s,a,s'). And p(z|s,a) is the prior distribution, which represents the probability distribution of the latent variable z given the conditional variables (s,a). Finally, adding the two parts together can obtain the CAVE loss.

[0108] Figure 2 This is the schematic diagram of the CVAE processing flow provided by the embodiment of the present application. As Figure 2 shown, the original data input form is a string of action states sa. In the first step, the data form is reorganized into sas', and it is input into the encoder of the CVAE to obtain the mean mean and variance std of the data. The sa is reorganized according to the data distribution to obtain the conditional state representation sz in the intermediate process. Subsequently, sz is input into the decoder of the CVAE to obtain the generated state s' at the next moment gen , and finally, using the original state s’ at the next moment and the generated state s' at the next moment gen calculate the value of the CVAE loss function, and the state transition probability of the target domain data can be obtained according to this value.

[0109] The purpose of this step is to obtain the state transition probability of the target domain data through the method of data modeling, so as to prepare for subsequent policy migration.

[0110] S105: Obtain the real-time interaction data of the robot action state, and use the double binary classifier architecture to determine the state transition probability of the data on the source domain according to the real-time interaction data.

[0111] It can be understood that since the action state data in the source domain is obtained through real-time interaction, unlike the finite data set in the target domain, if the source domain data is directly modeled frequently and at a high frequency, its computational cost will increase linearly with the data scale; while modeling with a low frequency and a small amount, it is difficult to accurately describe the state transition law on the dynamically growing data. Therefore, from the perspective of computational performance and modeling accuracy, it is not feasible to model the state transition probability in the source domain in real time multiple times.

[0112] In view of the above problems, in this embodiment, the same static modeling method is not adopted in the source domain to estimate the state transition probability, but a method based on a double binary classifier is used to indirectly estimate the state transition probability through the Bayes formula.

[0113] It can also be understood that in this embodiment, it is desired to obtain the state transition probability in the source domain, that is, p_source(s'|s,a) or p(s'|s,a,source). Through the derivation of the definition of conditional probability, Bayes formula, and total probability formula, the following results can be obtained:

[0114] logp_source(s′|s,a) = logp(source|s,a,s′) - logp(source|s,a)

[0115] Through the above transformation, the conditional state probability in the source domain is converted into the difference between two conditional probabilities. p(source|s,a,s') represents the probability that the environment is the source domain when the robot is in a specific state s, takes a certain action a, and transfers to a certain new state; p(source|s,a) represents the probability that the environment is the source domain when a specific state s appears and an action a is about to be taken.

[0116] Furthermore, this embodiment adopts a double binary classifier architecture: the first classifier takes (s,a,s') as the input and outputs the probability that it belongs to the source domain; the second classifier takes (s,a) as the input and also outputs the probability that it belongs to the source domain. By jointly training these two classifiers, the required conditional probability difference can be directly estimated from the online data in the source domain, so as to accurately approximate the state transition probability in the source domain.

[0117] Figure 3 For the double binary classifier model structure diagram provided by the embodiment of the present application, as Figure 3As shown, the two classifier neural networks have the same structure and are both used to classify source domain data and noise data (the noise data is directly generated data and is only used to train the classifier). The input data of one is in the form of (s,a), and the other is in the form of (s,a,s'). The input data first passes through two fully connected layers. The first fully connected layer uses an activation function to extract the non-linear representation of the input features, and the second fully connected layer is used to map the features of the previous layer to the output space. The scores for the two categories are obtained through the output layer, and finally the softmax function is used to convert the scores into the required conditional probabilities.

[0118] S106: Based on the state transition probability of the data in the target domain and the state transition probability of the data in the source domain, transfer the robot obstacle avoidance strategy trained on the source domain.

[0119] The state transition probabilities of the source domain and the target domain are obtained through the above steps, and the robot obstacle avoidance strategy trained on the source domain will be transferred based on these two values.

[0120] It can be understood that the goal of reinforcement learning is to optimize the policy π(a|s) so that the trajectory distribution p π (τ) maximizes the cumulative reward ∑r(s,a). When the state transition probabilities p source (s'|s,a) and p target (s'|s,a) in the source domain and the target domain are different, directly using the source domain reward function r source (s,a) will cause the trajectory distribution p π,target (τ) generated by the policy in the target domain to deviate from the expectation. Therefore, the essence of reward correction is to adjust the reward function so that the trajectory distribution p π,source (τ) optimized by the policy in the source domain is as close as possible to the true distribution p target (τ) in the target domain. Specifically, the corrected reward function satisfies the following formula:

[0121] r adapted (s,a,s′) = r source (s,a) + β · (log p target (s′|s,a) - log p source (s′|s,a))

[0122] Among them, r adapted (s,a,s') is the corrected reward value, r source (s,a) is the original reward value in the source domain, which is calculated according to the state action by a pre-specified reward function. β is a balance parameter used to control the intensity of cross-domain dynamic alignment. The two values in the parentheses in the formula represent the difference between the target domain state transition probability and the source domain state transition probability.

[0123] Based on the above formula expression, it can be intuitively found that the specific logic of reward correction can be divided into three cases, corresponding to different policy guidance goals: when the state transition probability is the same, the correction term approaches zero, and the policy can directly use the source domain reward to achieve migration, indicating that the dynamic characteristics of the source domain and the target domain are highly compatible, and the current optimal decision on the source domain can still play the best effect on the target domain; when the target domain transition probability is higher, the correction term is positive, indicating that the current action is more likely to achieve an efficient state transition in the target domain, and the action taken in this state can also play a positive role in completing the task of the target domain, encouraging the policy to be more inclined to transfer to such states and actions; when the target domain transition probability is lower, the correction term is negative, indicating that the current action may lead to an ineffective or dangerous transfer in the target domain. Although the action taken by the current policy is the optimal solution for the source domain, it is difficult to complete the expected task on the target domain. Therefore, a corresponding penalty is imposed on the reward to prevent the policy from completing the current action state transition, and the policy is forced to explore a more robust behavior pattern.

[0124] The robot obstacle avoidance method based on transfer reinforcement learning provided by the embodiments of the present application obtains a target domain data set, preprocesses the data set and inputs it into a trained Transformer model, and outputs new data samples through the Transformer model; mixes the original data samples and the new data samples to obtain a target domain synthetic data set; uses a conditional variational autoencoder to model the data distribution of the target domain according to the target domain synthetic data set to determine the state transition probability of the data on the target domain; uses a dual binary classifier architecture to determine the state transition probability of the data on the source domain according to the real-time interaction data of the robot state action; based on the state transition probability of the target domain and the state transition probability of the source domain, migrates the robot obstacle avoidance strategy trained on the source domain, reduces the cost loss in the training process, and increases the environmental adaptability of the obstacle avoidance strategy.

[0125] Figure 4 This is a schematic structural diagram of a robot obstacle avoidance device based on transfer reinforcement learning provided by the embodiments of the present application. As Figure 4 shown, the robot obstacle avoidance device 300 based on transfer reinforcement learning provided in this embodiment includes:

[0126] An acquisition module 301, configured to acquire a target domain data set, where the target domain data set includes: trajectory data;

[0127] A processing module 302, configured to preprocess the trajectory data;

[0128] The processing module 302 is further configured to input the preprocessed trajectory data as an original data sample into the trained Transformer network model, and output a new data sample through the Transformer network model;

[0129] The processing module 302 is further configured to mix the original data sample and the new data sample to obtain a target domain synthetic data set;

[0130] The determination module 303 is configured to model the data distribution of the target domain according to the target domain synthetic data set by using a conditional variational autoencoder, and determine the state transition probability of the data on the target domain;

[0131] The acquisition module 301 is further configured to acquire real-time interaction data of the robot action state;

[0132] The determination module 303 is further configured to determine the state transition probability of the data on the source domain according to the real-time interaction data by using a double binary classifier architecture;

[0133] The processing module 302 is further configured to migrate the robot obstacle avoidance strategy trained on the source domain based on the state transition probability of the data on the target domain and the state transition probability of the data on the source domain.

[0134] The robot obstacle avoidance device based on transfer reinforcement learning provided in this embodiment can execute the robot obstacle avoidance method based on transfer reinforcement learning provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here in this embodiment.

[0135] Figure 5 It is a schematic structural diagram of a robot obstacle avoidance device based on transfer reinforcement learning provided in an embodiment of the present application. As Figure 5 shown, the robot obstacle avoidance device based on transfer reinforcement learning provided in an embodiment of the present application, the robot obstacle avoidance device 400 based on transfer reinforcement learning includes: a receiver 401, a transmitter 402, a processor 403, and a memory 404.

[0136] The receiver 401 is configured to receive instructions and data;

[0137] The transmitter 402 is configured to send instructions and data;

[0138] The memory 404 is configured to store computer execution instructions;

[0139] The processor 403 is configured to execute the computer execution instructions stored in the memory 404 to implement each step performed by the robot obstacle avoidance method based on transfer reinforcement learning in the above embodiment. Specifically, reference can be made to the relevant descriptions in the foregoing embodiment of the robot obstacle avoidance method based on transfer reinforcement learning.

[0140] Optionally, the above-mentioned memory 404 can be either independent or integrated with the processor 403.

[0141] When the memory 404 is independently provided, the electronic device further includes a bus for connecting the memory 404 and the processor 403.

[0142] An embodiment of the present application further provides a computer storage medium, in which computer-executable instructions are stored. When the processor executes the computer-executable instructions, the robot obstacle avoidance method based on transfer reinforcement learning executed by the above-mentioned robot obstacle avoidance device based on transfer reinforcement learning is implemented.

[0143] An embodiment of the present application further provides a computer program product, including a computer program, which implements the above-mentioned robot obstacle avoidance method based on transfer reinforcement learning when executed by the processor.

[0144] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof. In the hardware implementation, the division of the functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be executed by several physical components in cooperation. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disc (DVD), or other optical disc storage, magnetic cassette, tape, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, a communication medium generally includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium.

[0145] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and examples are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0146] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A robot obstacle avoidance method based on transfer reinforcement learning, characterized in that, The method includes: Obtain a target domain dataset, where the target domain dataset includes trajectory data, and preprocess the trajectory data; Use the preprocessed trajectory data as the original data samples and input them into the trained Transformer network model, and output new data samples through the Transformer network model; Mix the original data samples and the new data samples to obtain a target domain synthetic dataset; Use a conditional variational autoencoder to model the data distribution of the target domain based on the target domain synthetic dataset, and determine the state transition probability of the data on the target domain; Obtain the real-time interaction data of the robot's action state, and use a double binary classifier architecture to determine the state transition probability of the data on the source domain based on the real-time interaction data; Based on the state transition probability of the data on the target domain and the state transition probability of the data on the source domain, transfer the robot obstacle avoidance strategy trained on the source domain.

2. The method according to claim 1, wherein The step of using a conditional variational autoencoder to model the data distribution of the target domain based on the target domain synthetic dataset to obtain the state transition probability of the data on the target domain includes: Decompose the trajectory data into the first state data at the next moment and the state action data at the current moment; Recombine the first state data and the state action data at the current moment to obtain recombined state action data; Input the recombined state action data into the encoder of the conditional variational autoencoder for analysis and processing to obtain the mean and variance of the recombined state action data; Recombine the state action data at the current moment according to the data distribution to obtain a conditional state representation, and input the conditional state representation into the decoder of the conditional variational autoencoder for analysis and processing to obtain the second state data at the next moment; Determine the loss function value of the conditional variational autoencoder according to the first state data and the second state data; Determine the state transition probability of the data on the target domain according to the loss function value of the conditional variational autoencoder.

3. The method according to claim 1, characterized in that The double binary classifier architecture includes: a first classifier and a second classifier. The step of using the double binary classifier architecture to determine the state transition probability of the data on the source domain based on the real-time interaction data includes: Analyze and process the state data, action data, and state data at the next moment of the robot at the current moment through the trained first classifier, and output a first conditional probability; Analyze and process the state data and action data of the robot at the current moment through the trained second classifier, and output a second conditional probability; Determine the state transition probability of the data on the source domain according to the first conditional probability and the second conditional probability.

4. The method according to claim 3, wherein The state transition probability of the data on the source domain satisfies the following formula: logp_source(s′|s,a)=logp(source|s,a,s′)-logp(source|s,a) Among them, p_source(s'|s,a) is the state transition probability of data in the source domain, s is the state at the current moment, a is the action at the current moment, s' is the state at the next moment, p(source|s,a,s') is the first conditional probability, and p(source|s,a) is the second conditional probability.

5. The method according to claim 1, characterized in that, Transferring the robot obstacle avoidance strategy trained on the source domain based on the state transition probability of data in the target domain and the state transition probability of data in the source domain includes: Determining whether the state transition probability of data in the target domain is consistent with the state transition probability of data in the source domain; When the state transition probability of data in the target domain is consistent with the state transition probability of data in the source domain, transferring the robot obstacle avoidance strategy according to the original reward value in the source domain; When the state transition probability of data in the target domain is inconsistent with the state transition probability of data in the source domain, correcting the original reward value in the source domain according to the difference between the state transition probability of data in the target domain and the state transition probability of data in the source domain, and transferring the robot obstacle avoidance strategy according to the corrected reward value.

6. The method according to claim 5, characterized in that Correcting the original reward value in the source domain according to the difference between the state transition probability of data in the target domain and the state transition probability of data in the source domain satisfies the following formula: r adapted (s,a,s′) = r source (s,a) + β·(log p target (s′|s,a) - log p source (s′|s,a)) where r adapted (s,a,s') is the corrected reward value, r source (s,a) is the original reward value in the source domain, β is the balance parameter, p target (s'|s,a) represents the state transition probability of the data in the target domain, p source (s'|s,a) represents the state transition probability of the data in the source domain, s is the state at the current moment, a is the action at the current moment, and s' is the state at the next moment.

7. A robot obstacle avoidance device based on transfer reinforcement learning, characterized in that, The device includes: An acquisition module, configured to acquire a target domain data set, where the target domain data set includes: trajectory data; A processing module, configured to preprocess the trajectory data; The processing module is further configured to input the preprocessed trajectory data as an original data sample into a trained Transformer network model, and output a new data sample through the Transformer network model; The processing module is further configured to mix the original data sample and the new data sample to obtain a target domain synthetic data set; A determination module, configured to model the target domain data distribution according to the target domain synthetic data set by using a conditional variational autoencoder, and determine the state transition probability of data in the target domain; The acquisition module is further configured to acquire real-time interaction data of the robot action state; The determination module is further configured to determine the state transition probability of data in the source domain according to the real-time interaction data by using a dual binary classifier architecture; The processing module is further configured to transfer the robot obstacle avoidance strategy trained on the source domain based on the state transition probability of data in the target domain and the state transition probability of data in the source domain.

8. A robot obstacle avoidance device based on transfer reinforcement learning, characterized in that, The device includes: A memory; A processor; Among them, the memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the robot obstacle avoidance method based on transfer reinforcement learning according to any one of claims 1-6.

9. A computer storage medium, characterized in that, Computer execution instructions are stored in the computer storage medium, and when the computer execution instructions are executed by a processor, they are used to implement the robot obstacle avoidance method based on transfer reinforcement learning according to any one of claims 1-6.

10. A computer program product, characterized in that, Comprising a computer program which, when executed by a processor, is used to implement the robot obstacle avoidance method based on transfer reinforcement learning according to any one of claims 1-6.

Images