Obstacle avoidance decision method and system for unmanned system

Abstract

The present invention discloses an obstacle avoidance decision method and system for an unmanned system, which belongs to the technical field of unmanned system decision making. A universal encoder is used to extract low-dimensional state representations shared in different environments from the current image sequence to obtain common visual features. Then, a task-specific encoder corresponding to the current environment type is used to further encode the common visual features into relevant feature representations for the current environment to obtain individual task features, thereby enabling the unmanned system to learn the common visual features and individual task features of the environment in a complex environment, and perform deep fusion to achieve feature reconstruction of the current image sequence. On this basis, the idea of ​​mutual simulation is introduced, and training is performed based on feature consistency loss and reward consistency loss, so that the universal encoder can accurately extract common visual features while focusing on individual task features, and has a selective focusing function, and then combined with a reinforcement learning model to achieve accurate obstacle avoidance decisions in complex environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned system decision-making, and more specifically, relates to an obstacle avoidance decision-making method and system for an unmanned system. Background Art

[0002] Unmanned system obstacle avoidance decision-making is a key link in determining whether the intelligent system can complete tasks safely and efficiently. It aims to enable the unmanned system to efficiently process information data and output the optimal obstacle avoidance decision-making strategy in complex dynamic environments. However, whether it is autonomous driving, drone swarms or unmanned boats, the complex dynamic characteristics of the environment (such as crowded pedestrians, complex terrain, ocean current changes, etc.) restrict the efficient output of the obstacle avoidance decision-making link. Therefore, it is of great significance to study an obstacle avoidance decision-making method for unmanned systems.

[0003] The existing obstacle avoidance decision-making methods for unmanned systems often use obstacle avoidance decision-making methods based on machine learning. This method uses data-driven training to obtain obstacle avoidance models to fit obstacle avoidance decision-making strategies. When the environment in which the unmanned system is located is complex and diverse, this method requires a large number of high-quality data samples from different environments for model training. Over-reliance on high-quality data will cause data bias, and if the data is missing or erroneous, it will also lead to wrong decisions, making it impossible to achieve accurate obstacle avoidance decisions in complex environments. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a decision-making method and system for an unmanned system, so as to solve the technical problem that the prior art cannot achieve accurate obstacle avoidance decision-making in complex environments.

[0005] To achieve the above objectives, in a first aspect, the present invention provides an obstacle avoidance decision method for an unmanned system, comprising:

[0006] The image sequence collected by the unmanned system in the environment at the current time k is input into the universal encoder for feature extraction to obtain the visual feature h at the time k. sh (k); h sh (k) is input into the reinforcement learning model to obtain the obstacle avoidance action a(k) of the unmanned system at time k and the reward value r(k) after executing a(k); where a(k) is the obstacle avoidance decision at time k; the reward value when no collision occurs after the unmanned system executes a(k) is greater than the reward value when a collision occurs;

[0007] After obtaining a(k), perform the following reinforcement training operations:

[0008] h sh (k) and a(k) are input into the first state transition model to predict the visual feature h at time k+1 sh (k+1);

[0009] h sh (k) is input into the environment classifier to obtain the category C of the environment where the unmanned system is located at time k; h sh (k) is input into the task-specific encoder corresponding to the environment category C to obtain the task feature h at time k i (k); h i (k) and a(k) are input into the second state transition model to predict the task feature h at time k+1 i (k+1);

[0010] h sh (k+1) and h i (k+1) is fused and input into the decoder, and the visual reconstruction feature h at time k+1 is obtained by decoding. rec (k+1); calculate h rec (k+1) and h rec The difference loss between (k) is used as feature consistency loss;

[0011] h sh (k) is input into the reward prediction model to predict the reward value r'(k+1) of the unmanned system at time k+1; the difference loss between r'(k+1) and r(k) is calculated as the reward consistency loss;

[0012] With the goal of minimizing the training loss including feature consistency loss and reward consistency loss and maximizing the cumulative reward value, the general encoder, reinforcement learning model, first state transfer model, task-specific encoder corresponding to environment category C, second state transfer model, decoder and reward prediction model are trained simultaneously;

[0013] Among them, the cumulative reward value is the sum of the reward values ​​output by the k reinforcement learning model from the initial moment to the current moment.

[0014] Further preferably, the above sh (k+1) and h i (k+1) After fusion, the input to the decoder is: h sh (k+1) and h i (k+1) are concatenated and input to the decoder.

[0015] Further preferably, the above reinforcement learning model is a Soft Actor-Critic model.

[0016] Further preferably, the environment classifier is pre-trained in the following manner:

[0017] Construct a training model including a cascaded feature extraction module and an environment classifier; input each image sequence sample in the pre-collected training set into the training model to obtain the prediction result of the corresponding environment category; train the training model with the goal of minimizing the difference loss between the prediction result of the environment category and the corresponding true category label to obtain a trained environment classifier;

[0018] The training set includes multiple image sequence samples collected by unmanned systems in different environments at different times.

[0019] Further preferably, the obstacle avoidance decision method of the unmanned system further includes: obtaining a true category label of the environment where the unmanned system is located at time k; calculating the difference loss between the category C of the environment where the unmanned system is located at time k obtained by the environment classifier and the true category label as the classification loss;

[0020] While minimizing the training loss and maximizing the cumulative reward value, the general encoder, reinforcement learning model, first state transition model, task-specific encoder corresponding to environment category C, second state transition model, decoder and reward prediction model are trained, and the environment classifier is also trained;

[0021] Among them, the above training loss also includes classification loss.

[0022] In a second aspect, the present invention provides an obstacle avoidance decision device for an unmanned system, comprising:

[0023] The universal encoder is used to extract features from the image sequence of the environment collected by the unmanned system at the current time k, and obtain the visual feature h at the time k. sh (k);

[0024] Reinforcement learning model for h-based sh (k) Obtain the obstacle avoidance action a(k) of the unmanned system at time k, and the reward value r(k) after executing a(k); where a(k) is the obstacle avoidance decision at time k; the reward value when no collision occurs after the unmanned system executes a(k) is greater than the reward value when a collision occurs;

[0025] The collaborative training module is used to perform enhanced training after obtaining a(k), including:

[0026] The first state transition model is used based on h sh (k) and a(k) predict the visual feature h at time k+1 sh (k+1);

[0027] Environment classifier for h-based sh (k) Obtain the category C of the environment where the unmanned system is located at time k;

[0028] The task-specific encoder group includes: task-specific encoders corresponding to different environment categories; wherein the task-specific encoder corresponding to environment category C is used to sh (k) Get the task feature h at time k i (k);

[0029] The second state transition model is used based on h i (k) and a(k) predict the task feature h at time k+1 i (k+1);

[0030] Decoder for h sh (k+1) and h i The fusion feature of (k+1) is decoded to obtain the task feature h at time k+1 i (k+1);

[0031] Reward prediction model for h sh (k) Predict the reward value r' (k+1) of the unmanned system at time k+1;

[0032] Training control unit for calculating h rec (k+1) and h rec (k) as the feature consistency loss; calculate the difference loss between r'(k+1) and r(k) as the reward consistency loss; with the goal of minimizing the training loss including the feature consistency loss and the reward consistency loss and maximizing the cumulative reward value, the general encoder, the reinforcement learning model, the first state transfer model, the task-specific encoder corresponding to the environment category C, the second state transfer model, the decoder and the reward prediction model are trained at the same time;

[0033] Among them, the cumulative reward value is the sum of the reward values ​​output by the k reinforcement learning model from the initial moment to the current moment.

[0034] Further preferably, the training control unit is further used to obtain the true category label of the environment where the unmanned system is located at time k; calculate the difference loss between the category C of the environment where the unmanned system is located at time k and the true category label as the classification loss; with the goal of minimizing the training loss and maximizing the cumulative reward value, while training the general encoder, the reinforcement learning model, the first state transition model, the task-specific encoder corresponding to the environment category C, the second state transition model, the decoder and the reward prediction model, the environment classifier is also trained;

[0035] Among them, the above training loss also includes classification loss.

[0036] In a third aspect, the present invention provides an obstacle avoidance decision system for an unmanned system, comprising: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the obstacle avoidance decision method for the unmanned system provided by the first aspect of the present invention is executed.

[0037] In a fourth aspect, the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the obstacle avoidance decision method for the unmanned system provided in the first aspect of the present invention.

[0038] In a fifth aspect, the invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the obstacle avoidance decision method for an unmanned system provided in the first aspect of the invention.

[0039] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0040] 1. The present invention provides an obstacle avoidance decision method for an unmanned system, which uses a universal encoder to extract low-dimensional state representations shared in different environments from the current image sequence to obtain common visual features; then, a task-specific encoder corresponding to the current environment type is used to further encode the common visual features into relevant feature representations for the current environment on the basis of the universal encoder to obtain individual task features, thereby enabling the unmanned system to learn the common visual features and individual task features of the environment in a complex environment, and perform deep fusion to achieve feature reconstruction of the current image sequence. On this basis, the idea of ​​mutual simulation is introduced, and training is performed based on feature consistency loss and reward consistency loss, so that the universal encoder can accurately extract common visual features while focusing on individual task features, and has a selective focusing function, and then combined with a reinforcement learning model to achieve obstacle avoidance decisions in complex environments; the present invention makes real-time decisions by learning the essential characteristics of the current environment, without relying on high-quality data samples, and the accuracy of the decision is high.

[0041] 2. The obstacle avoidance decision method for unmanned systems provided by the present invention obtains the abstract state representation of the image sequence through a universal encoder, which reduces the input dimension of the policy network and improves the sample utilization efficiency. At the same time, the abstract state representation often has strong semantic characteristics, which makes it easier for humans to understand the decision basis of the model and supports domain experts to evaluate the decision results by analyzing the abstract state.

[0042] 3. The obstacle avoidance decision method for unmanned systems provided by the present invention uses mutual simulation measurement to achieve state sharing of "different paths to reach the same goal", reduces the repeated learning of similar states by the strategy, and has high decision-making efficiency.

[0043] 4. Furthermore, in the obstacle avoidance decision-making method for unmanned systems provided by the present invention, the reinforcement learning model is a SoftActor-Critic model to better handle the complex dynamic environment and continuous control tasks faced by the unmanned system obstacle avoidance decision-making; specifically, 1) the Soft Actor-Critic model adopts a dual Q network and target smoothing to improve the convergence performance and has strong stability; 2) the Soft Actor-Critic model has high sample efficiency and is suitable for real-world tasks with expensive data; 3) the Soft Actor-Critic model can adapt to complex control tasks and support the acquisition of continuous actions; 4) the Soft Actor-Critic model adopts entropy regularization operation, which can improve the diversity and robustness of the strategy and has strong exploratory nature. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 An obstacle avoidance decision framework for an unmanned system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0046] To achieve the above objectives, firstly, Figure 1 As shown, the present invention provides an obstacle avoidance decision method for an unmanned system, comprising:

[0047] The image sequence of the environment collected by the unmanned system at the current time k is input into the universal encoder for feature extraction to obtain the visual feature h at the time k. sh (k); h sh (k) is input into the reinforcement learning model to obtain the obstacle avoidance action a(k) of the unmanned system at time k and the reward value r(k) after executing a(k); where a(k) is the obstacle avoidance decision at time k; the reward value when no collision occurs after the unmanned system executes a(k) is greater than the reward value when a collision occurs;

[0048] After obtaining a(k), perform the following reinforcement training operations:

[0049] h sh (k) and a(k) are input into the first state transition model to predict the visual feature h at time k+1 sh (k+1);

[0050] h sh (k) is input into the environment classifier to obtain the category C of the environment where the unmanned system is located at time k; h sh (k) is input into the task-specific encoder corresponding to the environment category C to obtain the task feature h at time k i (k); h i (k) and a(k) are input into the second state transition model to predict the task feature h at time k+1 i (k+1);

[0051] h sh (k+1) and h i (k+1) After fusion, it is input into the decoder, and the visual reconstruction feature h at time k+1 is obtained by decoding rec (k+1); calculate h rec (k+1) and h rec The difference loss between (k) is used as feature consistency loss;

[0052] h sh (k) is input into the reward prediction model to predict the reward value r'(k+1) of the unmanned system at time k+1; the difference loss between r'(k+1) and r(k) is calculated as the reward consistency loss;

[0053] With the goal of minimizing the training loss including feature consistency loss and reward consistency loss and maximizing the cumulative reward value, the general encoder, reinforcement learning model, first state transfer model, task-specific encoder corresponding to environment category C, second state transfer model, decoder and reward prediction model are trained simultaneously;

[0054] Among them, the cumulative reward value is the sum of the reward values ​​output by the k reinforcement learning model from the initial moment to the current moment.

[0055] It should be noted that Figure 1 h' sh represents the h at the next moment sh , that is, when h sh h sh (k), h' sh h sh (k+1); similarly, r' represents the reward value of the unmanned system at the next moment predicted by the reward prediction model, that is, r' (k+1).

[0056] It should be noted that h sh (k+1) and h iThere are many methods for (k+1) fusion, which can be directly spliced ​​(usually spliced ​​in the channel dimension), or fused by weighted average, attention mechanism, maximum pooling, average pooling, etc., which are not limited here. Preferably, in an optional implementation, splicing is used to fuse h sh (k+1) and h i (k+1).

[0057] It should be noted that the encoder can be a convolutional neural network (CNN), a self-supervised learning encoder (contrastive learning), a pre-trained model (ViT), etc., and there is no limitation here. The structures of the general encoder and the task-specific encoder can be the same or different, but the specific parameters are different.

[0058] The reinforcement learning model may be an Actor-Critic model, a Soft Actor-Critic model, a DDPG model, a TD3 model, a PPO model, an A3C model, an A2C model, etc., which are not limited here. Preferably, in an optional implementation, the reinforcement learning model is a Soft Actor-Critic model.

[0059] The state transfer model can be a Gaussian process model, a neural network model, a generative adversarial network, etc., which is not limited here. The structures of the first state transfer model and the second state transfer model can be the same or different, but the specific parameters are different.

[0060] The decoder can be a convolutional neural network (CNN), a self-supervised learning encoder (contrastive learning), a pre-trained model (ViT), etc., without limitation here.

[0061] The reward prediction model can be a linear regression reward prediction model, a neural network reward prediction model, a Gaussian process regression model, a reward prediction model based on time difference learning, etc., and there is no limitation here.

[0062] The environment classifier may be a K-nearest neighbor classifier, a support vector machine, a gradient boosting tree, a neural network, a Bayesian classifier, etc., and is not limited here.

[0063] In an optional implementation, the training loss is the weighted sum of feature consistency loss and reward consistency loss. Among them, the loss function for measuring feature consistency loss and reward consistency loss can be L1 loss function, L2 loss function, cosine similarity formula, Huber loss, cross entropy loss, KL divergence, etc., which are not limited here.

[0064] It should be noted that there are many ways to train the environment classifier, which are not limited here. The following are the implementation methods of different training methods:

[0065] In the first training mode, the environment classifier is a pre-trained environment classifier, which is pre-trained in the following way:

[0066] Construct a training model including a cascaded feature extraction module and an environment classifier; input each image sequence sample in the pre-collected training set into the training model to obtain the prediction result of the corresponding environment category; train the training model with the goal of minimizing the difference loss between the prediction result of the environment category and the corresponding true category label to obtain a trained environment classifier;

[0067] Among them, the training set includes image sequence samples collected by multiple unmanned systems in different environments at different times; the feature extraction module can be CNN, convolutional neural network (CNN), self-supervised learning encoder (contrastive learning), pre-trained model (ViT), etc., which is not limited here.

[0068] In the second training mode, the obstacle avoidance decision method of the unmanned system further includes: obtaining a true category label of the environment where the unmanned system is located at time k; calculating the difference loss between the category C of the environment where the unmanned system is located at time k obtained by the environment classifier and the true category label as the classification loss;

[0069] While minimizing the training loss and maximizing the cumulative reward value, the general encoder, reinforcement learning model, first state transition model, task-specific encoder corresponding to environment category C, second state transition model, decoder and reward prediction model are trained, and the environment classifier is also trained;

[0070] The above training loss also includes classification loss. The loss function for measuring classification loss can be cross entropy loss, L2 loss function, Huber loss, cross entropy loss, KL divergence, etc., which are not limited here.

[0071] In the third training method, the pre-trained environment classifier obtained in the first training method can be used as the initial environment classifier, and then in the actual application process, the second training method can be used to fine-tune the environment classifier to make the obstacle avoidance decision more accurate.

[0072] Since the second and third training methods require the real category labels of the current environment to be pre-labeled, when the user's requirement is to make the unmanned system more intelligent without affecting the accuracy of the obstacle avoidance decision, the environment classifier is preferably a pre-trained environment classifier, that is, the first training method. When the user's requirement is to make the obstacle avoidance decision more accurate, the second or third training method can be considered. The selection can be made according to the user's needs.

[0073] It should be noted that the unmanned system involved in the present invention may be an unmanned aerial vehicle, an unmanned vehicle, an unmanned boat or other unmanned systems, which is not limited here.

[0074] Aiming at the problem that unmanned systems are difficult to make flexible, efficient and accurate decisions in complex dynamic environments, the present invention draws inspiration from the information processing and decision-making methods of different organisms, and refers to the selective focusing function of eagle eyes and the multi-region division of labor and cooperation mode of the human brain to propose a cross-species bionic unmanned system decision-making method. Through deep reinforcement learning technology, a representation learning method is used to extract compact low-dimensional state representations from image sequences, and the idea of ​​mutual simulation is combined to ensure the strong correlation between state representation and tasks. At the same time, the consistency of representation before and after learning of the general encoder is designed, and an environmental classifier is constructed to determine the current environment type to call the appropriate task-specific encoder, so as to accurately ignore irrelevant background noise information and improve the efficiency of perceptual information processing, providing a new method and technical support for realizing efficient, generalized and accurate multi-task obstacle avoidance decision-making of unmanned systems.

[0075] In order to further illustrate the obstacle avoidance decision method of the unmanned system provided by the present invention, the following takes the autonomous obstacle avoidance decision of an unmanned vehicle in a complex dynamic environment as an example, and is described in detail as follows:

[0076] The specific decision-making method is mainly composed of sensors and core algorithm modules. Multi-task refers to the obstacle avoidance task of unmanned vehicles in the process of driving in different weather conditions (environments) such as rainy days, snowy days, sunsets, and sunny days. The important difficulty lies in the diversity of perception data brought by different weather conditions and the complexity of continuous action decision-making, which hinders the current development of obstacle avoidance decision-making performance of unmanned systems in complex environments. To this end, inspired by the selective focusing function of eagle eyes in the biological world and the multi-regional division of labor and cooperation of the human brain, we try to achieve the goal of enabling unmanned vehicles to selectively focus on useful information and accurately ignore task-irrelevant information, so as to find the commonality of obstacle avoidance tasks under different weather conditions and avoid repeated learning.

[0077] The specific contents of realizing the autonomous obstacle avoidance decision-making task of the unmanned vehicle are as follows:

[0078] 1) Data reception and preprocessing: The unmanned vehicle first obtains perception data through sensors. Specifically, it uses a camera to obtain an image sequence of the current environment (usually including 3 to 5 frames of images), and then performs preprocessing operations such as denoising and filtering.

[0079] 2) Framework design:

[0080] a. Construct a general encoder and multiple task-specific encoders; In this embodiment, the general encoder is composed of a convolutional layer and a multi-layer perceptron, so as to capture the shared spatial features and low-dimensional state representations under different weather conditions (different environmental conditions) and obtain the corresponding visual features; the task-specific encoder further encodes the state representation into task-related feature representations for different weather conditions on the basis of the general encoder. The general encoder and the task-specific encoder have similar structures, including convolutional layers, and add attention mechanisms, but the parameters are independent. Therefore, the general encoder and the task-specific encoder will generate a general state representation and a task-specific state representation respectively. The design of this module is inspired by the multi-regional division of labor and cooperation of the human brain, so that the unmanned vehicle can achieve deep integration in the commonality and individuality of task decision-making in different environments. Among them, there are multiple task-specific encoders, and each weather condition corresponds to a task-specific encoder. The structures of these task-specific encoders can be the same, but the parameters are different.

[0081] The universal encoder and task-specific encoder are inspired by the multi-regional division of labor and cooperation of the human brain, that is, when faced with new situations or unique situations, the brain will use existing common knowledge as a basis and adjust it in combination with specific environmental factors, needs and goals. The universal encoder is used to extract shared state representations that are not dependent on specific tasks. sh , providing consistent state representation for multi-task and multi-environment to reduce encoder redundancy. Task-specific encoders generate state representations related to specific tasks for multi-task and multi-environment problems. i , by context-dependent switching of task-specific encoders and fusing two types of encoders h t = cat(h sh , h i ) can enable unmanned systems to learn the common and individual characteristics of the environment in complex environments, so as to facilitate strong generalization decision-making.

[0082] b. Construct state transition model and reward prediction model: Introduce the idea of ​​mutual simulation and extend it to the cross-Markov decision process (MDP) environment. Based on the idea of ​​mutual simulation, in the cross-MDP environment, by defining the state equivalent mapping relationship of driving obstacle avoidance tasks under different weather conditions, construct a state transition model and a reward model, which are used to predict the state and reward value of the unmanned vehicle at the next moment, respectively, and establish the loss function of the state transition model and the reward model to ensure the universality of finding the same state-action pair in multiple MDPs. This process simulates the selective focusing function of eagle eyes to achieve the unmanned vehicle's focus on useful information under different weather conditions, and avoid the dynamic changes of the environment affecting the autonomous decision-making of the unmanned vehicle.

[0083] Specifically, for the bisimulation relation of a single MDP, that is, given an MDP, M =<S, A, P, R> , where S is the state space, A is the action space, P(s'|s, a) is the state transition probability, and R(s, a) is the reward function. The bisimulation relation B defines an equivalence relation , if for two states s', s∈S, two conditions are satisfied: (1) Same reward: if s ~ Bs', then for all actions a∈A, R(s, a) = R(s', a); (2) Consistent transition probabilities: for every action a∈A and state s''∈S, if s ~ Bs', then P(s''| s, a) = P(s''| s', a).

[0084] For the mutual simulation relationship across MDPs, assume there are two MDPs: M1=<S1, A1, P1, R1> ,M2=<S2, A2, P2,R2> , if a bisimulation relation is defined between two MDPs to make them behaviorally consistent, then the cross-MDP bisimulation relation is defined as, for M1 and M2 with equivalence relation, there exists a relation , and satisfies two conditions: (1) Reward consistency: for all (s1, s2)∈B and all actions a1∈A1, a2∈A2, R1(s1, a1) = R2(s2, a2) should be satisfied; (2) Transition consistency: for all (s1, s2)∈B and all actions a1∈A1, a2∈A2, the transition probability satisfies P1(s1' | s1, a1) = P2(s2' | s2, a2).

[0085] To this end, the first state transfer model and reward prediction model are constructed to learn the equivalence relationship between multiple environments to further optimize the shared representation learned by the general encoder. The second state transfer model is used to predict the task-related representation output by the task-specific encoder for the final reconstructed observation.

[0086] c. Constructing environment classifier and task-specific encoder: The environment classifier is mainly composed of several fully connected layers. The last layer uses the Softmax activation function to calculate the probability distribution of the unmanned vehicle's obstacle avoidance task inference under different weather conditions. The role of the environment classifier is to further infer the type of weather environment it belongs to by inputting the shared state representation, thereby calling the task-specific encoder to which it belongs, and constructing the cross entropy loss to calculate the difference between the classification result of the environment type and the corresponding true category label to ensure the accuracy of the environment classification, and use back propagation to optimize the environment classifier.

[0087] The idea of ​​mutual simulation is used to optimize the universal encoder, learn the universality between multiple tasks and multiple environments, and accurately ignore task-irrelevant noise. This step is to create a task-specific encoder for each environment; the environment classifier in this embodiment is a multi-layer neural network, which accepts state representation of visual features. h sh , and map and normalize them to obtain the probability that the current environment corresponding to the visual feature belongs to different environment types, and then calculate the classification loss through the cross entropy loss function to optimize the environment classifier, and then call the specific task encoder to achieve "selective focus".

[0088] d. Construction of Soft Actor-Critic learning architecture: The observation space is composed of observation vectors spliced ​​by general encoders and task-specific encoders, and the continuous action space uses heading angle and speed control as the output of the policy network. The reward space considers safety, efficiency, and energy consumption to guide the unmanned system to complete decision-making tasks. The SoftActor-Critic model includes a policy network and a value network. The policy network accepts the state representation generated by the encoder module to output the probability distribution of action selection, and the value network evaluates the value of the current state-action pair, guiding the policy network to gradually tend to the optimal.

[0089] Since the physical state space S is difficult to obtain, the observation space O provided by the sensor is used as the input of the policy network. The continuous action space A uses heading angle and speed control as the output of the policy network. The reward space R considers safety, efficiency and energy consumption, and is used to guide the unmanned system to complete the decision-making task. The policy network accepts the state representation generated by the encoder module to output the probability distribution of action selection. The value network evaluates the value of the current state-action pair and guides the policy network to gradually tend to the optimal. The details are as follows:

[0090] For the policy network, its objective function is:

[0091]

[0092] in, The action distribution output by the policy network; The value of the observation-action pair estimated by the value network; is the temperature parameter, which is used to adjust the randomness of the strategy; is the entropy of the policy, which is used to encourage the policy network to maintain randomness when selecting actions.

[0093] For the value network, the value network estimates the expected reward of taking an action under observation o, that is, the Q value. At the same time, the goal of the value network is to approach the target value .

[0094]

[0095] The objective function of the value network is:

[0096]

[0097] in, Used to estimate the value of observation-action pairs; is the target Q value; is the discount factor; D is the experience revisit pool.

[0098] e. Joint loss function design and back propagation optimization, which includes feature consistency loss and reward consistency loss. Feature consistency loss is the error between the reconstructed features restored by the decoder and the reconstructed features of the previous moment, which are represented by the feature concatenation output by the first state transfer model and the second state transfer model at the current moment. Reward consistency loss is the error between the reward prediction output by the reward prediction model and the actual reward, which is used to optimize the encoder. In order to prevent the overfitting problem, L1 regularization can be further introduced to sparsely select features, and L2 regularization can be introduced to reduce the extreme values ​​of weights. Specifically, by performing L1 regularization on the weight parameters in the universal encoder, L1 regularization imposes penalties on all weights of the universal encoder, encouraging some weights to become 0, so it has a sparse effect and reduces the complexity of the universal encoder. By performing L2 regularization on weights greater than a preset threshold (the value is 0.5 in this embodiment) in the universal encoder, L2 regularization imposes greater penalties on larger weights, so that the weights tend to smaller values, prevent the weights from being too large, and improve the smoothness of the universal encoder.

[0099] In summary, the present invention solves the problems of data dependence and rule dependence in traditional decision-making methods by drawing inspiration from the information processing and decision-making methods of different organisms, and improves the efficiency, generalization and accuracy of unmanned systems for multi-task and multi-environment decision-making. Inspired by the selective focusing function of Hawkeye and the multi-regional division of labor and cooperation of the human brain, the present invention proposes a strong generalization decision-making method that focuses on useful information. By using the state abstraction method and the idea of ​​mutual simulation, a more compact state representation is created for the perceived information to distinguish between task-related and irrelevant information in the environment, and its exclusive Markov process is constructed for multiple environments, and the state characteristics of "behavioral equivalence" between tasks in different environments are established to better extract common information between different environments, and cooperate with environmental classifiers and task-specific encoders to capture the differences between different environments. It can selectively focus on useful information and accurately ignore task-irrelevant information, greatly improving the efficiency and versatility of unmanned system decision-making. Specifically, a general encoder and a task-specific encoder, a general state transition model and a task-specific state transition model, and an environment classifier are constructed respectively to generate predicted state representations for multiple tasks and multiple environments. Combined with the reinforcement learning algorithm, multi-task decision-making in complex environments is achieved, which provides new ideas for the intelligent decision-making technology of unmanned systems and can be applied to autonomous decision-making tasks of unmanned systems such as drones, unmanned vehicles, and unmanned boats.

[0100] In a second aspect, the present invention provides an obstacle avoidance decision device for an unmanned system, comprising:

[0101] The universal encoder is used to extract features from the image sequence of the environment collected by the unmanned system at the current time k, and obtain the visual feature h at the time k. sh (k);

[0102] Reinforcement learning model for h-based sh (k) Obtain the obstacle avoidance action a(k) of the unmanned system at time k, and the reward value r(k) after executing a(k); where a(k) is the obstacle avoidance decision at time k; the reward value when no collision occurs after the unmanned system executes a(k) is greater than the reward value when a collision occurs;

[0103] The collaborative training module is used to perform enhanced training after obtaining a(k), including:

[0104] The first state transition model is used based on h sh (k) and a(k) predict the visual feature h at time k+1 sh (k+1);

[0105] Environment classifier for h-based sh (k) Obtain the category C of the environment where the unmanned system is located at time k;

[0106] The task-specific encoder group includes: task-specific encoders corresponding to different environment categories; wherein the task-specific encoder corresponding to environment category C is used to sh (k) Get the task feature h at time k i (k);

[0107] The second state transition model is used based on h i (k) and a(k) predict the task feature h at time k+1 i (k+1);

[0108] Decoder for h sh (k+1) and h i The fusion feature of (k+1) is decoded to obtain the task feature h at time k+1 i (k+1);

[0109] Reward prediction model for h sh (k) Predict the reward value r' (k+1) of the unmanned system at time k+1;

[0110] Training control unit for calculating h rec (k+1) and h rec (k) as the feature consistency loss; calculate the difference loss between r'(k+1) and r(k) as the reward consistency loss; with the goal of minimizing the training loss including the feature consistency loss and the reward consistency loss and maximizing the cumulative reward value, the general encoder, the reinforcement learning model, the first state transfer model, the task-specific encoder corresponding to the environment category C, the second state transfer model, the decoder and the reward prediction model are trained at the same time;

[0111] Among them, the cumulative reward value is the sum of the reward values ​​output by the k reinforcement learning model from the initial moment to the current moment.

[0112] The related technical scheme is the same as the obstacle avoidance decision method for the unmanned system provided in the first aspect of the present invention, and will not be described in detail here.

[0113] In a third aspect, the present invention provides an obstacle avoidance decision system for an unmanned system, comprising: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the obstacle avoidance decision method for the unmanned system provided by the first aspect of the present invention is executed.

[0114] The related technical scheme is the same as the obstacle avoidance decision method for the unmanned system provided in the first aspect of the present invention, and will not be described in detail here.

[0115] In a fourth aspect, the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the obstacle avoidance decision method for the unmanned system provided in the first aspect of the present invention.

[0116] The related technical scheme is the same as the obstacle avoidance decision method for the unmanned system provided in the first aspect of the present invention, and will not be described in detail here.

[0117] In a fifth aspect, the invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the obstacle avoidance decision method for an unmanned system provided in the first aspect of the invention.

[0118] The related technical scheme is the same as the obstacle avoidance decision method for the unmanned system provided in the first aspect of the present invention, and will not be described in detail here.

[0119] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An obstacle avoidance decision method for an unmanned system, characterized in that: include: The image sequence collected by the unmanned system in the environment at the current time k is input into the universal encoder for feature extraction to obtain the visual feature h at the time k. sh (k); h sh (k) is input into the reinforcement learning model to obtain the obstacle avoidance action a(k) of the unmanned system at time k and the reward value r(k) after executing a(k); where a(k) is the obstacle avoidance decision at time k; the reward value when no collision occurs after the unmanned system executes a(k) is greater than the reward value when a collision occurs; After obtaining a(k), perform the following reinforcement training operations: h sh (k) and a(k) are input into the first state transition model to predict the visual feature h at time k+1 sh (k+1); h sh (k) is input into the environment classifier to obtain the category C of the environment where the unmanned system is located at time k; h sh (k) is input into the task-specific encoder corresponding to the environment category C to obtain the task feature h at time k i (k); h i (k) and a(k) are input into the second state transition model to predict the task feature h at time k+1 i (k+1); h sh (k+1) and h i (k+1) is fused and input into the decoder, and the visual reconstruction feature h at time k+1 is obtained by decoding. rec (k+1); calculate h rec (k+1) and h rec The difference loss between (k) is used as feature consistency loss; h sh (k) is input into the reward prediction model to predict the reward value r'(k+1) of the unmanned system at time k+1; the difference loss between r'(k+1) and r(k) is calculated as the reward consistency loss; With the goal of minimizing the training loss including the feature consistency loss and the reward consistency loss and maximizing the cumulative reward value, the general encoder, the reinforcement learning model, the first state transition model, the task-specific encoder corresponding to the environment category C, the second state transition model, the decoder and the reward prediction model are trained simultaneously; The cumulative reward value is the sum of the reward values ​​output by the k reinforcement learning model from the initial moment to the current moment.

2. The obstacle avoidance decision method for an unmanned system according to claim 1, characterized in that: The h sh (k+1) and h i (k+1) is fused and input to the decoder as follows: sh (k+1) and h i (k+1) are concatenated and input to the decoder.

3. The obstacle avoidance decision method for an unmanned system according to claim 1, characterized in that: The reinforcement learning model is a Soft Actor-Critic model.

4. The obstacle avoidance decision method for an unmanned system according to any one of claims 1 to 3, characterized in that: The environment classifier is pre-trained in the following way: Construct a training model including a cascaded feature extraction module and an environment classifier; input each image sequence sample in the pre-collected training set into the training model to obtain a prediction result of the corresponding environment category; The training model is trained with the goal of minimizing the difference loss between the predicted result of the environment category and the corresponding true category label to obtain a trained environment classifier; The training set includes multiple image sequence samples collected by unmanned systems in different environments at different times.

5. The obstacle avoidance decision method for an unmanned system according to any one of claims 1 to 3, characterized in that: Also includes: Get the real category label of the environment where the unmanned system is located at time k; Calculate the difference loss between the category C of the environment where the unmanned system is located at time k obtained by the environment classifier and the true category label as the classification loss; While minimizing the training loss and maximizing the cumulative reward value, the general encoder, reinforcement learning model, first state transition model, task-specific encoder corresponding to environment category C, second state transition model, decoder and reward prediction model are trained, and the environment classifier is also trained; Wherein, the training loss also includes the classification loss.

6. An obstacle avoidance decision device for an unmanned system, characterized in that: include: The universal encoder is used to extract features from the image sequence of the environment collected by the unmanned system at the current time k, and obtain the visual feature h at the time k. sh (k); Reinforcement learning model for h-based sh (k) Obtain the obstacle avoidance action a(k) of the unmanned system at time k, and the reward value r(k) after executing a(k); where a(k) is the obstacle avoidance decision at time k; the reward value when no collision occurs after the unmanned system executes a(k) is greater than the reward value when a collision occurs; The collaborative training module is used to perform enhanced training after obtaining a(k), including: The first state transition model is used based on h sh (k) and a(k) predict the visual feature h at time k+1 sh (k+1); Environment classifier for h-based sh (k) Obtain the category C of the environment where the unmanned system is located at time k; The task-specific encoder group includes: task-specific encoders corresponding to different environment categories; wherein the task-specific encoder corresponding to environment category C is used to sh (k) Get the task feature h at time k i (k); The second state transition model is used based on h i (k) and a(k) predict the task feature h at time k+1 i (k+1); Decoder for h sh (k+1) and h i The fusion feature of (k+1) is decoded to obtain the task feature h at time k+1 i (k+1); Reward prediction model for h sh (k) Predict the reward value r' (k+1) of the unmanned system at time k+1; Training control unit for calculating h rec (k+1) and h rec (k) as the feature consistency loss; calculate the difference loss between r'(k+1) and r(k) as the reward consistency loss; with the goal of minimizing the training loss including the feature consistency loss and the reward consistency loss and maximizing the cumulative reward value, the general encoder, the reinforcement learning model, the first state transfer model, the task-specific encoder corresponding to the environment category C, the second state transfer model, the decoder and the reward prediction model are trained at the same time; The cumulative reward value is the sum of the reward values ​​output by the k reinforcement learning model from the initial moment to the current moment.

7. The obstacle avoidance decision device for an unmanned system according to claim 6, characterized in that: The training control unit is also used to obtain the true category label of the environment where the unmanned system is located at time k; calculate the difference loss between the category C of the environment where the unmanned system is located at time k and the true category label as the classification loss; with the goal of minimizing the training loss and maximizing the cumulative reward value, while training the general encoder, the reinforcement learning model, the first state transition model, the task-specific encoder corresponding to the environment category C, the second state transition model, the decoder and the reward prediction model, the environment classifier is also trained; Wherein, the training loss also includes the classification loss.

8. An obstacle avoidance decision system for an unmanned system, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the obstacle avoidance decision method for an unmanned system according to any one of claims 1 to 5 is executed.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the obstacle avoidance decision method for an unmanned system according to any one of claims 1 to 5.

10. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements the obstacle avoidance decision method for an unmanned system as described in any one of claims 1-5.

Images