Obstacle avoidance decision-making method and system for unmanned system

Abstract

The invention discloses an obstacle avoidance decision-making method and system of an unmanned system, and belongs to the technical field of unmanned system decision-making. Extracting low-dimensional state representation shared in different environments from the current image sequence by using a universal encoder to obtain common visual features; then, a task specific encoder corresponding to the current environment type is adopted to further encode the common visual features into related feature representation for the current environment, personalized task features are obtained, so that the unmanned system learns the common visual features and the personalized task features of the environment in a complex environment, deep fusion is carried out, and the environment quality is improved. And feature reconstruction of the current image sequence is realized. On the basis, a mutual simulation thought is introduced, and training is performed based on feature consistency loss and reward consistency loss, so that the universal encoder can accurately extract common visual features, can pay attention to personalized task features, has a selective focusing function, and further combines with a reinforcement learning model to improve the accuracy of the universal visual features. And an accurate obstacle avoidance decision in a complex environment is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned system decision-making, and more specifically, relates to an obstacle avoidance decision-making method and system for an unmanned system. Background Art

[0002] Obstacle avoidance decision-making for unmanned systems is a critical step in ensuring that intelligent systems can safely and efficiently complete their missions. Its goal is to enable unmanned systems to efficiently process information and data and output optimal obstacle avoidance strategies in complex and dynamic environments. However, whether it's autonomous driving, drone swarms, or unmanned boats, the complex dynamics of the environment (such as crowded pedestrians, complex terrain, and fluctuating ocean currents) hinder the efficient output of obstacle avoidance decision-making. Therefore, researching an obstacle avoidance decision-making method for unmanned systems is of great significance.

[0003] Existing obstacle avoidance decision-making methods for unmanned systems often use machine learning-based methods. These methods use data-driven training to develop obstacle avoidance models and then fit the obstacle avoidance decision-making strategy. Given the complex and diverse environments in which unmanned systems operate, these methods require a large amount of high-quality data samples from different environments for model training. However, over-reliance on high-quality data can lead to data bias, and missing or inaccurate data can also lead to incorrect decisions, making it impossible to achieve accurate obstacle avoidance decisions in complex environments. Summary of the Invention

[0004] In response to the above-mentioned defects or improvement needs of the existing technology, the present invention provides a decision-making method and system for an unmanned system to solve the technical problem that the existing technology cannot achieve accurate obstacle avoidance decisions in complex environments.

[0005] To achieve the above objectives, in a first aspect, the present invention provides an obstacle avoidance decision-making method for an unmanned system, comprising: The image sequence of the environment collected by the unmanned system at the current time k is input into the universal encoder for feature extraction to obtain the visual feature h at the time k. sh (k); h sh (k) is input into the reinforcement learning model to obtain the obstacle avoidance action a(k) of the unmanned system at time k and the reward value r(k) after executing a(k); where a(k) is the obstacle avoidance decision at time k; the reward value when the unmanned system executes a(k) without a collision is greater than the reward value when a collision occurs; After obtaining a(k), perform the following reinforcement training operations: h sh (k) and a(k) are input into the first state transition model to predict the visual feature h at time k+1 sh (k+1); h sh(k) is input into the environment classifier to obtain the category C of the environment where the unmanned system is located at time k; h sh (k) is input into the task-specific encoder corresponding to the environment category C, and the task feature h at time k is obtained. i (k); h i (k) and a(k) are input into the second state transition model to predict the task feature h at time k+1 i (k+1); h sh (k+1) and h i (k+1) is fused and input into the decoder, and the visual reconstruction feature h at time k+1 is obtained by decoding. rec (k+1); calculate h rec (k+1) and h rec The difference loss between (k) is used as feature consistency loss; h sh (k) is input into the reward prediction model to predict the reward value r'(k+1) of the unmanned system at time k+1; the difference loss between r'(k+1) and r(k) is calculated as the reward consistency loss; With the goal of minimizing the training loss including feature consistency loss and reward consistency loss and maximizing the cumulative reward value, the general encoder, reinforcement learning model, first state transition model, task-specific encoder corresponding to environment category C, second state transition model, decoder and reward prediction model are trained simultaneously; Among them, the cumulative reward value is the sum of the reward values ​​output by the k reinforcement learning model from the initial moment to the current moment.

[0006] More preferably, the above sh (k+1) and h i (k+1) After fusion, the input to the decoder is: h sh (k+1) and h i (k+1) are concatenated and input to the decoder.

[0007] Further preferably, the above reinforcement learning model is a Soft Actor-Critic model.

[0008] Further preferably, the environment classifier is pre-trained in the following manner: Construct a training model consisting of a cascaded feature extraction module and an environment classifier; input each image sequence sample in the pre-collected training set into the training model to obtain the corresponding environment category prediction result; train the training model with the goal of minimizing the difference loss between the predicted environment category result and the corresponding true category label to obtain a trained environment classifier; The training set includes multiple image sequence samples collected by unmanned systems in different environments at different times.

[0009] Further preferably, the obstacle avoidance decision method for the unmanned system further includes: obtaining a true category label of the environment where the unmanned system is located at time k; calculating the difference loss between the category C of the environment where the unmanned system is located at time k obtained by the environment classifier and the true category label as the classification loss; While minimizing the training loss and maximizing the cumulative reward, the general encoder, reinforcement learning model, first state transition model, task-specific encoder corresponding to environment category C, second state transition model, decoder and reward prediction model are trained simultaneously, and the environment classifier is also trained; Among them, the above training loss also includes classification loss.

[0010] In a second aspect, the present invention provides an obstacle avoidance decision-making device for an unmanned system, comprising: The universal encoder is used to extract features from the image sequence of the environment collected by the unmanned system at the current time k, and obtain the visual feature h at the time k. sh (k); Reinforcement learning model for h-based sh (k) Obtain the obstacle avoidance action a(k) of the unmanned system at time k, and the reward value r(k) after executing a(k); where a(k) is the obstacle avoidance decision at time k; the reward value when the unmanned system executes a(k) without a collision is greater than the reward value when a collision occurs; The collaborative training module is used to perform intensive training after obtaining a(k), specifically including: The first state transition model is used based on h sh (k) and a(k) predict the visual feature h at time k+1 sh (k+1); Environment classifier for h-based sh (k) Obtain the category C of the environment where the unmanned system is located at time k; The task-specific encoder group includes: task-specific encoders corresponding to different environment categories; wherein the task-specific encoder corresponding to environment category C is used to sh (k) Get the task feature h at time k i (k); The second state transition model is used based on h i (k) and a(k) predict the task feature h at time k+1 i (k+1); Decoder for h sh (k+1) and h iThe fusion feature of (k+1) is decoded to obtain the task feature h at time k+1 i (k+1); Reward prediction model for h sh (k) Predict the reward value r'(k+1) of the unmanned system at time k+1; Training control unit for calculating h rec (k+1) and h rec (k) as the feature consistency loss; calculate the difference loss between r'(k+1) and r(k) as the reward consistency loss; with the goal of minimizing the training loss including the feature consistency loss and the reward consistency loss and maximizing the cumulative reward value, the general encoder, reinforcement learning model, first state transition model, task-specific encoder corresponding to environment category C, second state transition model, decoder and reward prediction model are trained simultaneously; Among them, the cumulative reward value is the sum of the reward values ​​output by the k reinforcement learning model from the initial moment to the current moment.

[0011] Further preferably, the training control unit is further configured to obtain a true category label of the environment in which the unmanned system is located at time k; calculate the difference loss between the category C of the environment in which the unmanned system is located at time k and the true category label as the classification loss; and simultaneously train the general encoder, the reinforcement learning model, the first state transition model, the task-specific encoder corresponding to the environment category C, the second state transition model, the decoder, and the reward prediction model with the goal of minimizing the training loss and maximizing the cumulative reward value, and also train the environment classifier; Among them, the above training loss also includes classification loss.

[0012] In a third aspect, the present invention provides an obstacle avoidance decision system for an unmanned system, comprising: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it executes the obstacle avoidance decision method for the unmanned system provided by the first aspect of the present invention.

[0013] In a fourth aspect, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is run by the processor, the device where the storage medium is located is controlled to execute the obstacle avoidance decision method for the unmanned system provided in the first aspect of the present invention.

[0014] In a fifth aspect, the invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the obstacle avoidance decision method for the unmanned system provided in the first aspect of the invention.

[0015] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects: 1. The present invention provides an obstacle avoidance decision method for an unmanned system, which uses a universal encoder to extract low-dimensional state representations shared in different environments from the current image sequence to obtain common visual features; then, a task-specific encoder corresponding to the current environment type is used to further encode the common visual features into relevant feature representations for the current environment on the basis of the universal encoder to obtain individual task features, thereby enabling the unmanned system to learn the common visual features and individual task features of the environment in a complex environment, and perform deep fusion to achieve feature reconstruction of the current image sequence. On this basis, the idea of ​​mutual simulation is introduced, and training is performed based on feature consistency loss and reward consistency loss, so that the universal encoder can accurately extract common visual features while also focusing on individual task features, and has a selective focusing function, and then combined with a reinforcement learning model to achieve obstacle avoidance decisions in complex environments; the present invention makes real-time decisions by learning the essential characteristics of the current environment, without relying on high-quality data samples, and the accuracy of the decision is high.

[0016] 2. The obstacle avoidance decision-making method for unmanned systems provided by this invention uses a universal encoder to obtain an abstract state representation of the image sequence, reducing the input dimensionality of the policy network and thereby improving sample utilization efficiency. Furthermore, abstract state representations often have strong semantic properties, facilitating human understanding of the model's decision-making basis and enabling domain experts to evaluate decision outcomes by analyzing the abstract state.

[0017] 3. The obstacle avoidance decision-making method for unmanned systems provided by the present invention uses bisimulation metrics to achieve state sharing of "different paths to the same goal", reducing the repeated learning of similar states by the strategy and achieving higher decision-making efficiency.

[0018] 4. Furthermore, the obstacle avoidance decision-making method for unmanned systems provided by the present invention uses a SoftActor-Critic model as the reinforcement learning model to better handle the complex dynamic environment and continuous control tasks faced by unmanned system obstacle avoidance decisions. Specifically, 1) the Soft Actor-Critic model uses a dual Q network and target smoothing to improve convergence performance and has strong stability; 2) the Soft Actor-Critic model has high sample efficiency and is suitable for real-world tasks with expensive data; 3) the Soft Actor-Critic model can adapt to complex control tasks and support the acquisition of continuous actions; 4) the Soft Actor-Critic model uses entropy regularization operations, which can improve the diversity and robustness of strategies and has strong exploratory capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1An obstacle avoidance decision framework for an unmanned system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0021] To achieve the above objectives, firstly, Figure 1 As shown, the present invention provides an obstacle avoidance decision method for an unmanned system, comprising: The image sequence of the environment collected by the unmanned system at the current time k is input into the universal encoder for feature extraction to obtain the visual feature h at the time k. sh (k); h sh (k) is input into the reinforcement learning model to obtain the obstacle avoidance action a(k) of the unmanned system at time k and the reward value r(k) after executing a(k); where a(k) is the obstacle avoidance decision at time k; the reward value when the unmanned system executes a(k) without a collision is greater than the reward value when a collision occurs; After obtaining a(k), perform the following reinforcement training operations: h sh (k) and a(k) are input into the first state transition model to predict the visual feature h at time k+1 sh (k+1); h sh (k) is input into the environment classifier to obtain the category C of the environment where the unmanned system is located at time k; h sh (k) is input into the task-specific encoder corresponding to the environment category C, and the task feature h at time k is obtained. i (k); h i (k) and a(k) are input into the second state transition model to predict the task feature h at time k+1 i (k+1); h sh (k+1) and h i (k+1) After fusion, it is input to the decoder, and the visual reconstruction feature h at time k+1 is obtained by decoding rec (k+1); calculate h rec (k+1) and h rec The difference loss between (k) is used as feature consistency loss; h sh(k) is input into the reward prediction model to predict the reward value r'(k+1) of the unmanned system at time k+1; the difference loss between r'(k+1) and r(k) is calculated as the reward consistency loss; With the goal of minimizing the training loss including feature consistency loss and reward consistency loss and maximizing the cumulative reward value, the general encoder, reinforcement learning model, first state transition model, task-specific encoder corresponding to environment category C, second state transition model, decoder and reward prediction model are trained simultaneously; Among them, the cumulative reward value is the sum of the reward values ​​output by the k reinforcement learning model from the initial moment to the current moment.

[0022] It should be noted that Figure 1 h' sh represents the h at the next moment sh , that is, when h sh h sh (k), h' sh h sh (k+1); similarly, r' represents the reward value of the unmanned system at the next moment predicted by the reward prediction model, that is, r' (k+1).

[0023] It should be noted that h sh (k+1) and h i There are many methods for (k+1) fusion, which can be directly spliced ​​(usually spliced ​​in the channel dimension), or fused by weighted average, attention mechanism, maximum pooling, average pooling, etc., which are not limited here. Preferably, in an optional implementation, splicing is used to fuse h sh (k+1) and h i (k+1).

[0024] It should be noted that the encoder can be a convolutional neural network (CNN), a self-supervised learning encoder (contrastive learning), a pre-trained model (ViT), etc., without limitation here. The general encoder and task-specific encoder can have the same or different structures, but the specific parameters are different.

[0025] The reinforcement learning model can be an Actor-Critic model, a Soft Actor-Critic model, a DDPG model, a TD3 model, a PPO model, an A3C model, an A2C model, etc., without limitation herein. Preferably, in an optional embodiment, the reinforcement learning model is a Soft Actor-Critic model.

[0026] The state transition model can be a Gaussian process model, a neural network model, a generative adversarial network, etc., which is not limited here. The structures of the first state transition model and the second state transition model can be the same or different, but the specific parameters are different.

[0027] The decoder can be a convolutional neural network (CNN), a self-supervised learning encoder (contrastive learning), a pre-trained model (ViT), etc., without limitation here.

[0028] The reward prediction model can be a linear regression reward prediction model, a neural network reward prediction model, a Gaussian process regression model, a reward prediction model based on temporal difference learning, etc., and is not limited here.

[0029] The environment classifier may be a K-nearest neighbor classifier, a support vector machine, a gradient boosting tree, a neural network, a Bayesian classifier, etc., and is not limited here.

[0030] In one optional implementation, the training loss is a weighted sum of feature consistency loss and reward consistency loss. The loss functions used to measure feature consistency loss and reward consistency loss can be L1 loss, L2 loss, cosine similarity, Huber loss, cross entropy loss, KL divergence, etc., without limitation.

[0031] It should be noted that there are many ways to train the environment classifier, which are not limited here. The following are the implementation methods of different training methods: In the first training method, the environment classifier is a pre-trained environment classifier, which is pre-trained in the following way: Construct a training model consisting of a cascaded feature extraction module and an environment classifier; input each image sequence sample in the pre-collected training set into the training model to obtain the corresponding environment category prediction result; train the training model with the goal of minimizing the difference loss between the predicted environment category result and the corresponding true category label to obtain a trained environment classifier; The training set includes image sequence samples collected by multiple unmanned systems at different times and in different environments; the feature extraction module can be CNN, convolutional neural network (CNN), self-supervised learning encoder (contrastive learning), pre-trained model (ViT), etc., which are not limited here.

[0032] In the second training mode, the obstacle avoidance decision method for the unmanned system further includes: obtaining a true category label of the environment in which the unmanned system is located at time k; calculating a difference loss between the category C of the environment in which the unmanned system is located at time k obtained by the environment classifier and the true category label as a classification loss; While minimizing the training loss and maximizing the cumulative reward, the general encoder, reinforcement learning model, first state transition model, task-specific encoder corresponding to environment category C, second state transition model, decoder and reward prediction model are trained simultaneously, and the environment classifier is also trained; The aforementioned training loss also includes classification loss. Loss functions used to measure classification loss can include cross-entropy loss, L2 loss function, Huber loss, cross-entropy loss, KL divergence, etc., and are not limited here.

[0033] In the third training method, the pre-trained environment classifier obtained in the first training method can be used as the initial environment classifier. Then, in the actual application process, the second training method is used to fine-tune the environment classifier to make the obstacle avoidance decision more accurate.

[0034] Because the second and third training methods require pre-labeling of the current environment's true category labels, when the user's goal is to make the unmanned system more intelligent without compromising the accuracy of obstacle avoidance decisions, a pre-trained environment classifier (i.e., the first training method) is preferred. If the user's goal is to achieve even higher accuracy in obstacle avoidance decisions, the second or third training method can be considered. The choice can be made based on the user's specific needs.

[0035] It should be noted that the unmanned system involved in the present invention can be an unmanned aerial vehicle, an unmanned vehicle, an unmanned boat or other unmanned systems, which is not limited here.

[0036] This invention addresses the difficulty of unmanned systems in making flexible, efficient, and accurate decisions in complex dynamic environments. By drawing inspiration from the information processing and decision-making methods of different organisms and referring to the selective focusing function of eagle eyes and the multi-region division of labor and cooperation model of the human brain, a cross-species bionic unmanned system decision-making method is proposed. Through deep reinforcement learning technology, a representation learning method is used to extract compact low-dimensional state representations from image sequences, and the idea of ​​mutual simulation is combined to ensure the strong correlation between state representation and task. At the same time, the consistency of representation before and after learning of a general encoder is designed, and an environmental classifier is constructed to determine the current environment type to call the appropriate task-specific encoder. This allows for accurate ignoring of irrelevant background noise information, improving the efficiency of perceptual information processing, and providing a new method and technical support for achieving efficient, generalized, and accurate multi-task obstacle avoidance decision-making for unmanned systems.

[0037] To further illustrate the obstacle avoidance decision-making method for an unmanned system provided by the present invention, the following takes the autonomous obstacle avoidance decision-making of an unmanned vehicle in a complex dynamic environment as an example and describes it in detail as follows: The specific decision-making method primarily consists of sensors and core algorithm modules. Multi-tasking refers to the obstacle avoidance tasks performed by unmanned vehicles while driving in diverse weather conditions (environments), such as rain, snow, sunset, and clear skies. The key difficulty lies in the diversity of sensory data and the complexity of continuous action decision-making brought about by different weather conditions, which hinder the current development of unmanned systems' obstacle avoidance decision-making performance in complex environments. Therefore, inspired by the selective focusing function of eagle eyes in the biological world and the multi-regional division of labor and collaboration in the human brain, the goal is to enable unmanned vehicles to selectively focus on useful information and accurately ignore task-irrelevant information, thereby identifying common characteristics of obstacle avoidance tasks in different weather conditions and avoiding repeated learning.

[0038] The specific contents of the autonomous obstacle avoidance decision-making task for unmanned vehicles are as follows: 1) Data reception and preprocessing: The unmanned vehicle first acquires perception data through sensors. Specifically, it uses a camera to obtain an image sequence of the current environment (generally including 3 to 5 frames of images), and then performs preprocessing operations such as denoising and filtering.

[0039] 2) Framework design: a. Construct a universal encoder and multiple task-specific encoders. In this embodiment, the universal encoder consists of convolutional layers and a multi-layer perceptron, capturing shared spatial features and low-dimensional state representations across different weather conditions (and environmental conditions), generating corresponding visual features. The task-specific encoder, building on the universal encoder, further encodes the state representation into task-specific feature representations tailored to each weather condition. The universal and task-specific encoders have similar structures, including convolutional layers and an attention mechanism, but their parameters are independent. Therefore, the universal encoder and task-specific encoder generate universal and task-specific state representations, respectively. This module design, inspired by the multi-regional division of labor in the human brain, enables a deep fusion of commonalities and individualities in task decision-making for autonomous vehicles in diverse environments. Multiple task-specific encoders are provided, one for each weather condition. These task-specific encoders can share the same structure but differ in parameters.

[0040] The universal encoder and task-specific encoder are inspired by the multi-region division of labor and collaboration of the human brain. When faced with new or unique situations, the brain uses existing common knowledge as a basis and adjusts to specific environmental factors, needs, and goals. The universal encoder is used to extract shared state representations that are not dependent on specific tasks. sh , providing consistent state representation for multi-task and multi-environment to reduce encoder redundancy. Task-specific encoder generates state representation h related to specific tasks for multi-task and multi-environment problems. i , by context-dependent switching of task-specific encoders and fusing two types of encoders h t = cat(hsh , h i ) can enable unmanned systems to learn the common and individual characteristics of the environment in complex environments, so as to facilitate strong generalization decision-making.

[0041] b. Constructing a state transition model and reward prediction model: This approach introduces the concept of bisimulation and extends it to a cross-Markov decision process (MDP) environment. Based on bisimulation, by defining state equivalence mappings for obstacle avoidance tasks under different weather conditions, a state transition model and a reward model are constructed within the cross-MDP environment. These models are used to predict the next state and reward value of the autonomous vehicle, respectively. Loss functions for the state transition model and reward model are also established to ensure the universality of finding the same state-action pair across multiple MDPs. This process, by mimicking the selective focusing capabilities of an eagle's eye, enables the autonomous vehicle to focus on useful information in different weather conditions, preventing dynamic environmental changes from affecting its autonomous decision-making.

[0042] Specifically, for the bisimulation relation of a single MDP, that is, given an MDP, M =<S, A, P, R> , where S is the state space, A is the action space, P(s'|s, a) is the state transition probability, and R(s, a) is the reward function. The bisimulation relation B defines an equivalence relation , if for two states s', s∈S, two conditions are satisfied: (1) Same reward: if s ~ Bs', then R(s, a) = R(s', a) for all actions a∈A; (2) Uniform transition probabilities: for every action a∈A and state s''∈S, if s ~ Bs', then P(s''| s, a) = P(s''| s', a).

[0043] For the bisimulation relationship across MDPs, assume there are two MDPs: M1=<S1, A1, P1, R1> ,M2=<S2, A2, P2,R2> , if a bisimulation relation is defined between two MDPs to make them behaviorally consistent, then the cross-MDP bisimulation relation is defined as follows: for M1 and M2 with equivalence relation, there exists a relation , and satisfy two conditions: (1) Reward consistency: for all (s1, s2)∈B and all actions a1∈A1, a2∈A2, it should satisfy R1(s1, a1) = R2(s2, a2); (2) Transfer consistency: for all (s1, s2)∈B and all actions a1∈A1, a2∈A2, the transfer probability satisfies P1(s1' | s1, a1) = P2(s2' | s2, a2).

[0044] To this end, the first state transfer model and reward prediction model are constructed to learn the equivalence relationship between multiple environments to further optimize the shared representation learned by the universal encoder. The second state transfer model is used to predict the task-related representation output by the task-specific encoder for the final reconstructed observation.

[0045] c. Build an environment classifier and task-specific encoder: The environment classifier primarily consists of several fully connected layers, the last of which uses a softmax activation function to calculate the probability distribution of the unmanned vehicle's obstacle avoidance tasks under different weather conditions. The environment classifier uses the shared state representation as input to further infer the type of weather environment, thereby invoking the task-specific encoder. A cross-entropy loss is constructed to calculate the difference between the classified environment type and the corresponding true class label to ensure accurate environment classification. Backpropagation is then used to optimize the environment classifier.

[0046] The bisimulation idea is used to optimize the universal encoder, learn the universality between multiple tasks and multiple environments, and accurately ignore task-irrelevant noise. This step is achieved by creating a task-specific encoder for each environment; the environment classifier in this embodiment is a multi-layer neural network that accepts state representation of visual features. h sh , and map and normalize them to obtain the probability that the current environment corresponding to the visual feature belongs to different environment types, and then calculate the classification loss through the cross entropy loss function to optimize the environment classifier, and then call the specific task encoder to achieve "selective focus".

[0047] d. Construction of the Soft Actor-Critic Learning Architecture: The observation space consists of observation vectors spliced ​​from a general encoder and a task-specific encoder. The continuous action space uses heading angle and velocity control as the outputs of the policy network. The reward space considers safety, efficiency, and energy consumption to guide the unmanned system in completing its decision-making tasks. The Soft Actor-Critic model consists of a policy network and a value network. The policy network accepts the state representation generated by the encoder module and outputs a probability distribution of action choices. The value network evaluates the value of the current state-action pair, guiding the policy network towards the optimal state.

[0048] Because the physical state space S is difficult to obtain, the observation space O provided by sensors is used as the input of the policy network. The continuous action space A uses heading angle and speed control as the output of the policy network. The reward space R considers safety, efficiency, and energy consumption to guide the unmanned system in completing decision-making tasks. The policy network accepts the state representation generated by the encoder module and outputs the probability distribution of action selection. The value network evaluates the value of the current state-action pair, guiding the policy network towards the optimal state. The details are as follows: For the policy network, its objective function is: in, Action distribution output by the policy network; The value of the observation-action pair estimated by the value network; is the temperature parameter, which is used to adjust the randomness of the strategy; is the entropy of the policy, which is used to encourage the policy network to maintain randomness when selecting actions.

[0049] For the value network, the value network estimates the expected reward of taking action under observation o, that is, the Q value. At the same time, the goal of the value network is to approach the target value .

[0050] The objective function of the value network is: in, Used to estimate the value of observation-action pairs; is the target Q value; is the discount factor; D is the experience return pool.

[0051] e. Joint loss function design and backpropagation optimization, including feature consistency loss and reward consistency loss. Feature consistency loss is calculated by calculating the error between the reconstructed features (the concatenation of the feature representations output by the first and second state transition models at the current moment, restored by the decoder) and the reconstructed features at the previous moment. Reward consistency loss is calculated by calculating the error between the reward prediction output by the reward prediction model and the actual reward. This loss is used to optimize the encoder. To prevent overfitting, L1 regularization can be further introduced to sparsify features, and L2 regularization can be introduced to reduce extreme weight values. Specifically, L1 regularization is applied to the weight parameters in the universal encoder. L1 regularization penalizes all weights in the universal encoder, encouraging some weights to become zero. This has a sparsifying effect and reduces the complexity of the universal encoder. L2 regularization is applied to weights in the universal encoder that are greater than a preset threshold (0.5 in this embodiment). L2 regularization imposes a greater penalty on larger weights, causing them to tend toward smaller values, preventing them from becoming excessively large and improving the smoothness of the universal encoder.

[0052] In summary, the present invention draws inspiration from the information processing and decision-making methods of different organisms to solve the problems of data dependence and rule dependence in traditional decision-making methods, thereby improving the efficiency, generalization and accuracy of unmanned systems for multi-task and multi-environment decision-making. Inspired by the selective focusing function of Eagle Eye and the multi-regional division of labor and cooperation of the human brain, the present invention proposes a strong generalization decision-making method that focuses on useful information. By using the state abstraction method and the idea of ​​mutual simulation, a more compact state representation is created for the perception information to distinguish between task-related and irrelevant information in the environment, and its exclusive Markov process is constructed for multiple environments. The state characteristics of "behavioral equivalence" between tasks in different environments are established to better extract common information between different environments. The environment classifier and task-specific encoder are used to capture the differences between different environments. It can selectively focus on useful information and accurately ignore task-irrelevant information, greatly improving the efficiency and versatility of unmanned system decision-making. Specifically, a general encoder and a task-specific encoder, a general state transition model and a task-specific state transition model, and an environment classifier are constructed respectively to generate predicted state representations for multiple tasks and multiple environments. Combined with the reinforcement learning algorithm, multi-task decision-making in complex environments is achieved, which provides new ideas for the intelligent decision-making technology of unmanned systems and can be applied to autonomous decision-making tasks of unmanned systems such as drones, unmanned vehicles, and unmanned boats.

[0053] In a second aspect, the present invention provides an obstacle avoidance decision-making device for an unmanned system, comprising: The universal encoder is used to extract features from the image sequence of the environment collected by the unmanned system at the current time k, and obtain the visual feature h at the time k. sh (k); Reinforcement learning model for h-based sh (k) Obtain the obstacle avoidance action a(k) of the unmanned system at time k, and the reward value r(k) after executing a(k); where a(k) is the obstacle avoidance decision at time k; the reward value when the unmanned system executes a(k) without a collision is greater than the reward value when a collision occurs; The collaborative training module is used to perform intensive training after obtaining a(k), specifically including: The first state transition model is used based on h sh (k) and a(k) predict the visual feature h at time k+1 sh (k+1); Environment classifier for h-based sh (k) Obtain the category C of the environment where the unmanned system is located at time k; The task-specific encoder group includes: task-specific encoders corresponding to different environment categories; wherein the task-specific encoder corresponding to environment category C is used to sh (k) Get the task feature h at time ki (k); The second state transition model is used based on h i (k) and a(k) predict the task feature h at time k+1 i (k+1); Decoder for h sh (k+1) and h i The fusion feature of (k+1) is decoded to obtain the task feature h at time k+1 i (k+1); Reward prediction model for h sh (k) Predict the reward value r'(k+1) of the unmanned system at time k+1; Training control unit for calculating h rec (k+1) and h rec (k) as the feature consistency loss; calculate the difference loss between r'(k+1) and r(k) as the reward consistency loss; with the goal of minimizing the training loss including the feature consistency loss and the reward consistency loss and maximizing the cumulative reward value, the general encoder, reinforcement learning model, first state transition model, task-specific encoder corresponding to environment category C, second state transition model, decoder and reward prediction model are trained simultaneously; Among them, the cumulative reward value is the sum of the reward values ​​output by the k reinforcement learning model from the initial moment to the current moment.

[0054] The relevant technical solutions are the same as the obstacle avoidance decision method for unmanned systems provided in the first aspect of the present invention, and will not be described in detail here.

[0055] In a third aspect, the present invention provides an obstacle avoidance decision system for an unmanned system, comprising: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it executes the obstacle avoidance decision method for the unmanned system provided by the first aspect of the present invention.

[0056] The relevant technical solutions are the same as the obstacle avoidance decision method for unmanned systems provided in the first aspect of the present invention, and will not be described in detail here.

[0057] In a fourth aspect, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is run by the processor, the device where the storage medium is located is controlled to execute the obstacle avoidance decision method for the unmanned system provided in the first aspect of the present invention.

[0058] The relevant technical solutions are the same as the obstacle avoidance decision method for unmanned systems provided in the first aspect of the present invention, and will not be described in detail here.

[0059] In a fifth aspect, the invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the obstacle avoidance decision method for the unmanned system provided in the first aspect of the invention.

[0060] The relevant technical solutions are the same as the obstacle avoidance decision method for unmanned systems provided in the first aspect of the present invention, and will not be described in detail here.

[0061] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An obstacle avoidance decision method for an unmanned system, characterized in that: include: The image sequence collected by the unmanned system in the environment at the current time k is input into the universal encoder for feature extraction to obtain the visual feature h at the time k. sh (k); h sh (k) is input into the reinforcement learning model to obtain the obstacle avoidance action a(k) of the unmanned system at time k and the reward value r(k) after executing a(k); where a(k) is the obstacle avoidance decision at time k; the reward value when no collision occurs after the unmanned system executes a(k) is greater than the reward value when a collision occurs; After obtaining a(k), perform the following reinforcement training operations: h sh (k) and a(k) are input into the first state transition model to predict the visual feature h at time k+1 sh (k+1); h sh (k) is input into the environment classifier to obtain the category C of the environment where the unmanned system is located at time k; h sh (k) is input into the task-specific encoder corresponding to the environment category C to obtain the task feature h at time k i (k); h i (k) and a(k) are input into the second state transition model to predict the task feature h at time k+1 i (k+1); h sh (k+1) and h i (k+1) is fused and input into the decoder, and the visual reconstruction feature h at time k+1 is obtained by decoding. rec (k+1); calculate h rec (k+1) and h rec The difference loss between (k) is used as feature consistency loss; h sh (k) is input into the reward prediction model to predict the reward value r'(k+1) of the unmanned system at time k+1; the difference loss between r'(k+1) and r(k) is calculated as the reward consistency loss; With the goal of minimizing the training loss including the feature consistency loss and the reward consistency loss and maximizing the cumulative reward value, the general encoder, the reinforcement learning model, the first state transition model, the task-specific encoder corresponding to the environment category C, the second state transition model, the decoder and the reward prediction model are trained simultaneously; The cumulative reward value is the sum of the reward values ​​output by the k reinforcement learning model from the initial moment to the current moment.

2. The obstacle avoidance decision method for an unmanned system according to claim 1, characterized in that: The h sh (k+1) and h i (k+1) is fused and input to the decoder as follows: sh (k+1) and h i (k+1) are concatenated and input to the decoder.

3. The obstacle avoidance decision method for an unmanned system according to claim 1, characterized in that: The reinforcement learning model is a Soft Actor-Critic model.

4. The obstacle avoidance decision method for an unmanned system according to any one of claims 1 to 3, characterized in that: The environment classifier is pre-trained in the following way: Construct a training model including a cascaded feature extraction module and an environment classifier; input each image sequence sample in the pre-collected training set into the training model to obtain a prediction result of the corresponding environment category; The training model is trained with the goal of minimizing the difference loss between the predicted result of the environment category and the corresponding true category label to obtain a trained environment classifier; The training set includes multiple image sequence samples collected by unmanned systems in different environments at different times.

5. The obstacle avoidance decision method for an unmanned system according to any one of claims 1 to 3, characterized in that: Also includes: Get the real category label of the environment where the unmanned system is located at time k; Calculate the difference loss between the category C of the environment where the unmanned system is located at time k obtained by the environment classifier and the true category label as the classification loss; While minimizing the training loss and maximizing the cumulative reward value, the general encoder, reinforcement learning model, first state transition model, task-specific encoder corresponding to environment category C, second state transition model, decoder and reward prediction model are trained, and the environment classifier is also trained; Wherein, the training loss also includes the classification loss.

6. An obstacle avoidance decision device for an unmanned system, characterized in that: include: The universal encoder is used to extract features from the image sequence of the environment collected by the unmanned system at the current time k, and obtain the visual feature h at the time k. sh (k); Reinforcement learning model for h-based sh (k) Obtain the obstacle avoidance action a(k) of the unmanned system at time k, and the reward value r(k) after executing a(k); where a(k) is the obstacle avoidance decision at time k; the reward value when no collision occurs after the unmanned system executes a(k) is greater than the reward value when a collision occurs; The collaborative training module is used to perform enhanced training after obtaining a(k), including: The first state transition model is used based on h sh (k) and a(k) predict the visual feature h at time k+1 sh (k+1); Environment classifier for h-based sh (k) Obtain the category C of the environment where the unmanned system is located at time k; The task-specific encoder group includes: task-specific encoders corresponding to different environment categories; wherein the task-specific encoder corresponding to environment category C is used to sh (k) Get the task feature h at time k i (k); The second state transition model is used based on h i (k) and a(k) predict the task feature h at time k+1 i (k+1); Decoder for h sh (k+1) and h i The fusion feature of (k+1) is decoded to obtain the task feature h at time k+1 i (k+1); Reward prediction model for h sh (k) Predict the reward value r' (k+1) of the unmanned system at time k+1; Training control unit for calculating h rec (k+1) and h rec (k) as the feature consistency loss; calculate the difference loss between r'(k+1) and r(k) as the reward consistency loss; with the goal of minimizing the training loss including the feature consistency loss and the reward consistency loss and maximizing the cumulative reward value, the general encoder, the reinforcement learning model, the first state transfer model, the task-specific encoder corresponding to the environment category C, the second state transfer model, the decoder and the reward prediction model are trained at the same time; The cumulative reward value is the sum of the reward values ​​output by the k reinforcement learning model from the initial moment to the current moment.

7. The obstacle avoidance decision device for an unmanned system according to claim 6, characterized in that: The training control unit is also used to obtain the true category label of the environment where the unmanned system is located at time k; calculate the difference loss between the category C of the environment where the unmanned system is located at time k and the true category label as the classification loss; with the goal of minimizing the training loss and maximizing the cumulative reward value, while training the general encoder, the reinforcement learning model, the first state transition model, the task-specific encoder corresponding to the environment category C, the second state transition model, the decoder and the reward prediction model, the environment classifier is also trained; Wherein, the training loss also includes the classification loss.

8. An obstacle avoidance decision system for an unmanned system, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the obstacle avoidance decision method for an unmanned system as described in any one of claims 1 to 5 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the obstacle avoidance decision method for an unmanned system according to any one of claims 1 to 5.

10. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements the obstacle avoidance decision method for an unmanned system as described in any one of claims 1-5.

Images