Robotic dog collaborative decision-making method and system based on improved near-end strategy optimization algorithm

Abstract

The invention relates to the technical field of robot dog decision making, and provides a robot dog collaborative decision making method and system based on an improved near-end strategy optimization algorithm, and the method comprises the steps: S1, obtaining a robot dog local observation matrix based on the state information of a robot dog, the state information of an own robot dog, and the observation information of the robot dog; s2, carrying out coding processing on the local observation matrix of the robot dog to obtain an embedded matrix of a multi-dimensional space vector; s3, inputting the embedded matrix of the multi-dimensional space vector into an initial robot dog action decision network, and outputting an optimal action executed by a robot dog; s4, inputting the embedded matrix of the multi-dimensional space vector into the state value network, outputting state value parameters, and optimizing the robot dog action decision network and the state value network based on the state value parameters; and S5, iteratively executing the steps S1 to S4 to complete the collaborative decision-making task. According to the invention, cluster efficient collaborative decision can be realized, and task completion efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of robot dog decision-making technology, and in particular to a robot dog collaborative decision-making method and system based on an improved proximal policy optimization algorithm. Background Technology

[0002] As robot dog technology matures, its speed and flexibility have led to its widespread application in logistics, police patrols, home assistance for the elderly, and disaster search and rescue. Particularly in disaster search and rescue scenarios, the situations faced by robot dogs cannot be pre-planned through modeling. Robot dog swarms must rely on their individual perception capabilities and collaborative decision-making abilities to complete designated tasks. Therefore, collaborative decision-making methods based on known environmental information and information exchange between robot dog swarms are needed to address the decision-making problems of robot dogs in unknown environments.

[0003] Existing collaborative decision-making methods for robotic dogs are based on extracting environmental information to form a state-target node graph of the robotic dog. They then use a unified collaborative strategy for task decision-making and leverage a swarm of robotic dogs to iteratively explore unknown areas, thus completing the search task for a designated region. However, this approach requires the swarm to continuously explore and iterate over the location area, making it difficult to guarantee efficient exploration. Other collaborative decision-making methods utilize robotic dog state information and reinforcement learning to achieve collaborative decision-making tasks in simple scenarios, but they do not consider the collaborative decision-making problem of robotic dogs in complex and dynamic scenarios.

[0004] For complex and unknown task scenarios, robot dogs need to make behavioral decisions while perceiving their surroundings. Current methods based on directed graphs or decision trees are insufficient to cover all application scenarios, lack the ability to handle unexpected situations, and fail to achieve effective decision planning. Furthermore, in the collaborative decision-making problem of robot dogs in complex scenarios, decision trees or directed graphs cannot construct suitable decision chains to achieve optimal decision-making, resulting in long decision chains and low decision-making efficiency. Summary of the Invention

[0005] The purpose of this invention is to solve at least one technical problem in the background art and to provide a collaborative decision-making method and system for robot dogs based on an improved proximal policy optimization algorithm.

[0006] To achieve the above objectives, this invention provides a collaborative decision-making method for robot dogs based on an improved proximal policy optimization algorithm, comprising: S1. Obtain the robot dog's local observation matrix based on the robot dog's own state information, the friendly robot dog's state information, and its own observation information; S2. Encode the local observation matrix of the robot dog to obtain the embedding matrix of the multidimensional space vector; S3. Input the embedding matrix of the multidimensional space vector into the initial robot dog action decision network, output the probability distribution of available actions using action masking and normalization strategies, and use a greedy strategy to select the optimal action for the robot dog to execute from the probability distribution of available actions; after executing the action, obtain the training reward and the robot dog's own state information, the friendly robot dog's state information, and its own observation information at the next time step. S4. Input the embedding matrix of the multidimensional space vector into the state value network, output the state value parameters through the state value network, update the robot dog action decision network parameters based on the state value parameters and the action decision network loss function to optimize the robot dog action decision network, and update the state value network parameters based on the state value parameters and the state value network loss function to optimize the state value network. S5. Iteratively execute steps S1-S4 to optimize action strategies and achieve collaborative decision-making tasks.

[0007] According to one aspect of the present invention, obtaining the local observation matrix of the robot dog based on the robot dog's own state information, the user's own robot dog state information, and its own observation information includes: robot dog Local observation matrix acquired at each time step From its own state information Status information of our own robot dog and its own observation information It consists of three parts, the local observation matrix. Represented as: ; in 'i' represents the robot dog's serial number. Let x and y represent the horizontal and vertical coordinates of the position of robot dog i at time t, respectively. This represents the velocity component of the robot dog's current velocity at time t on the horizontal axis. This represents the velocity component of the robot dog's current velocity at time t on the vertical axis. This represents the azimuth information of the robot dog at time t, where h represents the robot's current orientation. This parameter represents the progress of the task being executed by the robot dog at time t. If the robot dog is idle, this parameter is 0. When the robot dog first receives a task, this parameter is 1. As the task progresses, the parameter gradually decreases proportionally until the task is completed. This indicates the type of task the robot dog performs: 0 for idle tasks, 1 for area search tasks, and 2 for fixed-point search and rescue tasks. ,in This represents the machine vision pixel matrix acquired by the robot dog's camera. This represents the point cloud matrix acquired by the robot dog's lidar.

[0008] According to one aspect of the present invention, the encoding process of the local observation matrix of the robot dog to obtain the embedding matrix of the multidimensional spatial vector includes: Using multilayer perceptrons to analyze local matrices Chinese robot dog status information set The data is encoded and converted into an observation matrix. The activation function used in the multilayer perceptron is ReLU, and its calculation formula is: ; in, }, For its own state information The vector representation after passing through a multilayer perceptron; Using convolutional neural networks to analyze machine vision pixel matrices The data is encoded and converted into an observation matrix. The activation function used in the convolutional neural network is ReLU, and its calculation formula is: ; Using Long Short-Time Memory Networks to Study LiDAR Point Cloud Matrix The data is encoded and converted into an observation matrix. The activation function used in the Long Short-Term Memory (LSTM) network is the Sigmoid function, calculated as follows: ; The acquired observation matrix is ​​encoded using a multi-head attention mechanism, and a query matrix representing the relationships between various state information is constructed using a multi-layer perceptron. Key matrix and value matrix Thus, the embedding matrix S of the multidimensional space vector is obtained; Among them, the query matrix Key matrix Value matrix The formula for calculating the embedding matrix S of a multidimensional vector space is: ; ; ; ; Among them, the query matrix Observation vectors generated using only their own state information Obtain the key matrix Sum matrix Generated using all observation matrices. Represents the vector dimension.

[0009] According to one aspect of the present invention, the update formula for the state value network parameters is as follows: ; in, For state-value network parameters, Let the state value network loss function be... , This represents the training reward value obtained by the robot dog after performing the action at time t. These represent the state-value network V in... Embedding matrix based on multidimensional space vectors at every moment Output state value parameters It is the learning rate of the behavioral decision network.

[0010] According to one aspect of the present invention, the update formula for the action policy network parameters is as follows: ; in, For action policy network parameters, Let the loss function be the action decision network. ,in, , ) represents the probability distribution of available actions output by the robot dog's action policy network at the current moment. This represents the probability distribution of available actions output by the robot dog's action policy network at the previous time step. For hyperparameters, Let represent the dominance function, where This is a hyperparameter.

[0011] To achieve the above objectives, the present invention also provides a robot dog collaborative decision-making system based on an improved proximal policy optimization algorithm, comprising: The local observation matrix acquisition module obtains the robot dog's local observation matrix based on the robot dog's own state information, the friendly robot dog's state information, and its own observation information. The embedding matrix calculation module for multidimensional spatial vectors encodes the local observation matrix of the robot dog to obtain the embedding matrix of the multidimensional spatial vectors. The action policy output module inputs the embedding matrix of the multidimensional space vector into the initial robot dog action decision network, outputs the probability distribution of available actions using action masking and normalization strategies, and uses a greedy strategy to select the optimal action for the robot dog to execute from the probability distribution of available actions; after executing the action, it obtains the training reward and the robot dog's own state information, the friendly robot dog's state information, and its own observation information at the next time step. The network optimization module inputs the embedding matrix of multidimensional space vectors into the state value network, outputs state value parameters through the state value network, updates the robot dog action decision network parameters based on the state value parameters and the action decision network loss function to optimize the robot dog action decision network, and updates the state value network parameters based on the state value parameters and the state value network loss function to optimize the state value network. The collaborative decision-making implementation module iteratively executes the local observation matrix acquisition module and the network optimization module to optimize the action strategy and realize the collaborative decision-making task.

[0012] To achieve the above objectives, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the robot dog cooperative decision-making method based on the improved proximal policy optimization algorithm as described above.

[0013] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the robot dog cooperative decision-making method based on the improved proximal policy optimization algorithm as described above.

[0014] According to the present invention, the proposed collaborative decision-making method for robot dog clusters based on an improved near-end policy optimization algorithm, by fusing local observation and global state information and using a reinforcement learning-based training method, enables the robot dog cluster to have the ability to cope with sudden states in multiple scenarios and make rapid decisions, thereby achieving efficient collaborative decision-making of the cluster and improving task completion efficiency.

[0015] According to the present invention, the decision-making efficiency of robot dog clusters under complex conditions is considered. Three types of neural networks are used to extract robot dog state information, which can ensure that the state data of robot dogs can be effectively extracted in complex environments.

[0016] This invention introduces an attention mechanism on the basis of the proximal policy optimization algorithm, and achieves rapid collaborative decision-making in complex environments through deep reinforcement learning training, avoiding the problems of excessively long decision-making time and the inability of traditional decision-making methods to solve sudden state decision-making problems.

[0017] The decision-making method provided by this invention does not require the construction of a directed graph or decision tree. It can directly generate decision strategies based on sensor information and the state information of other robot dogs, thereby improving the decision response time and enabling rapid response to task and emergency needs. It avoids the problem of low decision efficiency caused by an excessively long decision chain.

[0018] The algorithm used in this invention incorporates the state information of other robot dogs in the robot dog cluster, enabling the robot dogs to make full decisions based on global information. It achieves efficient cluster collaboration through a centralized training mode, avoiding resource waste and low search efficiency caused by untimely scheduling of robot dogs. Attached Figure Description

[0019] Figure 1 The flowchart illustrates a collaborative decision-making method for robot dogs based on an improved proximal policy optimization algorithm according to an embodiment of the present invention. Detailed Implementation

[0020] The invention will now be discussed with reference to exemplary embodiments. It should be understood that the described embodiments are merely intended to enable those skilled in the art to better understand and thus implement the invention, and are not intended to imply any limitation on the scope of the invention.

[0021] As used herein, the term "comprising" and its variations are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "at least partially based on". The terms "one embodiment" and "an embodiment" are to be interpreted as "at least one embodiment".

[0022] Figure 1 The flowchart schematically illustrates a robot dog cooperative decision-making method based on an improved proximal policy optimization algorithm according to an embodiment of the present invention. Figure 1 As shown, in this embodiment, the robot dog cooperative decision-making method based on the improved proximal policy optimization algorithm includes: S1. Obtain the robot dog's local observation matrix based on the robot dog's own state information, the friendly robot dog's state information, and its own observation information; S2. Encode the local observation matrix of the robot dog to obtain the embedding matrix of the multidimensional space vector; S3. Input the embedding matrix of the multidimensional space vector into the initial robot dog action decision network, output the probability distribution of available actions using action masking and normalization strategies, and use a greedy strategy to select the optimal action for the robot dog to execute from the probability distribution of available actions; after executing the action, obtain the training reward and the robot dog's own state information, the friendly robot dog's state information, and its own observation information at the next time step. S4. Input the embedding matrix of the multidimensional space vector into the state value network, output the state value parameters through the state value network, update the robot dog action decision network parameters based on the state value parameters and the action decision network loss function to optimize the robot dog action decision network, and update the state value network parameters based on the state value parameters and the state value network loss function to optimize the state value network. S5. Iteratively execute steps S1-S4 to optimize action strategies and achieve collaborative decision-making tasks.

[0023] In this implementation, it is necessary to confirm that the communication between the robot dogs is good and there is no communication interference, and that each robot dog can receive the status information of other robot dogs in the cluster; the robot dogs have local observation capabilities, and the robot dogs can only obtain the observation information collected by their own sensors.

[0024] In this embodiment, the robot dog's action decision network and state value network are commonly used neural networks, such as multilayer perceptrons and convolutional neural networks.

[0025] Furthermore, according to one embodiment of the present invention, a local observation matrix of the robot dog is obtained based on the robot dog's own state information, the robot dog's own state information, and its own observation information, including: robot dog Local observation matrix acquired at each time step From its own state information Status information of our own robot dog and its own observation information It consists of three parts, the local observation matrix. It can be represented as: ; in, 'i' represents the robot dog's serial number. Let x and y represent the horizontal and vertical coordinates of the position of robot dog i at time t, respectively. This represents the velocity component of the robot dog's current velocity at time t on the horizontal axis. This represents the velocity component of the robot dog's current velocity at time t on the vertical axis. This represents the azimuth information of the robot dog at time t, where h represents the robot's current orientation. This parameter represents the progress of the task being executed by the robot dog at time t. If the robot dog is idle, this parameter is 0. When the robot dog first receives a task, this parameter is 1. As the task progresses, the parameter gradually decreases proportionally until the task is completed. This indicates the type of task the robot dog performs: 0 for idle tasks, 1 for area search tasks, and 2 for fixed-point search and rescue tasks. ,in This represents the machine vision pixel matrix acquired by the robot dog's camera. This represents the point cloud matrix acquired by the robot dog's lidar.

[0026] Furthermore, according to one embodiment of the present invention, the local observation matrix of the robot dog is encoded to obtain an embedding matrix of a multidimensional spatial vector, including: (1) Using a multilayer perceptron (MLP) to process state information The data is encoded and converted into an observation matrix. The activation function used in the multilayer perceptron is ReLU (Rectified Linear Unit), and its calculation formula is as follows: ; in, }, For its own state information The vector representation after passing through the multilayer perceptron; this step mainly extracts features from the robot dog's state vector. The dimension of feature extraction and the number of multilayer perceptron layers can be dynamically adjusted according to the number of clusters, increasing or decreasing based on the dimension of the observed information. (2) Using a convolutional neural network (CNN) to process the pixel matrix of machine vision The data is encoded and converted into an observation matrix. The activation function used in the convolutional neural network is ReLU, and its calculation formula is: ; This step mainly extracts features from the machine vision pixel matrix captured by the robot dog camera. The size of the convolution kernel and pooling layer can be dynamically adjusted according to the camera parameters, but the dimension of the final output feature vector needs to be the same as the dimension of the vector output by the multilayer perceptron in step (1). (3) Using Long Short-Term Memory (LSTM) networks to analyze the lidar point cloud matrix The data is encoded and converted into an observation matrix. The activation function used in the Long Short-Term Memory (LSTM) network is the Sigmoid function, calculated as follows: ; This step mainly involves extracting features from the laser radar point cloud data matrix obtained by the robot dog's laser radar. Finally, the dimension of the output feature vector needs to be the same as the dimension of the vector output by the multilayer perceptron in step (1). (4) Using the multi-head attention mechanism (MHA), the observation matrix obtained in the first three steps is processed. Encode the data and use a multilayer perceptron to construct a query matrix in a multidimensional space that represents the relationships between various state information. Key matrix and value matrix Thus, the embedding matrix S of the multidimensional space vector is obtained; Among them, the query matrix Key matrix and value matrix The formula for calculating the embedding matrix S of a multidimensional vector space is: ; ; ; ; Among them, the query matrix Observation vectors generated using only their own state information Obtain the key matrix Sum matrix Generated using all observation matrices. The vector dimension is represented by this step, which makes the embedding matrix S of the generated multidimensional space vector contain more information about the robot dog's own state, actions, and decision-making layer.

[0027] Furthermore, according to one embodiment of the present invention, the action decision is judged based on prior information, and the action decision that will cause a collision is masked, and the action decision network is prohibited from selecting the action output.

[0028] Furthermore, according to one embodiment of the present invention, the update formula for the state value network parameters is as follows: ; in, For state-value network parameters, Let the state value network loss function be... , This represents the training reward value obtained by the robot dog after performing the action at time t. These represent the state-value network V in... Embedding matrix based on multidimensional space vectors at every moment Output state value parameters It is the learning rate of the behavioral decision network.

[0029] The update formula for the action policy network parameters is as follows: ;in, For action policy network parameters, Let the loss function be the action decision network. ,in, , ) represents the probability distribution of available actions output by the robot dog's action policy network at the current moment. This represents the probability distribution of available actions output by the robot dog's action policy network at the previous time step. For hyperparameters, Let represent the dominance function, where This is a hyperparameter.

[0030] Furthermore, according to one embodiment of the present invention, the collaborative decision-making task of the robot dog is completed by iteratively executing the above steps. Through continuous decision-making in complex environments, the robot dog swarm can achieve rapid decision-making based on its own observations and the states of other robot dogs. Furthermore, through continuous iterative learning, the robot dogs achieve autonomous learning and strategy optimization in complex environments, thereby realizing collaborative decision-making in unknown task scenarios.

[0031] According to the above-described scheme of the present invention, the proposed robot dog cluster collaborative decision-making method based on the improved near-end policy optimization algorithm, by fusing local observation and global state information and using reinforcement learning-based training, enables the robot dog cluster to have the ability to cope with sudden states in multiple scenarios and make rapid decisions, thereby achieving efficient collaborative decision-making of the cluster and improving task completion efficiency.

[0032] According to the above-described scheme of the present invention, the present invention considers the decision-making efficiency of robot dog clusters under complex conditions, and uses three kinds of neural networks to extract robot dog state information, which can ensure that the state data of robot dogs can be effectively extracted in complex environments.

[0033] This invention introduces an attention mechanism on the basis of the proximal policy optimization algorithm, and achieves rapid collaborative decision-making in complex environments through deep reinforcement learning training, avoiding the problems of excessively long decision-making time and the inability of traditional decision-making methods to solve sudden state decision-making problems.

[0034] The decision-making method provided by this invention does not require the construction of a directed graph or decision tree. It can directly generate decision strategies based on sensor information and the state information of other robot dogs, thereby improving the decision response time and enabling rapid response to task and emergency needs. It avoids the problem of low decision efficiency caused by an excessively long decision chain.

[0035] The algorithm used in this invention incorporates the state information of other robot dogs in the robot dog cluster, enabling the robot dogs to make full decisions based on global information. It achieves efficient cluster collaboration through a centralized training mode, avoiding resource waste and low search efficiency caused by untimely scheduling of robot dogs.

[0036] Furthermore, to achieve the above objectives, the present invention also provides a robot dog collaborative decision-making system based on an improved proximal policy optimization algorithm, comprising: The local observation matrix acquisition module obtains the robot dog's local observation matrix based on the robot dog's own state information, the friendly robot dog's state information, and its own observation information. The embedding matrix calculation module for multidimensional spatial vectors encodes the local observation matrix of the robot dog to obtain the embedding matrix of the multidimensional spatial vectors. The action policy output module inputs the embedding matrix of the multidimensional space vector into the initial robot dog action decision network, outputs the probability distribution of available actions using action masking and normalization strategies, and uses a greedy strategy to select the optimal action for the robot dog to execute from the probability distribution of available actions; after executing the action, it obtains the training reward and the robot dog's own state information, the friendly robot dog's state information, and its own observation information at the next time step. The network optimization module inputs the embedding matrix of multidimensional space vectors into the state value network, outputs state value parameters through the state value network, updates the robot dog action decision network parameters based on the state value parameters and the action decision network loss function to optimize the robot dog action decision network, and updates the state value network parameters based on the state value parameters and the state value network loss function to optimize the state value network. The collaborative decision-making implementation module iteratively executes the local observation matrix acquisition module and the network optimization module to optimize the action strategy and realize the collaborative decision-making task.

[0037] In this implementation, it is necessary to confirm that the communication between the robot dogs is good and there is no communication interference, and that each robot dog can receive the status information of other robot dogs in the cluster; the robot dogs have local observation capabilities, and the robot dogs can only obtain the observation information collected by their own sensors.

[0038] Furthermore, according to one embodiment of the present invention, a local observation matrix of the robot dog is obtained based on the robot dog's own state information, the robot dog's own state information, and its own observation information, including: robot dog Local observation matrix acquired at each time step From its own state information Status information of our own robot dog and its own observation information It consists of three parts, the local observation matrix. It can be represented as: ; in, 'i' represents the robot dog's serial number. Let x and y represent the horizontal and vertical coordinates of the position of robot dog i at time t, respectively. This represents the velocity component of the robot dog's current velocity at time t on the horizontal axis. This represents the velocity component of the robot dog's current velocity at time t on the vertical axis. This represents the azimuth information of the robot dog at time t, where h represents the robot's current orientation. This parameter represents the progress of the task being executed by the robot dog at time t. If the robot dog is idle, this parameter is 0. When the robot dog first receives a task, this parameter is 1. As the task progresses, the parameter gradually decreases proportionally until the task is completed. This indicates the type of task the robot dog performs: 0 for idle tasks, 1 for area search tasks, and 2 for fixed-point search and rescue tasks. ,in This represents the machine vision pixel matrix acquired by the robot dog's camera. This represents the point cloud matrix acquired by the robot dog's lidar.

[0039] Furthermore, according to one embodiment of the present invention, the local observation matrix of the robot dog is encoded to obtain an embedding matrix of a multidimensional spatial vector, including: (1) Using a multilayer perceptron (MLP) to process state information The data is encoded and converted into an observation matrix. The activation function used in the multilayer perceptron is ReLU (Rectified Linear Unit), and its calculation formula is as follows: ; in, }, For its own state information The vector representation after passing through the multilayer perceptron; this step mainly extracts features from the robot dog's state vector. The dimension of feature extraction and the number of multilayer perceptron layers can be dynamically adjusted according to the number of clusters, increasing or decreasing based on the dimension of the observed information. (2) Using a convolutional neural network (CNN) to process the pixel matrix of machine vision The data is encoded and converted into an observation matrix. The activation function used in the convolutional neural network is ReLU, and its calculation formula is: ; This step mainly extracts features from the machine vision pixel matrix captured by the robot dog camera. The size of the convolution kernel and pooling layer can be dynamically adjusted according to the camera parameters, but the dimension of the final output feature vector needs to be the same as the dimension of the vector output by the multilayer perceptron in step (1). (3) Using Long Short-Term Memory (LSTM) networks to analyze the lidar point cloud matrix The data is encoded and converted into an observation matrix. The activation function used in the Long Short-Term Memory (LSTM) network is the Sigmoid function, calculated as follows: ; This step mainly involves extracting features from the laser radar point cloud data matrix obtained by the robot dog's laser radar. Finally, the dimension of the output feature vector needs to be the same as the dimension of the vector output by the multilayer perceptron in step (1). (4) Using the multi-head attention mechanism (MHA), the observation matrix obtained in the first three steps is processed. Encode the data and use a multilayer perceptron to construct a query matrix in a multidimensional space that represents the relationships between various state information. Key matrix and value matrix Thus, the embedding matrix S of the multidimensional space vector is obtained; Among them, the query matrix Key matrix and value matrix The formula for calculating the embedding matrix S of a multidimensional vector space is: ; ; ; ; Among them, the query matrix Observation vectors generated using only their own state information Obtain the key matrix Sum matrix Generated using all observation matrices. The vector dimension is represented by this step, which makes the embedding matrix S of the generated multidimensional space vector contain more information about the robot dog's own state, actions, and decision-making layer.

[0040] Furthermore, according to one embodiment of the present invention, the action decision is judged based on prior information, and the action decision that will cause a collision is masked, and the action decision network is prohibited from selecting the action output.

[0041] Furthermore, according to one embodiment of the present invention, the update formula for the state value network parameters is as follows: ; in, For state-value network parameters, Let the state value network loss function be... , This represents the training reward value obtained by the robot dog after performing the action at time t. These represent the state-value network V in... Embedding matrix based on multidimensional space vectors at every moment Output state value parameters It is the learning rate of the behavioral decision network.

[0042] The update formula for the action policy network parameters is as follows: ; in, For action policy network parameters, Let the loss function be the action decision network. ,in, , ) represents the probability distribution of available actions output by the robot dog's action policy network at the current moment. This represents the probability distribution of available actions output by the robot dog's action policy network at the previous time step. For hyperparameters, Let represent the dominance function, where This is a hyperparameter.

[0043] Furthermore, according to one embodiment of the present invention, the collaborative decision-making task of the robot dog is completed by iteratively executing the above steps. Through continuous decision-making in complex environments, the robot dog swarm can achieve rapid decision-making based on its own observations and the states of other robot dogs. Furthermore, through continuous iterative learning, the robot dogs achieve autonomous learning and strategy optimization in complex environments, thereby realizing collaborative decision-making in unknown task scenarios.

[0044] According to the above-described scheme of the present invention, the proposed robot dog cluster collaborative decision-making method based on the improved near-end policy optimization algorithm, by fusing local observation and global state information and using reinforcement learning-based training, enables the robot dog cluster to have the ability to cope with sudden states in multiple scenarios and make rapid decisions, thereby achieving efficient collaborative decision-making of the cluster and improving task completion efficiency.

[0045] According to the above-described scheme of the present invention, the present invention considers the decision-making efficiency of robot dog clusters under complex conditions, and uses three kinds of neural networks to extract robot dog state information, which can ensure that the state data of robot dogs can be effectively extracted in complex environments.

[0046] This invention introduces an attention mechanism on the basis of the proximal policy optimization algorithm, and achieves rapid collaborative decision-making in complex environments through deep reinforcement learning training, avoiding the problems of excessively long decision-making time and the inability of traditional decision-making methods to solve sudden state decision-making problems.

[0047] The decision-making method provided by this invention does not require the construction of a directed graph or decision tree. It can directly generate decision strategies based on sensor information and the state information of other robot dogs, thereby improving the decision response time and enabling rapid response to task and emergency needs. It avoids the problem of low decision efficiency caused by an excessively long decision chain.

[0048] The algorithm used in this invention incorporates the state information of other robot dogs in the robot dog cluster, enabling the robot dogs to make full decisions based on global information. It achieves efficient cluster collaboration through a centralized training mode, avoiding resource waste and low search efficiency caused by untimely scheduling of robot dogs.

[0049] Furthermore, to achieve the above objectives, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the robot dog cooperative decision-making method based on the improved proximal policy optimization algorithm as described above.

[0050] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the robot dog cooperative decision-making method based on the improved proximal policy optimization algorithm as described above.

[0051] Those skilled in the art will recognize that the modules and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0052] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and equipment can be referred to the corresponding process in the foregoing method implementation, and will not be repeated here.

[0053] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0054] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs.

[0055] In addition, the functional modules in the embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0056] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the sending / receiving methods of various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0057] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this application.

[0058] It should be understood that the sequence number of each step in the invention and its embodiments does not absolutely imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

Claims

1. A collaborative decision-making method for robot dogs based on an improved proximal policy optimization algorithm, characterized in that, include: S1. Obtain the robot dog's local observation matrix based on the robot dog's own state information, the friendly robot dog's state information, and its own observation information; S2. Encode the local observation matrix of the robot dog to obtain the embedding matrix of the multidimensional space vector; S3. Input the embedding matrix of the multidimensional space vector into the initial robot dog action decision network, output the probability distribution of available actions using action masking and normalization strategies, and use a greedy strategy to select the optimal action for the robot dog to execute from the probability distribution of available actions; after executing the action, obtain the training reward and the robot dog's own state information, the friendly robot dog's state information, and its own observation information at the next time step. S4. Input the embedding matrix of the multidimensional space vector into the state value network, output the state value parameters through the state value network, update the robot dog action decision network parameters based on the state value parameters and the action decision network loss function to optimize the robot dog action decision network, and update the state value network parameters based on the state value parameters and the state value network loss function to optimize the state value network. S5. Iteratively execute steps S1-S4 to optimize action strategies and achieve collaborative decision-making tasks.

2. The robot dog cooperative decision-making method based on the improved proximal strategy optimization algorithm according to claim 1, characterized in that, The process of obtaining the robot dog's local observation matrix based on the robot dog's own state information, the friendly robot dog's state information, and its own observation information includes: robot dog Local observation matrix acquired at each time step From its own state information Status information of our own robot dog and its own observation information It consists of three parts, the local observation matrix. Represented as: ； in, 'i' represents the robot dog's serial number. Let x and y represent the horizontal and vertical coordinates of the position of robot dog i at time t, respectively. This represents the velocity component of the robot dog's current velocity at time t on the horizontal axis. This represents the velocity component of the robot dog's current velocity at time t on the vertical axis. This represents the azimuth information of the robot dog at time t, where h represents the robot's current orientation. This parameter represents the progress of the task being executed by the robot dog at time t. If the robot dog is idle, this parameter is 0. When the robot dog first receives a task, this parameter is 1. As the task progresses, the parameter gradually decreases proportionally until the task is completed. This indicates the type of task the robot dog performs: 0 for idle tasks, 1 for area search tasks, and 2 for fixed-point search and rescue tasks. ,in This represents the machine vision pixel matrix acquired by the robot dog's camera. This represents the point cloud matrix acquired by the robot dog's lidar.

3. The robot dog cooperative decision-making method based on the improved proximal strategy optimization algorithm according to claim 2, characterized in that, The encoding process of the robot dog's local observation matrix to obtain the embedding matrix of the multidimensional spatial vector includes: Using multilayer perceptrons to analyze local matrices Chinese robot dog status information set The data is encoded and converted into an observation matrix. The activation function used in the multilayer perceptron is ReLU, and its calculation formula is: ； in, }, For its own state information The vector representation after passing through a multilayer perceptron; Using convolutional neural networks to analyze machine vision pixel matrices Encode it and convert it into an observation matrix. The activation function used in the convolutional neural network is ReLU, and its calculation formula is: ； Using Long Short-Time Memory Networks to Study LiDAR Point Cloud Matrix Encode it and convert it into an observation matrix. The activation function used in the Long Short-Term Memory (LSTM) network is the Sigmoid function, calculated as follows: ； The acquired observation matrix is ​​encoded using a multi-head attention mechanism, and a query matrix representing the relationships between various state information is constructed using a multi-layer perceptron. Key matrix and value matrix Thus, the embedding matrix S of the multidimensional space vector is obtained; Among them, the query matrix Key matrix Value matrix The formula for calculating the embedding matrix S of a multidimensional vector space is: ； ； ； ； Among them, the query matrix Observation vectors generated using only their own state information Obtain the key matrix Sum matrix Generated using all observation matrices. Represents the vector dimension.

4. The robot dog cooperative decision-making method based on the improved proximal strategy optimization algorithm according to any one of claims 1-3, characterized in that, The update formula for the state value network parameters is as follows: ； in, For state-value network parameters, Let the state value network loss function be... , This represents the training reward value obtained by the robot dog after performing the action at time t. These represent the state-value network V in... Embedding matrix based on multidimensional space vectors at every moment Output state value parameters It is the learning rate of the behavioral decision network.

5. The robot dog cooperative decision-making method based on the improved proximal strategy optimization algorithm according to claim 4, characterized in that, The update formula for the action policy network parameters is as follows: ； in, For action policy network parameters, Let the loss function be the action decision network. ,in, , ) represents the probability distribution of available actions output by the robot dog's action policy network at the current moment. This represents the probability distribution of available actions output by the robot dog's action policy network at the previous time step. For hyperparameters, Let represent the dominance function, where This is a hyperparameter.

6. A collaborative decision-making system for robot dogs based on an improved proximal strategy optimization algorithm, characterized in that: include: The local observation matrix acquisition module obtains the robot dog's local observation matrix based on the robot dog's own state information, the friendly robot dog's state information, and its own observation information. The embedding matrix calculation module for multidimensional spatial vectors encodes the local observation matrix of the robot dog to obtain the embedding matrix of the multidimensional spatial vectors. The action policy output module inputs the embedding matrix of the multidimensional space vector into the initial robot dog action decision network, outputs the probability distribution of available actions using action masking and normalization strategies, and uses a greedy strategy to select the optimal action for the robot dog to execute from the probability distribution of available actions; after executing the action, it obtains the training reward and the robot dog's own state information, the friendly robot dog's state information, and its own observation information at the next time step. The network optimization module inputs the embedding matrix of multidimensional space vectors into the state value network, outputs state value parameters through the state value network, updates the robot dog action decision network parameters based on the state value parameters and the action decision network loss function to optimize the robot dog action decision network, and updates the state value network parameters based on the state value parameters and the state value network loss function to optimize the state value network. The collaborative decision-making implementation module iteratively executes the local observation matrix acquisition module and the network optimization module to optimize the action strategy and realize the collaborative decision-making task.

7. An electronic device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the robot dog cooperative decision-making method based on the improved proximal policy optimization algorithm as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the robot dog cooperative decision-making method based on the improved proximal policy optimization algorithm as described in any one of claims 1-5.

Images