Unmanned aerial vehicle cooperative control method, device and system

Abstract

The invention provides an unmanned aerial vehicle cooperative control method, device and system, and the method comprises the steps: obtaining a joint observation value based on the local observation values of a plurality of unmanned aerial vehicles; obtaining an actual reward value obtained by the unmanned aerial vehicle through interaction with the simulation environment; inputting the joint observation value and the actual reward value of the unmanned aerial vehicle into a first deep neural network, and outputting a task execution strategy of the unmanned aerial vehicle; and inputting the task execution strategy, the actual reward value and the local observation value of the unmanned aerial vehicle into a second deep neural network, and outputting an action instruction for controlling the unmanned aerial vehicle. According to the unmanned aerial vehicle cooperative control method, device and system provided by the invention, the reinforcement learning algorithm and the deep neural network are combined, the accuracy and intelligent level of unmanned aerial vehicle cooperative control decision are improved, and the decision quality and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device and system for cooperative control of unmanned aerial vehicles. Background Art

[0002] The current application status of UAVs in the military field is characterized by diversification, efficiency and intelligence.

[0003] First, with the advancement of science and technology and the growth of military needs, various types of military drones are playing an increasingly important role in wars and conflicts. Drones are widely used in the military field, including reconnaissance, attack, intelligence collection, navigation, etc. At the same time, with the development of technology, the application areas of drones are also expanding.

[0004] Secondly, drones have significant technical advantages. They can stay on standby for a long time and take off at any time, achieving full-time combat readiness and making combat readiness more efficient. In addition, drones also have the characteristics of carrying more weapons and equipment. Since they save cockpits, escape equipment, and some spare parts, they can carry more guided bombs, missiles and other weapons. In combat, drones can only carry one-way fuel, and the combat radius can be doubled.

[0005] Furthermore, the intelligence level of drones is also constantly improving. With the rapid development of microelectronics technology, micro-electromechanical system technology, sensor technology, and electrical automation technology, as well as the continuous improvement of related equipment performance and system integration, drones can carry out integrated reconnaissance, attack and assessment tasks, automatically identify multiple strategic targets, attack high-value hostile targets according to priority, and report back the attack results, make battlefield situation estimates, and formulate the next attack plan.

[0006] However, despite the significant progress in the application of UAVs in the military field, there are still some technical weaknesses and problems. For example, the existing UAV collaborative control system cannot meet the actual needs in terms of the intelligence and accuracy of auxiliary decision-making, and commanders still need to rely on a lot of manual judgment. Summary of the invention

[0007] The present invention provides a UAV collaborative control method, device and system, which are used to solve the defects of low intelligence and poor accuracy of UAV collaborative control decision-making in the prior art, and to improve the accuracy and intelligence level of UAV collaborative control decision-making.

[0008] The present invention provides a method for cooperative control of unmanned aerial vehicles, comprising the following steps: obtaining a joint observation value based on local observation values ​​of multiple unmanned aerial vehicles; obtaining an actual reward value obtained by the unmanned aerial vehicle through interaction with a simulation environment; inputting the joint observation value and the actual reward value of the unmanned aerial vehicle into a first deep neural network, and outputting a task execution strategy of the unmanned aerial vehicle; inputting the task execution strategy of the unmanned aerial vehicle, the actual reward value and the local observation value into a second deep neural network, and outputting an action instruction for controlling the unmanned aerial vehicle.

[0009] According to a drone collaborative control method provided by the present invention, before the task execution strategy of the drone, the actual reward value and the local observation value are input into a second deep neural network, the method further includes: calculating a time difference error based on an estimated value of a current state of the drone, an estimated value of a next state and the actual reward value; adjusting the weights of network parameters of the first deep neural network based on the time difference error, and obtaining a revised task execution strategy based on an updated output of the first deep neural network.

[0010] According to a UAV cooperative control method provided by the present invention, the time difference error is expressed as: δ=r+γV(s')-V(s)

[0011] Wherein, δ represents the time difference error, r represents the actual reward value, γ represents the discount factor, V(s) represents the estimated value of the current state of the drone, and V(s') represents the estimated value of the next state of the drone.

[0012] According to a UAV cooperative control method provided by the present invention, the UAV interacts with the simulation environment according to command information and the local observation values ​​of the multiple UAVs to obtain the actual reward value; wherein the command information includes the action instruction.

[0013] According to a UAV cooperative control method provided by the present invention, the command information also includes information on mission objectives and update information on the simulation environment.

[0014] According to a UAV cooperative control method provided by the present invention, before obtaining the joint observation value based on the local observation values ​​of multiple UAVs, the method also includes: using artificial intelligence technology to simulate the behavior of the mission target.

[0015] According to a drone collaborative control method provided by the present invention, before obtaining joint observation values ​​based on local observation values ​​of multiple drones, the method also includes: constructing the simulation environment including multiple simulation factors; wherein the multiple simulation factors include terrain factors, climate factors and electromagnetic interference factors.

[0016] The present invention also provides a UAV collaborative control device, comprising the following modules: a first acquisition module, used to obtain a joint observation value based on local observation values ​​of multiple UAVs; a second acquisition module, used to obtain an actual reward value obtained by the UAV through interaction with a simulation environment; a first output module, used to input the joint observation value and the actual reward value of the UAV into a first deep neural network, and output a task execution strategy of the UAV; a second output module, used to input the task execution strategy of the UAV, the actual reward value and the local observation value into a second deep neural network, and output action instructions for controlling the UAV.

[0017] The present invention also provides a UAV cooperative control system, comprising the UAV cooperative control device and a plurality of UAVs.

[0018] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the drone collaborative control method described above is implemented.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the drone collaborative control method as described in any one of the above is implemented.

[0020] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned drone collaborative control methods.

[0021] The UAV collaborative control method, device and system provided by the present invention obtain joint observation values ​​based on local observation values ​​of multiple UAVs, obtain actual reward values ​​obtained by each UAV through interaction with a simulation environment, input the joint observation values ​​and actual reward values ​​into a first deep neural network, output the task execution strategy of each UAV, input the task execution strategy, actual reward value and local observation values ​​into a second deep neural network, output action instructions for controlling each UAV, combine the reinforcement learning algorithm with the deep neural network, improve the accuracy and intelligence level of UAV collaborative control decision-making, and improve the decision-making quality and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 It is a flow chart of the UAV cooperative control method provided by the present invention.

[0024] Figure 2 It is a schematic diagram of the UAV cooperative control method provided by the present invention.

[0025] Figure 3 It is a structural schematic diagram of the UAV cooperative control device provided by the present invention.

[0026] Figure 4 It is a structural schematic diagram of the UAV cooperative control system provided by the present invention.

[0027] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Figure 1 FIG. 1 is a flow chart of the UAV cooperative control method provided by the present invention. Figure 1 As shown, the method includes: Step S1: Based on the local observation values ​​of multiple drones, a joint observation value is obtained.

[0030] The execution subject of the UAV cooperative control method provided by the present invention is a UAV cooperative control device, which is used to control the execution of cooperative tasks of multiple UAVs, and can also be called a command center. The command center obtains local observation values ​​of multiple UAVs and obtains joint observation values ​​by performing data fusion processing.

[0031] Figure 2 Schematic diagram of the UAV cooperative control method provided by the present invention. Figure 2As shown, in the execution of UAV collaborative tasks, taking the search and rescue task as an example, UAV 1 and UAV 2 perform preliminary searches in their respective assigned task areas. Each UAV is equipped with a highly sensitive sensor that can obtain local observations in the area in real time. UAV 1 flies in area A and collects environmental data of the area through sensors, which are recorded as local observations ob1. Similarly, UAV 2 flies in area B, collects environmental data of area B, and records them as local observations ob2. These observations may include sensor data, location information, flight status, and environmental information. Among them, sensor data may include, for example, camera images, lidar data, infrared sensor data, etc. Location information may include, for example, obstacle locations, target object locations, etc., which may be location coordinates provided by GPS data or other positioning systems. Flight status may include, for example, flight parameters such as speed, altitude, and attitude. Environmental information may include, for example, meteorological conditions, terrain data, etc.

[0032] After completing the preliminary search, UAV 1 and UAV 2 transmit the local observation data they have collected to the command center through the wireless communication system. After receiving the data transmitted by the UAV, the command center conducts joint observation processing. The purpose of joint observation is to integrate data from different UAVs to form a more comprehensive and accurate environmental perception. Through data fusion technology, the command center can eliminate redundant information and correct observation errors to obtain a complete environmental model, providing a reliable data basis for subsequent decision-making. By developing a more intelligent data cleaning algorithm, errors and noise in the data can be automatically identified and corrected, and data from multiple sources can be integrated to improve data quality.

[0033] After the local observation data collected by UAV 1 and UAV 2 are transmitted to the command center through the wireless communication system, the command center conducts joint observation and processing on these data. This process includes data integration and fusion, the purpose of which is to eliminate redundant information, correct observation errors, and ultimately generate a more comprehensive and accurate environmental model. The environmental model covers the observation results of multiple UAVs, combining these results to form a comprehensive and global perspective. This data fusion technology enables the model to reflect the overall situation of the current battlefield and provide a reliable data basis for subsequent decision-making. Therefore, it can be said that the environmental model is the result obtained by integrating and processing multiple local observations. It is a comprehensive perception and representation of the entire environment.

[0034] Step S2: Obtaining an actual reward value obtained by the drone through interaction with the simulation environment.

[0035] The UAV cooperative control method provided by the present invention utilizes a reinforcement learning algorithm combined with a deep neural network, which can autonomously learn and adjust strategies in a complex environment, thereby improving the intelligent level of decision-making. Among them, each UAV obtains an actual reward value by interacting with the simulation environment. The actual reward value is feedback on the result of the action, and usually directly reflects the change in the current state of the simulation environment. For example, if the UAV successfully avoids an obstacle, the simulation environment will feedback a positive reward. The simulation environment is responsible for providing instant feedback information to help the UAV update its strategy. Among them, the simulation environment includes all external information and factors in the simulation, such as: • Terrain and Buildings: Provides obstacle information.

[0036] •Weather conditions: wind speed, temperature, humidity, etc.

[0037] • Electromagnetic interference: Sources of interference that can affect communications and sensors.

[0038] •Enemy and Friendly Forces: Information about other participating units, including their locations and behaviors.

[0039] •Dynamic changes: such as the speed and direction of moving objects.

[0040] Step S3: input the joint observation value and the actual reward value of the drone into a first deep neural network, and output the task execution strategy of the drone.

[0041] The first deep neural network and the second deep neural network are deployed in the UAV cooperative control device, that is, the command center. The command center inputs the joint observation value and actual reward value of each UAV into the first deep neural network and outputs the task execution strategy of each UAV.

[0042] The command center uses the first deep neural network (first DNN) to analyze and optimize the joint observation results to output the optimized mission execution strategy. The input of the first deep neural network includes joint observation values ​​and reward values. After being processed by multiple layers of neural networks, the first deep neural network can identify complex environmental patterns and potential mission requirements, thereby generating the optimal mission execution strategy. This mission execution strategy will guide the drone on how to act in the next mission to maximize mission efficiency and success rate.

[0043] Step S4: input the task execution strategy of the drone, the actual reward value and the local observation value into a second deep neural network, and output an action instruction for controlling the drone.

[0044] The command center inputs the task execution strategy output by the first deep neural network, the actual reward value obtained by the drone through interaction with the simulation environment, and the local observation values ​​of multiple drones into the second deep neural network, and outputs the action instructions for controlling each drone. The drone cooperative control method provided by the present invention provides more intelligent and accurate decision-making suggestions by utilizing deep neural networks and reinforcement learning algorithms; it can monitor the battlefield situation in real time, provide accurate predictions and analysis, and help commanders make decisions quickly; it supports multi-drone cooperative operations, can better allocate tasks and resources, and improve combat efficiency and effectiveness.

[0045] The present invention can continuously optimize itself in a simulated environment through a reinforcement learning algorithm, accumulate a large amount of experience, and form an efficient strategy. This self-learning ability enables the platform to adapt to different tactical scenarios, adjust strategies in real time, and provide optimal decision support. The simulation algorithm and logic can be optimized to improve the accuracy and authenticity of the simulation. Parallel computing, distributed simulation and other technologies can be used to improve simulation efficiency while maintaining high accuracy. By optimizing the simulation logic structure and data processing flow, the simulation efficiency can be improved while ensuring high-precision simulation results, and the needs of large-scale scenarios can be met.

[0046] The UAV collaborative control method provided by the present invention obtains joint observation values ​​based on local observation values ​​of multiple UAVs, obtains actual reward values ​​obtained by each UAV through interaction with a simulation environment, inputs the joint observation values ​​and actual reward values ​​into a first deep neural network, outputs the task execution strategy of each UAV, inputs the task execution strategy, actual reward value and local observation values ​​into a second deep neural network, outputs action instructions for controlling each UAV, combines reinforcement learning algorithms with deep neural networks, improves the accuracy and intelligence level of UAV collaborative control decisions, and improves decision-making quality and efficiency.

[0047] According to a drone collaborative control method provided by the present invention, before the task execution strategy of the drone, the actual reward value and the local observation value are input into a second deep neural network, the method further includes: calculating a time difference error based on an estimated value of a current state of the drone, an estimated value of a next state and the actual reward value; adjusting the weights of network parameters of the first deep neural network based on the time difference error, and obtaining a revised task execution strategy based on an updated output of the first deep neural network.

[0048] Introduce a real-time data feedback mechanism to monitor errors and deviations during the simulation process, and adjust parameters and algorithms in a timely manner to ensure the accuracy of the simulation results. Machine learning technology can be combined to achieve adaptive adjustment and optimization. By establishing a real-time data monitoring and feedback system, data and error information generated during the simulation process are collected in real time. This solution combines machine learning technology to adaptively adjust simulation parameters and algorithms, optimize simulation models, and improve the accuracy and stability of simulation results. The real-time adjustment mechanism ensures that the simulation system can dynamically respond to environmental changes and the uncertainty of input data, providing more accurate simulation results.

[0049] To further improve the effectiveness of the strategy, the command center uses time difference error (TD-error) for strategy correction. TD-error is an error measurement method in reinforcement learning that adjusts the strategy by comparing the difference between expected rewards and actual rewards. The command center uses TD-error calculation to correct and optimize the strategy generated by the first DNN to ensure that the actual actions of the drone can achieve the expected results. In this way, the system can continuously improve the strategy, making the drone more intelligent and efficient in complex environments.

[0050] The command center obtains the estimated value of the current state and the next state of the drone based on the local observation value of the drone and the task execution result, calculates the time difference error based on the estimated value of the current state and the next state of the drone and the actual reward value, adjusts the weights of the network parameters of the first deep neural network based on the time difference error, and obtains the revised task execution strategy based on the output of the updated first deep neural network. The size and direction of TD-error determine the amplitude and direction of the strategy update. The system uses TD-error to update the weight of the first DNN and correct the strategy through the back propagation algorithm. In this way, the policy network can gradually tend to a better strategy, making the drone perform more intelligently and efficiently in complex environments. Among them, the estimated value of the current state of the drone and the estimated value of the next state are used to represent the expected reward.

[0051] like Figure 2 As shown in the figure, according to the optimization strategy and corrected action instructions output by the command center, UAV 1 and UAV 2 perform specific search and rescue tasks. During the execution of the task, the UAV continuously feeds back new local observation data and task execution results to the command center. This feedback mechanism ensures that the command center can grasp the progress of the task and environmental changes in real time, so as to make dynamic adjustments and optimizations. Among them, the task execution results include the changes in the environmental state of the UAV after the action is performed, the target discovery situation, the task completion situation, etc. These data can reflect the actual effect and execution of the current strategy.

[0052] The command center optimizes and adjusts the strategy based on the execution results and real-time feedback. Reinforcement learning methods such as TD-error are used to correct the parameters of the policy network and improve the decision logic to ensure that future actions are more effective. This process is a key link in reinforcement learning. Through continuous feedback and adjustment, the system can improve itself and make the performance of the drone more intelligent and efficient. The execution results help the command center optimize the subsequent task execution. Whether it is optimizing the search path in the search and rescue mission or adjusting the action strategy in other tasks, this feedback information can provide strong support for the next decision.

[0053] The command center continuously updates the first DNN through TD-error correction based on the latest feedback data to dynamically adjust strategies and actions. This cyclic optimization process enables the system to continuously learn and adapt to environmental changes to ensure the efficient completion of rescue missions. Through continuous iteration and optimization, the command center can gradually improve the overall performance of the system, making the drone more intelligent and reliable when performing tasks.

[0054] The UAV cooperative control method provided by the present invention calculates the time difference error according to the estimated value of the current state of the UAV, the estimated value of the next state and the actual reward value, adjusts the weight of the network parameter of the first deep neural network according to the time difference error, and obtains the revised task execution strategy according to the output of the updated first deep neural network. The task execution strategy is updated, which is conducive to obtaining more accurate action instructions, and further improves the accuracy and intelligence level of the UAV cooperative control decision-making.

[0055] According to a UAV cooperative control method provided by the present invention, the time difference error is expressed as: δ=r+γV(s')-V(s)

[0056] Where δ represents the time difference error, r represents the actual reward value, γ represents the discount factor, V(s) represents the estimated value of the current state of the drone, and V(s') represents the estimated value of the next state of the drone. The discount factor γ is a value between 0 and 1, which is used to measure the importance of future rewards.

[0057] The UAV cooperative control method provided by the present invention further improves the accuracy and intelligence level of UAV cooperative control decision-making by providing a calculation formula for the time difference error.

[0058] According to a UAV cooperative control method provided by the present invention, the UAV interacts with the simulation environment according to command information and the local observation values ​​of the multiple UAVs to obtain the actual reward value; wherein the command information includes the action instruction.

[0059] The drones interact with the simulation environment based on the command information and the local observation values ​​of multiple drones to obtain the actual reward value. The command information is the information given by the command center, including the action instructions output by the second deep neural network. The drones will share local observation values, such as Figure 2 Chinese T 2→1 represents the local observation value transmitted from UAV 2 to UAV 1, ob T 1→2 It indicates the local observation values ​​transmitted by UAV 1 to UAV 2. These local observation values ​​may include the sensor information of the UAV, environmental perception data, etc. The UAV can perform actions based on its own local observation values, the local observation values ​​obtained by other UAVs, and the command information given by the command center.

[0060] By designing and implementing more accurate sensor models, including radar and infrared sensors, the process of drones acquiring environmental information and target data can be accurately simulated. The solution combines actual sensor data with advanced algorithms to perform high-precision modeling of sensor response characteristics, ensuring the authenticity and reliability of data during the simulation process.

[0061] The UAV collaborative control method provided by the present invention obtains actual reward values ​​by interacting with a simulation environment based on action instructions and local observation values ​​of multiple UAVs, thereby improving the accuracy of the interaction between the UAV and the environment and the accuracy of the actual reward values, thereby further improving the accuracy and intelligence level of UAV collaborative control decisions.

[0062] According to a UAV cooperative control method provided by the present invention, the command information also includes information on mission objectives and update information on the simulation environment.

[0063] In addition to the action instructions output by the second deep neural network, the command information output by the command center to the drone may also include information about the mission target and updated information about the simulation environment. The mission target information may include target position, speed, type, etc. The updated information about the simulation environment may include new obstacles or enemy threats.

[0064] The UAV collaborative control method provided by the present invention improves the accuracy of the interaction between the UAV and the environment and the accuracy of the actual reward value by including the information of the mission target and the update information of the simulation environment in the command information, thereby further improving the accuracy and intelligence level of the UAV collaborative control decision.

[0065] According to a UAV cooperative control method provided by the present invention, before obtaining the joint observation value based on the local observation values ​​of multiple UAVs, the method also includes: using artificial intelligence technology to simulate the behavior of the mission target.

[0066] Artificial intelligence technology is introduced to simulate the intelligent behavior of the mission target, including identification, avoidance, counterattack, etc., to more realistically simulate the interaction process between the drone and the mission target. Methods such as reinforcement learning algorithms can be used to make the mission target have more intelligent behavior, truly reflect the interaction between the drone and the mission target, and enhance the authenticity of the simulation scene.

[0067] The UAV cooperative control method provided by the present invention simulates the behavior of the mission target by adopting artificial intelligence technology, truly reflects the interaction between the UAV and the mission target, and enhances the authenticity of the simulation scene.

[0068] According to a drone collaborative control method provided by the present invention, before obtaining joint observation values ​​based on local observation values ​​of multiple drones, the method also includes: constructing the simulation environment including multiple simulation factors; wherein the multiple simulation factors include terrain factors, climate factors and electromagnetic interference factors.

[0069] Build a more detailed and realistic environmental model, taking into account factors such as terrain, climate, and electromagnetic interference, to better simulate the behavior and performance of drones in complex environments and improve the authenticity of environmental models. Through simulation tools and algorithms, this solution accurately incorporates dynamic changes in complex environments (such as wind speed, temperature changes, electromagnetic interference, etc.) into the simulation process to improve the fidelity and accuracy of the simulation. By introducing a high-precision physics engine, physical phenomena such as ballistics, explosion shock waves, and vehicle dynamics can be accurately simulated to improve the authenticity of the simulation scene.

[0070] The UAV cooperative control method provided by the present invention further enhances the authenticity of the simulation scene by constructing a simulation environment including multiple simulation factors such as terrain factors, climate factors and electromagnetic interference factors.

[0071] The UAV cooperative control device provided by the present invention is described below. The UAV cooperative control device described below and the UAV cooperative control method described above can be referenced to each other.

[0072] Figure 3 Schematic diagram of the structure of the UAV cooperative control device provided by the present invention. Figure 3As shown, the UAV collaborative control device 1 includes a first acquisition module 10, a second acquisition module 20, a first output module 30 and a second output module 40, wherein: the first acquisition module 10 is used to: obtain a joint observation value based on local observation values ​​of multiple UAVs; the second acquisition module 20 is used to: obtain an actual reward value obtained by the UAV through interaction with the simulation environment; the first output module 30 is used to: input the joint observation value and the actual reward value of the UAV into a first deep neural network, and output a task execution strategy of the UAV; the second output module 40 is used to: input the task execution strategy of the UAV, the actual reward value and the local observation value into a second deep neural network, and output an action instruction for controlling the UAV.

[0073] The UAV collaborative control device provided by the present invention obtains a joint observation value based on local observation values ​​of multiple UAVs, obtains an actual reward value obtained by each UAV through interaction with a simulation environment, inputs the joint observation value and the actual reward value into a first deep neural network, outputs a task execution strategy for each UAV, inputs the task execution strategy, the actual reward value and the local observation value into a second deep neural network, outputs action instructions for controlling each UAV, combines a reinforcement learning algorithm with a deep neural network, improves the accuracy and intelligence level of UAV collaborative control decision-making, and improves decision-making quality and efficiency.

[0074] Figure 4 Schematic diagram of the structure of the UAV cooperative control system provided by the present invention. Figure 4 As shown, the UAV cooperative control system includes a UAV cooperative control device 1 and multiple UAVs 2. The UAV cooperative control device 1 controls multiple UAVs 2 to cooperatively perform tasks by outputting action instructions. The UAVs can also communicate with each other and share data. Figure 4 Not shown.

[0075] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the UAV cooperative control method, which includes: obtaining a joint observation value based on the local observation values ​​of multiple UAVs; obtaining the actual reward value obtained by the UAV through interaction with the simulation environment; inputting the joint observation value and the actual reward value of the UAV into the first deep neural network, and outputting the task execution strategy of the UAV; inputting the task execution strategy of the UAV, the actual reward value and the local observation value into the second deep neural network, and outputting the action instruction for controlling the UAV.

[0076] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0077] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the drone collaborative control method provided by the above-mentioned methods, which includes: obtaining a joint observation value based on local observation values ​​of multiple drones; obtaining an actual reward value obtained by the drone through interaction with a simulation environment; inputting the joint observation value and the actual reward value of the drone into a first deep neural network, and outputting a task execution strategy for the drone; inputting the task execution strategy of the drone, the actual reward value and the local observation value into a second deep neural network, and outputting action instructions for controlling the drone.

[0078] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the drone collaborative control method provided by the above-mentioned methods, the method comprising: obtaining a joint observation value based on local observation values ​​of multiple drones; obtaining an actual reward value obtained by the drone through interaction with a simulation environment; inputting the joint observation value and the actual reward value of the drone into a first deep neural network, and outputting a task execution strategy of the drone; inputting the task execution strategy of the drone, the actual reward value and the local observation value into a second deep neural network, and outputting action instructions for controlling the drone.

[0079] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A UAV cooperative control method, It is characterized in that include: Based on the local observations of multiple UAVs, joint observations are obtained; Obtaining an actual reward value obtained by the drone through interaction with the simulation environment; Inputting the joint observation value and the actual reward value of the drone into a first deep neural network, and outputting a task execution strategy of the drone; The task execution strategy of the drone, the actual reward value and the local observation value are input into a second deep neural network, and an action instruction for controlling the drone is output.

2. The UAV cooperative control method according to claim 1, It is characterized in that Before inputting the task execution strategy of the drone, the actual reward value, and the local observation value into the second deep neural network, the method further includes: Calculating a time difference error according to an estimated value of the current state of the drone, an estimated value of the next state, and the actual reward value; The weights of the network parameters of the first deep neural network are adjusted according to the time difference error, and the revised task execution strategy is obtained according to the updated output of the first deep neural network.

3. The UAV cooperative control method according to claim 2, It is characterized in that The time difference error is expressed as: δ=r+γV(s')-V(s) Wherein, δ represents the time difference error, r represents the actual reward value, γ represents the discount factor, V(s) represents the estimated value of the current state of the drone, and V(s') represents the estimated value of the next state of the drone.

4. The UAV cooperative control method according to claim 1, It is characterized in that The drone obtains the actual reward value by interacting with the simulation environment according to the command information and the local observation values ​​of the multiple drones; wherein the command information includes the action instruction.

5. The UAV cooperative control method according to claim 4, It is characterized in that The command information also includes information on mission objectives and update information on the simulation environment.

6. The UAV cooperative control method according to claim 5, It is characterized in that Before obtaining the joint observation value based on the local observation values ​​of the multiple drones, the method further includes: Artificial intelligence techniques are used to simulate the behavior of the mission objectives.

7. The UAV cooperative control method according to claim 1, It is characterized in that Before obtaining the joint observation value based on the local observation values ​​of the multiple drones, the method further includes: Constructing the simulation environment including multiple simulation factors; wherein the multiple simulation factors include terrain factors, climate factors and electromagnetic interference factors.

8. A UAV cooperative control device, It is characterized in that include: The first acquisition module is used to: acquire a joint observation value based on the local observation values ​​of multiple UAVs; The second acquisition module is used to: acquire an actual reward value obtained by the drone through interaction with the simulation environment; A first output module is used to: input the joint observation value and the actual reward value of the drone into a first deep neural network, and output a task execution strategy of the drone; The second output module is used to: input the task execution strategy of the drone, the actual reward value and the local observation value into a second deep neural network, and output an action instruction for controlling the drone.

9. A UAV cooperative control system, It is characterized in that It includes the UAV cooperative control device as described in claim 8 and multiple UAVs.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the drone cooperative control method as described in any one of claims 1 to 7 is implemented.

Images