Multi-patrolling device software collaborative upgrading method, device, equipment and medium

Abstract

The invention discloses a multi-patrolling device software collaborative upgrading method, device and equipment and a medium, and relates to the technical field of unmanned aerial vehicle intelligence. The method comprises the following steps: accessing a plurality of patrolling devices carrying to-be-upgraded software into the same network environment, and performing centralized management on the patrolling devices through a server; acquiring state information of each patrolling device, and inputting the state information into a pre-trained scheduling strategy model to output a scheduling strategy for a software upgrading sequence of the plurality of patrolling devices; the state information at least comprises electric quantity information and network quality information; the scheduling strategy model is constructed based on a Markov decision process and is obtained by training through a deep reinforcement learning algorithm; and on the basis of the scheduling strategy, software upgrading operation is executed on the patrol flight devices in sequence, and corresponding software upgrading states are fed back to the server. By means of the technical scheme, efficient, rapid and stable upgrading of the patrolling devices is achieved, and meanwhile the consistency and standardization of the patrolling devices after software upgrading are kept.

Description

Technical Field

[0001] This invention relates to the field of intelligent unmanned aerial vehicle (UAV) technology, and in particular to a method, apparatus, equipment, and medium for collaborative software upgrades of multiple patrol aircraft. Background Technology

[0002] With the rapid development of aviation technology and the continuous expansion of various application scenarios, the demand for drones is increasing daily. Among various types of drones, loitering drones have received increasing attention due to their unique functions and broad application prospects. However, due to the large number of loitering drones, they compete for limited network resources during software upgrades, greatly increasing the complexity of their use and leading to low mission efficiency. Multi-loitering drone software collaborative upgrades can globally schedule limited resources and optimize the queue of loitering drones requiring software upgrades through cooperative game theory, effectively solving the bottleneck of multi-loitering drone use. Currently, research on multi-loitering drone software collaborative upgrades still faces the following three pressing issues: (1) Low upgrade efficiency: Traditional upgrade methods are mostly carried out one by one. When faced with a large number of similar loitering drones, this method is not only time-consuming, but also prone to upgrade errors due to operator fatigue or negligence, which seriously affects the overall upgrade efficiency. (2) Poor network connection stability: In the process of collaborative software upgrade of multiple patrol aircraft, the stability of network connection is one of the key factors. Existing upgrade solutions often do not fully consider the impact of network fluctuations and interference on the upgrade process, which can easily lead to problems such as connection interruption and data loss during the upgrade process, thereby affecting the success rate and consistency of the upgrade; (3) Lack of intelligent scheduling mechanism: Most current upgrade operations rely on manual scheduling and management, lacking an intelligent scheduling mechanism. This not only increases labor costs, but also makes it easy for human factors to lead to unreasonable scheduling, failing to give full play to the advantages of multi-roaming aircraft software collaborative upgrades, and making it difficult to achieve standardization of the upgrade process. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for collaborative software upgrades of multiple loitering rovers, enabling efficient, rapid, and stable upgrades of multiple loitering rovers while maintaining consistency and standardization of the roitering rovers after the software upgrade. The specific solution is as follows: Firstly, this application discloses a method for collaborative software upgrades of multiple loitering vehicles, including: Multiple patrol drones equipped with the software to be upgraded are connected to the same network environment, and each patrol drone is centrally managed through a server. The status information of each of the loitering drones is acquired and input into a pre-trained scheduling strategy model to output a scheduling strategy for the software upgrade order of the multiple loitering drones; wherein, the status information includes at least power information and network quality information; the scheduling strategy model is constructed based on a Markov decision process and trained by a deep reinforcement learning algorithm; Based on the scheduling strategy, software upgrade operations are performed on each of the loitering drones in sequence, and the corresponding software upgrade status is fed back to the server.

[0004] Optionally, the step of connecting multiple patrol drones equipped with the software to be upgraded to the same network environment and centrally managing each patrol drone through a server includes: Multiple patrol drones equipped with the software to be upgraded will be connected to the same network environment via Ethernet ports. Each of the loitering drones is assigned a unique communication address using a dynamic host configuration protocol, and a network connection between the server and the loitering drone is established based on the communication address and a preset connection strategy. Determine whether the current software version of the loitering drone matches the record in the background; If there is a discrepancy, the difference is recorded in the background and the network connection of the patrol drone is disconnected to terminate the current software upgrade of the patrol drone. If they match, the target upgrade program file is transmitted in parallel from the server to the roaming vehicle based on multi-threaded distribution technology, and real-time data verification is performed during the transmission process to automatically retransmit any data with transmission errors.

[0005] Optionally, the training process of the scheduling strategy model includes: The server is treated as an intelligent agent, and a state space, action space, and reward function are defined to construct a Markov decision process model. The state space includes the state information and current state identifier of the patrol vehicle. The current state identifier represents the stage of the software to be upgraded during the scheduling of the software upgrade order, including not yet started, being adjusted, and adjusted completely. The action space includes scheduling rules for adjusting the software upgrade order. The reward function includes a reward for remaining battery power after the software upgrade is completed, and a penalty for battery depletion before the software upgrade is completed. The Markov decision process model is trained using a deep reinforcement learning algorithm to obtain the scheduling strategy model.

[0006] Optionally, the scheduling rules in the action space include: The first action is to maintain the current software upgrade sequence for the loitering drone; The second action is to advance the software upgrade order of the current patrol vehicle by one position. If the previous position of the current patrol vehicle is an idle time slot, it is occupied directly; if the previous position is another patrol vehicle, the order of the current patrol vehicle and the other patrol vehicle are swapped. The third action is to postpone the software upgrade order of the current patrol vehicle by one position. If the next slot of the current patrol vehicle is an idle time slot, it is occupied directly; if the next slot is occupied by another patrol vehicle, the order of the current patrol vehicle and the other patrol vehicle are swapped. Specifically, when the current loitering drone is at the head of the upgrade queue, the second action is prohibited; when the current loitering drone is at the end of the upgrade queue, the third action is prohibited.

[0007] Optionally, training the Markov decision process model using a deep reinforcement learning algorithm to obtain the scheduling policy model includes: Collect upgrade queue data of multiple sets of different numbers of loitering drones, and divide the upgrade queue data into training set, validation set and test set according to a preset ratio; Using a deep deterministic policy gradient algorithm, the software upgrade order of each upgrade queue in the training set is adjusted to train the Markov decision process model for multiple rounds, and the preset initial learning rate is gradually reduced during the training process; wherein, the deep deterministic policy gradient algorithm uses an Actor-Critic dual network structure, the Actor network is used to generate the scheduling rules, and the Critic network is used to evaluate the value of the scheduling rules generated by the Actor network. The trained model is validated using the validation set. When the average reward value of the validation set reaches a preset threshold, the model is determined to be successfully trained, and the current artificial neural network parameters are saved. Based on the parameters of the artificial neural network, the model is tested using the test set, and the scheduling strategy model is determined after the test is completed.

[0008] Optionally, the step of feeding back the corresponding software upgrade status to the server includes: The software upgrade status of each loitering drone is monitored in real time by the flight control computer, and the software upgrade status is fed back to the server for visualization. When a software upgrade anomaly is detected, the cause of the failure is automatically recorded, and the corresponding exception handling mechanism is triggered.

[0009] Optionally, after performing software upgrade operations on each of the loitering rovers sequentially based on the scheduling strategy, the method further includes: A consistency check is performed on each patrol aircraft that has completed the software upgrade operation to check whether there are any inconsistencies in the software version, functional modules and parameter settings of the upgraded software in the patrol aircraft. For the first patrol aircraft that passes the consistency check, the upgrade completion time of the software to be upgraded and the new software version information after the upgrade are recorded, and the first patrol aircraft is allowed to be powered down first to enter the standby state. For the second loitering rovers that fail the consistency check, a preset repair program is automatically initiated to re-upgrade the software to be upgraded until the consistency check of the second loitering rovers passes.

[0010] Secondly, this application discloses a multi-loitering vehicle software collaborative upgrade device, comprising: The network deployment module is used to connect multiple patrol drones equipped with the software to be upgraded to the same network environment, and to centrally manage each patrol drone through a server; The scheduling strategy determination module is used to acquire the status information of each of the loitering drones and input the status information into a pre-trained scheduling strategy model to output a scheduling strategy for the software upgrade order of the multiple loitering drones; wherein, the status information includes at least power information and network quality information; the scheduling strategy model is constructed based on a Markov decision process and trained by a deep reinforcement learning algorithm; The software collaborative upgrade module is used to perform software upgrade operations on each of the aforementioned patrol vehicles sequentially based on the scheduling strategy, and to feed back the corresponding software upgrade status to the server.

[0011] Thirdly, this application discloses an electronic device, which includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the multi-roaming aircraft software collaborative upgrade method as described above.

[0012] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned multi-roaming aircraft software collaborative upgrade method.

[0013] The beneficial effects of this application are as follows: This invention models the multi-loitering vehicle upgrade sequence optimization problem as a Markov decision process, and trains an agent using reinforcement learning algorithms to dynamically adjust the upgrade sequence based on the real-time status of each loitering vehicle, such as battery level and network quality. This mechanism transforms the scheduling problem into an intelligent decision-making problem, breaking through the limitations of traditional manual scheduling or fixed sequences. Furthermore, based on the Markov decision process modeling method and the algorithmic characteristics of deep reinforcement learning, the trained scheduling strategy model possesses excellent generalization ability, adapting to collaborative upgrade scenarios with different numbers of loitering vehicles without requiring redesign of scheduling rules for each scale, significantly improving the engineering applicability and scalability of the method. In addition, by monitoring the software upgrade status in real time and providing timely feedback, the system can automatically trigger corresponding countermeasures, such as retransmission or re-upgrade, when there are network fluctuations or upgrade failures, avoiding upgrade interruptions due to battery depletion or poor network conditions, and greatly improving upgrade stability and success rate.

[0014] Furthermore, the multi-loiter software collaborative upgrade device, equipment, and storage medium provided in this application correspond to the aforementioned multi-loiter software collaborative upgrade method and have the same effect. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 This application discloses a flowchart of a multi-loitering aircraft software collaborative upgrade method. Figure 2 This is an example diagram of the state transition of a Markov decision process disclosed in this application; Figure 3 This is a schematic diagram of a multi-loiter software collaborative upgrade device disclosed in this application; Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Currently, research on collaborative software upgrades for multiple loitering rovers faces three main problems: First, the upgrade efficiency is low. Traditional one-by-one upgrade methods are time-consuming when dealing with a large number of loitering rovers and are prone to errors due to human fatigue and negligence. Second, network connection stability is poor. Existing solutions do not fully consider fluctuation interference, leading to upgrade interruptions and data loss, which affects the success rate and consistency. Third, there is a lack of intelligent scheduling mechanisms, relying on manual management, which increases costs and results in unreasonable scheduling, making it difficult to leverage collaborative advantages and achieve standardized upgrades.

[0019] To this end, this application provides a multi-loitering vehicle software collaborative upgrade scheme, which can achieve efficient, rapid and stable upgrades of multiple loitering vehicles, while maintaining the consistency and standardization of the loitering vehicles after the software upgrade.

[0020] This invention discloses a method for collaborative software upgrades of multiple loitering rovers, see [link to relevant documentation]. Figure 1 As shown, the method includes: Step S11: Connect multiple patrol drones equipped with the software to be upgraded to the same network environment, and centrally manage each patrol drone through a server.

[0021] This step aims to build a unified network environment for subsequent collaborative software upgrades of multiple loitering rovers, ensuring a stable and reliable communication link between the server and each loitering rovers, and laying the foundation for status information collection, upgrade file distribution, and intelligent scheduling command issuance.

[0022] Before upgrading the software, all the patrol aircraft to be upgraded were placed in the designated upgrade area, and each patrol aircraft was ensured to have sufficient power to meet the energy consumption requirements of the entire upgrade process. The software to be upgraded in each patrol aircraft is flight management software with the ability to receive and send commands, enabling remote control and management of the patrol aircraft.

[0023] All patrol drones are connected to the same network environment via Ethernet ports. The server, as the core node in the network, is responsible for centralized management and upgrade operations throughout the entire upgrade process. The flight management computer, acting as a bridge between the server and the patrol drones, is a core device in the network. After the flight management computer starts up, it assigns a unique communication address (IP address) to each patrol drone via Dynamic Host Configuration Protocol (DHCP), ensuring that each patrol drone can be accurately identified and located within the network.

[0024] After IP address allocation is completed, the server receives IP requests from the patrol drones and establishes a stable network connection according to the preset connection strategy, ensuring that each patrol drone can successfully connect to the server and providing reliable communication support for subsequent software upgrade operations.

[0025] It's important to note that this step is more than just establishing a simple network connection. Its deeper technical significance lies in creating a data acquisition and command issuance channel for the subsequent reinforcement learning scheduling based on Markov decision processes, through the construction of a unified network access and stable communication link. Specifically: only with a stable communication connection can the server obtain the status information of each patrol aircraft in real time; the scheduling strategy generated after intelligent scheduling needs to be issued to each patrol aircraft in the form of commands, guiding them to perform software upgrades within a specified time window. A stable and reliable communication connection is a prerequisite for ensuring that scheduling commands are accurately delivered and executed; during the actual upgrade process, if network fluctuations or unexpected situations occur, the server needs to be able to detect and dynamically adjust the scheduling strategy in real time (such as reallocating time slots or adjusting the upgrade order), all of which depends on the continuous, two-way communication channel established in this step.

[0026] Furthermore, after completing the network environment setup and establishing communication connections, the core step is distributing upgrade program files to eligible loitering rovers. The purpose is to ensure that only loitering rovers with software versions consistent with the backend records can proceed with the upgrade process, and to lay the foundation for parallel upgrades of multiple loitering rovers through an efficient and reliable file transfer mechanism. Specifically, this includes the following steps: Step 1: Determine whether the current software version of the patrol drone matches the record in the background; Step 2: If there is a discrepancy, record the difference in the background and disconnect the network connection of the patrol drone to terminate the current software upgrade of the patrol drone; Step 3: If they match, the target upgrade program file is transmitted in parallel from the server to the roaming aircraft based on multi-threaded distribution technology, and real-time data verification is performed during the transmission process to automatically retransmit any data with transmission errors.

[0027] After establishing a stable network connection between the server and each patrol aircraft, the system first checks the current software version of each patrol aircraft using the unique identification code of each flight control computer. Specifically, the server maintains software version files for all patrol aircraft in the background. When a patrol aircraft connects to the network, the server compares its current software version with the version information recorded in the background. If the comparison results match, it indicates that the current software version of the patrol aircraft meets the pre-upgrade conditions and can be included in this round of collaborative upgrades. If the comparison results do not match, the server records the version difference information in the background and prompts the operator to intervene manually through an alarm. At the same time, in order not to affect the collaborative software upgrade of other patrol aircraft whose software versions have passed the check, the network connection of patrol aircraft with inconsistent software versions will be disconnected, and they will not be upgraded in this round. This pre-processing mechanism effectively avoids the situation where the entire upgrade process is blocked due to abnormal versions of some patrol aircraft, ensuring the continuity and efficiency of collaborative upgrades.

[0028] For patrol drones that pass the software version check, the server distributes the pre-prepared upgrade program file to each drone according to a specific strategy. To improve software upgrade efficiency and data transmission reliability, this invention employs multi-threaded distribution technology and a data verification mechanism. Multi-threaded distribution technology can send upgrade files to multiple patrol drones simultaneously, fully utilizing network bandwidth and significantly shortening upgrade file transmission time. Simultaneously, during file transmission, the data verification mechanism performs real-time verification of the transmitted data to ensure its integrity and accuracy. If an error is detected during data transmission, the system automatically retransmits the data until it is transmitted completely and error-free to each patrol drone.

[0029] As can be seen, the version checking mechanism excludes patrol aircraft with abnormal software versions from this round of upgrades, avoiding interruptions to the overall upgrade process due to individual device issues; the multi-threaded distribution technology fully utilizes network bandwidth to achieve parallel transmission of upgrade files to multiple patrol aircraft, significantly shortening file distribution time; and the data verification mechanism and automatic retransmission function effectively address data errors during network transmission, ensuring that the upgrade files received by each patrol aircraft are completely consistent with the source files.

[0030] Step S12: Obtain the status information of each of the loitering drones and input the status information into a pre-trained scheduling strategy model to output a scheduling strategy for the software upgrade order of the multiple loitering drones; wherein, the status information includes at least power information and network quality information; the scheduling strategy model is constructed based on a Markov decision process and trained by a deep reinforcement learning algorithm.

[0031] In this embodiment of the application, the server obtains the real-time status information of each patrol aircraft to be upgraded through communication with each patrol aircraft. The status information includes at least (1) power information: an integer from 0 to 100, corresponding to the power of the patrol aircraft from 0% to 100%, for example, 95 means 95% of the power remains; (2) network quality information: an integer from 0 to 100, indicating the quality of the network. The larger the number, the better the network quality. The better the network quality, the shorter the software upgrade time required.

[0032] To achieve intelligent optimization of the upgrade sequence of multiple loitering aircraft, this embodiment transforms the multi-loitering aircraft software upgrade scheduling problem into a trainable Markov decision process, and uses a deep reinforcement learning algorithm (Deep Deterministic Policy Gradient, DDPG) to intelligently schedule the software upgrade sequence, so as to reasonably arrange the software upgrade sequence and time of multiple loitering aircraft.

[0033] As can be seen, the trained scheduling strategy model is constructed based on a Markov decision process and trained using a deep reinforcement learning algorithm. It takes the state information of the patrol drones, including power and network quality information, as input, and dynamically optimizes the upgrade order of the patrol drones under limited network resources, outputting the corresponding scheduling strategy. This avoids the inefficiency caused by multiple patrol drones simultaneously competing for network resources, while ensuring that the patrol drones have sufficient remaining power after upgrades, preventing power depletion during the upgrade process. It achieves intelligent adaptation to complex scenarios, significantly improving the efficiency and reliability of multi-pattern patrol drone collaborative upgrades.

[0034] Step S13: Based on the scheduling strategy, perform software upgrade operations on each of the loitering drones in sequence, and feed back the corresponding software upgrade status to the server.

[0035] In this embodiment, the scheduling strategy output by the scheduling strategy model can quickly identify upgrade queues with a large remaining battery level and no loitering drones running out of battery before the software upgrade is completed. The system then executes the software upgrade operation in an orderly manner based on these queues. In this way, the theoretical scheduling result is transformed into a practically controllable upgrade process, ensuring that each loitering drone completes the software update in a predetermined order.

[0036] It should be noted that the multi-loitering aircraft software upgrade queue under the intelligent scheduling strategy output through the scheduling strategy model completes the software upgrade under ideal conditions. In practical applications, unforeseen circumstances may occur. Therefore, to ensure the reliability and traceability of the software upgrade process, this invention designs a real-time monitoring mechanism for the entire process. Throughout the software upgrade process, the flight control computer monitors the software upgrade status of each loitering aircraft in real time and feeds back relevant information to the server, enabling real-time monitoring and management of the entire software upgrade process.

[0037] The specific monitoring content for software upgrade status includes, but is not limited to: software upgrade progress (such as percentage completed, current execution stage), and abnormal events that may occur during the software upgrade process (such as upgrade failure, network connection interruption, etc.). The server gathers the software upgrade status of all patrol drones (including key information such as upgrade progress, number of successful upgrades, and number of failed upgrades) and presents it to the operators in a visual format, facilitating real-time monitoring and management of the entire upgrade process.

[0038] When the system detects an abnormal software upgrade event (such as multiple consecutive upgrade failures of the same patrol drone, or the disconnection of a critical patrol drone), it automatically triggers a tiered alarm and takes appropriate exception handling mechanisms according to the rules in the policy library, such as automatic retries, queuing suspension, and notification for manual intervention. For example, when a patrol drone's software upgrade fails, the system automatically records the reason for the failure and restarts the upgrade operation or performs manual intervention according to preset policies. In this way, problems can be detected and dealt with in a timely manner.

[0039] In one feasible implementation, after the software upgrade is completed, to ensure the consistency and standardization of all loitering pod software upgrades, the system automatically performs a consistency check on each loitering pod. Loitering pods found to be inconsistent are automatically upgraded again. Specifically: a consistency check is performed on each loitering pod that has completed the software upgrade operation to check for inconsistencies in the upgraded software version (e.g., comparing the upgraded software version number with the expected version number recorded by the server), functional modules (checking whether the core functional modules of the loitering pod's flight control software are loaded and responding normally, such as communication modules, navigation modules, and mission execution modules), and parameter settings (comparing the upgraded key parameter configurations, such as communication frequencies and flight restriction parameters, to ensure they are consistent with the standard configuration). For the first loitering pod that passes the consistency check, the upgrade completion time of the software to be upgraded and the new software version information after the upgrade are recorded, and the first loitering pod is allowed to be powered down first to enter standby mode. For the second loitering pod that fails the consistency check, a preset repair program is automatically started to re-upgrade the software to be upgraded until the second loitering pod passes the consistency check. Once all loitering drones in the queue have completed their software upgrades, passed the one-time verification, and the background records are updated, a multi-loitering drone software collaborative upgrade is complete. This demonstrates that the verification mechanism and real-time monitoring further ensure the consistency of the loitering drone software versions.

[0040] The beneficial effects of this application are as follows: This invention models the multi-loitering vehicle upgrade sequence optimization problem as a Markov decision process, and trains an agent using reinforcement learning algorithms to dynamically adjust the upgrade sequence based on the real-time status of each loitering vehicle, such as battery level and network quality. This mechanism transforms the scheduling problem into an intelligent decision-making problem, breaking through the limitations of traditional manual scheduling or fixed sequences. Furthermore, based on the Markov decision process modeling method and the algorithmic characteristics of deep reinforcement learning, the trained scheduling strategy model possesses excellent generalization ability, adapting to collaborative upgrade scenarios with different numbers of loitering vehicles without requiring redesign of scheduling rules for each scale, significantly improving the engineering applicability and scalability of the method. In addition, by monitoring the software upgrade status in real time and providing timely feedback, the system can automatically trigger corresponding countermeasures, such as retransmission or re-upgrade, when there are network fluctuations or upgrade failures, avoiding upgrade interruptions due to battery depletion or poor network conditions, and greatly improving upgrade stability and success rate.

[0041] Based on the foregoing embodiments, this embodiment will specifically describe step S12 in the above embodiments. Specifically, to achieve intelligent optimization of the upgrade sequence of multiple loitering aircraft, the training process of the scheduling strategy model may include the following steps: Step 1: Treat the server as an intelligent agent and define the state space, action space, and reward function to construct a Markov decision process model.

[0042] In this embodiment, the scheduling problem is abstracted as a Markov decision process, the server is regarded as an agent, and the following core elements are defined: (1) State space: used to observe the roaming queue of software to be upgraded in the server.

[0043] Each server can support collaborative software upgrades for 1 to 256 loitering rovers. This invention designs a large array of length 256 to represent the state. s t In the large array, each element represents a time slot, and each element is composed of smaller arrays: Empty subarray: indicates that the time slot is idle and currently not occupied by any rover; A non-empty subarray represents a patrol drone in the time slot that needs software upgrades. It contains three sub-elements: the first sub-element represents the patrol drone's current battery level (0-100); the second sub-element represents the patrol drone's network quality information (0-100); and the third sub-element is the patrol drone's current status identifier, used to characterize the stage of the software to be upgraded during the scheduling of the software upgrade order, with a value of 0, 1, or 2. 0 represents a patrol drone that has not yet started adjusting the software upgrade order, 1 represents a patrol drone that is adjusting the software upgrade order, and 2 represents a patrol drone that has completed adjusting the software upgrade order.

[0044] For example, state s t In this structure, time slot 1 may be [95, 90, 0], indicating that there is a patrol vehicle with 95% battery power, 90 network quality, and whose sequence has not yet been adjusted; time slot 2 may be [93, 95, 1], indicating that another patrol vehicle is being adjusted; and time slot 3 is empty ([]). This structure can fully reflect the distribution and adjustment progress of all patrol vehicles.

[0045] (2) Action Space: This includes scheduling rules for adjusting the software upgrade order. The agent gradually adjusts the software upgrade order of the patrol aircraft by selecting actions. Due to the large state space, to facilitate better learning, each action only changes a small portion of the state, i.e., each action only changes the software upgrade order of one patrol aircraft. Each patrol aircraft has 255 adjustment opportunities. The next patrol aircraft will only be adjusted after the previous one has finished adjusting its order. This design minimizes the changes between two adjacent states while still traversing all possible solutions, enabling the agent to gradually learn complex patrol aircraft software upgrade order scheduling strategies. For the patrol aircraft currently being adjusted, the scheduling rules include the following three actions: The first action (Action 0) is used to maintain the current software upgrade sequence of the loitering drone. The second action (Action 1) is used to advance the software upgrade order of the current patrol vehicle by one position. If the previous position of the current patrol vehicle is an idle time slot, it is occupied directly; if the previous position is another patrol vehicle, the order of the current patrol vehicle and the other patrol vehicle are swapped. The third action (Action 2) is used to postpone the software upgrade order of the current patrol vehicle by one position. If the next position of the current patrol vehicle is an idle time slot, it is occupied directly; if the next position is another patrol vehicle, the order of the current patrol vehicle and the other patrol vehicle are swapped.

[0046] To prevent overstepping boundaries, the second action is prohibited when the loitering drone is at the head of the upgrade queue; the third action is prohibited when the current loitering drone is at the end of the upgrade queue.

[0047] (3) Reward Function: Used to encourage agents to find the path to the maximum reward. If network quality remains constant, the time it takes for all patrol drones to complete the software upgrade is fixed. However, patrol drones that complete the upgrade first and pass the consistency check can be prioritized for power-off and enter standby mode after the background records the update time and the new software version. Therefore, the reward function includes the following: First part: a reward for remaining battery power after the software upgrade, encouraging a larger remaining battery power after the upgrade. Normalization is applied to help agents better handle software upgrades with varying numbers of patrol drones. Second part: a penalty for running out of battery power before the software upgrade is complete. If a patrol drone runs out of battery power before the upgrade is complete, a significant penalty is imposed on the agent. The specific definitions are as follows: (a) ; in, The i-th loitering rovers are awaiting software upgrades. For the collection of all loitering drones, n For the number of loitering drones, for After adjusting the software upgrade order, the upgrade will be completed and the remaining battery power will be used. for Complete the software upgrade before adjusting the upgrade order, using up the remaining battery power. This is a penalty item. Among them... and The calculation method is as follows: (b) In the formula: for The initial charge, For the power consumption rate of all loitering drones, for The time required for software upgrades is inversely related to network quality.

[0048] (c) In the formula: k for The penalty coefficient for running out of battery power before the software upgrade is complete is a relatively large constant value.

[0049] This reward function encourages agents to prioritize upgrading patrol drones with sufficient power and good network quality as early as possible, thereby maximizing the overall remaining power and improving mission availability while ensuring that all patrol drones are successfully upgraded.

[0050] The state transition diagram of the entire Markov decision process is as follows: Figure 2 As shown, n A roaming drone is placed in a server with a length of 256. The initial state before adjustment is: s 0 In time slot 1, there is a patrol drone with a current battery level of 95% and a network quality of 90%. In time slot 2, there is a patrol drone with a current battery level of 93% and a network quality of 95%. Time slot 3 is an idle time slot. The server intelligently identifies the first patrol drone in the time slots that has not yet adjusted its software upgrade order and makes adjustments accordingly. s 0 The current battery level of the patrol drone is 95%, and the network quality is 90%. At this time, the intelligent agent makes a decision. a 0 The action of =2 depends on the current state. s 0 and the actions performed a 0 The corresponding reward value was obtained. r 1 And entered the next state. s 1 ,in r 1 Calculated using formula (a). In state... s 1 In the middle, the patrol drone with a battery level of 95% and a network quality of 90% moves back one time slot, reaching time slot 2. At this point, the agent makes a decision. a 1 =0 actions. Therefore, in the state s 2 In the middle, the patrol drone with 95% battery and 90% network quality is still in time slot 2. After 255 action selections, the patrol drone with 95% battery and 90% network quality completes the software upgrade sequence adjustment. Next, the patrol drone with 93% battery and 95% network quality begins the software upgrade sequence adjustment. After 255... nFollowing this action selection, all loitering rovers adjusted their software upgrade order, resulting in a multi-loitering rovers software collaborative upgrade queue after intelligent scheduling.

[0051] Step 2: Train the Markov decision process model using a deep reinforcement learning algorithm to obtain the scheduling strategy model.

[0052] In this embodiment, the Deep Deterministic Policy Gradient (DDPG) algorithm is selected for model training. The DDPG algorithm combines deterministic policy gradient with deep neural networks, making it suitable for problems involving continuous action spaces and high-dimensional state spaces, and effectively handling the scheduling optimization task of this invention.

[0053] First, upgrade queue data for multiple groups of loitering rovers with varying numbers of aircraft are collected, and this data is divided into training, validation, and test sets according to a preset ratio. For example, a large amount of multi-loitering rovers software collaborative upgrade data is prepared as the initial queue before simulation adjustments. Each sample contains the initial sequence of a different number (1-256) of loitering rovers and their corresponding battery and network quality information. All samples are then divided into training, validation, and test sets in a 6:2:2 ratio.

[0054] The DDPG algorithm employs a dual-network structure of Actor and Critic. The Actor network is responsible for generating scheduling rules and selecting actions; its output is the probability of three actions after applying Softmax, and the action with the highest probability is selected as the final action. The Critic network is responsible for evaluating the value of the actions selected by the Actor network, further reducing training fluctuations and improving stability.

[0055] The agent is trained on the training set. In each round of training, one multi-roaming aircraft software upgrade queue is adjusted. Due to the large solution space, a large initial learning rate can be used, followed by a learning rate decay method to gradually reduce the learning rate for stable training. After adjusting the multi-roaming aircraft software upgrade queues in the training set, the model trained on the validation set is validated. In each round, only one multi-roaming aircraft software upgrade queue is trained and adjusted. After adjusting all multi-roaming aircraft software upgrade queues in the validation set, if the average reward value of the validation set reaches a preset threshold, the model is considered successfully trained, and the current artificial neural network parameters are saved. Finally, the saved neural network parameters are used to test on the test set. If the test passes, the final scheduling strategy model is determined.

[0056] Based on the scheduling strategy model, the server infers the real-time status of the current patrol drones and outputs a scheduling strategy for the software upgrade order of multiple patrol drones. This scheduling strategy can quickly identify queues with a large remaining battery level and no patrol drones running out of battery before the software upgrade is completed. This queue serves as the basis for sequentially executing software upgrades in subsequent steps.

[0057] As can be seen, a Markov decision process was designed based on the scheduling process of multi-loiter software collaborative upgrade, and the optimal scheduling strategy was autonomously discovered using a deep reinforcement learning algorithm. The trained agent has strong robustness and can intelligently schedule the software upgrade order of different numbers of loiterers, achieving intelligent adaptation to complex scenarios, strong generalization ability, and significantly improving the efficiency and reliability of multi-loiter collaborative upgrade.

[0058] Accordingly, this application also discloses a multi-loitering aircraft software collaborative upgrade device, see [link to relevant documentation]. Figure 3 As shown, the device includes: The network deployment module 11 is used to connect multiple patrol drones equipped with the software to be upgraded to the same network environment and to centrally manage each patrol drone through a server. The scheduling strategy determination module 12 is used to acquire the status information of each of the loitering drones and input the status information into a pre-trained scheduling strategy model to output a scheduling strategy for the software upgrade order of the multiple loitering drones; wherein, the status information includes at least power information and network quality information; the scheduling strategy model is constructed based on a Markov decision process and trained by a deep reinforcement learning algorithm; The software collaborative upgrade module 13 is used to perform software upgrade operations on each of the patrol aircraft in sequence based on the scheduling strategy, and to feed back the corresponding software upgrade status to the server.

[0059] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0060] Therefore, the above-described scheme in this embodiment models the multi-roaming aircraft upgrade sequence optimization problem as a Markov decision process. By training an agent using reinforcement learning algorithms, it can dynamically adjust the upgrade sequence based on the real-time status of each rover, such as battery level and network quality. This mechanism transforms the scheduling problem into an intelligent decision-making problem, overcoming the limitations of traditional manual scheduling or fixed sequences. Furthermore, based on the Markov decision process modeling method and the algorithmic characteristics of deep reinforcement learning, the trained scheduling strategy model possesses excellent generalization ability, adapting to collaborative upgrade scenarios with varying numbers of rover aircraft without requiring redesign of scheduling rules for each scale, significantly improving the method's engineering applicability and scalability. In addition, by monitoring the software upgrade status in real time and providing timely feedback, the system can automatically trigger corresponding countermeasures, such as retransmission or re-upgrade, when there are network fluctuations or upgrade failures, avoiding upgrade interruptions due to battery depletion or poor network conditions, and greatly improving upgrade stability and success rate.

[0061] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0062] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the multi-loitering aircraft software collaborative upgrade method disclosed in any of the foregoing embodiments.

[0063] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0064] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it can include an operating system 221, computer programs 222, and data 223, etc. The data 223 can include various types of data. The storage method can be temporary storage or permanent storage.

[0065] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the multi-roaming aircraft software collaborative upgrade method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0066] Furthermore, this application also discloses a computer-readable storage medium, which includes random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, magnetic disks, optical disks, or any other form of storage medium known in the art. The computer program, when executed by a processor, implements the aforementioned multi-roaming aircraft software collaborative upgrade method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0067] Furthermore, embodiments of this application also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements any of the above-described methods for collaborative software upgrades of multiple loitering rovers.

[0068] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0069] The steps of the multi-loitering vehicle software collaborative upgrade method or algorithm described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0070] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0071] The above provides a detailed description of a multi-loitering aircraft software collaborative upgrade method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for collaborative software upgrades of multiple loitering vehicles, characterized in that, include: Multiple patrol drones equipped with the software to be upgraded are connected to the same network environment, and each patrol drone is centrally managed through a server. The status information of each of the loitering drones is acquired and input into a pre-trained scheduling strategy model to output a scheduling strategy for the software upgrade order of the multiple loitering drones; wherein, the status information includes at least power information and network quality information; the scheduling strategy model is constructed based on a Markov decision process and trained by a deep reinforcement learning algorithm; Based on the scheduling strategy, software upgrade operations are performed on each of the loitering drones in sequence, and the corresponding software upgrade status is fed back to the server.

2. The multi-loitering vehicle software collaborative upgrade method according to claim 1, characterized in that, The process of connecting multiple patrol drones equipped with the software to be upgraded to the same network environment and centrally managing each patrol drone through a server includes: Multiple patrol drones equipped with the software to be upgraded will be connected to the same network environment via Ethernet ports. Each of the loitering drones is assigned a unique communication address using a dynamic host configuration protocol, and a network connection between the server and the loitering drone is established based on the communication address and a preset connection strategy. Determine whether the current software version of the loitering drone matches the record in the background; If there is a discrepancy, the difference is recorded in the background and the network connection of the patrol drone is disconnected to terminate the current software upgrade of the patrol drone. If they match, the target upgrade program file is transmitted in parallel from the server to the roaming vehicle based on multi-threaded distribution technology, and real-time data verification is performed during the transmission process to automatically retransmit any data with transmission errors.

3. The multi-loitering vehicle software collaborative upgrade method according to claim 1, characterized in that, The training process of the scheduling strategy model includes: The server is treated as an intelligent agent, and a state space, action space, and reward function are defined to construct a Markov decision process model. The state space includes the state information and current state identifier of the patrol vehicle. The current state identifier represents the stage of the software to be upgraded during the scheduling of the software upgrade order, including not yet started, being adjusted, and adjusted completely. The action space includes scheduling rules for adjusting the software upgrade order. The reward function includes a reward for remaining battery power after the software upgrade is completed, and a penalty for battery depletion before the software upgrade is completed. The Markov decision process model is trained using a deep reinforcement learning algorithm to obtain the scheduling strategy model.

4. The multi-loitering vehicle software collaborative upgrade method according to claim 3, characterized in that, The scheduling rules in the action space include: The first action is to maintain the current software upgrade sequence for the loitering drone; The second action is to advance the software upgrade order of the current patrol vehicle by one position. If the previous position of the current patrol vehicle is an idle time slot, it is occupied directly; if the previous position is another patrol vehicle, the order of the current patrol vehicle and the other patrol vehicle are swapped. The third action is to postpone the software upgrade order of the current patrol vehicle by one position. If the next slot of the current patrol vehicle is an idle time slot, it is occupied directly; if the next slot is occupied by another patrol vehicle, the order of the current patrol vehicle and the other patrol vehicle are swapped. Specifically, when the current loitering drone is at the head of the upgrade queue, the second action is prohibited; when the current loitering drone is at the end of the upgrade queue, the third action is prohibited.

5. The multi-loitering vehicle software collaborative upgrade method according to claim 3, characterized in that, The step of training the Markov decision process model using a deep reinforcement learning algorithm to obtain the scheduling policy model includes: Collect upgrade queue data of multiple sets of different numbers of loitering drones, and divide the upgrade queue data into training set, validation set and test set according to a preset ratio; Using a deep deterministic policy gradient algorithm, the software upgrade order of each upgrade queue in the training set is adjusted to train the Markov decision process model for multiple rounds, and the preset initial learning rate is gradually reduced during the training process; wherein, the deep deterministic policy gradient algorithm uses an Actor-Critic dual network structure, the Actor network is used to generate the scheduling rules, and the Critic network is used to evaluate the value of the scheduling rules generated by the Actor network. The trained model is validated using the validation set. When the average reward value of the validation set reaches a preset threshold, the model is determined to be successfully trained, and the current artificial neural network parameters are saved. Based on the parameters of the artificial neural network, the model is tested using the test set, and the scheduling strategy model is determined after the test is completed.

6. The multi-loitering vehicle software collaborative upgrade method according to claim 1, characterized in that, The step of feeding back the corresponding software upgrade status to the server includes: The software upgrade status of each loitering drone is monitored in real time by the flight control computer, and the software upgrade status is fed back to the server for visualization. When a software upgrade anomaly is detected, the cause of the failure is automatically recorded, and the corresponding exception handling mechanism is triggered.

7. The multi-loitering vehicle software collaborative upgrade method according to any one of claims 1 to 6, characterized in that, After performing software upgrade operations on each of the loitering rovers sequentially based on the scheduling strategy, the process further includes: A consistency check is performed on each patrol aircraft that has completed the software upgrade operation to check whether there are any inconsistencies in the software version, functional modules and parameter settings of the upgraded software in the patrol aircraft. For the first patrol aircraft that passes the consistency check, the upgrade completion time of the software to be upgraded and the new software version information after the upgrade are recorded, and the first patrol aircraft is allowed to be powered down first to enter the standby state. For the second loitering rovers that fail the consistency check, a preset repair program is automatically initiated to re-upgrade the software to be upgraded until the consistency check of the second loitering rovers passes.

8. A multi-loitering vehicle software collaborative upgrade device, characterized in that, include: The network deployment module is used to connect multiple patrol drones equipped with the software to be upgraded to the same network environment, and to centrally manage each patrol drone through a server; The scheduling strategy determination module is used to acquire the status information of each of the loitering drones and input the status information into a pre-trained scheduling strategy model to output a scheduling strategy for the software upgrade order of the multiple loitering drones; wherein, the status information includes at least power information and network quality information; the scheduling strategy model is constructed based on a Markov decision process and trained by a deep reinforcement learning algorithm; The software collaborative upgrade module is used to perform software upgrade operations on each of the aforementioned patrol vehicles sequentially based on the scheduling strategy, and to feed back the corresponding software upgrade status to the server.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the multi-roaming aircraft software collaborative upgrade method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein the computer programs, when executed by a processor, implement the multi-roaming aircraft software collaborative upgrade method as described in any one of claims 1 to 7.

Images