Spatiotemporal adaptive matching system and method based on multi-objective reinforcement learning

Abstract

A spatiotemporal adaptive matching system based on multi-objective reinforcement learning includes: a data fusion and cleaning module, an intelligent path planning module, a multi-objective scheduling and optimization module, an adaptive task optimization module, and a reinforcement learning-driven module. This invention achieves adaptive scheduling and path planning for UAVs in three-dimensional space through reinforcement learning and multi-objective optimization strategies, providing intelligent aircraft mission management, ensuring operational safety in complex airspaces, meeting the multiple requirements of the low-altitude industry for safety and efficiency, while reducing reliance on static rules and expert experience. It can dynamically adjust strategies based on real-time feedback and environmental changes, exhibiting high adaptability and meeting the core technical requirements of future low-altitude equipment in operation services, supervision, and the entire industry chain safety standardization system.

Description

Technical Field

[0001] This invention relates to a technology in the field of aircraft path planning, specifically a spatiotemporal adaptive matching system based on multi-objective reinforcement learning. Background Technology

[0002] Existing flight scheduling systems are based on static task planning, lacking dynamic adaptability and making it difficult to utilize real-time data for efficient scheduling. Existing path planning algorithms are mostly based on a single optimization objective, neglecting multi-dimensional resource efficiency and task priority. Furthermore, the A* algorithm cannot adaptively adjust in dynamic low-altitude environments, failing to cope with complex safety requirements and environmental changes. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes a spatiotemporal adaptive matching system based on multi-objective reinforcement learning. Through reinforcement learning and multi-objective optimization strategies, it enables adaptive scheduling and path planning of UAVs in three-dimensional space, providing intelligent aircraft mission management, ensuring operational safety in complex airspaces, and meeting the multiple requirements of the low-altitude industry for safety and efficiency. It also reduces reliance on static rules and expert experience, enabling dynamic strategy adjustments based on real-time feedback and environmental changes, exhibiting high adaptability, and meeting the core technical requirements of future low-altitude equipment in operation services, supervision, and the entire industry chain safety standardization system.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a spatiotemporal adaptive matching system based on multi-objective reinforcement learning, comprising: a data fusion and cleaning module, an intelligent path planning module, a multi-objective scheduling optimization module, an adaptive task optimization module, and a reinforcement learning-driven module. Specifically: the data fusion and cleaning module receives the aircraft's 3D coordinates, equipment status, and task priority data from multiple sources, user equipment, and a real-time monitoring system during the data reception phase. It then uses a spatiotemporal convolutional network (STGCN) to clean and fuse the data, standardizing data from different sources and removing outliers to obtain consistent spatiotemporal data. Spatiotemporal features are then extracted and output to the 3D intelligent path planning module and the reinforcement learning-driven module. The intelligent path planning module performs 3D spatial modeling of the flight area using grid partitioning technology, generates an initial path using an improved A* algorithm, and then uses reinforcement learning to optimize the matching system. The system further optimizes the path through reinforcement learning, enabling the aircraft to avoid obstacles and adapt its path in real time in complex dynamic environments. The multi-objective scheduling optimization module adopts the NSGA-II multi-objective optimization algorithm, comprehensively considering flight time, task priority, and resource consumption to form a scheduling optimization scheme. It generates a globally optimal solution set through Pareto front analysis and continuously optimizes scheduling efficiency by combining deep reinforcement learning. The adaptive task optimization module monitors the aircraft status and task progress in real time during the task execution phase. Based on the feedback information, it dynamically adjusts the task priority and scheduling strategy through reinforcement learning technology to optimize the path planning and resource allocation strategies. The reinforcement learning-driven module receives data and parameters from the data fusion and cleaning module and the adaptive task optimization module, and provides algorithmic support for the reinforcement learning parts of the three-dimensional intelligent path planning module and the multi-objective scheduling optimization module.

[0006] The consistent spatiotemporal data is obtained by parsing the data packet content, converting and standardizing the data from different sources, using STGCN to automatically detect and remove outliers, and using the task model and resource model as unique data sources to ensure that each time series data point corresponds to an accurate source. For source information based on process instances, the data is grouped and sorted according to timestamp order to obtain the time series data sequence of a single process instance.

[0007] The spatio-temporal convolutional network described herein employs, but is not limited to, the techniques described in "Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting" (IJCAI 2018), and extracts spatio-temporal features through the following steps:

[0008] Step 1: Model the multi-source sensor data (such as aircraft coordinates and equipment status) as a graph structure, where nodes represent data sources and edges represent spatiotemporal correlations. Use the spatial graph convolutional layer in the spatiotemporal convolutional network to perform graph convolution operations (GCN) to aggregate the features of adjacent nodes and capture the spatial dependencies.

[0009] Step 2: Use the temporal convolutional layer in the spatiotemporal convolutional network to perform dilated convolution, sliding along the time axis to extract temporal dynamic features, such as continuous changes in the trajectory of an aircraft or temporal fluctuations in mission priorities.

[0010] Step 3: By fusing spatial and temporal features through skip connections in the spatiotemporal convolutional network, the model's ability to learn complex spatiotemporal patterns is enhanced.

[0011] The improved A* algorithm specifically includes:

[0012] Step 1: Based on the 3D spatial grid model, the airspace is divided into equally spaced 3D grids. The grid cell is defined as Grid(x,y,z), where x,y,z are the 3D coordinates of the grid center point, respectively. The grid represents the space where the aircraft can travel and the distribution of obstacles in the airspace, transforming the path planning problem into a discretized path search problem in 3D space.

[0013] The three-dimensional spatial grid model includes: a spatial density sensing unit, a grid resolution adjustment unit, and a dynamic mapping unit. The spatial density sensing unit calculates the complexity of the local spatial domain based on real-time sensor data (such as obstacle distribution, aircraft density, and weather conditions). The grid resolution adjustment unit dynamically adjusts the grid resolution according to the complexity of the local spatial domain. The dynamic mapping unit updates the obstacle state in real time through a spatiotemporal convolutional network (STGCN), and the predicted trajectory of the dynamic obstacles is modeled by Kalman filtering and fed back into the grid model.

[0014] The complexity of the local spatial domain Where: N obs N represents the number of obstacles. uav V represents the number of aircraft. celL This represents the current grid cell volume.

[0015] The aforementioned dynamic adjustment refers to the following: using smaller grids (1m×1m×1m) in high-density areas (such as urban airspace) to improve obstacle avoidance accuracy; and using larger grids (10m×10m×10m) in low-density areas (such as open airspace) to reduce computational load.

[0016] Step 2: Construct a path. The goal of the search is to find the path P = {p1, p2, ..., p...} n}, such that the total cost function f(p) i The minimum value is f(p). i )=g(p i )+h(p i ), where: g(p i ) represents the distance from the starting point to the current node p. i The actual cost, h(p) i ) is the heuristically estimated cost from the current node to the target node;

[0017] Step 3: Combine reinforcement learning with h(p) i Dynamic optimization is performed, specifically: h ′ (p i )=α·h(p i )+β·Q(p i ,a), where: α and β are adjustment coefficients, satisfying α+β=1, which can be adaptively adjusted according to task priority to balance the impact of task priority and path cost of path nodes. Q(p i a) is the Q-value in reinforcement learning, representing the Q-value at node p. i The expected cost of the path after taking action a;

[0018] Step 4: During the path update process, Q-learning adjusts the path based on real-time feedback. At time t, given the environmental feedback, the Q-value of each node n is updated, specifically as follows: Where: η is the learning rate, γ is the discount factor, and r is the learning rate at node p. i The immediate reward at the location, where 'a' is the current action, 'a' is the next action, and 'p' is the next action. i+1 To perform action a ′ The next node reached afterward;

[0019] Step 5: Dynamically calculate task priorities and incorporate them into the weight adjustments for path planning to ensure that the aircraft prioritizes critical tasks while balancing resource allocation rationality and path planning efficiency. The dynamic adjustment of task priorities is based on the Analytic Hierarchy Process (AHP) and Bayesian optimization. Priority scores are determined by comprehensively evaluating the importance of the task, the aircraft's current state (e.g., battery level, payload capacity), and environmental changes (e.g., airspace congestion).

[0020] The aforementioned task priority Where: u i The priority score for the i-th task is obtained by weighted summation of the following factors: T i Task urgency (e.g., task time constraints), weighted by α T S i The aircraft's status (such as remaining battery power and payload capacity) is weighted by α.S E i Environmental complexity (e.g., high-density flow areas), with a weight of α. E The expression for priority scoring is: u i =α T T i +α S S i +α E e i Among them: T i +S i +E i =1, and the weights can be dynamically adjusted through Bayesian optimization to adapt to changes in the task scenario. The calculated task priority w i This weight will be mapped to the weights of each node in the path planning process, guiding the heuristic search of the A* algorithm and the reward allocation for reinforcement learning. Weight adjustment enables the system to balance multi-task demands in real time, improving the flexibility of path planning and the overall efficiency of task completion.

[0021] The NSGA-II multi-objective optimization algorithm specifically includes:

[0022] Step A: In the multi-objective optimization process, the following objective function is used for optimization:

[0023] a. Minimize task completion time: The goal is to reduce the execution time of each task as much as possible, specifically: Wherein: T i This represents the execution time of the i-th task, where n is the total number of tasks;

[0024] b. Maximizing resource utilization: This aims to improve the efficiency of resource use in the system, specifically: Where: R j U represents the available quantity of the j-th resource. j Let m be the utilization rate of the j-th resource, and m be the total number of resources.

[0025] c. Path Optimization and Energy Minimization: Considering energy optimization in UAV path planning, the goal is to select the path with the minimum energy consumption, specifically: Among them: E k Let p represent the energy consumption of the k-th path, and p be the number of all possible paths.

[0026] Step B: NSGA-II is combined with Pareto front analysis to solve the problem and achieve multi-objective optimization. The Pareto front analysis includes:

[0027] a. Initial population generation: The system first generates an initial population containing N individuals, and randomly initializes the solution for each individual.

[0028] b. Non-dominated ranking: Ranking the individuals in the population using non-dominated methods and calculating the dominance degree of each individual. An individual A dominates another individual B if and only if: and This method calculates the Pareto front solution set, making each individual as close as possible to the optimal solution.

[0029] c. Crowding distance calculation: Calculate the crowding distance for each individual. i To assess its relative distribution with other individuals, specifically: in: and Let f represent the function values ​​of adjacent individuals in the objective function f.

[0030] d. Selection and crossover mutation: Using tournament selection, individuals with higher fitness and lower crowding are selected from the parents for crossover and mutation to generate the next generation population.

[0031] Step C: By introducing Deep Reinforcement Learning (DRL), the scheduling strategy can be adaptively adjusted to cope with complex and dynamic task requirements, including:

[0032] a. State representation: The state space s is represented by indicators such as task completion time, resource usage, and path energy consumption.

[0033] b. Reward Function: Define the reward function. Where: w i This represents the weight of the i-th objective. The system continuously updates the weights through reinforcement learning to achieve a dynamic balance among different objectives.

[0034] Step D: The final scheduling optimization scheme is a diverse set of schemes P generated by the Pareto front solution technique. Each scheme contains optimization results for task scheduling, resource allocation, and path selection. For practical applications of system scheduling, the system can select the optimal solution from the Pareto solution set based on specific application requirements (such as the urgency of tasks or the current resource usage) to meet the current scheduling objectives.

[0035] The reinforcement learning techniques specifically include:

[0036] Step i: Define the system state space S and action space A, and collect the current state P and system feedback data at each task progress. Based on the state and feedback data, select action a. t This updates the scheduling strategy.

[0037] Step ii: Update the Q value according to the Q-learning algorithm defined in the intelligent path planning module, and set r to the value at node p. i The immediate reward at time t is specifically set as the value of the reward function R at time t, R(s). t ,a t In each feedback, the strategy is adjusted based on the Q-value of the current state and action;

[0038] Step iii: Perform a weighted summation of historical states and feedback data to update the strategy weights for task priority and path selection. Let the set of historical states be {s}. t-n ,…,s t The corresponding feedback weight is {w}. t-n ,…,w t If}, then the priority adjustment value of a certain task at the current time t is... Where: w i It is obtained by normalizing the distance from the current time. Feedback data that is closer to the current moment has a higher weight;

[0039] Step iv: By updating the Q value, the system continuously optimizes the scheduling parameters, enabling it to achieve adaptive optimization in the face of environmental changes and adjustments to task requirements.

[0040] Technical effect

[0041] This invention proposes a path planning and reinforcement learning collaborative optimization mechanism based on a dynamic adaptive 3D grid model: It suggests an adaptive modeling method that dynamically adjusts the resolution of the 3D spatial grid, combining spatial density (obstacle distribution, number of aircraft, weather conditions) to adjust the grid size in real time (1m in high-density areas). 3 Low-density area 10m 3 This model is used to improve the heuristic function of the A* algorithm. The Q-value weights (h) of path nodes are dynamically optimized through reinforcement learning. ′ (p i )=α·h(p i )+β·Q(p i This paper proposes a method that deeply integrates task priority (AHP score) with path cost to achieve real-time obstacle avoidance and path optimization in a dynamic 3D environment. Furthermore, it combines the NSGA-II multi-objective optimization algorithm with deep reinforcement learning (DRL) through dynamic weight adjustment. The Pareto frontier solution set is updated in real time. The objective function weights (w) are driven by a reinforcement learning feedback mechanism (the state space includes the aircraft position, mission priority, and obstacle distribution). iAdaptive optimization generates a globally optimal scheduling scheme that adapts to environmental changes. This invention applies STGCN to UAV multi-source data cleaning and dynamic task priority mapping. It extracts the spatiotemporal dependencies of multi-source data (coordinates, state, task) through spatiotemporal graph convolution (aggregating features of adjacent nodes in the spatial dimension) and temporal dilated convolution (capturing temporal fluctuations), and dynamically adjusts the task priority weights (u) based on Bayesian optimization. i =α T T i +α S S i +α E E i This system enables real-time linkage between task priority and path planning, and constructs an adaptive closed-loop optimization system based on reinforcement learning. By monitoring the aircraft's status (battery level, position, load), environmental changes (airspace congestion, obstacle movement), and mission progress in real time, it dynamically adjusts path planning and resource allocation strategies. A weighted fusion mechanism based on historical feedback data is introduced. By combining Q-learning with an online update strategy, self-learning and self-optimization of task scheduling can be achieved.

[0042] Compared to existing technologies, this invention does not rely on fixed rules or static models. Instead, it optimizes the scheduling scheme based on real-time feedback and task progress through an adaptive adjustment mechanism, significantly improving the system's adaptability and real-time response capabilities. Through a multi-objective optimization algorithm, the system can simultaneously handle multiple scheduling objectives and generate multiple optimized schemes, ensuring efficient task execution and maximized resource utilization. Intelligent data processing and anomaly detection technologies further enhance the accuracy and reliability of scheduling decisions. Overall, the method of this invention not only improves the efficiency and accuracy of task scheduling but also possesses strong applicability and flexibility, enabling its widespread application in various complex low-altitude economic mission scenarios and meeting ever-changing needs. Attached Figure Description

[0043] Figure 1 This is the main flowchart of the method of the present invention;

[0044] Figure 2 This is a system main diagram topology module structure diagram of the present invention;

[0045] Figure 3 This is a schematic diagram comparing the path planning time of embodiments of the present invention;

[0046] Figure 4 This is a schematic diagram of the three-dimensional distribution of the Pareto front solution set in an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram comparing the performance of STGCN and Kalman filtering in an embodiment of the present invention. Detailed Implementation

[0048] like Figure 1 As shown in the figure, this embodiment relates to a spatiotemporal adaptive matching system based on multi-objective reinforcement learning, which includes: a data fusion and cleaning module, an intelligent path planning module, a multi-objective scheduling optimization module, an adaptive task optimization module, and a reinforcement learning driven module.

[0049] The data fusion and cleaning module includes: a multi-source data receiving unit, a spatiotemporal standardization unit, an anomaly detection and cleaning unit, and a multimodal fusion unit. Specifically: the multi-source data receiving unit collects and performs preliminary classification processing based on input information from multi-source sensors, user equipment, and the real-time monitoring system to obtain raw spatiotemporal data streams (including heterogeneous data such as aircraft 3D coordinates, equipment status, and task priorities); the spatiotemporal standardization unit performs format parsing and unified standardization processing (such as coordinate system transformation and timestamp alignment) based on data format differences to obtain intermediate data with consistent spatiotemporal dimensions; the anomaly detection and cleaning unit performs outlier detection and noise removal processing (such as outlier filtering and missing value imputation) based on the spatiotemporal correlation features of the spatiotemporal convolutional network (STGCN) to obtain a cleaned, high-quality spatiotemporal dataset; and the multimodal fusion unit performs multi-source data fusion based on the label identification information of the task model and resource model (including spatial graph convolution aggregating adjacent node features and temporal dilated convolution extracting temporal dependencies), outputting the fused consistent spatiotemporal data, and distributing it to the intelligent path planning module and the reinforcement learning-driven module.

[0050] The intelligent path planning module includes: a 3D grid modeling unit, an initial path generation unit, a reinforcement learning optimization unit, and a path dynamic adjustment unit. The 3D grid modeling unit performs dynamic adaptive grid division processing (high-density area 1m) based on airspace density indicators (number of obstacles, aircraft distribution, weather conditions) and mission requirements. 3 Low-density area 10m 3 This yields a 3D spatial rasterized model. The initial path generation unit, based on the starting point, target point, and raster model information, uses an improved A* algorithm (integrating task priority weights h(p)) to generate the path. i Heuristic search is performed to generate an initial feasible path. The reinforcement learning optimization unit dynamically adjusts the Q-values ​​of the path nodes based on real-time environmental feedback (dynamic obstacle trajectories, aircraft status) through Q-learning. The optimized path with real-time obstacle avoidance and optimal energy consumption is obtained. The path dynamic adjustment unit performs path weight redistribution processing based on changes in task priority (AHP score) and resource allocation constraints, and outputs the final adaptively adjusted three-dimensional path scheme.

[0051] The multi-objective scheduling optimization module includes: a multi-objective definition unit, an NSGA-II solution unit, a deep reinforcement learning collaborative unit, and a scheme selection unit. Specifically: the multi-objective definition unit models the objective function based on the optimization requirements of task completion time f1, resource utilization f2, and path energy consumption f3 to obtain a multi-objective optimization problem description; the NSGA-II solution unit generates a Pareto front solution set based on the initial population (randomly generated solution set) and non-dominated sorting rules to obtain a globally non-dominated solution set; and the deep reinforcement learning collaborative unit selects the scheme based on the real-time state space S (aircraft position, task progress, and environmental changes) through a reward function. Dynamically adjust target weight w i The NSGA-II iterative optimization is driven to obtain a dynamically updated Pareto optimal solution set. The scheme selection unit performs solution set screening based on the current task urgency and resource constraints through Bayesian optimization and analytic hierarchy process (AHP) to output the final scheduling scheme (such as the shortest time scheme or the lowest energy consumption scheme).

[0052] The adaptive task optimization module includes: a real-time status monitoring unit, a priority dynamic evaluation unit, a strategy dynamic adjustment unit, and a closed-loop feedback unit. Specifically, the real-time status monitoring unit collects and extracts features from aircraft sensor data (battery level, position, load) and environmental monitoring information (obstacle movement, airspace congestion) to obtain a real-time status vector. The priority dynamic evaluation unit adjusts the strategy based on the task urgency T. i Aircraft Status S i Environmental complexity E i Weighted score (u i =α T T i +α S S i +α E E i The task priority is dynamically calculated to obtain the real-time priority weight w. i The strategy dynamic adjustment unit adjusts the strategy based on priority weights and historical feedback data (V). t =Σw i ·Q(s i The system performs online optimization of path planning and resource allocation strategies, outputs dynamically adjusted scheduling instructions, and the closed-loop feedback unit evaluates the effectiveness of the strategies and corrects parameters based on the task execution results and environmental changes, forming a closed-loop optimization mechanism to ensure continuous system self-adaptation.

[0053] The reinforcement learning-driven module includes a data integration unit, an algorithm configuration unit, a model training unit, and a real-time parameter update unit. Specifically: the data integration unit integrates state-action pairs based on the spatiotemporal data output from the data fusion and cleaning module and the feedback information from the adaptive task optimization module to obtain a reinforcement learning training dataset; the algorithm configuration unit initializes and configures Q-learning parameters (learning rate η, discount factor γ) and the deep Q-network (DQN) structure according to task scenario requirements (such as obstacle avoidance sensitivity and energy consumption constraints) to obtain a customized reinforcement learning model; the model training unit trains the model using historical data and real-time interaction data through experience replay and policy gradient descent to obtain an optimized Q-value function and policy network; and the real-time parameter update unit performs online policy updates and model parameter synchronization based on dynamic environmental changes (such as sudden obstacles or sudden changes in task priority) to ensure that the reinforcement learning algorithms of each module are always in optimal condition.

[0054] Through specific practical experiments, Figure 2 In the application scenario shown, the learning rate η = 0.01 and the discount factor γ = 0.9 are set for the reinforcement learning-driven module; the NSGA-II population size N = 200 and the number of iterations is 100; the STGCN anomaly detection threshold is set to 3σ (standard deviation). Running the above method, the experimental data obtained are: the improved A* algorithm in a dynamic raster model (high-density area 1m)... 3 The average time to generate the initial path is 0.8 seconds, a 68% improvement over the static A* algorithm (2.5 seconds). In scenarios with sudden obstacles, the reinforcement learning optimization unit reduces the dynamic path adjustment response time to 0.2 seconds, achieving an obstacle avoidance success rate of 95.6% (compared to 78.4% for traditional methods). In a simulated 72-hour continuous operation, the system's task completion rate reached 99.2%, an 11.7 percentage point improvement over the existing static scheduling system (87.5%). The average resource utilization rate reached 88.7% (compared to 72.4% for traditional systems), and redundant resource consumption was reduced by 23%. The system comprises a web application layer, a business processing layer, and a data layer.

[0055] The web-based application layer provides user interaction with the system, offering functions such as task order placement, status viewing, path display, and result feedback. Users input basic flight mission information through the web interface and view the system's path planning results and mission execution progress in real time. A dynamic, responsive user interface is built using HTML5, CSS3, and React.js, supporting the input of flight mission information (such as flight target, mission priority, resource requirements, etc.). Users can upload files (JSON) or directly input mission parameters via forms; these inputs are submitted to the service layer through a RESTful API. Furthermore, by visually displaying the aircraft's path and mission allocation, users can clearly see the mission's execution status. Flight mission path planning results, mission execution status, resource utilization, and other information are visualized using Matplotlib and D3.js. Path information is displayed graphically, showing the aircraft's position, status, and resource allocation in real time.

[0056] The business processing layer is the core of the method, implementing the core algorithms for data processing from input to output, task scheduling, and path planning, and completing the adaptive matching and optimized scheduling of flight missions.

[0057] This layer comprises several modules: a data fusion and cleaning module, an intelligent path planning module, a multi-objective scheduling optimization module, an adaptive task optimization module, and a reinforcement learning-driven module. Specifically, the data fusion and cleaning module receives data from multiple sources within the web application layer and implements a Spatiotemporal Convolutional Network (STGCN) using PyTorch. It convolves both temporal and spatial features to extract spatiotemporal dependencies. The data from different sources is denoised, outliers are removed, and then Z-score normalization is performed using NumPy, mapping the data to a standard normal distribution with a mean of 0 and a variance of 1. The cleaned and fused spatiotemporal data is output. The intelligent path planning module models the 3D space using a grid-based approach. It receives the fused and cleaned spatiotemporal data, including the aircraft's starting and target positions, obstacle information in the flight area, and task priorities. The 3D space of the flight area is divided into uniformly sized cells, with each cell representing a feasible region. A heuristic function is then used to evaluate the cost of the path, and the A* algorithm is implemented using the SciPy library to find the path with the lowest cost from the starting point to the target in a known gridded environment, thus generating an initial shortest path. Then, a reinforcement learning-driven module is invoked to optimize the generated initial path. In each state, the aircraft selects actions based on the current environment and learns from the reward function. During flight, the aircraft updates its state using real-time sensor data (such as position, velocity, and battery status) and re-evaluates the path based on environmental changes. The reinforcement learning module optimizes the path in real-time based on the latest state information. The optimized path is adjusted through continuous state feedback, enabling the aircraft to adapt to environmental changes in real time.

[0058] The multi-objective scheduling optimization module uses NSGA-II, implemented based on the DEAP library, as its core algorithm. The initial solution is constructed by receiving preliminary paths generated by the intelligent path planning module. Through selection, crossover, and mutation of each generation of the population, the task scheduling scheme is optimized. Each optimization gradually approaches the Pareto front based on objectives such as aircraft resource consumption, task completion time, and priority, ultimately generating multiple scheduling schemes that meet different needs. The adaptive task optimization module receives real-time data such as the aircraft's status, path progress, and resource consumption, and feeds back parameters such as task priority, path planning, and resource allocation strategies to the multi-objective scheduling optimization module and the reinforcement learning-driven module based on factors such as task execution progress, aircraft status, and environmental changes. This allows them to continuously adjust the objective function and update the Q-value function, achieving dynamic optimization of the scheme. The reinforcement learning-driven module implements a Deep Q-Network (DQN) in PyTorch, using a Deep Neural Network (CNN) to approximate the Q-value function. The state space S is defined to include the aircraft's current position, velocity, task priority, and the distribution of environmental obstacles: S = {(x,y,z),ν} x ,ν y ,ν z ,t,o}, where: (x,y,z) is the position of the spacecraft, ν x ,v y ,v z The velocity component of the aircraft in three-dimensional space is denoted by t, where t is the remaining time of the mission, and o is the position and state of the obstacle. The action space defines each action of the aircraft in the gridded environment at each moment as a movement to the left, right, up, down, forward, or backward: A = {left, right, up, down, forward, backward}. The reward function is designed considering path length, obstacle avoidance capability, resource consumption, and task priority: R = -α·path_length + β·avoided_obstacle - γ·resource_consumption + δ·task_priority, where α, β, γ, and δ are weight coefficients that control the influence of each factor on the final reward. After model initialization, it interacts with the environment, takes actions, observes the results, and collects data on state, action, reward, and the next state. Based on the collected experience, the Q-value is updated or the policy network is optimized to gradually approach the optimal policy. It also collaborates and interacts with other modules, continuously learning and adapting to mission requirements and environmental changes, making flight mission scheduling more intelligent and efficient.

[0059] The data layer stores and manages various types of data generated in the system, ensuring not only data persistence and efficient access, but also supporting real-time data querying, updating, and historical data storage. Based on the characteristics of different data types, the data layer employs different database storage solutions. InfluxDB is used as the storage system for spatiotemporal data streams. These streams are collected through the web application layer or external sensor modules and submitted to the backend service via a RESTful API. The backend service writes the data to InfluxDB using the InfluxDB client library. PostgreSQL is used as a relational database to store system status data and historical task records. The required data is obtained from the web application through a RESTful API interface, and the backend service stores the obtained structured data in the PostgreSQL database. MongoDB is used to store external system data. External system data (such as real-time spatial domain data) is transmitted to the system through an Open API interface and stored in MongoDB. The service layer interacts with the MongoDB database via a RESTful API, requesting real-time spatial domain data for task scheduling and path optimization.

[0060] Table 1 Comparison of Technical Features

[0061]

[0062]

[0063]

[0064] Compared to existing technologies, this method demonstrates significant advantages in several key technical characteristics, particularly in terms of reliability, efficiency, adaptability, and effectiveness. These improvements stem from the comprehensive approach employed in this invention, which not only introduces deep reinforcement learning and multi-objective optimization algorithms into existing path planning and task scheduling, but also innovates in multiple dimensions, including data processing, model optimization, and real-time decision-making.

[0065] This invention significantly improves reliability. Existing task scheduling and path planning systems often rely on simple data cleaning and receiving mechanisms, which can lead to data loss, inconsistency, and latency when processing multi-source heterogeneous data, thus affecting the accuracy of scheduling decisions and the overall reliability of the system. This invention introduces a Spatiotemporal Convolutional Network (STGCN) to eliminate noise and outliers in multi-source data during the data fusion and cleaning stages, ensuring data consistency and accuracy in both time and space dimensions. Figure 5As shown, STGCN achieves an anomaly detection accuracy of 98.5%, compared to 89.2% for the traditional method (Kalman filtering); STGCN has a false alarm rate of 1.3%, compared to 8.7% for the traditional method. STGCN can efficiently process large-scale spatiotemporal data, minimizing the impact of data loss and anomaly interference, providing reliable input data for subsequent path planning and task scheduling, and ensuring the high reliability of the entire system.

[0066] This invention demonstrates a significant improvement in efficiency. Existing path planning methods, especially the A* algorithm, are typically only applicable to two-dimensional static environments, and their efficiency drops considerably when encountering dynamic obstacles or complex environments. This invention combines the A* algorithm with reinforcement learning, using deep reinforcement learning to optimize the path planning results in real time, enabling the system to quickly calculate the optimal path in complex and dynamic environments. This method offers a significant improvement in both computational speed and path optimization efficiency compared to existing algorithms. Figure 3 As shown, when a high-density area uses 1m 3 When using a grid, obstacle avoidance accuracy is improved by 40%, while the overall computational load only increases by 15% (compared to a 50% increase in computational load for traditional fixed grids).

[0067] Furthermore, the combined effect of reinforcement learning mechanisms in 3D grid modeling and path planning improves the system's computational efficiency, ensuring the aircraft can respond quickly and make optimal decisions in changing environments, exhibiting strong real-time performance. Adaptability is another key feature of this invention. Existing path planning and task scheduling systems typically rely on preset rules or manual adjustment strategies, lacking the ability to respond in real-time to environmental changes and task requirements. In contrast, this invention introduces an adaptive reinforcement learning mechanism, enabling the system to dynamically adjust task priorities, resource allocation, and path planning strategies based on environmental changes, aircraft status, and task progress. The system can not only adjust the aircraft's task scheduling in real time but also flexibly and adaptively adjust the optimal solution based on changes in task requirements and fluctuations in the external environment. This dynamic adjustment capability significantly enhances the system's adaptability in dealing with emergencies and complex tasks, enabling the aircraft to better cope with different scenarios and task requirements, ensuring the successful completion of tasks.

[0068] In terms of effectiveness, this invention effectively solves the problem that existing optimization methods struggle to balance multiple objectives by employing the NSGA-II multi-objective optimization algorithm and Pareto front solving. Existing multi-objective optimization algorithms often provide only a single scheduling scheme, making it difficult to meet the multiple optimization needs of complex tasks regarding time, resources, and task priority. By introducing NSGA-II, this invention can not only handle multiple optimization objectives but also generate multiple feasible optimal solutions through Pareto front solving, providing users with more choices. Figure 4As shown, this invention generates 15 non-dominated solutions, while the traditional NSGA-II method only generates 8; moreover, the task completion time f1 is reduced from 20 minutes to 15 minutes, and the resource utilization rate f2 is increased from 75% to 92%. This multi-scheme selection and flexibility enables the system to dynamically generate suitable scheduling schemes according to different task requirements, ensuring that the system maximizes scheduling efficiency and resource utilization in various application scenarios, and fully meets user needs.

[0069] In summary, this invention innovatively combines reinforcement learning, spatiotemporal convolutional networks, and multi-objective optimization algorithms to improve the system's performance in terms of reliability, efficiency, adaptability, and effectiveness. Unlike existing solutions that rely on preset rules and simple algorithms, this invention's system can perform real-time optimization and adaptive scheduling in dynamic and complex environments, providing a more flexible and efficient task scheduling and path planning solution. Through these improvements in technical characteristics, this invention can provide higher-quality intelligent services in multiple fields such as aircraft task scheduling, path planning, and resource allocation, meeting the needs of different tasks and environments.

[0070] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A spatiotemporal adaptive matching system based on multi-objective reinforcement learning, characterized in that, include: The system comprises a data fusion and cleaning module, an intelligent path planning module, a multi-objective scheduling optimization module, an adaptive task optimization module, and a reinforcement learning-driven module. Specifically: The data fusion and cleaning module receives 3D coordinates, equipment status, and task priority data from multiple sensors, user equipment, and the real-time monitoring system during the data reception phase. It then uses a spatiotemporal convolutional network (STGCN) to clean and fuse the data, standardizing data from different sources and removing outliers to obtain consistent spatiotemporal data. Spatiotemporal features are then extracted and output to the 3D intelligent path planning module and the reinforcement learning-driven module. The intelligent path planning module performs 3D spatial modeling of the flight area using grid partitioning technology. After generating an initial path using an improved A* algorithm, it further optimizes the path through reinforcement learning, enabling the aircraft to... In complex and dynamic environments, it can avoid obstacles and adaptively adjust paths in real time. The multi-objective scheduling optimization module adopts the NSGA-II multi-objective optimization algorithm, which comprehensively considers flight time, task priority, and resource consumption to form a scheduling optimization scheme. It generates a globally optimal solution set through Pareto front analysis and continuously optimizes scheduling efficiency by combining deep reinforcement learning. The adaptive task optimization module monitors the aircraft status and task progress in real time during the task execution phase. Based on the feedback information, it dynamically adjusts the task priority and scheduling strategy through reinforcement learning technology. The reinforcement learning-driven module receives data and parameters from the data fusion and cleaning module and the adaptive task optimization module, and provides algorithmic support for the reinforcement learning part of the three-dimensional intelligent path planning module and the multi-objective scheduling optimization module. The improved A* algorithm specifically includes: Step 1: Based on the 3D spatial grid model, divide the spatial domain into equally spaced 3D grids, and define the grid unit as... ,in: The three-dimensional coordinates of the grid center point are respectively used. The grid represents the space where the aircraft can travel and the distribution of obstacles in the airspace, transforming the path planning problem into a discretized path search problem in three-dimensional space. Step 2: Constructing a path search aims to find a path. This makes the total cost function The smallest, specifically: ,in: It is from the starting point to the current node. The actual cost, It is a heuristic estimate of the cost from the current node to the target node; Step 3: Combine reinforcement learning with Dynamic optimization is performed, specifically as follows: ,in: and The adjustment coefficient satisfies It can adaptively adjust according to task priority to balance the impact of task priority and path cost on path nodes. It is in reinforcement learning The value represents the value at the node. Take action Expected cost of the subsequent path; Step 4: During the path update process, Q-learning adjusts the path based on real-time feedback, within a given timeframe. Update each node based on real-time environmental feedback. The Q value is as follows: ,in: For learning rate, As a discount factor, For the node Instant rewards at the location For the current action, For the next step, To perform the action The next node reached afterward; Step 5: By dynamically calculating task priorities and incorporating them into the weight adjustment of path planning, it is ensured that the aircraft can prioritize the execution of critical tasks, while taking into account the rationality of resource allocation and the efficiency of path planning. The dynamic adjustment of task priorities is based on the Analytic Hierarchy Process (AHP) and Bayesian optimization. The priority score is determined by comprehensively evaluating the importance of the task, the current state of the aircraft, and environmental changes.

2. The spatiotemporal adaptive matching system based on multi-objective reinforcement learning according to claim 1, characterized in that, The consistent spatiotemporal data is obtained by parsing the data packet content, converting and standardizing the data from different sources, using STGCN to automatically detect and remove outliers, and using the task model and resource model as unique data sources to identify labels, thereby ensuring that each time series data point can correspond to an accurate source. Based on the source information of the process instance, the data is grouped and sorted according to the timestamp order to obtain the time-series data sequence of a single process instance; The spatiotemporal convolutional network described above extracts spatiotemporal features through the following steps: Step 1: Model the multi-source sensor data as a graph structure, where nodes represent data sources and edges represent spatiotemporal correlations. Use the spatial graph convolutional layer in the spatiotemporal convolutional network (GCN) to perform graph convolution operations to aggregate the features of adjacent nodes and capture the spatial dependencies. Step 2: Use the temporal convolutional layer in the spatiotemporal convolutional network to perform dilated convolutions that slide along the time axis to extract temporal dynamic features; Step 3: By using skip connections in the spatiotemporal convolutional network, spatial and temporal features are fused to enhance the model's ability to learn complex spatiotemporal patterns.

3. The spatiotemporal adaptive matching system based on multi-objective reinforcement learning according to claim 1, characterized in that, The NSGA-II multi-objective optimization algorithm specifically includes: Step A: In the multi-objective optimization process, the following objective function is used for optimization: a1. Minimize task completion time: The goal is to reduce the execution time of each task as much as possible, specifically: ,in: Indicates the first The execution time of each task The total number of tasks; a2. Maximizing resource utilization: This aims to improve the efficiency of resource utilization in the system, specifically: ,in: For the first The number of available resources, For the first The utilization rate of each resource The total quantity of resources; a3. Path Optimization and Energy Minimization: Considering energy optimization in UAV path planning, the goal is to select the path with the minimum energy consumption, specifically: ,in: Indicates the first Energy consumption of each path The number of all possible paths; Step B: NSG-II is combined with Pareto front solution to achieve multi-objective optimization. The Pareto front analysis includes: b1. Initial Population Generation: The system first generates a population containing... An initial population of individuals, with solutions randomly initialized for each individual; b2. Non-dominated ranking: The individuals in the population are ranked non-dominated, and the dominance of each individual is calculated. Dominating another individual If and only if: and This method is used to calculate the Pareto front solution set, making each individual as close as possible to the optimal solution; b3. Crowding distance calculation: Calculate the crowding distance for each individual. To assess its relative distribution with other individuals, specifically: ,in: They represent the objective functions respectively. The function values ​​of adjacent individuals in the array; b4. Selection and Crossover Mutation: Using tournament selection, individuals with higher fitness and lower crowding are selected from the parents for crossover and mutation to generate the next generation population; Step C: By introducing Deep Reinforcement Learning (DRL), the scheduling strategy can be adaptively adjusted to cope with complex and dynamic task requirements, including: c1. State Representation: Using metrics such as task completion time, resource usage, and path energy consumption as the state space. The representation of; c2. Reward Function: Define the reward function ,in: Indicates the first The system continuously updates the weights of each objective through reinforcement learning to achieve a dynamic balance among different objectives. Step D: The final scheduling optimization scheme is a diverse set of schemes generated using the Pareto front solution technique. Each solution includes optimization results for task scheduling, resource allocation, and path selection. For practical applications of system scheduling, the system can select the optimal solution from the Pareto solution set according to specific application requirements to meet the current scheduling objectives.

4. The spatiotemporal adaptive matching system based on multi-objective reinforcement learning according to claim 1, characterized in that, The reinforcement learning techniques specifically include: Step i: Define the system state space and action space And collect the current status during each task progress. Based on the system's feedback data and the status, select the appropriate action. This updates the scheduling strategy. Step ii: Update the Q-value according to the Q-learning algorithm defined in the intelligent path planning module. For the node The immediate reward at that location is specifically set as a reward function. exist value of time In each feedback, the strategy is adjusted based on the Q value of the current state and action; Step iii: Perform a weighted summation of historical states and feedback data to update the strategy weights for task priority and path selection. Let the set of historical states be... The corresponding feedback weight is Then a certain task at the current time Priority adjustment value ,in: It is obtained by normalizing the distance from the current time. Feedback data that is closer to the current moment has a greater weight; Step iv: By updating the Q value, the system continuously optimizes the scheduling parameters, enabling it to achieve adaptive optimization in the face of environmental changes and adjustments to task requirements.

5. The spatiotemporal adaptive matching system based on multi-objective reinforcement learning according to claim 1, characterized in that, The data fusion and cleaning module includes: a multi-source data receiving unit, a spatiotemporal standardization unit, an anomaly detection and cleaning unit, and a multimodal fusion unit. Specifically: the multi-source data receiving unit collects and performs preliminary classification processing based on input information from multi-source sensors, user equipment, and a real-time monitoring system to obtain the original spatiotemporal data stream; the spatiotemporal standardization unit performs format parsing and unified standardization processing based on data format differences to obtain intermediate data with consistent spatiotemporal dimensions; the anomaly detection and cleaning unit performs outlier detection and noise removal based on the spatiotemporal correlation features of a spatiotemporal convolutional network (STGCN) to obtain a cleaned, high-quality spatiotemporal dataset; and the multimodal fusion unit performs multi-source data fusion based on the label identification information of the task model and resource model, outputting the fused consistent spatiotemporal data, which is then distributed to the intelligent path planning module and the reinforcement learning-driven module.

6. The spatiotemporal adaptive matching system based on multi-objective reinforcement learning according to claim 1, characterized in that, The intelligent path planning module includes: a 3D grid modeling unit, an initial path generation unit, a reinforcement learning optimization unit, and a path dynamic adjustment unit. Specifically: the 3D grid modeling unit performs dynamic adaptive grid division based on spatial density indicators and task requirements to obtain a 3D spatial gridded model; the initial path generation unit uses an improved A* algorithm to perform heuristic search based on the starting point, target point, and grid model information to generate an initial feasible path; the reinforcement learning optimization unit dynamically adjusts the Q-values ​​of path nodes based on real-time environmental feedback using Q-learning to obtain an optimized path with optimal real-time obstacle avoidance and energy consumption; and the path dynamic adjustment unit performs path weight redistribution based on changes in task priority and resource allocation constraints, outputting the final adaptively adjusted 3D path scheme.

7. The spatiotemporal adaptive matching system based on multi-objective reinforcement learning according to claim 1, characterized in that, The multi-objective scheduling optimization module includes: a multi-objective definition unit, an NSGA-II solution unit, a deep reinforcement learning collaboration unit, and a scheme selection unit, wherein: the multi-objective definition unit determines the task completion time based on the task completion time. Resource utilization rate Path energy consumption To meet the optimization requirements, objective function modeling is performed to obtain a multi-objective optimization problem description. The NSGA-II solver unit generates the Pareto front solution set based on the initial population and non-dominated sorting rules, resulting in a global non-dominated solution set. The deep reinforcement learning collaborative unit then calculates the solution set based on the real-time state space. Through the reward function Dynamically adjust target weights The NSGA-II iterative optimization is driven to obtain a dynamically updated Pareto optimal solution set. The scheme selection unit performs solution set screening based on the current task urgency and resource constraints through Bayesian optimization and analytic hierarchy process (AHP) to output the final scheduling scheme.

8. The spatiotemporal adaptive matching system based on multi-objective reinforcement learning according to claim 1, characterized in that, The adaptive task optimization module includes: a real-time status monitoring unit, a priority dynamic evaluation unit, a strategy dynamic adjustment unit, and a closed-loop feedback unit. Specifically, the real-time status monitoring unit performs status acquisition and feature extraction processing based on aircraft sensor data and environmental monitoring information to obtain a real-time status vector; the priority dynamic evaluation unit adjusts the strategy according to the task urgency... Aircraft status Environmental complexity The weighted score is used to dynamically calculate task priority and obtain real-time priority weights. The strategy dynamic adjustment unit performs online optimization of path planning and resource allocation strategies based on priority weights and historical feedback data, and outputs dynamically adjusted scheduling instructions. The closed-loop feedback unit evaluates the strategy effect and corrects parameters based on task execution results and environmental changes, forming a closed-loop optimization mechanism to ensure the system's continuous self-adaptation.

9. The spatiotemporal adaptive matching system based on multi-objective reinforcement learning according to claim 1, characterized in that, The reinforcement learning-driven module includes a data integration unit, an algorithm configuration unit, a model training unit, and a parameter real-time update unit. Specifically: the data integration unit integrates state-action pairs based on the spatiotemporal data output from the data fusion and cleaning module and the feedback information from the adaptive task optimization module to obtain a reinforcement learning training dataset; the algorithm configuration unit initializes and configures Q-learning parameters and the deep Q-network (DQN) structure according to task scenario requirements to obtain a customized reinforcement learning model; the model training unit trains the model using historical data and real-time interaction data through experience replay and policy gradient descent to obtain an optimized Q-value function and policy network; and the parameter real-time update unit performs online policy updates and model parameter synchronization based on dynamic environmental changes to ensure that the reinforcement learning algorithms of each module are always in optimal condition.

Images