Unmanned aerial vehicle control method and system based on large model intelligent driving

Abstract

The invention relates to the technical field of AI large models, in particular to an unmanned aerial vehicle control method based on intelligent driving of a large model, and the method comprises the steps: carrying out the preprocessing of multi-modal data, operating a pre-trained Transform architecture large model, carrying out the fusion analysis of the preprocessed multi-modal data, and extracting the spatial structure characteristics and dynamic target information in an environment; providing the output environment characteristics of the Transform architecture large model and the optical image information to a semantic segmentation model, outputting a pixel-level segmentation image by using the semantic segmentation model based on the UNet segmentation architecture, and labeling image semantic classification labels of the environment characteristics; based on the output pixel-level segmentation image, an initial obstacle avoidance path is generated by adopting an RRT algorithm, and static obstacle avoidance is covered; according to the method, deep semantic understanding can be carried out on image and structure information in a complex environment, spatial layout and target dynamic characteristics are extracted, and intelligent path generation and global scheduling control are carried out on the basis.

Description

Technical Field

[0001] This invention relates to a drone control method, and more particularly to a drone control method and system based on large-scale model intelligent drive, belonging to the field of AI large-scale model technology. Background Technology

[0002] With the rapid development of artificial intelligence and unmanned systems technology, drones have been widely used in fields such as inspection, surveying, logistics, and disaster relief. To improve mission efficiency and operational flexibility, researchers are constantly exploring more intelligent human-machine interaction methods. Among these, using natural language to issue commands for drone missions has become one of the current research hotspots. This trend has driven the exploration of intelligent control systems based on language models and speech recognition technology in the field of drones.

[0003] However, traditional drone control systems still have many problems. On the one hand, existing technologies mainly rely on manual operation or simple rule-based automated control logic, which has a high operating threshold and is difficult to adapt to complex dynamic environments. For example, in low-light conditions or severely obstructed scenarios, operators need to frequently adjust the flight attitude, resulting in low mission efficiency. On the other hand, although some drones with automation capabilities can achieve basic path planning, they still perform poorly in dynamic obstacle avoidance, scene understanding, and flight strategy optimization. Their path optimization capabilities are limited, and they lack intelligent mechanisms for global scheduling and control, making it difficult to meet the needs of complex practical applications.

[0004] In view of the above, in order to overcome the above technical problems, this invention designs a UAV control method based on large-scale intelligent driving, which solves the above technical problems. Summary of the Invention

[0005] The technical objective of this invention is to propose a UAV control method based on large-model intelligent driving, in order to solve the problems of high complexity of manual operation, insufficient target detection accuracy, limited image segmentation capability and low path planning efficiency of traditional systems, and improve the autonomous decision-making ability and mission execution efficiency of UAVs.

[0006] To achieve the above-mentioned technical objectives, the present invention provides the following technical solution: The UAV control method based on large-model intelligent drive provided by this invention includes the following steps: Multimodal environmental data is collected using multimodal sensors mounted on drones; The multimodal data is preprocessed and a pre-trained Transformer architecture large model is run. The preprocessed multimodal data is then fused and analyzed to extract spatial structure features and dynamic target information in the environment. Finally, environmental features with the above features and information are output. The environmental features and optical image information output by the large model of the Transformer architecture are provided to the semantic segmentation model. The semantic segmentation model based on the UNet segmentation architecture outputs a pixel-level segmentation map, and the large model is trained by the image semantic classification labels of the pre-annotated environmental features. Based on the output pixel-level segmentation map, the RRT algorithm is used to generate an initial obstacle avoidance path covering static obstacle avoidance; the initial obstacle avoidance path is optimized by combining a reinforcement learning framework to form an optimized path; the Transformer architecture large model sends path planning instructions to the UAV based on the optimized path. By leveraging the inference capabilities of the Transformer large model, multimodal environmental data, pixel-level segmentation maps, and path planning instructions are jointly analyzed to output global optimization scheduling and real-time control decisions for UAV flight.

[0007] The process of using a semantic segmentation model based on the UNet segmentation architecture to output pixel-level segmentation maps includes the following steps: Initialize the UNet segmentation architecture, which has multiple category output channels, each corresponding to a preset image semantic classification label. The image semantic classification labels include ground, obstacles, and flyable areas. Construct an image tensor with dimensions [1,3,256,256] as the model input, where: the first dimension represents the batch size; the second dimension is the number of image channels, using RGB three channels; the third and fourth dimensions are the height and width of the input image, both 256 pixels; With gradient computation disabled, the input image tensor is fed into the semantic segmentation model. An improved Softmax activation function is integrated into the semantic segmentation model. The image tensor of the original environmental features is processed through context information enhancement and attention mechanism to enhance the discriminative power between categories and obtain the original output feature map. Perform a Softmax operation along the category dimension on the original output feature map to obtain the probability value of each pixel belonging to each category, and use torch.argmax to find the category index with the highest probability to obtain the final semantic classification mask for each pixel; Traverse the predicted mask image and calculate the number of pixels corresponding to each category to verify the segmentation effect and model accuracy.

[0008] Furthermore, an improved Softmax activation function is integrated into the semantic segmentation model to process the original features through contextual information enhancement and attention mechanisms. The steps include: The image to be processed is input into a neural network to obtain the multi-class logits output corresponding to each pixel. The logits is a four-dimensional tensor with dimensions [batch, number of classes, height, width]. A softmax operation is performed on the logits along the category dimension to obtain a standard category probability distribution map for subsequent context information extraction. The softmax class probability distribution map is subjected to context convolution operation. The context convolution operation is a group convolution operation, in which the number of input channels, the number of output channels, and the number of groups are all equal to the number of classes. The convolution kernel size is a preset context size, and padding is added in the spatial dimension to extract the context information of each class channel. The aggregated context feature map is input into the attention convolution module, and channel-wise convolution is performed to extract spatial attention information. Then, it is normalized by the Sigmoid function to obtain a pixel-level attention weight map. The original logits are weighted and enhanced by taking the logarithm of the attention weight map and adding it to the original logits to improve the model’s response to important regions and suppress the response intensity of irrelevant regions. A constant term of 1e-8 is added to the logarithm operation to avoid numerical instability. A softmax operation is performed on the enhanced logits divided by a temperature coefficient less than 1 to enhance the sharpness of the distribution.

[0009] The steps for generating the initial obstacle avoidance path based on the RRT algorithm include: The initial obstacle avoidance path is generated using the goal-oriented adaptive RRT algorithm; The input parameters and return values ​​of the target-oriented adaptive RRT algorithm are as follows: Input parameters: start_pos is the starting position coordinate [x,y,z], representing the initial three-dimensional position of the UAV; goal_pos is the target position coordinates [x,y,z], representing the target position reached by the drone; `max_iterations` is the maximum number of iterations, with a default of 1000, which controls the search depth of the algorithm. convergence_threshold is the convergence threshold, which defaults to 1.0 meter and is the distance standard for determining whether the target has been reached. Return value: optimal_path is the optimized 3D path point sequence, in the format [[x1,y1,z1],[x2,y2,z2],...]; The initial obstacle avoidance path is input into a reinforcement learning framework for iterative optimization to minimize flight latency and improve obstacle avoidance success rate. In multi-drone collaborative scenarios, a distributed path planning protocol is introduced, in which each drone periodically exchanges path information and adjusts its own path based on the conflict detection results, thereby avoiding group flight conflicts.

[0010] Furthermore, the target-oriented adaptive RRT algorithm includes: (a) Based on the current iteration number iter and the maximum iteration number max_iter, use the formula: P_goal=α+β×(1-exp(-γ×iter / max_iter)) The target bias probability P_goal is dynamically calculated, where α, β, and γ are the initial target bias probability, the dynamic adjustment coefficient, and the convergence control parameter, respectively. (b) Sample the target point directly with probability P_goal; otherwise, sample uniformly and randomly in free space C_free. (c) For each sampling point, find the node with the smallest distance in the RRT search tree, and apply the formula ρ_local = N_obstacle / N_total based on the local obstacle density: step_size=step_base×(1-ρ_local×λ) Adaptive adjustment of expansion step size; Where step_base is the base step size; ρ_local is the local obstacle density; λ is the step size adjustment coefficient; N_obstacle is the number of obstacle grids within the detection radius; and N_total is the total number of grids within the detection radius. (d) Expand new nodes from the nearest node toward the sampling point by step_size, and use an enhanced linear interpolation collision detection algorithm with a resolution of 0.1 m to detect path connectivity; (e) When the Euclidean distance between the new node and the target point is less than the convergence threshold and there is no collision when they are directly connected, the target node is added to the search tree to generate the initial path.

[0011] Preferably, the parameters α, β, and γ are set to 0.1, 0.2, and 2.0, respectively.

[0012] In step (d), detection points are generated by linear interpolation at intervals of 0.1 m between the start and end nodes, and a raster map query is performed on each detection point.

[0013] This invention also provides a large-scale intelligent driving-based unmanned aerial vehicle (UAV) control system, characterized in that it includes: The environmental data acquisition engine collects environmental data through multimodal sensors mounted on the drone. The multimodal sensors acquire image data in real time to obtain multimodal data. The data preprocessing engine is used to preprocess multimodal data and run pre-trained Transformer architecture large models to perform fusion analysis on the preprocessed multimodal data and extract spatial structure features and dynamic target information in the environment. The large model inference engine is used to provide the output environment features and optical image information of the Transformer architecture large model to the semantic segmentation model. The semantic segmentation model based on the UNet segmentation architecture outputs pixel-level segmentation maps and labels the category of the environment features. The path planning engine is used to generate initial obstacle avoidance paths from the output pixel-level segmented map using the RRT algorithm, and to cover static obstacle avoidance; it combines a reinforcement learning framework to optimize the obstacle avoidance paths and sends path planning instructions to the large Transformer architecture model. Among them, the large model inference engine also uses the reasoning capabilities of the large model to jointly analyze perception information, target position, semantic segmentation results and path planning data, so as to realize global optimization scheduling and real-time control decision-making for UAV flight.

[0014] The UAV control system based on large-scale intelligent drive also includes a flight control output engine, which is used to adjust the attitude of the UAV through motor control. The flight control output engine provides output feedback to the data preprocessing engine and accepts inputs from the data flow and control flow from the large model inference engine and the path planning engine.

[0015] The present invention also provides a computer device, comprising: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the UAV control method based on large model intelligent drive as described in any one of claims 1 to 7.

[0016] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the UAV control method based on large model intelligent drive as described in any one of claims 1 to 7.

[0017] The beneficial effects of this invention are as follows: 1. This invention constructs a large model based on the Transformer architecture and integrates multimodal perception, semantic segmentation, and reinforcement learning path optimization to form an intelligent control method for unmanned aerial vehicles (UAVs). This method enables deep semantic understanding of image and structural information in complex environments, extracting spatial layout and target dynamic features, and then performing intelligent path generation and global scheduling control. Leveraging the reasoning and task understanding capabilities of the large model, it significantly improves the UAV's ability to understand natural language task commands and its robustness in task execution. This effectively solves problems in existing technologies such as insufficient path planning accuracy, poor interaction flexibility, and lack of environmental awareness, thereby enhancing the autonomous flight capabilities of UAVs in complex scenarios.

[0018] 2. Furthermore, the UNet semantic segmentation model integrated with the improved Softmax activation function and context enhancement mechanism in this invention can fully mine semantic context information and strengthen the response of important regions in image feature processing. Through attention weighting and temperature coefficient adjustment mechanisms, the model's ability to classify complex image regions is enhanced, thereby improving the accuracy and stability of semantic segmentation. This optimization scheme ensures accurate segmentation of semantic labels for ground, obstacles, and winged areas in the pixel-level segmentation image, providing higher-quality basic data for subsequent path planning and effectively reducing the misjudgment rate and collision risk in path planning.

[0019] 3. This invention introduces a goal-oriented adaptive RRT path planning algorithm, which combines dynamic adjustment of the expansion step size based on local obstacle density with an enhanced interpolation collision detection and reinforcement learning optimization framework, significantly improving the flexibility and real-time performance of path generation. Especially in dynamic or dense obstacle environments, this mechanism can converge to a safe and efficient path more quickly. Furthermore, in multi-UAV collaborative tasks, it utilizes path exchange and conflict detection strategies to achieve distributed dynamic path coordination control, effectively reducing the probability of multi-UAV path conflicts and improving the system's collaborative operation efficiency and safety. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] The above and other aspects of the invention will now be described by way of example only, with reference to the accompanying drawings, in which: Figure 1 This is a flowchart of the control method of the present invention; Figure 2This is a schematic diagram of the overall architecture of a drone control system based on large-scale intelligent driving, showing the integration relationship of core components such as the drone body, controller, high-performance computing unit, camera, infrared sensor, and lidar. Figure 3 This is a schematic diagram of the system's software architecture, illustrating the operational logic of the input data flow, data processing engine, large model inference engine, path planning engine, and flight control output. Detailed Implementation

[0022] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0023] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0024] Example 1: Figure 1 This is a flowchart of a UAV control method based on large-model intelligent drive, as described in Embodiment 1 of the present invention. This flowchart merely illustrates the logical sequence of the method described in this embodiment. Provided there are no conflicts, different methods may be used in other possible embodiments of the present invention. Figure 1 Complete the steps shown or described in the order indicated.

[0025] The UAV control method based on large-scale intelligent driving provided in this embodiment can be applied to a terminal and executed by a UAV control system based on large-scale intelligent driving. This system can be implemented in software and / or hardware and can be integrated into the terminal, such as any smartphone, tablet, or computer device with communication capabilities. See also... Figure 1 The method implemented in this way specifically includes the following steps: Step 1: Collect environmental data using the multimodal sensors mounted on the drone. The multimodal sensors collect image data in real time to obtain multimodal data. Step 2: Preprocess the multimodal data and run the pre-trained Transformer architecture large model to perform fusion analysis on the preprocessed multimodal data and extract spatial structure features and dynamic target information in the environment; Step 3: Provide the output environment features and optical image information of the Transformer architecture large model to the semantic segmentation model, use the UNet-based segmentation architecture to output pixel-level segmentation map, and label the image semantic classification labels of the environment features; Step 4: Based on the output pixel-level segmentation map, the RRT algorithm is used to generate an initial obstacle avoidance path and cover static obstacle avoidance; the obstacle avoidance path is optimized by combining the reinforcement learning framework, and path planning instructions are sent to the Transformer architecture large model; Step 5: By leveraging the reasoning capabilities of the large model, jointly analyze the perceived information, target location, semantic segmentation results, and path planning data to achieve global optimization scheduling and real-time control decisions for UAV flight.

[0026] Specifically, the multimodal sensor acquisition process is as follows: the camera captures visible light images at a rate of 30 frames per second with a resolution of 1920×1080; the infrared sensor supplements thermal imaging data under low light conditions, with a detection range covering a horizontal viewing angle of 60°; the lidar scans the three-dimensional space at a frequency of 10Hz to generate an environmental model with a point cloud density of 100 points per square meter.

[0027] like Figure 3 As shown, the implementation process of the large model inference module includes: the data preprocessing unit is responsible for cleaning different types of data; the encoder of the Transformer architecture extracts the relationship between different data, and combines visible light images, thermal imaging and point cloud data to understand the relationship between them; the attention mechanism helps the model pay more attention to dynamic targets (such as moving vehicles and pedestrians), and finally outputs environmental feature information.

[0028] In this embodiment, the optimized semantic segmentation model includes: a context aggregation module that combines global feature vectors to optimize the boundary segmentation accuracy of flyable areas; and outputting a pixel-level segmentation map labeled with category labels for the ground, obstacles, and flyable areas.

[0029] The implementation of the path planning algorithm includes: the RRT algorithm generates the initial obstacle avoidance path, covering static obstacle avoidance; the reinforcement learning framework can optimize the obstacle avoidance path and formulate the optimal path; in the case of multiple UAVs, a distributed path planning protocol is introduced to avoid path conflicts in group flight.

[0030] Example 2: This embodiment is an optimization based on the above embodiments. For details not described in this embodiment, please refer to Embodiment 1. The UAV control method based on large-model intelligent driving provided in this embodiment uses a semantic segmentation model based on the UNet segmentation architecture to output pixel-level segmentation maps. The specific algorithm includes: Step 1: Initialize the improved U-Net semantic segmentation model: model = UNet(n_classes=3) # n_classes=3: correspond to the three classes: ground, obstacles, and flyable areas respectively. model.eval() Step 2: Input Tensor Preprocessing input_tensor = torch.randn(1, 3, 256, 256) # Parameter description: # 1: Batch size # 3: Input the number of image channels (RGB three channels) # 256,256: Input image dimensions (height × width) Step 3: Model Reasoning Process with torch.no_grad(): # Disable gradient computation to save memory and speed up inference. # Forward propagation to obtain the original output feature map raw_output = model(input_tensor) # Multi-scale feature fusion and context aggregation optimization # Apply the softmax activation function to enhance the distinction between categories. softmax_output = F.softmax(raw_output, dim=1) # dim=1: Perform softmax in the channel dimension # Get the maximum probability class index for each pixel predicted_mask = torch.argmax(softmax_output, dim=1) # 0: Ground region #1: Obstacle Area #2: Flightable Area Step 4: Verify the output results print("Predicted pixel classification mask shape:", predicted_mask.shape) print("Segmentation category statistics:") for class_id in range(3): pixel_count = torch.sum(predicted_mask == class_id).item() class_name = ["Ground","Obstacles","Flyable Area"][class_id] print(f"{class_name}pixel count: {pixel_count}") In this embodiment, the specific algorithm for generating the initial obstacle avoidance path using the RRT algorithm is as follows: def generate_initial_path(self, start_pos, goal_pos, max_iterations=1000, convergence_threshold=1.0): """ Goal-oriented adaptive RRT path planning algorithm Algorithm input parameter description: - start_pos: Starting position coordinates [x, y, z], representing the initial three-dimensional position of the UAV. - goal_pos: Target position coordinates [x, y, z], representing the target position reached by the drone. - max_iterations: Maximum number of iterations, defaults to 1000, controls the search depth of the algorithm. - convergence_threshold: Convergence threshold, defaults to 1.0 meter, the distance standard for determining whether the target has been reached. Return value: - optimal_path: The optimized 3D path point sequence, in the format [[x1,y1,z1], [x2,y2,z2], ...] """ # Initialize the RRT planner rrt_planner = RRTPlanner(self.map_data) # 3D environment map data structure # Node tree initialization node_tree = [start_pos] # All nodes in the RRT search tree `parent_map = {tuple(start_pos): None}` # Records the parent node of each node, used for path backtracking. """ Adaptive target-oriented sampling strategy The probability distribution function for sampling point selection: P(x_rand) = { P_goal, # If a target point is selected P_uniform(x), # If uniform sampling is performed in free space C_free } The target orientation probability is: P_goal = α + β × (1 - exp(-γ × iter / max_iter)) α: Initial target orientation probability (0.1) β: Dynamic adjustment coefficient (0.2) γ: Convergence control parameter (2.0) iter: Current iteration number """ goal_bias_probability = 0.1 # Goal-oriented sampling probability, dynamically adjusted to balance exploration and development. adaptive_step_size = 0.5 # Adaptive step size, dynamically adjusted according to obstacle density for iteration_count in range(max_iterations): # Standard-oriented adaptive sampling strategy if np.random.random() <goal_bias_probability: # Goal-oriented sampling: Directly select the target point to accelerate convergence random_sample_point = goal_pos else: # Free-space random sampling: Random sampling within the feasible region random_sample_point = self._sample_random_point() # Find the nearest neighbor node nearest_idx = self._find_nearest_node(node_tree, random_sample_point) nearest_node_pos = node_tree[nearest_idx] # The node in the current search tree that is closest to the sampling point """ Local environment-aware adaptive step size strategy Step size adjustment formula based on local obstacle density: step_size = step_base × (1 - ρ_local × λ) The local obstacle density is calculated as follows: ρ_local = N_obstacle / N_total step_base: Base step size (0.5m) ρ_local: Local obstacle density [0,1] λ: Step size adjustment factor (0.5) N_obstacle: Number of obstacle grid cells within the detection radius N_total: Total number of grid cells within the detection radius """ # Dynamically adjust step size based on local obstacle density density = self._calculate_local_obstacle_density(nearest_node_pos) dynamic_step_size = adaptive_step_size * (1.0 - density * 0.5) """ # Expand new nodes from the nearest node to the sampling point new_extended_node = self._steer(nearest_node_pos, random_sample_point, step_size=dynamic_step_size) Enhanced collision detection algorithm Calculation of detection points for trajectory linear interpolation: P_check(t) = P_start + t × (P_end - P_start), t ∈ [0,1] P_start, P_end: Start and end detection points """ if not self._enhanced_collision_check(nearest_node_pos, new_extended_node): # Adding new nodes to the search tree node_tree.append(new_extended_node) parent_map[tuple(new_extended_node)] = tuple(nearest_node_pos) # Intelligent Termination Condition Judgment distance = self._calculate_euclidean_distance(new_extended_node,goal_pos) if distance <convergence_threshold: # Check if a straight path from the new node to the target is feasible. if not self._enhanced_collision_check(new_extended_node, goal_pos): # A feasible path was successfully found node_tree.append(goal_pos) parent_map[tuple(goal_pos)] = tuple(new_extended_node) break def _enhanced_collision_check(self, start_node, end_node): """ Enhanced collision detection algorithm, key variables: - start_node: The coordinates of the starting detection node [x, y, z] - end_node: The coordinates of the terminated detection node [x, y, z] - collision_resolution: Collision detection resolution, affecting detection accuracy. """ collision_resolution = 0.1 # Collision detection resolution (meters) trajectory_length = self._calculate_euclidean_distance(start_node,end_node) check_points_count = int(trajectory_length / collision_resolution) +1 for i in range(check_points_count): # Generate detection points by linear interpolation along the trajectory interpolation_ratio = i / (check_points_count - 1) check_point = [ start_node[0] + interpolation_ratio * (end_node[0]- start_node[0]), start_node[1] + interpolation_ratio * (end_node[1]- start_node[1]), start_node[2] + interpolation_ratio * (end_node[2]- start_node[2]) ] # Check if the current point has collided with an obstacle. if self._is_point_in_obstacle(check_point): return True # Collision occurred return False # No collision This further optimizes the generation of initial obstacle avoidance paths, covering static obstacle avoidance; combined with a reinforcement learning framework, it can optimize obstacle avoidance paths and formulate the optimal path.

[0031] Example 3: Building upon Example 2, this example provides a detailed explanation of the semantic segmentation model's inference process. Specifically, an improved Softmax activation function is integrated into the semantic segmentation model, and the original features are processed through contextual information enhancement and attention mechanisms. The specific algorithm is as follows: #Inherited from nn.Module, specifically a neural network module used to improve softmax output and enhance the discriminative power of the distribution. class ContextAwareImprovedSoftmax(nn.Module): """ Context-Aware Improved Softmax: Enhancing Discrimination by Incorporating Spatial Context Information """ #num_classes represents the number of categories; in this example, there are 3 categories. #context_size: The size of the context kernel, which determines the receptive field. The 3 indicates a 3x3 convolution. def __init__(self, num_classes=3, context_size=3): super(ContextAwareImprovedSoftmax, self).__init__() self.num_classes = num_classes self.context_size = context_size # Context Aggregation Network self.context_conv = nn.Conv2d(num_classes, num_classes, # The number of input and output channels is num_classes, representing a set of features for each class). kernel_size=context_size, # Convolution kernel size padding=context_size / / 2, # Padding size to maintain feature map size `groups=num_classes)` # Number of groups = number of classes. Each class undergoes independent convolution to avoid information clutter. # Attention Mechanism self.attention_conv = nn.Conv2d(num_classes,num_classes,1)# 1x1 convolution kernel def forward(self, x): # Standard softmax, - transforms logits into a probability distribution base_prob = F.softmax(x, dim=1) # Value range: [0, 1], Sum of probabilities for each class for each pixel = 1 # Contextual Information Aggregation context_features = self.context_conv(base_prob) # Calculate attention weights attention_weights =torch.sigmoid(self.attention_conv(context_features)) # Perform attention convolution on the context information, then compress it to [0,1] using sigmoid to obtain an "attention map". This map guides the model on which regions to focus on; larger values ​​indicate greater importance. # Context-enhanced logits #Add log(attention value) to the original logits as an adjustment; a higher weight strengthens the class, while a lower weight suppresses it. enhanced_logits = x + torch.log(attention_weights + 1e-8) # 1e-8: to prevent numerical instability caused by log(0) # Applying temperature scaling further enhances differentiation temperature = 0.8 # Use a lower temperature to enhance peak performance return F.softmax(enhanced_logits / temperature, dim=1) # Temperature parameter < 1.0, making the distribution sharper. This embodiment enhances the model's ability to classify complex image regions by using attention weighting and temperature coefficient adjustment mechanisms, thereby improving the accuracy and stability of semantic segmentation.

[0032] Example 4: Based on embodiments one to three above, this embodiment also provides a UAV control system based on large-scale intelligent drive, including: The environmental data acquisition engine collects environmental data through multimodal sensors mounted on the drone. The multimodal sensors acquire image data in real time to obtain multimodal data. The data preprocessing engine is used to preprocess multimodal data and run pre-trained Transformer architecture large models to perform fusion analysis on the preprocessed multimodal data and extract spatial structure features and dynamic target information in the environment. The large model inference engine is used to provide the output environment features and optical image information of the Transformer architecture large model to the semantic segmentation model. The semantic segmentation model based on the UNet segmentation architecture outputs pixel-level segmentation maps and labels the category of the environment features. The path planning engine is used to generate initial obstacle avoidance paths from the output pixel-level segmented map using the RRT algorithm, and to cover static obstacle avoidance; it combines a reinforcement learning framework to optimize the obstacle avoidance paths and sends path planning instructions to the large Transformer architecture model. Among them, the large model inference engine also uses the reasoning capabilities of the large model to jointly analyze perception information, target position, semantic segmentation results and path planning data, so as to realize global optimization scheduling and real-time control decision-making for UAV flight.

[0033] like Figure 3 As shown, the UAV control system based on large-scale intelligent drive also includes a flight control output engine, which is used to adjust the attitude of the UAV through motor control. The flight control output engine provides output feedback to the data preprocessing engine and accepts inputs from the data flow and control flow from the large model inference engine and the path planning engine.

[0034] The UAV control system based on large-scale intelligent driving provided in the embodiments of the present invention can execute the UAV control method based on large-scale intelligent driving provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0035] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0036] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0037] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0038] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0039] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A UAV control method based on large-scale model intelligent drive, characterized in that, Includes the following steps: Multimodal environmental data is collected using multimodal sensors mounted on drones; The system preprocesses multimodal environmental data, runs a pre-trained Transformer architecture large model, performs fusion analysis on the preprocessed multimodal data, and outputs environmental features. The environmental features and multimodal environmental data are sent to a semantic segmentation model based on the UNet segmentation architecture to output a pixel-level segmentation map; Based on the output pixel-level segmentation map, the RRT algorithm is used to generate an initial obstacle avoidance path covering static obstacle avoidance. By combining a reinforcement learning framework, the initial obstacle avoidance path is optimized to form an optimized path; The Transformer architecture model sends path planning commands to drones based on optimized paths. By leveraging the inference capabilities of the Transformer large model, multimodal environmental data, pixel-level segmentation maps, and path planning instructions are jointly analyzed to output global optimization scheduling and real-time control decisions for UAV flight.

2. The UAV control method based on large-model intelligent driving according to claim 1, characterized in that, The steps involved in using a semantic segmentation model based on the UNet segmentation architecture to output pixel-level segmentation maps include: Initialize the UNet segmentation architecture, which has multiple category output channels, each corresponding to a preset image semantic classification label. The image semantic classification labels include ground, obstacles, and flyable areas. Construct an image tensor with dimensions [1,3,256,256] as the model input, where: the first dimension represents the batch size; the second dimension is the number of image channels, using RGB three channels; the third and fourth dimensions are the height and width of the input image, both 256 pixels; With gradient computation disabled, the image tensor is input into the semantic segmentation model. An improved Softmax activation function is integrated into the semantic segmentation model. The image tensor is processed through context information enhancement and attention mechanism to enhance the discriminative power between categories and obtain the original output feature map. Perform a Softmax operation along the category dimension on the original output feature map to obtain the probability value of each pixel belonging to each category, and use torch.argmax to find the category index with the highest probability to obtain the final semantic classification mask for each pixel; Traverse the predicted mask image and calculate the number of pixels corresponding to each category to verify the segmentation effect and model accuracy.

3. The UAV control method based on large-model intelligent driving according to claim 2, characterized in that, An improved Softmax activation function is integrated into the semantic segmentation model. Image tensors are processed through contextual information enhancement and attention mechanisms. The steps include: The image to be processed is input into a neural network to obtain the multi-class logits output corresponding to each pixel. The logits is a four-dimensional tensor with dimensions [batch, number of classes, height, width]. A softmax operation is performed on the logits along the category dimension to obtain a standard category probability distribution map for subsequent context information extraction. The softmax class probability distribution map is subjected to context convolution operation. The context convolution operation is a group convolution operation, in which the number of input channels, the number of output channels, and the number of groups are all equal to the number of classes. The convolution kernel size is a preset context size, and padding is added in the spatial dimension to extract the context information of each class channel. The aggregated context feature map is input into the attention convolution module, and channel-wise convolution is performed to extract spatial attention information. Then, it is normalized by the Sigmoid function to obtain a pixel-level attention weight map. The original logits are weighted and enhanced by taking the logarithm of the attention weight map and adding it to the original logits to improve the model’s response to important regions and suppress the response intensity of irrelevant regions. A constant term of 1e-8 is added to the logarithm operation to avoid numerical instability. A softmax operation is performed on the enhanced logits divided by a temperature coefficient less than 1 to enhance the sharpness of the distribution.

4. The UAV control method based on large-model intelligent driving according to claim 1, characterized in that, The steps for generating an initial obstacle avoidance path based on the RRT algorithm include: The initial obstacle avoidance path is generated using the goal-oriented adaptive RRT algorithm; The input parameters and return values ​​of the target-oriented adaptive RRT algorithm are as follows: Input parameters: start_pos is the starting position coordinate [x,y,z], representing the initial three-dimensional position of the UAV; goal_pos is the target position coordinates [x,y,z], representing the target position reached by the drone; `max_iterations` is the maximum number of iterations, with a default of 1000, which controls the search depth of the algorithm. convergence_threshold is the convergence threshold, which defaults to 1.0 meter and is the distance standard for determining whether the target has been reached. Return value: optimal_path is the optimized 3D path point sequence, in the format [[x1,y1,z1],[x2,y2,z2],...]; The initial obstacle avoidance path is input into a reinforcement learning framework for iterative optimization to minimize flight latency and improve obstacle avoidance success rate. In multi-drone collaborative scenarios, a distributed path planning protocol is introduced, in which each drone periodically exchanges path information and adjusts its own path based on the conflict detection results, thereby avoiding group flight conflicts.

5. The UAV control method based on large-model intelligent driving according to claim 4, characterized in that, The target-oriented adaptive RRT algorithm includes: (a) Based on the current iteration number iter and the maximum iteration number max_iter, use the formula: P_goal=α+β×(1-exp(-γ×iter / max_iter)) The target bias probability P_goal is dynamically calculated, where α, β, and γ are the initial target bias probability, the dynamic adjustment coefficient, and the convergence control parameter, respectively. (b) Sample the target point directly with probability P_goal; otherwise, sample uniformly and randomly in free space C_free. (c) For each sampling point, find the node with the smallest distance in the RRT search tree, and apply the formula ρ_local = N_obstacle / N_total based on the local obstacle density: step_size=step_base×(1-ρ_local×λ) Adaptive adjustment of expansion step size; Where step_base is the base step size; ρ_local is the local obstacle density; λ is the step size adjustment coefficient; N_obstacle is the number of obstacle grids within the detection radius; and N_total is the total number of grids within the detection radius. (d) Expand new nodes from the nearest node toward the sampling point by step_size, and use an enhanced linear interpolation collision detection algorithm with a resolution of 0.1 m to detect path connectivity; (e) When the Euclidean distance between the new node and the target point is less than the convergence threshold and there is no collision when they are directly connected, the target node is added to the search tree to generate the initial path.

6. The UAV control method based on large-model intelligent driving according to claim 5, characterized in that: The parameters α, β, and γ take values ​​of 0.1, 0.2, and 2.0, respectively.

7. The UAV control method based on large-model intelligent driving according to claim 5, characterized in that, In step (d): detection points are generated by linear interpolation at intervals of 0.1 m between the start and end nodes, and a raster map query is performed on each detection point.

8. A UAV control system based on large-scale intelligent driving, characterized in that, include: The environmental data acquisition engine collects multimodal environmental data through multimodal sensors mounted on the drone; The data preprocessing engine is used to preprocess multimodal environmental data and run pre-trained Transformer architecture large models to perform fusion analysis on the preprocessed multimodal data and output environmental features. The large model inference engine is used to send the environmental features and multimodal environmental data to the semantic segmentation model based on the UNet segmentation architecture, and output pixel-level segmentation maps. The path planning engine is used to generate an initial obstacle avoidance path covering static obstacle avoidance from the output pixel-level segmentation map using the RRT algorithm. The initial obstacle avoidance path is optimized by combining a reinforcement learning framework to form an optimized path; the large Transformer architecture model sends path planning instructions to the drone based on the optimized path. Among them, the large model inference engine also uses the inference capabilities of the Transformer large model to jointly analyze multimodal environmental data, pixel-level segmentation maps, and path planning instructions, and output global optimization scheduling and real-time control decisions for UAV flight.

9. A computer device, characterized in that, include: memory for storing computer programs; A processor for executing the computer program to implement the steps of the UAV control method based on large model intelligent drive as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the UAV control method based on large model intelligent drive as described in any one of claims 1 to 7.

Images