Unmanned aerial vehicle scheduling method, device and equipment based on farmland lodging identification and medium

Abstract

The invention relates to the technical field of intelligent agriculture, and provides an unmanned aerial vehicle scheduling method and device based on farmland lodging recognition, equipment and a medium, and the method comprises the steps: obtaining the flight position data, flight attitude data and image data of an unmanned aerial vehicle; the image data comprises RGB image data, multispectral image data and near-infrared image data; inputting the flight position data, the flight attitude data and the image data into a farmland lodging recognition model to obtain a lodging recognition result; determining an initial waypoint based on the lodging identification result, and adjusting waypoint density distribution of the initial waypoint based on the area complexity of the lodging area in the initial waypoint; and performing unmanned aerial vehicle task scheduling based on the waypoint density distribution diagram. According to the method, the detection precision of the farmland lodging area is effectively improved by introducing multi-channel input of the RGB image, the multispectral image and the near-infrared image and combining position embedding of accurate geographic position information provided by the flight position data and the flight attitude data.

Description

Technical Field

[0001] The present invention relates to the field of intelligent agricultural technology, and in particular to a method, device, equipment and medium for dispatching unmanned aerial vehicles (UAVs) based on farmland lodging identification. Background Art

[0002] With the development of modern agriculture, the demand for more refined and intelligent agricultural production management is growing. Traditional crop management relies on manual inspections, which is time-consuming and labor-intensive, making it difficult to implement efficiently on large-scale or complex farmlands. This is particularly true for crop lodging monitoring, as manual inspections are often delayed and prone to missed or misjudged detections. Lodging can affect crop growth and yield, making timely detection and treatment crucial. To this end, drone technology is being used for agricultural monitoring. However, aerial photography alone is insufficient. Traditional image processing techniques lack precision, and they can't accurately identify lodging areas under complex lighting conditions. Furthermore, the safety of coordinated flight of multiple drones in complex farmland environments remains a challenge.

[0003] With the development of deep learning and computer vision, deep learning-based image processing methods are being applied to agricultural monitoring, improving the accuracy of lodging detection. However, large-scale, real-time farmland inspections still face key technical challenges, such as flight path planning, flight safety, and image data transmission and processing. While existing technologies have achieved some application in drone-based agricultural inspections, most systems still suffer from the following issues: first, low inspection efficiency, often requiring manual intervention; and second, insufficient accuracy and robustness in detecting lodging areas. Summary of the Invention

[0004] The present invention provides a drone scheduling method, device, equipment and medium based on farmland lodging identification, which is used to solve the defects of the existing technology in the field of drone agricultural inspection, such as low inspection efficiency, frequent need for manual intervention, and insufficient accuracy and robustness of lodging area detection.

[0005] The present invention provides a method for dispatching unmanned aerial vehicles (UAVs) based on farmland lodging identification, comprising the following steps: Acquire flight position data, flight attitude data, and image data of the UAV; the image data includes RGB image data, multispectral image data, and near-infrared image data; Inputting the flight position data, the flight attitude data, and the image data into a farmland lodging recognition model to obtain a lodging recognition result output by the farmland lodging recognition model; the lodging recognition result includes the coordinates of the lodging area and the severity result of the lodging area; Determining an initial waypoint based on the lodging identification result, and adjusting a waypoint density distribution of the initial waypoint based on the regional complexity of the lodging area in the initial waypoint; Based on the waypoint density distribution map, UAV mission scheduling is performed.

[0006] According to a drone scheduling method based on farmland lodging recognition provided by the present invention, the farmland lodging recognition model is obtained by replacing all standard convolutions in the U-Net model with dilated convolutions and replacing the attention mechanism in the U-Net model with a channel attention mechanism and a spatial attention mechanism.

[0007] According to a drone scheduling method based on farmland lodging recognition provided by the present invention, the training steps of the farmland lodging recognition model include: Obtain sample multi-sensor data, the label classification result of the sample multi-sensor data, and the regression label; the sample multi-sensor data includes sample flight position data, sample flight attitude data, and sample image data; Obtain an initial model; the initial model includes an initial encoding layer, and an initial segmentation branch and an initial regression branch respectively connected to the initial encoding layer; Input the sample multi-sensor data into the initial model to obtain a predicted probability map output by the initial model; Based on the initial segmentation branch, perform segmentation recognition on the predicted probability map to obtain a segmentation prediction result of the sample multi-sensor data; Based on the initial regression branch, perform regression prediction on the predicted probability map to obtain a regression prediction result of the sample multi-sensor data; Based on the segmentation prediction result and the label classification result, as well as the regression prediction result and the regression label, perform parameter iteration on the initial model to obtain the farmland lodging recognition model.

[0008] According to a drone scheduling method based on farmland lodging recognition provided by the present invention, the drone task scheduling based on the waypoint density distribution map includes: Based on the lodging area severity result and the waypoint density distribution map, perform drone task scheduling.

[0009] According to a drone scheduling method based on farmland lodging recognition provided by the present invention, the drone task scheduling based on the lodging area severity result and the waypoint density distribution map includes: Determine the task priority corresponding to the lodging area severity result, as well as the flight duration and energy consumption of the drone; Based on the flight duration, the energy consumption, and the task priority, determine the task price; Based on the task prices of each drone, determine the task allocation target of the waypoint density map, and construct a drone scheduling model with the minimum task allocation target as the goal; Based on the drone scheduling model, perform drone task scheduling.

[0010] A drone scheduling method based on farmland lodging recognition provided by the present invention, the step of determining the regional complexity includes: Obtain the slope of the lodging area, the obstacle density, and the severity result of the lodging area; Based on the slope, the obstacle density, and the severity result of the lodging area, determine the regional complexity.

[0011] A drone scheduling method based on farmland lodging recognition provided by the present invention, adjusting the waypoint density distribution of the initial waypoints based on the regional complexity of the lodging areas in the initial waypoints, includes: Based on the regional complexity, determine the waypoint density; Based on the waypoint density, adjust the waypoint density distribution of the initial waypoints.

[0012] The present invention also provides a drone scheduling system based on farmland lodging recognition, including the following units: An acquisition unit, configured to acquire the flight position data, flight attitude data, and image data of the drone; the image data includes RGB image data, multispectral image data, and near-infrared image data; An input unit, configured to input the flight position data, the flight attitude data, and the image data into a farmland lodging recognition model to obtain a lodging recognition result output by the farmland lodging recognition model; the lodging recognition result includes the lodging area coordinates and the severity result of the lodging area; An adjustment unit, configured to determine initial waypoints based on the lodging recognition result, and adjust the waypoint density distribution of the initial waypoints based on the regional complexity of the lodging areas in the initial waypoints; A drone scheduling unit, configured to perform drone task scheduling based on the waypoint density distribution map.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it implements the drone scheduling method based on farmland lodging recognition as described in any one of the above.

[0014] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the drone scheduling method based on farmland lodging recognition as described in any one of the above.

[0015] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the drone scheduling method based on farmland lodging recognition as described in any one of the above.

[0016] The drone scheduling method, device, equipment and medium based on farmland lodging recognition provided by the present invention obtain the flight position data, flight attitude data and image data of the drone; the image data includes RGB image data, multispectral image data and near-infrared image data; then, the flight position data, flight attitude data and image data are input into the farmland lodging recognition model to obtain the lodging recognition result output by the farmland lodging recognition model; the lodging recognition result includes the lodging area coordinates and the lodging area severity result; then, based on the lodging recognition result, the initial waypoints are determined, and based on the regional complexity of the lodging area in the initial waypoints, the waypoint density distribution of the initial waypoints is adjusted. Finally, based on the waypoint density distribution map, drone task scheduling is performed. This method effectively improves the detection accuracy of the farmland lodging area by introducing multi-channel input of RGB images, multispectral images and near-infrared images and position embedding that combines the accurate geographical location information provided by the flight position data and flight attitude data; moreover, the combination of these multi-source information not only helps the model extract physical features in more dimensions, but also enhances the geographical positioning ability of the lodging area through position embedding, significantly improving the positioning accuracy of the lodging area, especially in scenarios with complex terrain or limited perspectives, and the effect is more significant. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a schematic flowchart of the drone scheduling method based on farmland lodging recognition provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the waypoint densification algorithm provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the drone scheduling hardware system based on farmland lodging recognition provided by the present invention.

[0021] Figure 4 It is a schematic diagram of the drone scheduling software system based on farmland lodging recognition provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the farmland lodging recognition model provided by the present invention.

[0023] Figure 6 It is an architecture diagram of multi-drone collaborative scheduling provided by the present invention.

[0024] Figure 7It is a schematic diagram of the ground station system provided by the present invention.

[0025] Figure 8 It is a schematic structural diagram of the UAV scheduling system based on farmland lodging recognition provided by the present invention.

[0026] Figure 9 It is a schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0027] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without making creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0028] The present invention provides a UAV scheduling method based on farmland lodging recognition. Figure 1 It is a schematic flow diagram of the UAV scheduling method based on farmland lodging recognition provided by the present invention. As Figure 1 shown, the method includes step 110, step 120, step 130 and step 140.

[0029] Step 110, obtaining the flight position data, flight attitude data and image data of the UAV; the image data includes RGB image data, multispectral image data and near-infrared image data.

[0030] Specifically, the flight position data, flight attitude data and image data of the UAV can be obtained. Among them, the flight position data of the UAV can be obtained based on the RTK-GPS (Real-Time Kinematic Global Positioning System) system equipped on the UAV, and the flight attitude data can be obtained based on the IMU (Inertial Measurement Unit) equipped on the UAV. The embodiments of the present invention do not make specific limitations on this.

[0031] Here, the image data can include RGB image data, multispectral image data and near-infrared image data. The RGB image data, multispectral image data and near-infrared image data can be collected by an RGB camera, a multispectral camera and an infrared camera respectively.

[0032] In the UAV image acquisition and precise positioning, in order to eliminate the influence of various factors such as camera perspective, flight altitude, shooting angle, etc. on the extraction of the lodging area and coordinate mapping, on the one hand, a variety of sensors are used, including infrared cameras, multispectral cameras and RGB cameras to collect data, and on the other hand, RTK-GPS and IMU systems are used to ensure that each photo contains accurate geographical location information. In this way, the geographical information of each picture can be automatically corresponding to the image content. Further, RTK-GPS is used to interact with the ground base station to correct the errors in the satellite signal in real time. At the same time, when the RTK-GPS signal fails or is unstable, IMU is used to compensate for the positioning errors caused by external factors, improving the positioning accuracy and flight stability.

[0033] It should be noted that during the flight, multi-angle shooting and high-resolution cameras are used for image acquisition to ensure that each image is accurately corresponding to the actual geographical coordinates.

[0034] It should be noted that real-time data is transmitted to the UAV using a low-latency 5G communication network to ensure that the data transmission delay between the UAV and the ground operation center is less than 50 ms.

[0035] Step 120, input the flight position data, the flight attitude data and the image data into the farmland lodging recognition model to obtain the lodging recognition result output by the farmland lodging recognition model; the lodging recognition result includes the lodging area coordinates and the lodging area severity result.

[0036] Specifically, the flight position data, the flight attitude data and the image data can be input into the farmland lodging recognition model to obtain the lodging recognition result output by the farmland lodging recognition model. The formula is as follows: Among them, represents the lodging recognition result, are RGB image data, multispectral image data and near-infrared image data respectively, represents the deviation between the RGB image data, the multispectral image data and the near-infrared image data, is the parameter of the farmland lodging recognition model, which is a deep learning function for lodging area extraction.

[0037] Here, the farmland lodging recognition model is obtained by improving the traditional U-Net grid to meet the output requirements of the embodiments of the present invention.

[0038] In the input layer of the farmland lodging recognition model, the improved network supports multi-channel input, including RGB, multi-spectral, and near-infrared images. At the same time, position embedding is added. The RTK-GPS and IMU sensors equipped on the above-mentioned drones can provide accurate geographical location information, and this location information can be input into the network together with image features to help the model improve the positioning accuracy of the lodging area.

[0039] Specifically, convolutional operations are performed on each image data channel to extract different physical features. At the same time, position embedding is added. The RTK-GPS signal provides location information, and the IMU provides the current attitude information of the drone. These information are mapped into a vector for input into the network.

[0040] Furthermore, in the encoder part, in order to enhance the edge detail restoration ability of the lodging area, deeper convolutional layers are used in the convolutional layers of the network, and dilated convolution is introduced to expand the receptive field to capture more extensive context information. To further help the model focus on the detailed parts of the lodging area, a spatial attention mechanism and a channel attention mechanism are added in the embodiments of the present invention to improve the detail restoration ability of the model. The spatial attention mechanism mainly calculates the spatial attention map of the feature map, adjusts the attention area of the feature, and thus calculates the importance of each pixel. At the same time, a channel attention mechanism is added to focus on the channel dimension of the feature map, adjust the weights of different channels, so that the model can focus on more important feature channels, thereby improving the learning ability of key features.

[0041] In the output stage of the improved U-Net network model, it is not only necessary to identify and segment the lodging area in the image, but also to predict the severity or degree of the lodging area. That is, the lodging recognition result includes the coordinates of the lodging area and the severity result of the lodging area. Therefore, a multi-task learning framework can be adopted to complete the segmentation task and regression task of the lodging area respectively.

[0042] In addition, considering the computing power requirements of the on-board computer carried by the drone, the network is further improved. To ensure that the model can run efficiently on edge devices, it is necessary to reduce the number of model parameters and the amount of computation. Common methods include depthwise separable convolution, pruning, quantization, etc. In the embodiments of the present invention, depthwise separable convolution operations are added to the network to achieve this effect. This is mainly used when convolutional operations are used to extract multi-channel input features, and the standard convolution is decomposed into depthwise convolution and pointwise convolution to reduce the amount of computation.

[0043] Step 130, based on the lodging recognition result, determine the initial waypoints, and adjust the waypoint density distribution of the initial waypoints based on the regional complexity of the lodging area in the initial waypoints.

[0044] Specifically, after obtaining the lodging recognition result, the initial waypoints can be determined based on the lodging recognition result. Considering that the flight route of this preliminary waypoint does not take into account terrain changes and regional complexity, it is necessary to intelligently adjust the waypoint density according to the actual situation of the lodging area.

[0045] Furthermore, the waypoint density distribution of the initial waypoints can be adjusted according to the regional complexity of the lodging area in the initial waypoints.

[0046] Step 140, perform UAV mission scheduling based on the waypoint density distribution map.

[0047] Specifically, after obtaining the waypoint density distribution map, UAV mission scheduling can be performed based on the waypoint density distribution map.

[0048] Here, UAV mission scheduling can combine the lodging area severity result and the waypoint density distribution map.

[0049] The method provided by the embodiments of the present invention obtains the flight position data, flight attitude data, and image data of the UAV; the image data includes RGB image data, multispectral image data, and near-infrared image data; then, the flight position data, flight attitude data, and image data are input into the farmland lodging recognition model to obtain the lodging recognition result output by the farmland lodging recognition model; the lodging recognition result includes the lodging area coordinates and the lodging area severity result; then, based on the lodging recognition result, the initial waypoints are determined, and the waypoint density distribution of the initial waypoints is adjusted based on the regional complexity of the lodging area in the initial waypoints. Finally, UAV mission scheduling is performed based on the waypoint density distribution map. This method effectively improves the detection accuracy of the lodging area by introducing multi-channel input of RGB images, multispectral images, and near-infrared images, as well as position embedding that combines the accurate geographical location information provided by the flight position data and flight attitude data; moreover, the combination of these multi-source information not only helps the model extract physical features in more dimensions, but also enhances the geographical positioning ability of the lodging area through position embedding, significantly improving the positioning accuracy of the lodging area, especially in scenarios with complex terrain or limited perspectives, the effect is more significant.

[0050] Based on the above embodiments, the farmland lodging recognition model is obtained by replacing all standard convolutions in the U-Net model with dilated convolutions and replacing the attention mechanism in the U-Net model with a channel attention mechanism and a spatial attention mechanism.

[0051] Specifically, the farmland lodging recognition model is obtained by replacing all standard convolutions in the U-Net model with dilated convolutions and replacing the attention mechanism in the U-Net model with a channel attention mechanism and a spatial attention mechanism.

[0052] In the method provided by the embodiments of the present invention, the introduction of the spatial attention mechanism and the channel attention mechanism enables the model to adaptively adjust the attention area and feature channels, thereby enhancing the model's learning ability for key areas and features. The spatial attention mechanism effectively helps the network focus on the specific spatial positions of the lodging areas, and the channel attention mechanism strengthens the learning of important feature channels, enabling the model to adaptively adjust in complex scenarios and improving the effects of the segmentation and regression tasks of the lodging areas.

[0053] Based on the above embodiments, the training steps of the farmland lodging recognition model include: Step 210, obtaining sample multi-sensor data, the label classification result of the sample multi-sensor data, and the regression label; the sample multi-sensor data includes sample flight position data, sample flight attitude data, and sample image data; Step 220, obtaining an initial model; the initial model includes an initial encoding layer, and an initial segmentation branch and an initial regression branch respectively connected to the initial encoding layer; Step 230, inputting the sample multi-sensor data into the initial model to obtain a predicted probability map output by the initial model; Step 240, based on the initial segmentation branch, performing segmentation recognition on the predicted probability map to obtain a segmentation prediction result of the sample multi-sensor data; Step 250, based on the initial regression branch, performing regression prediction on the predicted probability map to obtain a regression prediction result of the sample multi-sensor data; Step 260, based on the segmentation prediction result and the label classification result, and the regression prediction result and the regression label, performing parameter iteration on the initial model to obtain the farmland lodging recognition model.

[0054] Specifically, in order to better obtain the farmland lodging recognition model, it can be trained based on the following steps: First, sample multi-sensor data, the label classification result of the sample multi-sensor data, and the regression label can be obtained, where the sample multi-sensor data includes sample flight position data, sample flight attitude data, and sample image data.

[0055] Then, an initial model is obtained. The initial model includes an initial encoding layer, and an initial segmentation branch and an initial regression branch respectively connected to the initial encoding layer. Among them, the initial encoding layer can be an improved U-Net model.

[0056] Here, the parameters of the initial model can be preset or randomly generated, and the embodiments of the present invention do not make specific limitations on this.

[0057] After obtaining the initial model, the sample multi-sensor data can be input into the initial model to obtain the predicted probability map output by the initial model.

[0058] Then, based on the initial segmentation branch, the predicted probability map is segmented and recognized to obtain the segmentation prediction result of the sample multi-sensor data.

[0059] Furthermore, based on the initial regression branch, regression prediction is performed on the predicted probability map to obtain the regression prediction result of the sample multi-sensor data.

[0060] Finally, based on the segmentation prediction result and the label classification result, as well as the regression prediction result and the regression label, the target loss is determined, and the parameters of the initial model are iterated based on the target loss. The initial model after parameter iteration is used as the farmland lodging recognition model.

[0061] Among them, the formula for the target loss is as follows: Among them, represents the target loss, is the loss of the segmentation task, is the loss of the regression task, and are weighting coefficients that control the contributions of the segmentation task and the regression task to the target loss.

[0062] In the actual operation process, the flight height of the drone for data collection directly affects the proportion of the lodging area in the image. When collecting data at a height of 10 - 15M, the proportion of the lodging area and the non-lodging area in the image is unbalanced. Therefore, the cross-entropy loss is used here to represent the loss function of the segmentation task. In the regression branch for predicting the severity or degree of the lodging area, the mean squared error loss function is used to measure the difference between the predicted value (regression prediction result) and the true value (regression label).

[0063] For the overall loss function, generally speaking, the selected weighting coefficient can be adjusted through cross-validation or set empirically according to the importance of the task and the characteristics of the data.

[0064] In the related technology, in the field of drone agricultural inspection, the flight path planning is not intelligent enough, resulting in incomplete coverage of some areas.

[0065] Based on the above embodiments, step 140 includes: Step 141, performing drone task scheduling based on the lodging area severity result and the waypoint density distribution map.

[0066] Specifically, based on the severity results of the lodging area and the waypoint density distribution map, UAV task scheduling is performed. This operation corresponds to the task scheduling module. The main input of the task scheduling module is the waypoint density distribution map processed by the densified waypoint algorithm and the lodging area severity results of each waypoint area after being processed by the farmland lodging identification model. The output is a single task group executed by each UAV. The core algorithm of this module is the auction algorithm.

[0067] Based on the above embodiment, step 141 includes: Step 1411, determining the task priority corresponding to the severity result of the fallen area, as well as the flight time and energy consumption of the UAV; Step 1412: determining a task price based on the flight duration, the energy consumption, and the task priority; Step 1413: Determine the task allocation target of the waypoint density map based on the task quotations of each drone, and construct a drone scheduling model with the goal of minimizing the task allocation target. Step 1414: Perform UAV task scheduling based on the UAV scheduling model.

[0068] Specifically, the improved auction algorithm in this task is mainly reflected in three stages: bidding, auctioning, and execution. In the bidding stage: each drone in the multi-UAV system will evaluate the "bid" for executing the task based on the task priority of the current task (the task priority corresponding to the severity result of the fallen area) as well as its own flight time and energy consumption (such as remaining power, distance, task timeliness, etc.). The drone's bid is defined here as a weighted cost function that includes three factors: flight time, energy consumption, and task priority. In the auction stage: for a given single waypoint or regional waypoint task, the auction algorithm selects the drone with the lowest bid to execute the waypoint task based on the bids of all drones. The final execution stage: a single drone executes the waypoint task obtained through the auction and provides real-time feedback.

[0069] For the improved auction algorithm, the loss function is defined as follows: T ={ p 1, p 2,…, pn} and multiple drones U ={ U 1, U 2,…, Um}, the task allocation goal is to minimize the overall cost C , that is, the drone scheduling model is: Among them, 1 is the indicator function, indicating that the drone Ui Whether assigned to a taskpj 。

[0070] If some waypoint tasks are not assigned, enter the next round of bidding. The selected drones start to execute the assigned tasks, and the drones update their own states (position, power, task completion progress, etc.) during the task execution. The system feeds back to the drones according to the execution states of the drones (whether the task is completed, power consumption, etc.) and adjusts according to the progress of the task, and the entire system is thus completed.

[0071] In the whole system, the lodging area identified by the improved U-net network is used as the first-level identification, and the multi-drone system is used to take further detailed photos of the lodging area, increasing the accuracy of the identification of the farmland lodging area and providing effective support for the development of agriculture.

[0072] The method provided by the embodiment of the present invention can dynamically allocate tasks according to multiple factors such as the task priority, remaining power, and flight ability of the drones by introducing multi-drone collaborative operation and auction algorithm, ensuring that each waypoint task can be allocated and executed among the most suitable drones. The multi-drone collaborative optimization algorithm can greatly improve the execution efficiency of the task and the accuracy of task completion, and can adapt to the dynamic task scheduling with different numbers and capabilities of drones participating, that is, automatically allocate tasks through the improved auction algorithm to ensure that the lodging area is fully covered and avoid blind spots.

[0073] Based on the above embodiments, the determining step of the regional complexity includes: Step 310, obtaining the slope, obstacle density of the lodging area and the result of the severity of the lodging area; Step 320, determining the regional complexity based on the slope, the obstacle density and the result of the severity of the lodging area.

[0074] Specifically, first define a regional complexity function , used to quantify the complexity of a region, and the factors of complexity include the following items. Slope: The degree of undulation of the terrain. Areas with larger slopes require more waypoints to ensure accuracy. Obstacle density: The quantity and distribution of obstacles (such as trees, rocks, etc.) in the region. Areas with dense obstacles require more waypoints to avoid collisions. Lodging severity: The severity of the lodging area. Areas with severe lodging require denser waypoints for detailed monitoring.

[0075] Among them, slope: The degree of undulation of the terrain. Areas with larger slopes require more waypoints to ensure accuracy, and the slope The formula for is: Among them, ZRepresents the height value of the terrain, and are respectively the gradients of the terrain in the x and y directions.

[0076] Obstacle density: According to the map or sensor data, count the number of obstacles in each small area and calculate the obstacle density. The formula for the obstacle density is: Lodging severity : That is, the severity of the lodging area obtained from the output of the previous network. By integrating these factors, the regional complexity function can be expressed as: where α, and are weight coefficients used to adjust the influence of different factors.

[0077] Based on the above embodiments, in step 130, adjusting the waypoint density distribution of the initial waypoints based on the regional complexity of the lodging areas in the initial waypoints includes: Step 131, determining the waypoint density based on the regional complexity; Step 132, adjusting the waypoint density distribution of the initial waypoints based on the waypoint density.

[0078] Specifically, Figure 2 is a schematic diagram of the waypoint densification algorithm provided by the present invention. As shown in Figure 2 , the purpose of the waypoint densification algorithm is to dynamically adjust the waypoint density according to the environmental information of the multi - UAV system sensors, the coordinates of the lodging areas, the regional complexity of the waypoint areas, and the lodging area severity results, and obtain a waypoint distribution map, thereby optimizing the flight efficiency and accuracy of the UAVs in the mission. This method ensures that the waypoint density adapts to different geographical features and mission requirements by analyzing the terrain, lodging severity, and the distribution of obstacles. The embodiments of the present invention are improved based on the waypoint densification algorithm.

[0079] The waypoint density is determined by a densification function in the following specific form: where, fIt is a non-linear mapping function. Here, the sigmoid function is used. Through such a mapping relationship, more waypoints will be obtained in complex areas (such as areas with large slopes, many obstacles, and severe lodging). The above-mentioned waypoint densification algorithm can achieve that for complex areas such as areas with large slopes or severe lodging, the waypoint densification algorithm will automatically reduce the waypoint spacing, thereby improving the flight accuracy and mission completion rate of the unmanned aerial vehicle. For simple areas, the waypoint density will be appropriately reduced to improve flight efficiency and save time and resources.

[0080] The method provided by the embodiment of the present invention. This layer calculates the complexity of each area according to the input data (slope, obstacle density, lodging severity). Then, the complexity value of the area is passed into the densification function and mapped through a sigmoid function. In this way, a higher waypoint density will be obtained in high-complexity areas. According to the calculated waypoint density, the waypoint distribution is dynamically adjusted in different areas. For complex areas (such as areas with large slopes, many obstacles, or severe lodging), the waypoint density is increased to ensure high-precision monitoring. For simple areas, the waypoint density is reduced to improve flight efficiency. Finally, a waypoint distribution map required for the mission, the adjusted waypoint spacing, and the flight path are optimized. Provide a more efficient and accurate flight route, reduce resource waste, and ensure the efficient completion of the mission.

[0081] Based on any of the above embodiments, Figure 3 is a schematic diagram of the unmanned aerial vehicle scheduling hardware system based on farmland lodging recognition provided by the present invention, Figure 4 is a schematic diagram of the unmanned aerial vehicle scheduling software system based on farmland lodging recognition provided by the present invention. As Figure 3 、 Figure 4 shown, the unmanned aerial vehicle and the NVIDIA on-board computer platform are integrated to build a subsequent algorithm operation platform. At the same time, each unmanned aerial vehicle is connected to the ground station through a low-latency signal. The improved U-Net network runs on this hardware system, while the task scheduling module, that is, the auction algorithm, runs on the ground station system. In large farmlands, a multi-unmanned aerial vehicle system is adopted. The task extraction module used by a single unmanned aerial vehicle runs in parallel on multiple unmanned aerial vehicle systems and jointly inputs the improved U-Net network module to output an overall waypoint distribution map, and then the multi-unmanned aerial vehicle scheduling system performs route allocation and scheduling.

[0082] Specifically, the hardware platform of the UAV system mainly includes the integration of the following modules. (1) Data acquisition module, which consists of a near-infrared camera, a multispectral camera, and an RGB camera. The main function of this module is to provide image data for the entire system in three channels, laying the foundation for subsequent image recognition. (2) Positioning module, which consists of a GPS positioning chip and an IMU high-precision sensor. The main function of this module is to provide position and attitude information for the image data collected by the data acquisition module, facilitating the processing by the subsequent improved U-net network. (3) Main control center module, which consists of a NVIDIA chip and its corresponding peripherals. This chip controls the data transmission to the ground station and runs the core algorithm of the entire system, providing powerful computing power for the entire system to ensure its stable operation.

[0083] Based on any of the above embodiments, Figure 5 is a schematic diagram of the farmland lodging recognition model provided by the present invention. As Figure 5 shown, in the input layer, vector integration is carried out by means of position information embedding. The input size of the RGB image (L_rgb) is H × W ×3 (height, width, number of channels), the input image size of the multispectral (L_ms) is H × W ×4 (height, width, number of spectral channels), and the input size of the near-infrared image (L_ir) is H × W ×1 (height, width, single-channel grayscale image). The position information includes GPS information, GPSx , GPSy which are longitude and latitude coordinates, being scalars with a dimension of 1. IMU information: attitude information (such as angle θ ), assumed to be a scalar with a dimension of 1. The GPS and IMU information are mapped to a position embedding vector P , with a dimension of Dp . Then, broadcast summation is performed on the above vectors. Then, the standard convolution in the original U-Net network becomes dilated convolution with a dilation rate of d, and the extracted feature vector is output. Channel attention and spatial attention mechanisms are added to the original U-Net network to make the division details of the lodging area more accurate. Finally, multi-task output is performed to output the results of image segmentation and image regression.

[0084] Based on any of the above embodiments, Figure 6 is an architecture diagram of the multi-UAV collaborative scheduling provided by the present invention. As Figure 6As shown, the task generation module generates a waypoint distribution map of the farmland area based on the improved U-Net network. The priority of each waypoint (such as the lodging degree) is also output by the U-Net and used to determine the urgency of the task. Each waypoint contains its position coordinates and priority at the same time, indicating the priority execution order of the task at that position. Then, the drone gives a waypoint quotation according to its own power, distance, carried equipment, its own attitude and other information. Subsequently, the task scheduling center starts an auction bid to select the most suitable drone to execute the current waypoint task. When auctioning, a single waypoint can be auctioned, or a flight route or flight surface composed of multiple waypoints can be auctioned. The scheduling center schedules multiple drones to work collaboratively, automatically allocates inspection tasks, coordinates flight paths, ensures the efficient execution of tasks and avoids repeated inspections.

[0085] Figure 7 is a schematic diagram of the ground station system provided by the present invention, as Figure 7 shown, the ground station system is used for route visualization, visualization of the severity of the lodging area, and visualization of the drone status, etc. During the flight, image data and lodging area information are quickly transmitted to the operation center, and at the same time, the data is processed on the on-board computer platform to ensure real-time feedback and decision support. The detection results of the lodging area are visually processed through the ground station system to generate detailed area coordinates and lodging information, and are fed back to the ground operators. The operators can view real-time monitoring, waypoint information and crop status through the mobile application, and adjust subsequent inspection tasks and path planning in a timely manner.

[0086] The multi-drone system based on farmland lodging recognition includes at least one drone equipped with an RTK-GPS system, an IMU system and a high-resolution camera for image acquisition and recording the flight position through the positioning system; an image processing module for preprocessing the images collected by the drone and detecting and segmenting the lodging area using the improved U-Net network; a task scheduling system for generating a waypoint distribution map according to the lodging area recognition result and allocating tasks to multiple drones through an auction algorithm; a communication module for using 5G communication technology to transmit drone data to the ground operation center in real time, and the ground operation center is used for processing, visualizing the data and feeding back and adjusting the tasks.

[0087] In summary, the system integrates drones, NVIDIA on-board computer platforms, and ground station communication systems, combines image processing, deep learning, and multi-drone collaborative work to achieve efficient and accurate identification of lodging areas and task execution. The specific implementation process includes the following steps. Drone image acquisition and precise positioning: The drone is equipped with a high-precision RTK-GPS and IMU system to collect images in real-time and perform precise flight positioning to ensure that each image corresponds to the actual geographical coordinates. Image preprocessing and lodging area extraction: The images collected by the drone in real-time are preprocessed by the on-board computer, and an improved U-Net network is used for automatic identification of the lodging area, and multi-spectral and infrared images are combined to enhance the recognition accuracy, especially under adverse lighting conditions. Task allocation and multi-drone collaboration: According to the real-time recognition results of the lodging area, a waypoint distribution map is generated by the densified waypoint algorithm system, and task scheduling is achieved through the auction algorithm. The drones bid for tasks according to their respective battery levels, distances, equipment, etc., and the dispatching center assigns tasks according to the priority and drone status to ensure efficient execution. Real-time data transmission and processing: The system realizes real-time data transmission between the drone and the ground operation center through 5G low-latency communication technology. During the flight, the image data and lodging area information are quickly transmitted and processed to generate visual area coordinates and lodging information for the operator to adjust subsequent tasks and path planning.

[0088] Through a highly integrated hardware platform, advanced image processing technology, and intelligent scheduling system, the present invention realizes precise detection of farmland lodging areas and collaborative task execution of drones, providing a new solution for automated inspection and refined management in the agricultural field.

[0089] The following describes the drone scheduling system provided by the present invention for farmland lodging recognition. The drone scheduling system for farmland lodging recognition described below can be correspondingly referred to with the drone scheduling method for farmland lodging recognition described above.

[0090] Based on any of the above embodiments, the present invention provides a drone scheduling system for farmland lodging recognition, Figure 8 which is a schematic structural diagram of the drone scheduling system provided by the present invention for farmland lodging recognition. As Figure 8 shown, the system includes: An acquisition unit 810, configured to acquire flight position data, flight attitude data, and image data of the drone; the image data includes RGB image data, multi-spectral image data, and near-infrared image data; An input unit 820, configured to input the flight position data, the flight attitude data, and the image data into a farmland lodging recognition model to obtain a lodging recognition result output by the farmland lodging recognition model; the lodging recognition result includes lodging area coordinates and a lodging area severity result; An adjustment unit 830, configured to determine an initial waypoint based on the lodging recognition result, and adjust the waypoint density distribution of the initial waypoint based on the regional complexity of the lodging area in the initial waypoint; A UAV scheduling unit 840, configured to perform UAV task scheduling based on the waypoint density distribution map.

[0091] The system provided by the embodiment of the present invention obtains the flight position data, flight attitude data and image data of the UAV; the image data includes RGB image data, multispectral image data and near-infrared image data; then inputs the flight position data, flight attitude data and image data into the farmland lodging recognition model to obtain the lodging recognition result output by the farmland lodging recognition model; the lodging recognition result includes the lodging area coordinates and the lodging area severity result; then, based on the lodging recognition result, determine the initial waypoint, and adjust the waypoint density distribution of the initial waypoint based on the regional complexity of the lodging area in the initial waypoint, and finally, perform UAV task scheduling based on the waypoint density distribution map. This method effectively improves the detection accuracy of the lodging area by introducing the multi-channel input of RGB images, multispectral images, and near-infrared images and the position embedding that combines the accurate geographical location information provided by the flight position data and flight attitude data; moreover, the combination of these multi-source information not only helps the model extract physical features in more dimensions, but also enhances the geographical positioning ability of the lodging area through position embedding, significantly improving the positioning accuracy of the lodging area, especially in scenarios with complex terrain or limited perspectives, the effect is more significant.

[0092] Based on any of the above embodiments, the farmland lodging recognition model is obtained by replacing all standard convolutions in the U-Net model with dilated convolutions and replacing the attention mechanism in the U-Net model with a channel attention mechanism and a spatial attention mechanism.

[0093] Based on any of the above embodiments, it further includes a training unit, and the training unit is specifically configured to: Obtain sample multi-sensor data, the label classification result of the sample multi-sensor data and the regression label; the sample multi-sensor data includes sample flight position data, sample flight attitude data and sample image data; Obtain an initial model; the initial model includes an initial encoding layer, and an initial segmentation branch and an initial regression branch respectively connected to the initial encoding layer; Input the sample multi-sensor data into the initial model to obtain a predicted probability map output by the initial model; Based on the initial segmentation branch, perform segmentation recognition on the predicted probability map to obtain the segmentation prediction result of the sample multi-sensor data; Based on the initial regression branch, perform regression prediction on the predicted probability map to obtain the regression prediction result of the sample multi-sensor data; Based on the segmentation prediction result, the label classification result, as well as the regression prediction result and the regression label, perform parameter iteration on the initial model to obtain the farmland lodging recognition model.

[0094] Based on any of the above embodiments, the drone scheduling unit 840 specifically includes: A drone scheduling subunit, configured to perform drone task scheduling based on the lodging area severity result and the waypoint density distribution map.

[0095] Based on any of the above embodiments, the drone task scheduling subunit specifically is used for: Determine the task priority corresponding to the lodging area severity result, as well as the flight duration and energy consumption of the drone; Based on the flight duration, the energy consumption, and the task priority, determine the task quotation; Based on the task quotations of each drone, determine the task assignment target of the waypoint density map, and construct a drone scheduling model with the goal of minimizing the task assignment target; Based on the drone scheduling model, perform drone task scheduling.

[0096] Based on any of the above embodiments, it further includes a region complexity determination unit, and the region complexity determination unit specifically is used for: Obtain the slope of the lodging area, the obstacle density, and the lodging area severity result; Based on the slope, the obstacle density, and the lodging area severity result, determine the region complexity.

[0097] Based on any of the above embodiments, the adjustment unit 830 specifically is used for: Based on the region complexity, determine the waypoint density; Based on the waypoint density, adjust the waypoint density distribution of the initial waypoints.

[0098] Figure 9 It is a schematic structural diagram of the electronic device provided by the present invention, as Figure 9As shown in the figure, the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communication bus 940. Among them, the processor 910, the communications interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logical instructions in the memory 930 to execute the UAV scheduling method based on farmland lodging recognition. The method includes: obtaining the flight position data, flight attitude data, and image data of the UAV; the image data includes RGB image data, multispectral image data, and near-infrared image data; inputting the flight position data, the flight attitude data, and the image data into a farmland lodging recognition model to obtain the lodging recognition result output by the farmland lodging recognition model; the lodging recognition result includes the lodging area coordinates and the lodging area severity result; based on the lodging recognition result, determining an initial waypoint, and adjusting the waypoint density distribution of the initial waypoint based on the regional complexity of the lodging area in the initial waypoint; based on the waypoint density distribution map, performing UAV task scheduling.

[0099] In addition, when the logical instructions in the above-mentioned memory 930 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0100] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the drone scheduling method based on farmland lodging recognition provided by the above-mentioned various methods. The method includes: obtaining the flight position data, flight attitude data, and image data of the drone; the image data includes RGB image data, multispectral image data, and near-infrared image data; inputting the flight position data, flight attitude data, and image data into a farmland lodging recognition model to obtain a lodging recognition result output by the farmland lodging recognition model; the lodging recognition result includes the lodging area coordinates and the lodging area severity result; based on the lodging recognition result, determining an initial waypoint, and adjusting the waypoint density distribution of the initial waypoint based on the area complexity of the lodging area in the initial waypoint; based on the waypoint density distribution map, performing drone task scheduling.

[0101] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the drone scheduling method based on farmland lodging recognition provided by the above-mentioned various methods. The method includes: obtaining the flight position data, flight attitude data, and image data of the drone; the image data includes RGB image data, multispectral image data, and near-infrared image data; inputting the flight position data, flight attitude data, and image data into a farmland lodging recognition model to obtain a lodging recognition result output by the farmland lodging recognition model; the lodging recognition result includes the lodging area coordinates and the lodging area severity result; based on the lodging recognition result, determining an initial waypoint, and adjusting the waypoint density distribution of the initial waypoint based on the area complexity of the lodging area in the initial waypoint; based on the waypoint density distribution map, performing drone task scheduling.

[0102] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0103] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention .

Claims

1. A method for scheduling unmanned aerial vehicles based on the recognition of farmland lodging, characterized in that, Including: Obtain the flight position data, flight attitude data, and image data of the drone; the image data includes RGB image data, multispectral image data, and near-infrared image data; Input the flight position data, the flight attitude data, and the image data into the farmland lodging recognition model to obtain the lodging recognition result output by the farmland lodging recognition model; the lodging recognition result includes the lodging area coordinates and the lodging area severity result; Based on the lodging recognition result, determine the initial waypoints, and adjust the waypoint density distribution of the initial waypoints based on the area complexity of the lodging area in the initial waypoints; Based on the waypoint density distribution map, perform drone task scheduling.

2. The drone scheduling method based on farmland lodging recognition according to claim 1, wherein, The farmland lodging recognition model is obtained by replacing all standard convolutions in the U-Net model with dilated convolutions and replacing the attention mechanism in the U-Net model with a channel attention mechanism and a spatial attention mechanism.

3. The drone scheduling method based on farmland lodging recognition according to claim 2, wherein, The training steps of the farmland lodging recognition model include: Obtain sample multi-sensor data, the label classification result of the sample multi-sensor data, and the regression label; the sample multi-sensor data includes sample flight position data, sample flight attitude data, and sample image data; Obtain an initial model; the initial model includes an initial encoding layer, and an initial segmentation branch and an initial regression branch respectively connected to the initial encoding layer; Input the sample multi-sensor data into the initial model to obtain the predicted probability map output by the initial model; Based on the initial segmentation branch, perform segmentation recognition on the predicted probability map to obtain the segmentation prediction result of the sample multi-sensor data; Based on the initial regression branch, perform regression prediction on the predicted probability map to obtain the regression prediction result of the sample multi-sensor data; Based on the segmentation prediction result and the label classification result, and the regression prediction result and the regression label, perform parameter iteration on the initial model to obtain the farmland lodging recognition model.

4. The UAV scheduling method based on farmland lodging recognition according to any one of claims 1 to 3, characterized in that The performing drone task scheduling based on the waypoint density distribution map includes: Perform drone task scheduling based on the lodging area severity result and the waypoint density distribution map.

5. The method for scheduling an unmanned aerial vehicle based on farmland lodging recognition according to claim 4, wherein The performing drone task scheduling based on the lodging area severity result and the waypoint density distribution map includes: Determine the task priority corresponding to the lodging area severity result, as well as the flight duration and energy consumption of the drone; Based on the flight duration, the energy consumption, and the task priority, determine the task quotation; Based on the task quotations of each drone, determine the task assignment target of the waypoint density map, and construct a drone scheduling model with the minimum task assignment target as the goal; Based on the drone scheduling model, perform drone task scheduling.

6. The method for scheduling an unmanned aerial vehicle based on farmland lodging recognition according to any one of claims 1 to 3, characterized in that The determination steps of the area complexity include: Obtain the slope, obstacle density, and the lodging area severity result of the lodging area; Based on the slope, the obstacle density, and the lodging area severity result, determine the area complexity.

7. The method for dispatching an unmanned aerial vehicle based on farmland lodging recognition according to any one of claims 1 to 3, characterized in that, Adjusting the waypoint density distribution of the initial waypoints based on the regional complexity of the lodging areas in the initial waypoints includes: Determining the waypoint density based on the regional complexity; Adjusting the waypoint density distribution of the initial waypoints based on the waypoint density.

8. An unmanned aerial vehicle scheduling system based on farmland lodging recognition, characterized in that, Including: An acquisition unit configured to acquire the flight position data, flight attitude data, and image data of the unmanned aerial vehicle; the image data includes RGB image data, multispectral image data, and near-infrared image data; An input unit configured to input the flight position data, the flight attitude data, and the image data into a farmland lodging recognition model to obtain a lodging recognition result output by the farmland lodging recognition model; the lodging recognition result includes lodging area coordinates and a lodging area severity result; An adjustment unit configured to determine initial waypoints based on the lodging recognition result and adjust the waypoint density distribution of the initial waypoints based on the regional complexity of the lodging areas in the initial waypoints; A UAV scheduling unit configured to perform UAV task scheduling based on the waypoint density distribution map.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the UAV scheduling method based on farmland lodging recognition according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the UAV scheduling method based on farmland lodging recognition according to any one of claims 1 to 7.

Images