Method and device for determining flight path and nonvolatile storage medium

Abstract

The invention discloses a method and device for determining a flight path and a nonvolatile storage medium. The method comprises the steps that a path planning instruction sent by a ground control station is received, and the path planning instruction is used for instructing to determine the flight path of the unmanned aerial vehicle according to position information of a to-be-detected target; in response to the path planning instruction, position information of a to-be-detected target is determined through a sensing system installed on the unmanned aerial vehicle, the sensing system comprises a radar and a camera device, and the position information of the to-be-detected target is jointly determined according to data detected by the radar and data detected by the camera device; the flight path of the unmanned aerial vehicle is determined according to the position information of the to-be-detected target and the current information of the unmanned aerial vehicle, and the current information of the unmanned aerial vehicle comprises the current position and the current situation information of the unmanned aerial vehicle. The technical problem that the target object cannot be accurately tracked due to deviation between the preset flight path of the unmanned aerial vehicle and the motion path of the target object is solved.

Description

Technical Field

[0001] The present application relates to the field of drone technology, and more specifically, to a method and device for determining a flight path, and a non-volatile storage medium. Background Art

[0002] Unmanned aerial vehicles (UAVs) offer autonomy, flexibility, and high mobility, making them suitable for a variety of business scenarios, including data collection, emergency rescue, and cargo transportation. With the development of drone technology, drones can also be used to track targets. Related technologies use a thermal infrared camera on a drone to capture images of a target and transmit them to a ground station for image processing, thereby obtaining information about the target's trajectory. However, this method cannot accurately determine the specific form and characteristics of the target, resulting in low target location accuracy and an inability to accurately track the target.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a method and device for determining a flight path, and a non-volatile storage medium, to at least solve the technical problem of being unable to accurately track a target object due to the deviation between the preset flight path of a drone and the motion path of the target object.

[0005] According to one aspect of an embodiment of the present application, a method for determining a flight path is provided, comprising: receiving a path planning instruction sent by a ground control station, wherein the path planning instruction is used to instruct determination of a flight path of a UAV based on position information of a target to be detected, and the ground control station includes: multiple ground base stations for communicating with the UAV; in response to the path planning instruction, determining the position information of the target to be detected by a perception system installed on the UAV, wherein the perception system includes: a radar and a camera device, and the position information of the target to be detected is jointly determined based on data detected by the radar and data detected by the camera device; determining the flight path of the UAV based on the position information of the target to be detected and the current information of the UAV, wherein the current information of the UAV includes: the current position and current situation information of the UAV, and the current situation information is used to describe environmental information of the area to which the current position belongs, and the environmental information includes: a no-fly zone.

[0006] Optionally, the position information of the target to be detected is determined by a perception system installed on the drone, including: detecting the position of the target to be detected by radar to obtain a first type of data for indicating the position information of the target to be detected; determining the adjustment information and initial flight path of the drone based on the first type of data, wherein the adjustment information is used to adjust the detection field of view of the drone to a maximized coverage field of view, the maximized coverage field of view is the detection field of view with the largest area containing the target area among multiple detection fields corresponding to the drone, the target area is the area containing the target to be detected, and the initial flight path is used to instruct the drone to fly towards the target area; after the drone reaches the target area, the target area is detected by a camera device according to different camera parameters to obtain a second type of data, wherein the second type of data includes: images containing the target to be detected and different detection scenes, and different detection scenes correspond to different camera parameters; using a target detection model to process and analyze the second type of data to obtain a detection result output by the target detection model, wherein the detection result includes: bounding box information of the target to be detected, and the target detection model is obtained by training a neural network model using image data containing the bounding box as training data; determining the position information of the target to be detected based on the first type of data and the detection result.

[0007] Optionally, the target detection model is trained by the following method: obtaining a predicted bounding box output by the target detection model for the training data; for each predicted bounding box, determining a true bounding box corresponding to the predicted bounding box in the training data; determining the union area and intersection area of the predicted bounding box and the true bounding box, and determining the target area ratio corresponding to the predicted bounding box based on the union area and the intersection area; determining a weight value corresponding to the predicted bounding box according to the type of the predicted bounding box, and determining an independent loss function corresponding to the predicted bounding box based on the weight value and the target area ratio, wherein the types of the predicted bounding boxes include: first-class bounding boxes, second-class bounding boxes, third-class bounding boxes, and fourth-class bounding boxes, and the area ratios corresponding to the first-class bounding boxes, the second-class bounding boxes, the third-class bounding boxes, and the fourth-class bounding boxes respectively decrease in sequence, and the area ratios are used to indicate the ratio of the area of the predicted bounding box to the area of the target image, and the target image is the image to which the predicted bounding box belongs; determining a total loss function of the target detection model in this training process based on multiple independent loss functions corresponding to multiple predicted bounding boxes, and determining the training progress of the target detection model based on the total loss function, wherein the training process includes: continuing training and stopping training.

[0008] Optionally, a target detection model is used to process and analyze the second type of data, wherein the target detection model processes and analyzes the second type of data, including: performing feature extraction processing on the second type of data in a convolutional layer containing multiple convolutional networks to obtain a feature image, wherein a first preset step size is used in the first convolutional network to perform feature extraction processing on the second type of data, and a second preset step size is used in other convolutional networks to perform feature extraction processing on the second type of data, and the first preset step size is greater than the second preset step size; for each feature image, determining the original image corresponding to the feature image, wherein the original image is an image contained in the second type of data; determining the area ratio of the bounding box of the feature image according to the area of the bounding box of the feature image and the area of the original image, and determining the type of the bounding box of the feature image according to the area ratio of the bounding box of the feature image; determining the detection head corresponding to the feature image according to the type of the bounding box of the feature image, and using the detection head to process and analyze the feature image, wherein the detection head is a neural network in the target detection model used to detect the target to be detected, and the feature images contained in different types of bounding boxes correspond to different detection heads.

[0009] Optionally, before determining the position information of the target to be detected based on the first category of data and the detection results, it also includes: obtaining airborne position information, wherein the airborne position information is the current position of the drone detected by a positioning device installed on the drone; determining calibration information based on the first category of data and the airborne position information, and correcting the first category of data based on the calibration information.

[0010] Optionally, the flight path of the UAV is determined based on the position information of the target to be detected and the current information of the UAV, including: generating a map model based on the current information of the UAV and the position information of the target to be detected, wherein the map model is composed of multiple nodes, each node represents a part of the target area, and the target area is the area to which the target to be detected belongs; determining the starting point of the flight path, the end point of the flight path, and the flight path planning area in the map model, wherein the starting point is a node determined to represent the current position of the UAV, the end point is a node representing the position of the target to be detected, and the flight path planning area is an area in the target area that includes the starting point and the end point but does not include a no-fly zone; determining the flight path in the flight path planning area based on the starting point and the end point of the flight path.

[0011] Optionally, the flight path is determined in the flight path planning area based on the starting point and the end point of the flight path, including: classifying the flight path planning area into a first area and a second area, wherein the distance value between the first-type nodes contained in the first area and the end point is greater than a preset distance value, and the distance between the second-type nodes contained in the second area and the end point is less than a preset distance value; determining multiple first-type waypoints in the first area according to a first search step length, and determining multiple second-type waypoints in the second area according to a second search step length, wherein the first search step length is greater than the second search step length; determining the flight path based on the starting point, multiple first-type waypoints, multiple second-type waypoints and the end point.

[0012] Optionally, a plurality of first-class waypoints are determined in the first area according to the first search step, including: executing a first-class waypoint screening method in the first area to obtain a plurality of first-class waypoints, wherein the first-class waypoint screening method includes: S1, determining the starting point as the starting point, and determining the node whose distance value from the starting point is the first search step as the first-class node to be screened; determining the flight cost of each first-class node to be screened, wherein the flight cost is used to measure the resource consumption of the drone flying from the starting point to the end point via the first-class node to be screened; determining the first-class node to be screened corresponding to the smallest flight cost as the adjacent node of the starting point; S2, determining the adjacent node as the new starting point in the first-class waypoint screening method, and executing the first-class waypoint screening method again until the determined new adjacent node belongs to the second area, and stopping executing the first-class waypoint screening method; wherein each adjacent node determined by executing the first-class waypoint screening method is a first-class waypoint, and each adjacent node determined by executing the first-class waypoint screening method includes: all adjacent nodes screened in S1 and S2.

[0013] Optionally, determining a plurality of second-category waypoints in the second area according to the second search step length includes: executing a second-category waypoint screening method in the second area to obtain a plurality of second-category waypoints, wherein the second-category waypoint screening method includes: S3, determining a target first-category waypoint as a starting point, wherein the target first-category waypoint is the last first-category waypoint determined according to the first search step length; determining a node whose distance value from the starting point is the second search step length as a second-category node to be screened; determining a flight cost for each second-category node to be screened, wherein the flight cost is used to measure the flight time of the drone from the target first-category waypoint via the first search step length. The resource consumption of the second type of nodes to be screened flying to the destination; the adjacent node of the starting point of the second type of node to be screened corresponding to the smallest flight cost is determined; S4, the adjacent node is determined as the new starting point in the second type of route node screening method, and the second type of route node screening method is executed again until the determined second type of nodes to be screened include the destination, and the second type of waypoint screening method is stopped; wherein, each adjacent node determined by executing the second type node screening method is a second type waypoint, and each adjacent node determined by executing the second type waypoint screening method includes: all adjacent nodes screened in S3 and S4.

[0014] According to another aspect of an embodiment of the present application, a device for determining a flight path is also provided, including: a receiving module for receiving a path planning instruction sent by a ground control station, wherein the path planning instruction is used to instruct the determination of the flight path of the UAV based on the position information of the target to be detected, and the ground control station includes: multiple ground base stations for communicating with the UAV; a positioning module for responding to the path planning instruction to determine the position information of the target to be detected through a perception system installed on the UAV, wherein the perception system includes: radar and camera equipment, and the position information of the target to be detected is jointly determined based on the data detected by the radar and the data detected by the camera equipment; a determination module for determining the flight path of the UAV based on the position information of the target to be detected and the current information of the UAV, wherein the current information of the UAV includes: the current position and current situation information of the UAV, and the current situation information is used to describe the environmental information of the area to which the current position belongs, and the environmental information includes: a no-fly zone.

[0015] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a computer program is stored. The device where the non-volatile storage medium is located executes the above-mentioned method for determining a flight path by running the computer program.

[0016] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned method for determining a flight path through the computer program.

[0017] According to another aspect of an embodiment of the present application, a computer program product is further provided, comprising computer instructions, which implement the steps of the above-mentioned method for determining a flight path when executed by a processor.

[0018] In an embodiment of the present application, a path planning instruction is received from a ground control station, wherein the path planning instruction is used to instruct the determination of a flight path of a UAV based on the position information of a target to be detected. The ground control station includes: multiple ground base stations for communicating with the UAV; in response to the path planning instruction, the position information of the target to be detected is determined by a perception system installed on the UAV, wherein the perception system includes: radar and camera equipment, and the position information of the target to be detected is jointly determined based on data detected by the radar and data detected by the camera equipment; the flight path of the UAV is determined based on the position information of the target to be detected and the current information of the UAV, wherein the current information of the UAV includes: the current position of the UAV and current situation information, the current situation information is used to describe the environmental information of the area to which the current position belongs, and the environmental information includes: a no-fly zone. By combining the radar and the gimbal camera, the association of multimodal information is achieved. By combining the radar detection information and the visual information, the image extraction accuracy and clarity of the target to be tracked are improved, and the purpose of accurately locating the target to be tracked is achieved, thereby achieving the technical effect of improving the accuracy of identifying the target to be tracked and improving the accuracy of tracking the target object, thereby solving the technical problem of being unable to accurately track the target object due to the deviation between the preset flight path of the UAV and the motion path of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a hardware structure block diagram of a computer terminal for implementing a method for determining a flight path according to an embodiment of the present application;

[0021] Figure 2 is a flowchart of the steps of a method for determining a flight path according to an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of deployment of a ground base station according to an embodiment of the present application;

[0023] Figure 4 This is a structural diagram of a device for determining a flight path according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0027] Pod attitude: This refers to the attitude of a sensor or payload pod mounted on a drone, aircraft, or other aircraft relative to the main body of the aircraft or to a geographic coordinate system. This attitude primarily includes pitch, yaw, and roll angles, representing the pod's rotation angles about the vertical, horizontal, and roll axes, respectively.

[0028] In related technologies, the target to be tracked is located by capturing images with a thermal infrared camera and then performing image processing on the images. Consequently, there is a problem of not being able to know the target to be tracked and accurately locate the target to be tracked. Furthermore, when the target to be tracked is not accurately located, the target's motion path will deviate from the drone's flight path, making it impossible to accurately track the target. To address this issue, the present application provides a solution in the embodiments, which is described in detail below.

[0029] According to an embodiment of the present application, an embodiment of a method for determining a flight path is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal for implementing a method for determining a flight path. Figure 1 As shown, the computer terminal 10 may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0031] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for determining the flight path in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned method for determining the flight path. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0034] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0035] The present application provides a method for determining a flight path that can be applied in the above operating environment. Figure 2 is a flowchart of the steps of the method for determining the flight path provided in an embodiment of the present application, such as Figure 2 As shown, the method includes the following steps:

[0036] Step S202: Receive a path planning instruction sent by a ground control station, wherein the path planning instruction is used to instruct the determination of the flight path of the UAV according to the position information of the target to be detected. The ground control station includes: multiple ground base stations that communicate with the UAV.

[0037] The present invention provides a method for locating a target using a wireless network using a drone, which is used to address the problem of low positioning and tracking accuracy caused by the use of thermal infrared cameras in the prior art, where the outline and details of the target to be tracked in the image are unclear. In the method provided in the present invention, the drone can communicate with a ground control station, and the user can activate the drone's function of tracking the target to be detected through the ground control station. In step S202, the ground control station can be composed of multiple ground base stations capable of communicating with the user terminal, and the ground base stations can simultaneously communicate with the drone. Therefore, the user can send a path planning instruction to the drone through the ground control station. The path planning instruction records the location information of the target to be detected, instructing the drone to plan a flight path based on the location information of the target to be detected. Therefore, whether the target to be detected is a real-time moving object or a static object, the method provided in the present invention can improve the positioning accuracy of the target to be detected and improve the accuracy of tracking the target to be detected. For example, when the application scenario of the method provided in the present invention is a forest fire detection task, the ground control station can command the drone to go to the fire source area to obtain real-time fire information and disaster situation. In this case, the target to be detected is the fire source area, which is a dynamic target to be detected.

[0038] In an embodiment of the present application, the service base station (i.e., ground base station) constituting the ground control station realizes two-way communication with the UAV through a networking protocol; wherein, the networking protocol is generally divided into competitive and non-competitive types. The competitive networking protocol has the characteristics of decentralized control, flexibility, randomness, and conflict resolution, and can effectively manage data transmission on the shared channel and improve the performance of the system; the non-competitive protocol is a non-conflicting and non-competitive protocol, and its channel resources are allocated to each node (each node represents a ground base station) according to certain rules, thereby avoiding competition and conflict. According to the difference in channel resources (such as time domain, code domain, frequency domain, and spatial domain), this type of protocol can be further divided into time division multiple access (TDMA), code division multiple access (CDMA), frequency division multiple access (FDMA), and space division multiple access (SDMA) protocols. Since code division multiple access (CDMA) cannot support large-scale networks and the interference problem of the code division multiple access (CDMA) system is relatively serious, the protocol followed by the drone and the ground control station when communicating in the embodiment of the present application does not include the code division multiple access protocol; in addition, since the resources of frequency division multiple access (FDMA) are always occupied once allocated, resulting in low channel utilization and unable to support large-scale drone swarm networks, the protocol followed by the drone and the ground control station when communicating in the embodiment of the present application does not include the frequency division multiple access protocol.

[0039] In the embodiment of the present application, the communication between the drone and the ground control station follows a networking protocol that mainly considers the uplink. The ground control station is a single-layer network deployed at a certain density. The drone moves in a hemispherical area with the service base station as the center and R as the radius. The proportion of the service base station connected to the ground control station or the drone user is the same as the proportion of the ground control station and drone users with uplink requirements to all uplink users. The same frequency band is reused between the service base stations, so the interference between adjacent cells needs to be considered. The average network coverage of the drone users and the ground control station is the key indicator. According to the relationship between the indicator and the service base station movement range and the power control factor, the overall network performance of the two types of users and the partial path loss reverse power control method are considered, and finally the optimal power control factor is obtained to complete the parameter setting. When the density of the ground base stations is relatively high, there is a worst R to minimize the uplink coverage probability of the ground user, because the increase in the radius of the drone user movement range leads to a higher line-of-sight propagation probability of the interference link, but then the interference is reduced due to the increase in path loss. As the density of serving base stations decreases, this phenomenon becomes less obvious. When the density of serving base stations is relatively low, the coverage probability of ground users decreases linearly with the increase of R. A larger power control factor will lead to higher uplink performance for ground users. A lower power control factor should be used for ground users. Figure 3 This is a schematic diagram of the deployment of ground base stations, such as Figure 3As shown, in an embodiment of the present application, a method for deploying a serving base station is as follows: the number of drones is determined based on actual user needs and the distribution of obstructions in the area where the target to be detected belongs. The number of drones guides the deployment of serving base stations in the actual network based on the actual conditions of the target area. The serving base stations dynamically adjust their deployment positions based on a random mobility model to achieve dynamic coverage of the target area. The speed and direction of drone movement can be dynamically adjusted based on the spatial distribution of obstructions to avoid collisions between drones and obstructions. Changes in obstructions during drone movement can cause changes in transmission distance and link line-of-sight and non-line-of-sight states. Therefore, the drone's air-to-ground channel experiences temporal link fluctuations, causing user performance fluctuations. Because obstructions can cause obstruction effects on channels in multiple directions, the obstruction states of drones at different spatial locations are not independent but rather correlated, resulting in correlation between the channels at different drone locations. The channel fluctuations caused by the high dynamic characteristics of drones make the instantaneous performance at the user's location time-varying, making it impossible to accurately represent the communication quality of the network. Therefore, the method provided in the embodiment of the present application introduces a large-scale temporal performance indicator to calculate the multi-slot joint coverage ratio of the user, effectively solving the problem of invalid instantaneous performance indicators. In the embodiment of the present application, occlusion correlation parameters in the target scene are counted, and occlusion correlation is introduced into the multi-time slot joint coverage analysis, so as to obtain more accurate coverage results.

[0040] In step S204, in response to the path planning instruction, the position information of the target to be detected is determined by the perception system installed on the drone, wherein the perception system includes: radar and camera equipment, and the position information of the target to be detected is determined based on the data detected by the radar and the data detected by the camera equipment.

[0041] In step S204, after receiving the path planning instruction, the UAV starts the function of tracking the target to be detected, locates the target to be detected through the perception system installed on the UAV, and obtains the position information of the target to be detected; in this embodiment, the perception system on the UAV is composed of a radar and a camera device, and the position information of the target to be detected is determined based on the data detected by the radar and the information returned by the camera device, wherein the camera device can be a high-definition camera.

[0042] According to some optional embodiments of the present application, the position information of the target to be detected is determined by a perception system installed on a drone, including: detecting the position of the target to be detected by radar to obtain first type of data for indicating the position information of the target to be detected; determining the adjustment information and initial flight path of the drone based on the first type of data, wherein the adjustment information is used to adjust the detection field of view of the drone to a maximized coverage field of view, the maximized coverage field of view being the detection field of view with the largest area including the target area among multiple detection fields corresponding to the drone, the target area being the area including the target to be detected, and the initial flight path being used to instruct the drone to fly toward the target area; after the drone arrives at the target area, the target area is detected by a camera device according to different camera parameters to obtain second type of data, wherein the second type of data includes: images including the target to be detected and different detection scenes, and different detection scenes correspond to different camera parameters; using a target detection model to process and analyze the second type of data to obtain a detection result output by the target detection model, wherein the detection result includes: bounding box information of the target to be detected, and the target detection model is obtained by training a neural network model using image data including the bounding box as training data; determining the position information of the target to be detected based on the first type of data and the detection result.

[0043] The method provided in the embodiment of the present application improves the accuracy of positioning by combining multimodal information to determine the position of the target to be tracked. For example, when the method provided in the embodiment of the present application is applied to a forest fire detection task, the data obtained when the radar detects the fire source position and the data obtained when the camera equipment detects the fire source position are used to jointly determine the specific location of the fire source. In this embodiment, the perception system is used to achieve accurate matching of the target to be detected within the field of view, and the specific position of the target to be detected is solved to perform positioning estimation; wherein, when locating the target to be detected by radar detection technology, the field of view coverage is maximized as a constraint condition, so that when locating the target to be detected, the drone position and pod posture are adjusted to maximize the coverage field of view of the drone, so that when the drone locates the target to be detected, its detection field of view can maximize the coverage of the area where the target to be detected is located (i.e., the target area); therefore, after obtaining the position information of the target to be detected returned by the radar (i.e., the first type of data), it is necessary to determine the path of the drone to the target area (i.e., the initial flight path) based on the first type of data, as well as the adjustment information for adjusting the detection field of the drone; in this embodiment, the factors affecting the detection field of the drone include: pod posture and camera parameters (such as the angle and focal length of the camera device), etc. Therefore, the above-mentioned adjustment information at least includes: information for adjusting the pod posture, information for adjusting the angle of the camera device, and information for adjusting the focal length of the camera device. After determining the path (i.e., the initial flight path) of the drone to the target area based on the first type of data, the drone is controlled to fly along the flight path to the target area. After determining that the drone has arrived at the target area, the detection field of view of the drone is adjusted according to the adjustment information. After confirming that the adjustment is completed, the image data (i.e., the second type of data) of the target area is obtained by the camera device. When the image data of the target area is obtained by the camera device, the angle and focal length of the pan-tilt camera are controlled to obtain image data of the target to be detected in different detection scenes. That is, in this embodiment, when the camera device detects the target area with different camera parameters, the detection scenes to which the target to be detected belongs in the obtained image data are different. Further, the target detection model is used to process and analyze the information (i.e., the second type of data) obtained by the camera device for locating the target to be detected to obtain the detection result output by the target detection model. In this embodiment, the detection result includes information (i.e., bounding box information) for selecting the rectangular box of the target to be detected, for example, the coordinates of each vertex of the rectangular bounding box. Therefore, the image data obtained by the camera device using the target detection model can be processed to locate the target to be detected. The above-mentioned target detection model has the function of determining the target object in the image (the target object is surrounded by a bounding box), and is obtained by training the neural network model using image data containing the target object as training data and target detection as the training task.After successfully locating the target to be detected, the target to be detected can be distinguished from other objects in the field of view by assigning numbers to the target to be detected, which is also conducive to tracking the target to be detected. Through the method provided in this embodiment, the visual detection information of the drone (i.e., the second type of data) and the information returned by the radar to the ground control station (i.e., the first type of data) are combined; the information returned by the radar (i.e., the first type of data) provides global spatial position coordinates for locating the target to be detected, and the data collected by the camera equipment in different detection scenarios is processed and analyzed in real time according to the visual target detection algorithm, and the bounding box information of all the targets to be detected in the image is returned to the ground control station in real time, achieving the technical effect of accurately locating the target to be detected in the field of view.

[0044] Furthermore, before the target detection model is used to process and analyze the second type of data, cleaning the second type of data can improve the accuracy of positioning. Therefore, a method for cleaning image data is also provided in the embodiment of the present application. However, even if the second type of data is not processed, the method provided in the embodiment of the present application can still be implemented. The cleaning method of the second type of data provided in the embodiment of the present application is as follows: when the camera device acquires video data, a video segmentation tool is used to extract the video data and decompose the video data into data of the same length, for example, decompose it into image data of 60 frames per second; next, the image data is processed to remove blurred or unclear images of the target, thereby completing the cleaning of the second type of data. In addition, the training data of the target detection model can be obtained by the following method: using a video segmentation tool to extract the video data acquired by the camera device before the current task is executed, decomposing the video data into data of the same length, for example, decomposing it into image data of 60 frames per second; next, the image data is processed to remove blurred or unclear images of the target, and then annotating the image using an automatic image annotation tool; the annotated video data can be used as training data, and a deep learning model is trained based on the annotated image data.

[0045] In this embodiment, the target detection model can be loaded into the memory. For example, the raw data of the target detection model can be loaded from the non-volatile memory into the volatile memory so that the processor can run the target detection model. The raw data of the target detection model refers to unprocessed data, which generally includes parameters and structural data of the target detection model. The structural data can be a calculation relationship based on the parameters, such as the forward propagation calculation relationship between intermediate layers and neurons. Specifically, the target detection model can include code related to the structure of the target detection model, such as code for performing related calculations between intermediate layers and neurons.

[0046] In one embodiment, an area for loading the target detection model can be divided into two parts in memory, including a structure data storage area and a parameter storage area. The structure data storage area is used to store structure-related code, and the parameters referenced by it can point to the addresses of specific parameters in the parameter storage area through pointers. During the training process of the target detection model, it may be necessary to frequently update the parameters, and the parameter values in the parameter storage area can be simply updated.

[0047] Optionally, the target detection model is trained by the following method: obtaining a predicted bounding box output by the target detection model for the training data; for each predicted bounding box, determining a true bounding box corresponding to the predicted bounding box in the training data; determining the union area and intersection area of the predicted bounding box and the true bounding box, and determining the target area ratio corresponding to the predicted bounding box based on the union area and the intersection area; determining a weight value corresponding to the predicted bounding box according to the type of the predicted bounding box, and determining an independent loss function corresponding to the predicted bounding box based on the weight value and the target area ratio, wherein the types of the predicted bounding boxes include: first-class bounding boxes, second-class bounding boxes, third-class bounding boxes, and fourth-class bounding boxes, and the area ratios corresponding to the first-class bounding boxes, the second-class bounding boxes, the third-class bounding boxes, and the fourth-class bounding boxes respectively decrease in sequence, and the area ratios are used to indicate the ratio of the area of the predicted bounding box to the area of the target image, and the target image is the image to which the predicted bounding box belongs; determining a total loss function of the target detection model in this training process based on multiple independent loss functions corresponding to multiple predicted bounding boxes, and determining the training progress of the target detection model based on the total loss function, wherein the training process includes: continuing training and stopping training.

[0048] The target detection model proposed in the previous embodiment has the following adjustments compared to the target detection model used in the related art: 1) A detection head for detecting small targets (for example, targets to be detected with image pixels of 4x4) is added; 2) The loss function is improved, and a focusing mechanism is introduced in the calculation process of the loss function, so that small targets are taken into account more when calculating the loss function, and the detection of small targets by the target detection model is fully optimized. The target detection model adopts a multi-stage training mechanism. First, a large amount of labeled image data (i.e., training data) is used for preliminary training to obtain the predicted bounding box output by the target detection model (in the training stage) for the training data; during the training process, the loss function is calculated based on the intersection over union (IoU) of the real bounding box in the training data and the predicted bounding box output by the model, and the loss function is used to determine whether the target detection model has been trained. Among them, when calculating the loss function, first determine the predicted bounding box and the true bounding box corresponding to the same training data, then calculate the area of the same image in the image contained in the predicted bounding box and the image contained in the true bounding box (i.e., the intersection area), and calculate the area of this part of the image that contains both the image contained in the predicted bounding box and the image contained in the true bounding box (i.e., the union area). The ratio of the above intersection area to the union area is determined as the effective area ratio of the predicted bounding box of this training data (i.e., the target area ratio). The effective area ratio can be used to evaluate whether the target detection model can accurately identify the target to be detected. When the effective area ratio is larger, it means that the training effect of the target detection model is better, and the target to be detected can be identified more accurately. In addition, due to the introduction of the focusing mechanism, different weight values are defined for different types of prediction bounding boxes when calculating the loss function. It is pre-defined that the weight value of the prediction bounding box containing small targets (such as the target to be detected with an image pixel of 4x4) is larger, so that the detection ability of the target detection model for small targets is improved. Then, in order to use the weight value to calculate the loss function, it is necessary to first determine the type of the prediction bounding box. In this embodiment, the type of the prediction bounding box is determined according to the area of the image surrounded by the prediction bounding box. Specifically, the ratio of the area of the image surrounded by the prediction bounding box to the total area of the image to which the image surrounded by the prediction bounding box belongs (i.e., the target image) is determined as the area ratio corresponding to the prediction bounding box. The type of the predicted bounding box can be determined according to the interval to which the area ratio belongs. For example, three area ratios (0.01, 0.05, and 0.1) are pre-set. The type of the predicted bounding box with an area ratio less than or equal to 0.01 is defined as the bounding box of an extremely small target (i.e., the fourth type of bounding box), the type of the predicted bounding box with an area ratio greater than 0.01 and less than or equal to 0.05 is defined as the bounding box of a small target (i.e., the third type of bounding box), the type of the predicted bounding box with an area ratio greater than 0.05 and less than or equal to 0.1 is defined as the bounding box of a large target (i.e., the second type of bounding box), and the bounding box with an area ratio greater than 0.1 is positioned as the bounding box of an extremely large target (i.e., the first type of bounding box).Since different types of predicted bounding boxes correspond to different weight values, the weight value corresponding to the predicted bounding box can be determined after the type of predicted bounding box is determined. In this embodiment, the first type of bounding box (larger target) is set with a lower weight value because it is relatively easy to detect; the weight value corresponding to the second type of bounding box is greater than the weight value corresponding to the first type of bounding box to ensure accurate detection of medium-sized targets; the weight value corresponding to the third type of bounding box is greater than the weight value corresponding to the second type of bounding box to enhance the detection capability of small targets; the weight value corresponding to the fourth type of bounding box (extremely small target) is the largest, ensuring that even the smallest targets can be accurately detected. The above method can determine the loss function (i.e., independent loss function) of the target detection model on each predicted bounding box, and the loss function (i.e., total loss function) that determines whether the target detection model training is completed is jointly determined by the above multiple independent loss functions; after the total loss function of the target detection model is determined by the above method, the training process decision is made based on the total loss function. The training process of the target detection model is determined based on the change of the total loss function and the preset training stop condition. If the total loss function no longer decreases significantly in several consecutive iterations or reaches a predetermined training round, the training is stopped. Otherwise, continue training steps until the stopping condition is met.

[0049] According to some other optional embodiments of the present application, a target detection model is used to process and analyze the second category of data, wherein the target detection model processes and analyzes the second category of data, including: performing feature extraction processing on the second category of data in a convolutional layer containing multiple convolutional networks to obtain a feature image, wherein a first preset step size is used in the first convolutional network to perform feature extraction processing on the second category of data, and a second preset step size is used in other convolutional networks to perform feature extraction processing on the second category of data, and the first preset step size is greater than the second preset step size; for each feature image, determining the original image corresponding to the feature image, wherein the original image is an image contained in the second category of data; determining the area ratio of the bounding box of the feature image according to the area of the bounding box of the feature image and the area of the original image, and determining the type of the bounding box of the feature image according to the area ratio of the bounding box of the feature image; determining the detection head corresponding to the feature image according to the type of the bounding box of the feature image, and using the detection head to process and analyze the feature image, wherein the detection head is a neural network in the target detection model used to detect the target to be detected, and the feature images contained in different types of bounding boxes correspond to different detection heads.

[0050] As mentioned in the above embodiment, the improvement of the target detection model lies in the addition of a detection head for detecting small targets. In addition, the target detection model has been improved in the following aspects: the step size used in the convolution processing (i.e., the preset step size) has been increased, especially the step size used in the convolution processing of the first convolution layer network has been increased, and a large step size is used to reduce the size of the image data, so that the target detection model can process data in real time. Due to the above improvements in the target detection model, its data processing process is different from the data processing process of the traditional detection model. When the target detection model performs data processing, it extracts features from the image data through a multi-stage convolutional network, combined with different preset step sizes and detection heads, to improve the accuracy and efficiency of detecting targets of different sizes. The specific steps are as follows: In the convolutional layer of the target detection model, the image data (i.e., the second type of data) obtained by the camera device is subjected to feature extraction through multiple convolutional networks. Among them, the first convolutional network adopts a larger step size (i.e., the first preset step size), such as a step size of 4 or 8, to reduce the amount of model calculation, which is suitable for detecting larger or very small targets; while in the convolutional layer, in other convolutional networks, a smaller step size (i.e., the second preset step size), such as a step size of 2, is adopted to extract more abstract features, which is suitable for detecting medium-sized targets. The settings of these step sizes are determined experimentally during the training phase of the model to achieve the best detection effect. For each feature image processed by feature extraction, its corresponding original image is determined so that the area ratio and bounding box type corresponding to the feature image can be subsequently calculated; the original image is an unprocessed image collected from the drone gimbal camera (i.e., the camera device) and included in the second type of data. Next, a corresponding detection head is determined for each feature image, and the detection head is used to process and analyze the feature image. Specifically, the ratio of the area of the predicted bounding box surrounding the target to be detected in the feature image to the area of the original image corresponding to the feature image is determined as the area ratio corresponding to the feature image. The area ratio is an important indicator for evaluating the size and positioning accuracy of the target to be detected. The area ratio can be used to further determine the type of bounding box to select a suitable detection head to process the feature image. The detection head is a neural network included in the target detection model, and each detection head is designed to detect targets of different sizes to be detected. Which detection head is specifically used for each feature image can be determined by the area ratio corresponding to the feature image. For example, in the previous embodiment, the predicted bounding boxes are divided into four categories: first-class bounding boxes, second-class bounding boxes, third-class bounding boxes, and fourth-class bounding boxes. In this embodiment, corresponding detection heads are set for each type of bounding box. Therefore, the detection head corresponding to the feature image can be determined according to the type of bounding box surrounding the target to be detected in the feature image. Through the method provided in this embodiment, the most suitable detection head is automatically selected for processing targets of different sizes, which significantly improves the target detection model's detection ability for various types of targets to be tracked, especially small targets and extremely small targets.From a drone's perspective, this multi-scale feature extraction and targeted detection head enable more effective target identification and location, overcoming the limitations of single-scale detection when dealing with objects of varying sizes. The preset step size used in the convolution process described above refers to the distance the convolution kernel moves across the input image. This preset step size determines the size of the output feature map and the level of refinement of feature extraction.

[0051] Step S206, determining the flight path of the drone based on the location information of the target to be detected and the current information of the drone, wherein the current information of the drone includes: the current location of the drone, current situation information, and the current situation information is used to describe the environmental information of the area to which the current location belongs. The environmental information includes: no-fly zones.

[0052] In step S206, the flight path of the drone is planned based on the position information of the target to be detected determined in step S204 and the current information of the drone, so that the drone can accurately track the target to be detected; the above-mentioned current information of the drone includes the current position of the drone and the current situation information of the drone, wherein the above-mentioned current situation information refers to the relevant information of the environment to which the current position of the drone belongs, for example, the no-fly zone in the environment to which the current position of the drone belongs, the position of obstacles in the environment to which the current position of the drone belongs, etc.

[0053] According to some optional embodiments of the present application, before determining the position information of the target to be detected based on the first category of data and the detection results, it also includes: obtaining airborne position information, wherein the airborne position information is the current position of the drone detected by a positioning device installed on the drone; determining calibration information based on the first category of data and the airborne position information, and correcting the first category of data based on the calibration information.

[0054] In this embodiment, in order to improve the radar detection accuracy, a calibration method for radar positioning data (i.e., the first type of data) is also provided. Specifically, according to the spatial position coordinates (X s ,Y s ,Z s ) and the drone's spatial position coordinates (X GPS ,Y GPS ,Z GPS ) (i.e. record the position information) to perform differential calibration. According to the formula Determine the calibration information (δX, δY, δZ), and calculate the current position (X) of the drone detected by the radar based on the calibration information (δX, δY, δZ). s ,Y s ,Z s) (i.e. the first type of data) is corrected. In the above formula, m is the number of drones, i represents the i-th data point, and each data point represents the coordinates of the target to be detected in scene i.

[0055] Optionally, the flight path of the UAV is determined based on the position information of the target to be detected and the current information of the UAV, including: generating a map model based on the current information of the UAV and the position information of the target to be detected, wherein the map model is composed of multiple nodes, each node represents a part of the target area, and the target area is the area to which the target to be detected belongs; determining the starting point of the flight path, the end point of the flight path, and the flight path planning area in the map model, wherein the starting point is a node determined to represent the current position of the UAV, the end point is a node representing the position of the target to be detected, and the flight path planning area is an area in the target area that includes the starting point and the end point but does not include a no-fly zone; determining the flight path in the flight path planning area based on the starting point and the end point of the flight path.

[0056] In an embodiment of the present application, after the positioning of the target to be detected is completed, a two-dimensional incremental real-time map model will be quickly generated based on the real-time position information, real-time situation information and position information of the target to be detected of the UAV; and on the basis of the map model, a smart search algorithm is used to quickly and in real time generate waypoints for guiding the UAV. In this embodiment, a map model consisting of multiple nodes is constructed based on the geographic information and obstacle distribution of the area to which the target to be detected belongs (i.e., the target area). Each node in the map model represents a geographic location in the target area (or a part of the range or area in the target area). In addition, the map model also marks no-fly zones that UAVs cannot enter, such as buildings, forests, waters, etc. When planning the flight path of a drone, in the constructed map model, the starting point of the flight path is determined according to the current information of the drone (including position, altitude, orientation, etc.), that is, the node corresponding to the current position of the drone; the end point of the flight path is determined according to the position information of the target to be detected, that is, the node corresponding to the position of the target to be detected; the flyable area in the target area other than the no-fly zone is clearly defined as the flight path planning area, and an intelligent search algorithm is used in the path planning area based on the starting point and end point of the flight path to determine the flight path of the drone to track the target to be detected, thereby ensuring the safety of the flight path.

[0057] According to some optional embodiments of the present application, a flight path is determined in a flight path planning area based on the starting point and the end point of the flight path, including: classifying the flight path planning area into a first area and a second area, wherein a distance value between the first-category nodes contained in the first area and the end point is greater than a preset distance value, and a distance between the second-category nodes contained in the second area and the end point is less than a preset distance value; determining a plurality of first-category waypoints in the first area according to a first search step length, and determining a plurality of second-category waypoints in the second area according to a second search step length, wherein the first search step length is greater than the second search step length; determining the flight path based on the starting point, the plurality of first-category waypoints, the plurality of second-category waypoints and the end point.

[0058] In this embodiment, an improved intelligent search algorithm is used to determine the flight path of the drone tracking the target to be detected in the flight path planning area. The specific improvement is that an adaptive search step size is used to plan the flight path, that is, when the current point is far away from the target point (such as the position of the target to be detected), the algorithm uses a larger search step size, and when the current point is near the target point, the algorithm uses a smaller step size for a refined search. Therefore, in this embodiment, when determining the flight path in the path planning area, the path planning area is first partitioned, and the path planning area is further classified into two areas. The distance between all nodes (i.e., first-class nodes) contained in one area (i.e., the first area) and the node representing the current position of the target to be detected (i.e., the end point) is greater than or equal to a preset distance value; and the distance between all nodes (i.e., second-class nodes) contained in the other area (i.e., the second area) and the node representing the current position of the target to be detected (i.e., the end point) is less than the preset distance value. After partitioning the path planning area, in the first area, a larger search step size (i.e., the first search step size) is used to quickly determine the waypoints of the drone in the first area (i.e., the first type of waypoints). The selection of this step size is intended to improve the efficiency of the path search and reduce unnecessary consumption of computing resources. In the second area, a smaller search step size (i.e., the second search step size) is switched to more finely determine the waypoints of the drone in the second area (i.e., the second type of waypoints). The above-mentioned first type of waypoints, second type of waypoints, and the starting point and end point together determine the flight path of the drone, ensuring that the drone can perform more accurate trajectory planning when approaching the target to be detected, thereby improving the accuracy of target positioning. The above-mentioned search step size (including the first search step size and the second search step size) refers to the moving distance from a node to its adjacent node when constructing or searching for a path. In path planning, especially on a grid map (such as the two-dimensional map model in the embodiment of the present application), this step size represents a fixed unit distance, and the unit distance is, for example, the side length of a grid unit.

[0059] Optionally, a plurality of first-class waypoints are determined in the first area according to the first search step, including: executing a first-class waypoint screening method in the first area to obtain a plurality of first-class waypoints, wherein the first-class waypoint screening method includes: S1, determining the starting point as the starting point, and determining the node whose distance value from the starting point is the first search step as the first-class node to be screened; determining the flight cost of each first-class node to be screened, wherein the flight cost is used to measure the resource consumption of the drone flying from the starting point to the end point via the first-class node to be screened; determining the first-class node to be screened corresponding to the smallest flight cost as the adjacent node of the starting point; S2, determining the adjacent node as the new starting point in the first-class waypoint screening method, and executing the first-class waypoint screening method again until the determined new adjacent node belongs to the second area, and stopping executing the first-class waypoint screening method; wherein each adjacent node determined by executing the first-class waypoint screening method is a first-class waypoint, and each adjacent node determined by executing the first-class waypoint screening method includes: all adjacent nodes screened in S1 and S2.

[0060] The algorithm flow of the improved intelligent search algorithm used in planning the flight path of the drone in the embodiment of the present application is as follows: M1, initialize the map model G, the starting point S, the target point (i.e., the end point) T, the untraversed node table Open, the traversed node table Close, and store the starting point S in Open; M2, determine whether there is a node in Open, if there is no node, the search ends; M3, calculate the flight cost of each node in Open, and arrange all the flight costs; select the node K with the smallest flight cost in the Open table, and move K to the Close table; M4, determine whether K is a representative node. Table represents the node of the target to be detected (i.e., the end point). If so, output the optimal path from the starting point to K; M5, if K is not the node representing the target to be detected (i.e., the end point), generate a child node of K. If the child node is in the Close table, delete the child node; M6, if the child node is in the Open table, determine the relationship between the flight cost from the parent node to the child node and the flight cost from K to the child node. If the flight cost from K to the child node is smaller, update the parent node of the child node to K; M7, if the child node is not in the Close table or the Open table, add it to the Open table and go to step M2. Based on the above, in this embodiment, the first region is the portion of the path planning area that includes the starting point after classification. When planning the UAV's flight path in the first region, starting from the node representing the UAV's current location (i.e., the starting point), a search step size (i.e., the first search step size) with a larger value among the preset step sizes is used to search for nodes in the first region whose distance from the starting point meets the step size requirement. These nodes whose distance from the starting point is the first search step size are marked as first-class nodes to be screened. Next, the waypoints (i.e., adjacent nodes) closest to the starting point are determined from the first-class nodes to be screened. The setting of the first search step size is based on a combination of factors such as the UAV's maximum flight speed, terrain complexity, and computing resource availability. When determining the waypoints (i.e., adjacent nodes) closest to the starting point from the first-class nodes to be screened, the flight cost of the UAV from the starting point to the destination via each first-class node to be screened is first determined. The flight cost is a quantitative indicator that reflects the resource consumption required for the UAV to fly from the starting point to the destination via the first-class node to be screened, including flight distance, fuel / power consumption, time cost, and possible terrain obstacles encountered. Generally, the lower the flight cost of a node, the more likely it is to be a component of a path. According to the above rules, in this embodiment, the first-category node to be screened with the smallest flight cost is determined as the adjacent node of the starting point. This adjacent node will be used as the starting point (i.e., the new starting point) when the adjacent node is determined next time. The above-mentioned first-category route screening node method will continue to be executed on the new starting point, and the adjacent nodes of each new starting point will be continuously determined and used as the first-category waypoint.Since the first area does not contain an end point, when determining waypoints in the first area, there will not be a situation where the end point exists in the first type of nodes to be filtered. That is, when executing the intelligent search algorithm for the first area, it is impossible to determine whether the intelligent search algorithm has been completed in the first area based on whether there is an end point in the first type of nodes to be filtered. Instead, the stopping condition is whether there is a node belonging to the second area in the first type of nodes to be filtered. In summary, determining the first type of waypoints in the first area is an iterative process. When determining the waypoints for the first time, the node representing the current position of the drone (i.e., the starting point) is used as the starting point, and the intelligent search algorithm is executed on the starting point. The distance value and the flight cost are used to determine the adjacent nodes of the starting point. Each subsequent execution uses the adjacent node determined in the last execution of the intelligent search algorithm as the starting point. The adjacent nodes determined in each execution of the intelligent search algorithm are all waypoints in the flight path of the drone. In this embodiment, the adjacent nodes belonging to the first area are called first type waypoints. For example, when screening the first type of waypoints, in the first area, starting from the current position of the drone, expand to the surrounding nodes each time with the first search step (for example, 50 meters), calculate the flight cost of each node, and select the node with the lowest flight cost as the next waypoint until entering the second area.

[0061] According to some other optional embodiments of the present application, a plurality of second-category waypoints are determined in the second area according to the second search step, including: executing a second-category waypoint screening method in the second area to obtain a plurality of second-category waypoints, wherein the second-category waypoint screening method includes: S3, determining a target first-category waypoint as a starting point, wherein the target first-category waypoint is the last first-category waypoint determined according to the first search step; determining a node whose distance value from the starting point is the second search step as a second-category node to be screened; determining a flight cost for each second-category node to be screened, wherein the flight cost is used to measure the distance of the drone from the target first-category waypoint to the starting point. The resource consumption of the waypoint flying to the destination via the second type of nodes to be screened; the second type of nodes to be screened corresponding to the smallest flight cost are used to determine the adjacent node of the starting point; S4, the adjacent node is determined as the new starting point in the second type of route node screening method, and the second type of route node screening method is executed again until the determined second type of nodes to be screened include the destination, and the second type of waypoint screening method is stopped; wherein, each adjacent node determined by executing the second type of node screening method is a second type waypoint, and each adjacent node determined by executing the second type of waypoint screening method includes: all adjacent nodes screened in S3 and S4.

[0062] The difference between planning a flight path in the first area and planning a flight path in the second area using an intelligent search algorithm is that the search step size used is different. When planning the flight path of the drone in the second area, a smaller search step size (for example, 20 meters) is used to screen waypoints, and the flight cost is also used as the judgment criterion until the (second type) nodes to be screened before determining the adjacent nodes contain the end point, thereby ensuring the accuracy and safety of the flight path. In this embodiment, the steps of executing the intelligent search algorithm on the second area are the same as those of executing the intelligent search algorithm on the first area. Both are to first screen out multiple nodes that may serve as adjacent nodes based on the distance to the starting point (i.e., the second type of nodes to be screened), and then determine the node with the smallest flight cost among the second type of nodes to be screened as the adjacent node of the starting point, that is, the waypoint of the drone when flying in the second area (i.e., the second type of waypoint). The difference is that the second area does not contain a node representing the current position of the drone (i.e., the starting point). Therefore, when the intelligent search algorithm is executed on the second area for the first time, the adjacent node determined when the intelligent search algorithm was last executed on the first area (i.e., the target first-class waypoint) is used as the starting point. In addition, the second area contains a node representing the position of the target to be detected (i.e., the end point). Therefore, the stopping condition for executing the intelligent search algorithm on the second area is that the end point is included in the second-class node to be screened. When the end point is included in the second-class node to be screened, the adjacent node is no longer judged by the flight cost, but the end point is directly used as the adjacent node. However, in the process of iteratively executing the intelligent search algorithm, each time it is executed, the adjacent node determined last time is still used as the starting point for the next execution of the intelligent search algorithm. In this embodiment, the adjacent nodes belonging to the second area are called second-class waypoints, that is, the waypoints located in the second area among the multiple waypoints included in the flight path of the drone.

[0063] By combining radar and cameras through the above steps, multimodal information can be correlated. The position of the target being tracked is determined by combining radar and camera information, improving positioning accuracy. The drone's flight path is replanned based on the target's location, preventing deviations between the drone's flight path and the target's movement path, thus improving tracking accuracy.

[0064] Figure 4 is a structural diagram of a device for determining a flight path according to an embodiment of the present application, such as Figure 4As shown, the flight path device includes: a receiving module 40, which is used to receive a path planning instruction sent by a ground control station, wherein the path planning instruction is used to indicate that the flight path of the UAV is determined according to the position information of the target to be detected, and the ground control station includes: multiple ground base stations that communicate with the UAV; a positioning module 42, which is used to respond to the path planning instruction and determine the position information of the target to be detected through a perception system installed on the UAV, wherein the perception system includes: radar and camera equipment, and the position information of the target to be detected is jointly determined based on the data detected by the radar and the data detected by the camera equipment; a determination module 44, which is used to determine the flight path of the UAV based on the position information of the target to be detected and the current information of the UAV, wherein the current information of the UAV includes: the current position and current situation information of the UAV, and the current situation information is used to describe the environmental information of the area to which the current position belongs, and the environmental information includes: a no-fly zone.

[0065] Figure 4When the flight path device shown executes the method provided by the embodiment of the present application, a receiving module 40 receives a path planning instruction sent by a user through a ground control station. The path planning instruction is used to instruct the drone to plan a tracking path based on the location of the target to be detected, thereby achieving real-time tracking of the target to be detected. The ground control station includes multiple base stations installed on the ground. These multiple base stations communicate bidirectionally with the drone according to a preset networking protocol to ensure that the drone can receive the path planning instruction sent by the ground control station. After receiving the path planning instruction, the positioning module 42 determines the current location of the target to be detected. The positioning module 42 improves the accuracy of the positioning of the target to be detected by combining radar positioning information and camera positioning information. Specifically, the radar and camera are installed in the drone's perception system. The positioning module 42 uses the radar and camera in the perception system to perceive the position information of the target to be detected. The radar can provide the precise three-dimensional coordinates of the target to be detected, while the camera captures the visual features of the target to be detected through image analysis. The combination of these two data can verify and enhance the accuracy and reliability of target positioning. Furthermore, after locating the target to be detected, the flight path of the drone is planned by determining module 44 to avoid the flight path of the drone deviating from the motion trajectory of the target to be detected when tracking the target to be detected. The determining module 44 plans the flight path according to the position information of the target to be detected and the current information of the drone (including the current position of the drone and the no-fly zone in the environment, etc.); when planning the flight path, a map model of the target area is first generated, and the map model is divided into a first area that includes the starting point but does not include the end point and a second area that includes the end point but does not include the end point. The intelligent search algorithm is executed in different areas to plan the flight path, wherein, when executing the intelligent search algorithm in different areas, different search steps (first search step and second search step) are used to screen out a series of waypoints in the two areas respectively, and the flight path of the drone is determined based on the screened waypoints. According to the method provided in the embodiment of the present application, efficient target tracking of drones in complex environments can be achieved through multimodal perception and intelligent path planning.

[0066] It should be noted that Figure 4 The preferred implementation of the embodiment shown can be found in Figure 2 The relevant description of the illustrated embodiment will not be repeated here.

[0067] An embodiment of the present application further provides a non-volatile storage medium, in which a computer program is stored. The device where the non-volatile storage medium is located executes the above method for determining a flight path by running the computer program.

[0068] The above-mentioned non-volatile storage medium is used to store programs that perform the following functions: receiving path planning instructions sent by a ground control station, wherein the path planning instructions are used to instruct the determination of the flight path of the UAV based on the position information of the target to be detected, and the ground control station includes: multiple ground base stations that communicate with the UAV; in response to the path planning instructions, determining the position information of the target to be detected through a perception system installed on the UAV, wherein the perception system includes: radar and camera equipment, and the position information of the target to be detected is jointly determined based on the data detected by the radar and the data detected by the camera equipment; determining the flight path of the UAV based on the position information of the target to be detected and the current information of the UAV, wherein the current information of the UAV includes: the current position and current situation information of the UAV, and the current situation information is used to describe the environmental information of the area to which the current position belongs, and the environmental information includes: no-fly zone.

[0069] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above method for determining a flight path through the computer program.

[0070] The processor in the above-mentioned electronic device is used to run a program that performs the following functions: receiving a path planning instruction sent by a ground control station, wherein the path planning instruction is used to instruct the determination of the flight path of the UAV based on the position information of the target to be detected, and the ground control station includes: multiple ground base stations that communicate with the UAV; in response to the path planning instruction, determining the position information of the target to be detected through a perception system installed on the UAV, wherein the perception system includes: radar and camera equipment, and the position information of the target to be detected is jointly determined based on the data detected by the radar and the data detected by the camera equipment; determining the flight path of the UAV based on the position information of the target to be detected and the current information of the UAV, wherein the current information of the UAV includes: the current position of the UAV, current situation information, and the current situation information is used to describe the environmental information of the area to which the current position belongs, and the environmental information includes: no-fly zone.

[0071] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the steps of the above method for determining a flight path when executed by a processor.

[0072] It should be noted that the various modules in the above-mentioned device for determining the flight path can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0073] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0074] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0075] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0076] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0077] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0078] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0079] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for determining a flight path, characterized in that: include: Receiving a path planning instruction sent by a ground control station, wherein the path planning instruction is used to instruct determination of a flight path of the UAV based on location information of a target to be detected, the ground control station including: a plurality of ground base stations communicating with the UAV; In response to the path planning instruction, determining the position information of the target to be detected by a perception system installed on the drone, wherein the perception system includes: a radar and a camera device, and the position information of the target to be detected is determined based on data detected by the radar and data detected by the camera device; The flight path of the drone is determined based on the position information of the target to be detected and the current information of the drone, wherein the current information of the drone includes: the current position and current situation information of the drone, the current situation information is used to describe the environmental information of the area to which the current position belongs, and the environmental information includes: a no-fly zone.

2. The method according to claim 1, characterized in that Determining the position information of the target to be detected by a perception system installed on the drone includes: detecting the position of the target to be detected by the radar, and obtaining first type of data indicating position information of the target to be detected; Determining adjustment information and an initial flight path for the drone based on the first type of data, wherein the adjustment information is used to adjust the drone's detection field of view to a maximized coverage field of view, the maximized coverage field of view being the detection field of view with the largest area containing a target area among multiple detection fields corresponding to the drone, the target area being an area containing the target to be detected, and the initial flight path being used to instruct the drone to fly toward the target area; After the drone arrives at the target area, the target area is detected by the camera device according to different camera parameters to obtain second-type data, wherein the second-type data includes: images containing the target to be detected and different detection scenes, and different detection scenes correspond to different camera parameters; Processing and analyzing the second type of data using a target detection model to obtain a detection result output by the target detection model, wherein the detection result includes bounding box information of the target to be detected, and the target detection model is obtained by training a neural network model using image data containing the bounding box as training data; The location information of the target to be detected is determined according to the first type of data and the detection result.

3. The method according to claim 2, characterized in that The target detection model is trained using the following method: Obtaining a predicted bounding box output by the object detection model for the training data; For each of the predicted bounding boxes, determining a ground-truth bounding box corresponding to the predicted bounding box in the training data; Determining a union area and an intersection area of the predicted bounding box and the true bounding box, and determining a target area ratio corresponding to the predicted bounding box based on the union area and the intersection area; Determining a weight value corresponding to the predicted bounding box according to a type of the predicted bounding box, and determining an independent loss function corresponding to the predicted bounding box according to the weight value and the target area ratio, wherein the types of the predicted bounding box include: a first-class bounding box, a second-class bounding box, a third-class bounding box, and a fourth-class bounding box, and the area ratios corresponding to the first-class bounding box, the second-class bounding box, the third-class bounding box, and the fourth-class bounding box, respectively, decrease in sequence, and the area ratio is used to indicate a ratio of the area of the predicted bounding box to the area of a target image, and the target image is an image to which the predicted bounding box belongs; Determine the total loss function of the target detection model in this training process according to the multiple independent loss functions corresponding to the multiple predicted bounding boxes, and determine the training process of the target detection model according to the total loss function, wherein the training process includes: continue training and stop training.

4. The method according to claim 2, characterized in that The target detection model is used to process and analyze the second type of data, wherein the target detection model processes and analyzes the second type of data, including: Performing feature extraction processing on the second type of data in a convolutional layer comprising multiple convolutional networks to obtain a feature image, wherein a first preset step size is used in a first convolutional network to perform feature extraction processing on the second type of data, and a second preset step size is used in other convolutional networks to perform feature extraction processing on the second type of data, and the first preset step size is greater than the second preset step size; For each of the feature images, determining an original image corresponding to the feature image, wherein the original image is an image included in the second type of data; determining an area ratio of the bounding box of the feature image according to the area of the bounding box of the feature image and the area of the original image, and determining a type of the bounding box of the feature image according to the area ratio of the bounding box of the feature image; The detection head corresponding to the feature image is determined according to the type of the bounding box of the feature image, and the feature image is processed and analyzed using the detection head, wherein the detection head is a neural network in the target detection model for detecting the target to be detected, and the feature images contained in different types of bounding boxes correspond to different detection heads.

5. The method according to claim 2, characterized in that Before determining the location information of the target to be detected according to the first type of data and the detection result, the method further includes: Acquiring airborne position information, wherein the airborne position information is the current position of the UAV detected by a positioning device installed on the UAV; Calibration information is determined based on the first type of data and the airborne position information, and correction processing is performed on the first type of data based on the calibration information.

6. The method according to claim 1, characterized in that Determining a flight path of the drone based on the position information of the target to be detected and the current information of the drone includes: Generate a map model based on the current information of the drone and the location information of the target to be detected, wherein the map model is composed of a plurality of nodes, each of which represents a portion of a target area, and the target area is the area to which the target to be detected belongs; Determining a starting point of the flight path, an end point of the flight path, and a flight path planning area in the map model, wherein the starting point is a node representing the current position of the UAV, the end point is a node representing the position of the target to be detected, and the flight path planning area is an area within the target area that includes the starting point and the end point but does not include the no-fly zone; The flight path is determined in the flight path planning area according to a starting point of the flight path and an end point of the flight path.

7. The method according to claim 6, characterized in that Determining the flight path in the flight path planning area according to the starting point of the flight path and the end point of the flight path includes: Classifying the flight path planning area into a first area and a second area, wherein the distance between the first type of nodes included in the first area and the end point is greater than a preset distance value, and the distance between the second type of nodes included in the second area and the end point is less than the preset distance value; determining a plurality of first-category waypoints in the first area according to a first search step size, and determining a plurality of second-category waypoints in the second area according to a second search step size, wherein the first search step size is greater than the second search step size; The flight path is determined according to the starting point, a plurality of the first-category waypoints, a plurality of the second-category waypoints, and the end point.

8. The method according to claim 7, characterized in that Determining a plurality of first-category waypoints in the first area according to a first search step size includes: A first-category waypoint screening method is executed in the first area to obtain a plurality of first-category waypoints, wherein the first-category waypoint screening method comprises: S1, determining the starting point as the starting point, and determining nodes whose distance from the starting point is the first search step as first-category nodes to be screened; determining a flight cost for each first-category node to be screened, wherein the flight cost is used to measure resource consumption of the UAV flying from the starting point to the end point via the first-category nodes to be screened; determining the first-category node to be screened corresponding to the smallest flight cost as an adjacent node of the starting point; S2, determining the adjacent node as a new starting point in the first type of waypoint screening method, and executing the first type of waypoint screening method again until the determined new adjacent node belongs to the second area, and then stopping executing the first type of waypoint screening method; Among them, each adjacent node determined by executing the first type of waypoint screening method is the first type of waypoint, and each adjacent node determined by executing the first type of waypoint screening method includes: all adjacent nodes screened in S1 and S2.

9. The method according to claim 7, characterized in that Determining a plurality of second-category waypoints in the second area according to a second search step size includes: Executing a second-category waypoint screening method in the second area to obtain a plurality of second-category waypoints, wherein the second-category waypoint screening method comprises: S3, determining a target first-category waypoint as a starting point, wherein the target first-category waypoint is the last first-category waypoint determined according to the first search step; determining a node whose distance value from the starting point is the second search step as a second-category node to be screened; determining a flight cost of each second-category node to be screened, wherein the flight cost is used to measure resource consumption of the UAV flying from the target first-category waypoint to the end point via the second-category node to be screened; and determining the second-category node to be screened corresponding to the smallest flight cost as an adjacent node of the starting point; S4, determining the adjacent node as a new starting point in the second-category route node screening method, and executing the second-category route node screening method again until the determined second-category node to be screened includes the end point, and then stopping executing the second-category route node screening method; Among them, each adjacent node determined by executing the second type of node screening method is the second type of waypoint, and each adjacent node determined by executing the second type of waypoint screening method includes: all adjacent nodes screened in S3 and S4.

10. A device for determining a flight path, characterized in that: include: a receiving module, configured to receive a path planning instruction sent by a ground control station, wherein the path planning instruction is used to instruct determination of a flight path of the UAV based on location information of a target to be detected, the ground control station including: a plurality of ground base stations communicating with the UAV; a positioning module, configured to determine, in response to the path planning instruction, the position information of the target to be detected by a perception system installed on the drone, wherein the perception system includes: a radar and a camera device, and the position information of the target to be detected is determined based on data detected by the radar and the data detected by the camera device; A determination module is used to determine the flight path of the drone based on the position information of the target to be detected and the current information of the drone, wherein the current information of the drone includes: the current position and current situation information of the drone, the current situation information is used to describe the environmental information of the area to which the current position belongs, and the environmental information includes: a no-fly zone.

11. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, wherein the device where the non-volatile storage medium is located executes the method for determining a flight path according to any one of claims 1 to 9 by running the computer program.

12. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method for determining a flight path according to any one of claims 1 to 9 through the computer program.

13. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method for determining a flight path according to any one of claims 1 to 9 are implemented.

Images