Unmanned aerial vehicle target searching method and device and medium

Abstract

The invention discloses an unmanned aerial vehicle target searching method and device and a medium, and relates to the technical field of unmanned aerial vehicles. The unmanned aerial vehicle target searching method comprises the steps of planning a region full-coverage searching path of an unmanned aerial vehicle based on a pre-designed path searching algorithm capable of estimating steering time cost in flight of the unmanned aerial vehicle; when the unmanned aerial vehicle flies according to the planned path, the target is recognized through a pre-constructed target recognition model, and the target recognition model is configured to be provided with a multi-scale feature fusion model used for fusing target features of different scales and a multi-head self-attention mechanism used for recognizing the target in the shielding area. According to the method, efficient path planning with optimal time consumption is realized through a path search algorithm capable of estimating the steering time cost in the flight of the unmanned aerial vehicle, and the recognition effect can still be ensured under the conditions of small targets, target deformation, shielding and the like through the target recognition model with the multi-scale detection capability and the multi-head self-attention mechanism.

Description

Technical Field

[0001] The present application relates to the field of drone technology, and in particular to a drone target search method, device and medium. Background Art

[0002] In the field of emergency rescue, the search and rescue of life targets has always been a difficult problem in the field of emergency rescue due to its timeliness and geographical complexity. With the rapid development of drone technology, emergency rescue has brought new development trends. The aerial operation of drones can break the constraints of complex regions on the application of technology, and the visual pods on drones can greatly improve the search field of view. Therefore, the use of drones to achieve intelligent search and rescue of high-altitude life targets has become a new trend in the development of emergency equipment.

[0003] However, the current use of drones to search for life targets relies on the operator to manually control the drone and manually observe the video data transmitted back to the ground station to determine whether there is a life target. This method of search operation is cumbersome and has great limitations, making it difficult to meet the current trend of intelligent development of drones. In this regard, there is currently a life target visual recognition technology represented by deep neural networks, but this technology still has the following problems when it comes to specific applications:

[0004] 1) Due to the pitch posture of the drone, the outline of the life form is not complete, making it difficult to identify;

[0005] 2) Due to the high-altitude viewing angle of the drone, the area where the living object exists is small and obscured;

[0006] 3) During the UAV search process, how to achieve efficient path planning, ensure communication status and stable image acquisition quality are important requirements to ensure the efficiency of the search operation.

[0007] Therefore, in order to improve the promotion and effectiveness of the practical application of high-altitude life target search and rescue technology based on drones, an embodiment of the present application proposes a new drone target search solution. Summary of the invention

[0008] The purpose of the embodiments of the present application is to provide a drone target search method, device and medium to at least partially solve the above-mentioned technical problems.

[0009] In order to achieve the above-mentioned objectives, the first aspect of the present application provides a UAV target search method, comprising: planning a regional full-coverage search path for the UAV based on a pre-designed path search algorithm capable of estimating the turning time cost during UAV flight; and identifying the target through a pre-built target recognition model when the UAV flies along the planned path, wherein the target recognition model is configured to have a multi-scale feature fusion model for fusing target features of different scales and a multi-head self-attention mechanism for identifying targets in occluded areas.

[0010] In an embodiment of the present application, the planning of the area full coverage search path of the drone includes: constructing a path search algorithm, the comprehensive cost function adopted by the path search algorithm includes a turning time cost function for calculating the turning time cost between the current node and the adjacent node, and the path search algorithm aims at the optimal path that can cover the entire target search area; and for the target search area, in each path planning from the set planning starting point to the set planning end point, iteratively executing the path search algorithm to obtain the optimal path as the area full coverage search path.

[0011] In the embodiment of the present application, the turning time cost function is designed as:

[0012] d(n)=r*abs(θ n -θ n-1 )

[0013] Where d(n) is the turn time cost function of node n, r is the weight used to adjust the turn time, and θ n is the heading angle of the drone at node n, θ n-1 is the heading angle of the drone at the neighboring node of node n.

[0014] In the embodiment of the present application, the comprehensive cost function is designed as:

[0015] f(n)=g(n)+w*h(n)+k*d(n)

[0016] Among them, f(n) is the comprehensive cost function of node n, g(n) is the path cost function from the planning starting point to node n, h(n) is the heuristic function used to estimate the cost from node n to the planning end point, d(n) is the turning time cost function, and w and k are weight coefficients.

[0017] In the embodiment of the present application, the path cost function is designed as:

[0018] g(n)=w1(n)+w2(n)

[0019] Among them, w1(n) and w2(n) represent the distance cost and height change cost respectively.

[0020] In an embodiment of the present application, the path search algorithm is an optimized A-star algorithm or an optimized greedy best-first search algorithm that adopts the turning time cost function.

[0021] In an embodiment of the present application, the path search algorithm also uses a path interval parameter as a constraint condition, wherein the path interval parameter is configured to limit any one of the following: a row-by-row or column-by-column search coverage method; and a search path interval that can cooperate with the search coverage method to make the search range cover the entire target search area.

[0022] In an embodiment of the present application, planning the area-wide coverage search path of the drone also includes: determining the flight altitude and / or obstacle avoidance strategy of the drone when flying along the planned path based on the path planning result.

[0023] In the embodiment of the present application, the planning is performed based on a 2.5D map with elevation information.

[0024] In an embodiment of the present application, the UAV target search method further includes: using a millimeter wave radar to control the UAV to perform fixed-altitude flight.

[0025] In the embodiment of the present application, the target features of different scales include features based on visible light images and features based on infrared images.

[0026] In an embodiment of the present application, identifying a target through a pre-constructed target recognition model includes: generating feature maps of different scales through the multi-scale feature fusion model to extract features of different scales, and performing preliminary fusion on the extracted features; performing deep fusion on the features after the preliminary fusion through the multi-head self-attention mechanism to enhance the target features of the occluded area; based on the features after the deep fusion, using a VIT architecture encoder to construct a target recognition model; optimizing the target recognition model; and using the optimized target recognition model to identify targets in the target search area.

[0027] In an embodiment of the present application, the multi-scale feature fusion model uses a feature pyramid FPN network structure to generate the feature map.

[0028] In an embodiment of the present application, the optimizing target recognition model includes: performing target classification, loss estimation, effect evaluation, pruning processing and / or inference quantization deployment on the target recognition model.

[0029] A second aspect of the present application provides a drone target search device, comprising: a memory configured to store instructions; and a processor configured to call the instructions from the memory and to implement any of the above-mentioned drone target search methods when executing the instructions.

[0030] A third aspect of the present application provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute any of the above-mentioned drone target search methods.

[0031] Through the above technical scheme, on the one hand, the present application plans the path by means of a path search algorithm that can estimate the turning time cost during the flight of the UAV, thereby achieving high-efficiency path planning with optimal time consumption; on the other hand, through a target recognition model with multi-scale detection capabilities and a multi-head self-attention mechanism, high-altitude life target search and rescue is carried out, and good recognition effects can still be guaranteed in the case of small targets, target deformation, occlusion, etc.

[0032] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0034] Figure 1 A schematic diagram of a process flow of a drone target search method according to an embodiment of the present application is shown schematically;

[0035] Figure 2 A schematic diagram of a process of planning a regional full coverage search path for a drone according to an embodiment of the present application is schematically shown;

[0036] Figure 3 A schematic diagram of a process for optimizing the A-star algorithm structure according to an embodiment of the present application is shown;

[0037] Figure 4 The schematic diagram shows a flow chart of iterative execution optimization of the A-star algorithm according to an embodiment of the present application;

[0038] Figure 5 The following schematically shows a process flow diagram of target identification according to an embodiment of the present application;

[0039] Figure 6 A schematic diagram of a process for constructing a target recognition model according to an embodiment of the present application is schematically shown; and

[0040] Figure 7 The present invention schematically shows a process flow diagram of an intelligent search and rescue operation of a high-altitude life form target by a UAV according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0042] It should be noted that the technical solutions between the various embodiments can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0043] Here, some terms involved in the embodiments of the present application are first introduced to facilitate understanding of the embodiments of the present application:

[0044] 1. A star algorithm: commonly known as the A* search algorithm, it is one of the more popular heuristic search algorithms, which is used to find the lowest cost path from the starting point to the end point on the graph plane. The workflow of the traditional A* algorithm includes starting from the starting point, searching for all possible states of nodes in a certain order, and evaluating them. Based on the traditional A* algorithm, the embodiment of the present application further proposes an optimized A star algorithm, and its main optimization direction is to enable the algorithm to characterize the turning time cost of the drone flight, which will be specifically introduced below and will not be repeated here.

[0045] 2. Greedy best-first search algorithm, which is also a heuristic search algorithm that selects nodes that appear to be closest to the target for expansion at each step. The comprehensive cost function used by the traditional greedy best-first search algorithm is the same as the above-mentioned formula (1), and the embodiment of the present application adds a turning time cost function for estimating the turning time of the drone on the basis of the traditional greedy best-first search algorithm. That is, the turning time cost of the drone in flight is estimated by an optimized greedy best-first search algorithm to more accurately reflect the actual flight situation of the drone. This will also be described in detail below, so it will not be repeated here.

[0046] 3. 2.5D map: In the embodiment of the present application, it refers to a map model that adds elevation information to a conventional two-dimensional map, which is suitable for path planning of aircraft such as drones. Among them, the elevation information can reflect the changes in the ups and downs of the ground in a certain area, and can be understood interchangeably with the altitude information in the embodiment of the present application. In the embodiment of the present application, the drone path planning is performed based on the 2.5D map instead of the 3D map, mainly because the 2.5D map with elevation information is sufficient to reflect the location, flight altitude and other information of the drone, thereby avoiding the huge computational cost brought to the drone controller when rendering a three-dimensional image when using a 3D map. In addition, 3D maps are generally applicable to known areas covered by satellites, and when the embodiment of the present application uses drones for search and rescue of life targets, they usually face unknown areas. In this case, the use of 2.5D maps is more convenient for drone control decisions.

[0047] 4. VIT architecture encoder: Vision Transformer (VIT) is a deep learning model based on the Transformer architecture for image recognition and computer vision tasks. Unlike traditional convolutional neural networks (CNNs), VIT directly treats images as serialized inputs and uses a self-attention mechanism to process pixel relationships in images. In an embodiment of the present application, the VIT architecture encoder is suitable for training target recognition models, and the specific training steps will be introduced below and will not be repeated here.

[0048] 5. Multi-Head Self-Attention Mechanism: It refers to applying the self-attention mechanism to multiple different representations (i.e., "heads") of the input sequence in parallel, and then concatenating these representations to obtain the final output through a linear transformation. The self-attention mechanism is shown in the following formula:

[0049]

[0050] Among them, Q, K and V are three input representation vectors, representing the query vector, key vector and value vector respectively. k represents the dimension of the key vector, K T In the self-attention mechanism, based on the query vector, by calculating the similarity between the query vector and all key vectors K, a weight distribution is obtained for weighted summing of the associated data vector V.

[0051] Based on formula (1), the multi-head self-attention mechanism is as follows:

[0052] MultiHeadAttention(Q,K,V)=Concat(head 1,...,head h )W 0 (2)

[0053] Among them, head i represents the i-th head, and each head is calculated using the following formula:

[0054]

[0055] From the above formulas, it can be seen that the multi-head self-attention mechanism performs multiple self-attention calculations on the input sequence, each calculation uses a different linear transformation matrix (called a "head"), and then merges these results to obtain a more comprehensive information representation. Therefore, the multi-head self-attention mechanism can capture rich contextual information, and the embodiment of the present application uses this advantage to solve the target occlusion problem.

[0056] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0057] Figure 1 The following schematically shows a flow chart of a method for searching a target by a drone according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a method for searching a target by a UAV, which may include the following steps S100-S200.

[0058] Step S100, based on a pre-designed path search algorithm capable of estimating the turning time cost of the UAV during flight, a regional full coverage search path of the UAV is planned.

[0059] Since the search area selected manually may be irregular or scattered, the conventional round-trip (BF) and spiral (SP) search methods are difficult to achieve good results when targeting relatively complex search areas. Therefore, the embodiment of the present application constructs a path search algorithm that can estimate the turning time cost of the UAV in flight to plan the area coverage search path of the UAV. Based on this, Figure 2 As shown, in a preferred embodiment, planning a regional full coverage search path for a drone may include steps S110-S120.

[0060] Step S110, constructing a path search algorithm.

[0061] The comprehensive cost function used by the path search algorithm includes a turning time cost function for calculating the turning time cost between the current node and the adjacent node, and the path search algorithm aims at an optimal path that can cover the entire target search area.

[0062] For example, considering that the angular velocity of the drone is consistent during flight, the direction angle of the drone can be used to characterize its turning time cost to obtain the turning time cost function. Specifically, the turning time cost function is shown as follows:

[0063] d(n)=r*abs(θ n -θ n-1 ) (4)

[0064] Here, d(n) represents the turn time cost function of node n; r is a constant used to adjust the weight of the turn time; θ n is the heading angle of the drone at node n, θ n-1 is the heading angle of the drone at the neighboring node of node n.

[0065] Further, based on formula (4), the comprehensive cost function of the path search algorithm such as the traditional A-star algorithm or the traditional greedy best first search algorithm is optimized, and the corresponding optimized A-star algorithm or optimized greedy best first search algorithm can be obtained. That is, the path search algorithm can be the optimized A-star algorithm or the optimized greedy best first search algorithm using the turning time cost function, and its comprehensive cost function is shown in the following formula:

[0066] f(n)=g(n)+w*h(n)+k*d(n) (5)

[0067] Wherein, f(n) is the comprehensive cost function of node n; g(n) is the path cost from the planning starting point to node n; h(n) is the heuristic function used to estimate the cost from node n to the planning end point; d(n) represents the path interval parameter cost. Combined with formula (4), in the embodiment of the present invention, d(n) is the turning time cost function; w and k are weight coefficients used to balance the importance of the heuristic function and the path interval parameter.

[0068] For example, the optimized A-star algorithm takes the comprehensive cost, path interval parameter and fixed-altitude flight of the UAV as constraints, and aims at the optimal path that can cover the entire target search area. As shown in formula (5), the comprehensive cost includes the path cost from the planning starting point to the current node, the path cost from the current node to the planning end point, and the path interval parameter cost based on the turning time cost between the current node and the adjacent node.

[0069] In the example, the path spacing parameter can also be configured to define any of the following: a row-by-row or column-by-column search coverage method; and a search path spacing that can cooperate with the search coverage method to make the search range cover the entire target search area. Among them, the specific search coverage method and search path spacing are conducive to ensuring coverage of the entire area. In addition, the millimeter wave radar can be used to control the drone to perform fixed-altitude flight, and the flight altitude can be determined using a 2.5D map. For example, the map data and elevation information of the target search area are first obtained, and processed to convert into a grid map, and then the altitude information is determined.

[0070] In actual operation, the interval size of the drone's search path can be determined by the drone's target recognition field of view. The flight altitude of the path point is controlled by the millimeter wave radar, so that the drone maintains a fixed altitude above the ground to ensure the accuracy of the drone's target search. Therefore, when constructing the path search algorithm, the drone's path interval parameter is given by the drone's search range field of view, and when the drone maintains terrain-simulating flight, the distance from the ground is fixed, so the search path interval is a fixed value.

[0071] As described above, the path search algorithm may be an optimized A-star algorithm or an optimized greedy best-first search algorithm. The implementation of the optimized A-star algorithm and the optimized greedy best-first search algorithm are respectively specifically introduced through two examples.

[0072] In Example 1, for step S110, Figure 3 As shown, the following steps S111-S115 can be used to implement the basic framework and data structure of the optimized A-star algorithm.

[0073] Step S111, after initializing the map, define the starting point of the drone and the location of the target search area, and obtain the elevation information of the area.

[0074] Step S112, initialize the open list and the closed list, add the starting point to the open list, and consider the comprehensive cost.

[0075] Among them, the open list is used to store the nodes to be detected, and the closed list is used to store the checked nodes. In addition, for each path point, the above formula (5) is used to calculate the comprehensive cost, but for the starting point, it needs to satisfy:

[0076] g(n)=0 (6)

[0077] And, for h(n) in formula (5), the following formula is used for calculation:

[0078]

[0079] Among them, (x n ,y n ,z n) is the position coordinate of node n, (x goal ,y goal ,z goal ) is the location coordinate of the planned end point. However, it should be noted that since the path planning is segmented, the end point refers to the planned target point corresponding to the current road section, rather than the end point of the entire target search area.

[0080] Step S113, take out the node with the smallest comprehensive cost from the open list as the current node, and check whether the current node is the final planning node (i.e., the planning end point). If it is, end the algorithm; if not, transfer the current node to all neighboring nodes for expansion, traverse the current node and all adjacent nodes, and calculate their comprehensive cost function.

[0081] Specifically, based on equations (5)-(7), the positions of adjacent nodes can be determined as:

[0082] (x new ,y new ,z new )=(x n +Δx,y n +Δy,z n +Δz) (8)

[0083] Among them, (x new ,y new ,z new ) represents the position coordinates of the neighboring nodes of node n, and (Δx, Δy, Δz) represents the distance between node n and its neighboring nodes.

[0084] Therefore, when m is used to represent the adjacent nodes, the cost function of the adjacent nodes can be updated by the following formula:

[0085]

[0086] Among them, for the heuristic function h(m), it is easy to know that the Euclidean distance is used as the heuristic function.

[0087] Step S114, starting from the key node, backtracking to the parent node of each node, guiding back to the starting point, and rebuilding the path.

[0088] Step S115, displaying the flight path and coverage area of ​​the drone on a 2.5D map.

[0089] In this way, according to the above steps S111-S115, the optimized A-star algorithm is used to calculate the optimal path from the current node to the next node each time, gradually covering the entire area.

[0090] In Example 2, for step S110, the basic framework and data structure of the optimized greedy best first search algorithm are similar to those of the optimized A-star algorithm in Example 1 above, and the main difference lies in the design of the path cost function. For the comprehensive cost function of formula (5), the path cost function of the optimized greedy best first search algorithm needs to satisfy:

[0091] g(n)= w1(n)+w2(n) (10)

[0092] In the optimization greedy best-first search algorithm, the path cost represented by g(n) is the flight distance cost w(n), which is the actual flight distance of the drone moving from one node to another. The flight distance cost w(n) consists of two parts: the distance cost w1(n) and the height change cost w2(n).

[0093] Among them, w1(n) = distance(current, neighbor), using Euclidean distance or Manhattan distance calculation, current represents the current node, neighbor represents the adjacent node, and distance() is the distance calculation function. w2(n) = |z_current-z_neighbor|, is designed to take into account the impact of altitude changes on the drone.

[0094] It should be noted that the above formula (10) can also be used to optimize the path cost function of the optimized A-star algorithm.

[0095] Back to Figure 2 After completing the construction of the path search algorithm, further perform the following step S120.

[0096] Step S120, for the target search area, in each path planning from the set planning starting point to the set planning end point, iteratively execute the path search algorithm to obtain an optimal path as the search path that fully covers the area.

[0097] For step S120, Figure 4 As shown, taking the optimized A-star algorithm as an example, iteratively executing the optimized A-star algorithm may include executing the following steps S121-S126 until a path is found or it is determined that the path does not exist.

[0098] Step S121, converting the coordinates of the starting point and the end point into grid unit coordinates and initializing them.

[0099] Step S122, creating an empty open list and a closed list for storing nodes to be expanded and nodes that have been expanded.

[0100] Step S123, adding the starting point node to the open list, and setting its path cost g(n) to 0.

[0101] Step S124, repeatedly execute the following A* algorithm association steps until the open list is empty: select the node with the minimum comprehensive cost function f(n) as the current node current; move the current node current to the closed list and delete it from the open list; if the current node current is the end point, the path search is completed and the loop is exited; if the current node current is not the end point, expand its adjacent nodes.

[0102] Step S125, for the adjacent node neighbor, perform the following update steps: calculate the path cost g(neighbor) and update path from the starting point to the adjacent node neighbor; consider the adjacent cells in the horizontal direction and the impact of height changes in the vertical direction; for each adjacent node neighbor, calculate the comprehensive cost function f(neighbor), and check whether the cost needs to be updated; if the neighbor is not in the open list, add it to the open list and update the cost; if the neighbor is already in the open list, compare the new cost with the old cost, and select the smaller cost to update the path.

[0103] Step S126, check whether the path exists. After the loop ends, if the open list is empty and the end point is not found, it means that the path does not exist.

[0104] Thus, through the above steps S121-S126, the preliminary path planning is completed. It should be noted that the steps of using the optimized greedy best-first search algorithm to perform preliminary path planning are similar to steps S121-S126, and will not be repeated here.

[0105] In the example, back to Figure 2 Planning the area-wide coverage search path of the drone may also include the following step S130:

[0106] Step S130, determining the flight altitude and / or obstacle avoidance strategy of the UAV when flying along the planned path according to the path planning result.

[0107] This step S130 is intended to ensure the consistency of path planning, and specifically includes: determining the altitude of the UAV when flying along the path based on the path planning results; using millimeter-wave radar to perform fixed-altitude flight, maintain a fixed distance from the ground, and leave redundant parameters; combining a preset obstacle avoidance strategy, controlling the UAV to fly in the vertical direction according to the obstacle avoidance strategy to avoid collision or interference.

[0108] So, back to Figure 1, as described above, for step S100, the embodiment of the present application essentially provides a 2.5D UAV area full coverage path planning method based on a path search algorithm that can estimate the turning time cost of the UAV flight, that is, by optimizing the A-star algorithm or optimizing the greedy best-first search algorithm, combined with the turning time cost function, the optimal path planning of the UAV is realized. In this way, the corresponding path planning focuses on the optimal path and turning time, combined with millimeter-wave radar and elevation information, etc., and can be effectively applied to the UAV search path planning in large outdoor scenes.

[0109] Further, continue to refer to Figure 1 The drone target search method of the embodiment of the present application also includes the following step S200.

[0110] Step S200, when the UAV flies along the planned path, the target is identified by using a pre-built target recognition model.

[0111] Among them, the target recognition model is configured to have a multi-scale feature fusion model for fusing target features of different scales and a multi-head self-attention mechanism for identifying targets in occluded areas.

[0112] Regarding step S200, the design idea is: in the drone's life target search, the target state is prone to being obscured, the target area is small, etc., which makes it difficult for the common neural network recognition model trained based on the public data set of human targets to achieve a good recognition effect, and the frequency of missed detection and false detection is high. Therefore, the embodiment of the present application aims to construct a new target recognition model to improve the target recognition accuracy through a multi-scale feature fusion model and to deal with the target occlusion problem through a multi-head self-attention mechanism.

[0113] Among them, the target features of different scales targeted by the multi-scale feature fusion model include features based on visible light images and features based on infrared images. Based on this, the corresponding target recognition model with the multi-scale feature fusion model may also be understood as a dual-light target recognition model that combines frame dynamic information, using the operation of fusing visible light and infrared multi-frame information to improve the accuracy of life target recognition.

[0114] In addition, the multi-head self-attention mechanism is shown in the above formulas (1)-(3). On the basis of the preliminary fusion of features by the multi-scale feature fusion model, it further performs deep fusion of features.

[0115] Accordingly, in the example, Figure 5 As shown, for step S200, identifying the target through the pre-built target recognition model includes the following steps S210-S230.

[0116] Step S210: generating feature maps of different scales through the multi-scale feature fusion model to extract features of different scales, and preliminarily fusing the extracted features.

[0117] For example, the following formula is used to extract feature maps from visible light and infrared images:

[0118] F vis =Conv vis (I vis ) (11)

[0119] F ir =Con vir (I ir ) (12)

[0120] Among them, I vis and I ir Represent visible light and infrared images respectively, Conv vis and Con vir The conventional convolutional neural network feature extraction module for extracting features from the corresponding image, F vis and F ir Represent the corresponding extracted visible light and infrared features respectively.

[0121] After extracting the features, the following formula is used to further perform feature splicing:

[0122] F concat =Concat(F vis ,F ir ) (13)

[0123] Among them, F concat It is the concatenated feature, and concat is the regular feature concatenation module.

[0124] Furthermore, multiple frames of information are picked up and concatenated along the channel dimension to form a large input sequence, as shown below:

[0125]

[0126] Among them, cat is the channel dimension splicing method, is the concatenated feature information of the i-th single frame, and num_frames is the number of frames.

[0127] The multi-scale feature fusion model can enhance the detection capability of the target recognition model for small targets, deformed targets, etc.

[0128] Step S220, for the initially fused features, deep fusion is performed through the multi-head self-attention mechanism to enhance the target features of the occluded area.

[0129] Based on the above formula (14), in order to further extract features, the channel attention mechanism is introduced to achieve the feature extraction shown in the following formula:

[0130] F attn =σ(FC 2 (δ(FC 1 (GAP(F mul_concat ))))))*F mul_concat (15)

[0131] Among them, GAP represents global average pooling, FC 1 and FC 2 is a fully connected layer, σ and δ are Sigmoid and ReLU activation functions respectively.

[0132] Further using the self-attention mechanism to improve the expressiveness of the fusion features, we can obtain the following equations (2) to (4):

[0133] F fusion =Attention(F attn ) (16)

[0134] In this way, the target features of the occluded area are enhanced through the deep fusion of the multi-head self-attention mechanism.

[0135] Step S230, based on the deeply fused features, a target recognition model is constructed using a VIT architecture encoder.

[0136] In the example, Figure 6 As shown, the target recognition model is constructed through the following steps S231-S235.

[0137] In step S231, the fused feature map is cut into patches of a fixed size and embedded into a high-dimensional vector first.

[0138] For example, cutting and embedding are accomplished by:

[0139] E i =Linear(Patch i ) (17)

[0140] Among them, E i Represents the embedding vector of the i-th patch.

[0141] Step S232, adding information coding to each patch to store the spatial relationship and adding time position coding.

[0142] For example, each patch is processed as follows:

[0143]

[0144] Among them, parameter is a trainable proof parameter, zeros represents the initialization matrix to zero, num_frames is the number of frames, dim represents the feature vector dimension, TE represents the temporal position encoding, P i Represents information encoding, E' i Represents the embedding vector after adding information and temporal position encoding.

[0145] Step S233, using a multi-layer Transformer encoder to process the embedded vector.

[0146] For example, the following is used for processing:

[0147] Z l = TransformerLayer(Z l-1 ) (19)

[0148] Among them, Z l Represents the output of the lth layer. The input of the first layer Transformer consists of E' of all single patches i Combined.

[0149] In this way, a target recognition model with a multi-scale feature fusion model and a multi-head self-attention mechanism is obtained, so that the robustness of the model can be enhanced by using multi-scale feature fusion and multi-head self-attention mechanisms for small target detection and occlusion problems. For example, multi-scale feature fusion is used to enhance the model's ability to detect small targets, and a feature pyramid (FPN) network structure is used to generate feature maps of different scales, as shown in the following formula:

[0150] F multi-scale =FPN(F fusion ) (20)

[0151] Among them, F fusion represents the fusion feature, and FPN further enhances the feature map through the top-level bottom-up path and fuses features of different scales through lateral connections, as shown in the following formula:

[0152] P i =Conv(C i )+Upsample(P i+1 ) (twenty one)

[0153] Among them, C i is the feature map of the i-th layer, P i Represents the fused feature map.

[0154] For example, to solve the occlusion problem, a multi-head self-attention mechanism is introduced to focus on the unoccluded feature areas, which can be expressed as follows:

[0155] F Mul_attn =MultiHeadAttention(F input ) (twenty two)

[0156] Step S234, optimizing the target recognition model so that the target recognition model can achieve a preset recognition effect.

[0157] In an example, the optimization includes target classification, loss estimation, effect evaluation, pruning processing and / or inference quantization deployment for the target recognition model.

[0158] For target classification, for example, the encoder output is passed through the classification head to classify human targets, and then the regression head is used to regress the bounding box of the human target. Following equation (19), the rules used can be described as follows:

[0159] Class=Softmax(Linear(Z L )) (twenty three)

[0160] BBox=Linear(Z L ) (twenty four)

[0161] Training on losses, such as joint training using classification loss and bounding box loss.

[0162] Moreover, based on the loss training, the model can be further trained using stochastic gradient descent (SGD) or Adam optimization, and the learning rate scheduling strategy of learning rate scaling can be set for training.

[0163] Furthermore, after evaluating the model's effectiveness, you can consider pruning the model and deploying it for inference quantization to improve the model's real-time performance. Model pruning can effectively reduce the number of model parameters and the amount of calculation by accurately identifying and removing parameters or connections that contribute less to performance, making the model more efficient and lightweight; inference quantization can compress the model size, thereby increasing the model's inference speed, and quantization can also reduce memory usage during the model inference process, thereby reducing the cost of model deployment.

[0164] Step S235, using the optimized target recognition model to recognize targets in the target search area.

[0165] Through the above steps S231-S235, the embodiment of the present application effectively utilizes the complementary information of visible light and infrared images, inter-frame temporal dynamic information, etc., and designs an innovative target recognition model applied to the Transfomer architecture. The model fully integrates the characteristics of different spectra and can stably identify living targets even in complex environments.

[0166] So, back to Figure 1 For step S200, the embodiment of the present application constructs a drone life target recognition model framework based on dual-light fusion and transformer architecture, and the application of multi-scale feature fusion model and multi-head self-attention mechanism is equivalent to adding a small target attention mechanism and an occluded target response module, which better promotes the drone's recognition and tracking of life targets.

[0167] On the basis of completing the path planning in step S100 and completing the target identification in step S200, as Figure 7 As shown, the drone target search method of the embodiment of the present application can be used to perform drone high-altitude life target intelligent search and rescue operations. Figure 7 First, the path planning function is used on the operating system of the drone to plan the route (equivalent to the path) on the map and automatically generate the search path; the drone is equipped with a three-light pod to perform terrain-simulating flight according to the planned route. During the flight, the life-based target recognition function is turned on. After the target is recognized, the gimbal control and the pod zoom strategy are combined to complete the target positioning through pod attitude adjustment and lens zoom, and the recognition image and positioning information are transmitted back to the control center. The control center decides whether to start the drone tracking and throwing, shouting and other tasks according to the command of the control center.

[0168] It should be noted that the current field drone life target recognition technology does not carefully consider the situation where there is severe occlusion of human targets in the field search and the situation where the human targets occupy a small number of pixels in the image, and does not propose a targeted solution strategy for how to achieve optimal path search, stable detection, and tracking in the target recognition process. Figure 7 As shown, we can make full use of drones and 2.5D map platforms, combine visible light and infrared data with high-performance, multi-scale detection results of excellent target recognition models, realize accurate recognition of living targets, and combine target tracking technology with pod control strategy. While ensuring the target recognition and detection effect, we can effectively realize further rescue work, improve the actual application ability of target recognition technology, and can effectively cope with complex environments.

[0169] Accordingly, the drone target search method of the embodiment of the present application plans the path through a path search algorithm that can estimate the turning time cost of the drone in flight, achieving time-optimized and efficient path planning on a 2.5D map. On the other hand, a target recognition model with multi-scale detection capabilities and a multi-head self-attention mechanism is used to search and rescue high-altitude life targets, and a good recognition effect can still be guaranteed in the case of small targets, target deformation, occlusion, etc. Therefore, the drone target search method of the embodiment of the present application improves the practical application promotion and effectiveness of the drone-based high-altitude life target search and rescue technology, can meet the search of life targets in conventional general scenarios and small targets, target deformation, and target occlusion scenarios, and improves the level of emergency rescue intelligence.

[0170] The present application also provides a drone target search device, comprising: a memory configured to store instructions; and a processor configured to call the instructions from the memory and implement any of the above drone target search methods when executing the instructions. In an example, the drone target search device is, for example, an onboard computer in a drone flight control system or a computing device in a ground control station.

[0171] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the drone target search involved in the embodiment of the present application can be achieved by adjusting the kernel parameters.

[0172] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory may include at least one memory chip.

[0173] The embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned drone target search method. The machine is, for example, a separately configured controller or an onboard controller or a ground station controller of a drone.

[0174] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0175] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0178] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0179] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0180] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0181] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0182] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for searching a target by an unmanned aerial vehicle, characterized in that: include: Based on a pre-designed path search algorithm that can estimate the turning time cost of the UAV during flight, a regional full coverage search path for the UAV is planned; as well as When the UAV flies along the planned path, the target is identified by a pre-built target recognition model, wherein the target recognition model is configured to have a multi-scale feature fusion model for fusing target features of different scales and a multi-head self-attention mechanism for identifying targets in occluded areas.

2. The drone target search method according to claim 1, characterized in that: The planning of the regional full coverage search path of the UAV includes: Construct a path search algorithm that satisfies the following conditions: The comprehensive cost function used by the path search algorithm includes a turning time cost function for calculating the turning time cost between the current node and the adjacent node; and The path search algorithm aims at an optimal path that can cover the entire target search area; and For the target search area, in each path planning from a set planning starting point to a set planning end point, the path search algorithm is iteratively executed to obtain an optimal path as a search path that fully covers the area.

3. The drone target search method according to claim 2, characterized in that: The turning time cost function is designed as: d(n)=r*abs(θ n -θ n-1 ) Where d(n) is the turn time cost function of node n, r is the weight used to adjust the turn time, and θ n is the heading angle of the drone at node n, θ n-1 is the orientation angle of the drone at the neighboring node of node n.

4. The drone target search method according to claim 2, characterized in that: The comprehensive cost function is designed as: f(n)=g(n)+w*h(n)+k*d(n) Among them, f(n) is the comprehensive cost function of node n, g(n) is the path cost function from the planning starting point to node n, h(n) is the heuristic function used to estimate the cost from node n to the planning end point, d(n) is the turning time cost function, and w and k are weight coefficients.

5. The drone target search method according to claim 2, characterized in that: The path cost function is designed as: g(n)=w1(n)+w2(n) Among them, w1(n) and w2(n) represent the distance cost and height change cost respectively.

6. The method for searching for a target by an unmanned aerial vehicle according to claim 2, characterized in that: The path search algorithm is an optimized A-star algorithm or an optimized greedy best-first search algorithm using the turn time cost function.

7. The drone target search method according to claim 2, characterized in that: The path search algorithm also uses a path spacing parameter as a constraint, wherein the path spacing parameter is configured to limit any one of the following: a row-by-row or column-by-column search coverage method; and a search path spacing that can cooperate with the search coverage method to make the search range cover the entire target search area.

8. The method for searching for a target by an unmanned aerial vehicle according to claim 2, characterized in that: The planning of the regional full coverage search path of the drone also includes: According to the path planning result, the flight altitude and / or obstacle avoidance strategy of the UAV when flying along the planned path is determined.

9. The method for searching for a target by an unmanned aerial vehicle according to any one of claims 1 to 8, characterized in that: The planning is performed based on a 2.5D map with elevation information.

10. The drone target search method according to any one of claims 1 to 8, characterized in that: The UAV target search method also includes: using a millimeter wave radar to control the UAV to perform fixed-altitude flight.

11. The method for searching for a target by an unmanned aerial vehicle according to claim 1, characterized in that: The target features of different scales include features based on visible light images and features based on infrared images.

12. The method for searching for a target by an unmanned aerial vehicle according to claim 1, characterized in that: The identifying of the target by using the pre-built target recognition model comprises: By using the multi-scale feature fusion model, feature maps of different scales are generated to extract features of different scales, and the extracted features are preliminarily fused; For the initially fused features, deep fusion is performed through the multi-head self-attention mechanism to enhance the target features of the occluded area; Based on the deeply fused features, a target recognition model is constructed using a VIT architecture encoder; Optimizing the target recognition model; and The optimized target recognition model is used to recognize targets in the target search area.

13. The method for searching for a target by an unmanned aerial vehicle according to claim 12, characterized in that: The multi-scale feature fusion model uses a feature pyramid FPN network structure to generate the feature map.

14. The drone target search method according to claim 12, characterized in that: The optimizing the target recognition model includes: performing target classification, loss estimation, effect evaluation, pruning processing and / or inference quantization deployment on the target recognition model.

15. A drone target search device, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and implement the drone target search method according to any one of claims 1 to 14 when executing the instructions.

16. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for enabling a machine to execute the drone target search method according to any one of claims 1 to 14.

Images