Unmanned aerial vehicle control method, computer device and storage medium
Patent Information
- Application Number
- CN202610992364.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-03
- Publication Date
- 2026-09-18
Smart Images
Figure CN122776816A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of unmanned aerial vehicle (UAV) technology, and more particularly to UAV control methods, computer equipment, and storage media. Background Technology
[0002] With the development of drone technology, drones equipped with visual sensors have been widely used in security patrols, public safety monitoring, and intelligent traffic management. In these scenarios, drones often need to identify and track specific targets within a group of people on the ground. How to effectively control drones in group target tracking scenarios is a key technical problem that needs to be solved. Summary of the Invention
[0003] This application provides a method for controlling unmanned aerial vehicles (UAVs), computer equipment, and storage media.
[0004] Firstly, a method for controlling an unmanned aerial vehicle (UAV) is provided, including: Target identification information from multiple drones is acquired to obtain multiple target identification information, wherein the target identification information includes multimodal information of at least one identified target; Determine descriptive information for each of the identified targets, the descriptive information being determined based on the multimodal information of the corresponding identified target; Based on preset filtering conditions and the description information, the target is identified and filtered to obtain a first target set; Determine the evaluation value of each target in the first target set, and filter the targets in the first target set based on the evaluation values to obtain a second target set; The identified targets in the second target set are used as tracking targets. The control command for at least one of the multiple drones is determined, and the corresponding control command is sent to the at least one drone.
[0005] In this technical solution, by aggregating target identification information from multiple UAVs, the overall perception range of ground-based group targets is expanded, providing a more comprehensive target information foundation for the rational control of UAVs. A first target set is obtained by matching and filtering identified targets based on preset screening conditions, and a second target set is obtained by prioritizing the first target set based on evaluation values. This achieves two-level filtering and quantitative assessment of the importance of group targets, ensuring that UAV control decisions are based on a reasonable judgment of target importance, avoiding the indiscriminate allocation of UAV resources to group targets. Furthermore, the identified targets in the second target set are used as tracking targets. By comprehensively considering the status of multiple UAVs, control commands for at least one UAV are determined and issued for execution, achieving a reasonable matching and dynamic scheduling between the tracking target and the executing UAV.
[0006] In conjunction with the first aspect, in one possible implementation, the multimodal information includes the appearance feature information, identity feature information, and behavioral feature information of the target being identified; For each of the identified targets, descriptive information of the identified target is determined, including: The appearance feature information, the identity feature information, and the behavioral feature information are fused to obtain the description information of the identified target.
[0007] In conjunction with the first aspect, in one possible implementation, the step of identifying targets based on preset filtering conditions and the descriptive information to obtain a first target set includes: The preset filtering conditions are matched with the description information of each of the identified targets; The first target set is obtained by filtering the identification targets whose description information matches the preset filtering conditions.
[0008] In conjunction with the first aspect, in one possible implementation, for each identified target in the first target set, determining the evaluation value of the identified target includes: Acquire the identity feature information, behavioral feature information, and target density of the area where the target is located; A first evaluation value is determined based on the identity feature information, a second evaluation value is determined based on the behavioral feature information, and a third evaluation value is determined based on the target density. The evaluation value of the identified target is determined based on the first evaluation value, the second evaluation value, and the third evaluation value.
[0009] In conjunction with the first aspect, in one possible implementation, the step of identifying and filtering targets in the first target set based on the evaluation value to obtain a second target set includes: The second target set is obtained by filtering out targets in the first target set whose evaluation values are greater than or equal to preset values; Alternatively, each of the identified targets in the first target set can be sorted in descending order according to the evaluation value; The first K targets are selected and arranged to obtain the second target set, where K is a positive integer.
[0010] In conjunction with the first aspect, in one possible implementation, determining the control command for at least one of the multiple drones by using the identified targets in the second target set as tracking targets includes: Based on the location information of each of the drones and the location information of each of the tracking targets, at least one target drone is determined from the plurality of drones, and the tracking target to be tracked for each target drone is determined; For each target UAV, based on the position information of the target being tracked by the target UAV, the flight control command and gimbal control command of the target UAV are determined, and the flight control command and the gimbal control command are used as the control command of the target UAV.
[0011] In conjunction with the first aspect, in one possible implementation, determining at least one target drone from the plurality of drones based on the location information of each of the drones and the location information of each of the tracking targets, and determining the tracking target to be tracked for each of the target drones, includes: Based on the location information of each UAV and the location information of each tracked target, the distance between each UAV and each tracked target is determined; Based on the distance, at least one drone is assigned to each of the multiple drones to obtain the at least one target drone and the target drone to be tracked for each target drone; wherein the distance between the target drone and the corresponding target drone to be tracked meets a preset condition.
[0012] In conjunction with the first aspect, in one possible implementation, determining the flight control commands and gimbal control commands of the target UAV based on the position information of the target UAV to be tracked includes: Based on the position information of the target drone to be tracked and the position information of the target drone, the flight speed vector of the target drone, the gimbal pitch angle adjustment amount of the target drone, and the gimbal yaw angle adjustment amount of the target drone are determined. The flight speed vector is used as the flight control command, and the gimbal pitch angle adjustment and the gimbal yaw angle adjustment are used as the gimbal control commands.
[0013] Secondly, a drone control device is provided, comprising: The acquisition module is used to acquire target identification information from multiple drones to obtain multiple target identification information, wherein the target identification information includes multimodal information of at least one identified target; A determining module is used to determine the description information of each of the identified targets, the description information being determined based on the multimodal information of the corresponding identified target; The filtering module is used to filter the identified targets based on preset filtering conditions and the description information to obtain a first target set; The determining module is further configured to determine the evaluation value of each identified target in the first target set; The filtering module is further configured to identify and filter targets in the first target set based on the evaluation value to obtain a second target set; The determining module is further configured to use the identified targets in the second target set as tracking targets to determine the control commands for at least one of the multiple drones; The communication module is used to send corresponding control commands to the at least one drone.
[0014] Thirdly, a computer device is provided, including a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor causing the computer device to implement the unmanned aerial vehicle control method of the first aspect described above when executing the one or more computer programs.
[0015] Fourthly, a computer-readable storage medium is provided, which stores a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the unmanned aerial vehicle control method of the first aspect.
[0016] Fifthly, a computer program product is provided that, when run on a computer device, causes any of the methods provided in the first aspect to be executed. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 A schematic diagram of a system architecture provided for an embodiment of this application; Figure 2 A flowchart illustrating a drone control method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a drone control device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0020] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.
[0021] The technical solution of this application is described in detail below.
[0022] See Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of this application, such as... Figure 1 As shown, the system architecture includes: computer equipment, multiple drones, and a security management and monitoring terminal. The computer equipment is communicatively connected to the multiple drones and the security management and monitoring terminal.
[0023] Among them, computer equipment is the core processing and control unit of the entire system. Computer equipment can be servers (such as ground control station servers, cloud servers), workstations, or electronic devices with corresponding data processing capabilities (such as laptops, industrial computers, desktop industrial control hosts, etc.). It plays a pivotal role in data aggregation, analysis, decision-making, and command issuance, and is the key execution entity for realizing multi-machine collaborative identification and dynamic matching of tracking targets.
[0024] A drone is an unmanned aerial vehicle equipped with aerial cameras, infrared thermal imaging, ranging, environmental sensors, and wireless communication modules. Drones are responsible for real-time photography, detection, and identification of ground-based targets, generating target identification information, and uploading this information to computer equipment. Simultaneously, as the executor of control commands, the drone adjusts its flight attitude, heading, speed, and other parameters accordingly upon receiving control commands from the computer equipment to continuously track designated targets. The collaborative operation of multiple drones compensates for the limited field of view and the risk of missing targets inherent in a single drone.
[0025] A security management and monitoring terminal is a human-machine interface device used by personnel in security, public security, or traffic management sectors. It displays in real-time target identification results obtained from UAV processing, descriptive information and target selection results from computer processing, UAV flight status, and control command execution status. This facilitates monitoring, verification, and necessary manual intervention or confirmation of the overall tracking situation. The terminal can be a large monitoring screen, tablet computer, command and dispatch console, or other devices with display and interactive functions, playing a role in human-machine collaboration, decision support, and supervision, ensuring that the system remains under control and monitorable by personnel while operating automatically.
[0026] See Figure 2 , Figure 2 This is a flowchart illustrating a drone control method provided in an embodiment of this application, as shown below. Figure 2 As shown, this method is applied to the aforementioned computer device, and the method includes the following steps: S201, acquire target identification information from multiple drones to obtain multiple target identification information, wherein the target identification information includes multimodal information of at least one identified target.
[0027] Before step S201, the operator sets task parameters through the visual operation interface of the computer device. The task parameters include at least one of the following: geofence coordinates of the flight area, preset patrol route, preset screening conditions, and weight coefficients α, β, and γ associated with the evaluation value.
[0028] Among them, the flight area geofence coordinates refer to the electronic boundary that defines the area where drones are allowed to fly, which is a closed area enclosed by latitude and longitude coordinates.
[0029] Pre-set patrol routes refer to a series of waypoints planned in advance inside the fence, and the drone will then automatically patrol according to the order of the waypoints.
[0030] The preset filtering conditions refer to the criteria used to filter and identify targets, performing preliminary screening. When the preset filtering conditions are empty, it means no filtering is performed, and all identified targets participate in the subsequent evaluation. When the preset filtering conditions are not empty, it means filtering is performed, and only identified targets that meet the preset filtering conditions are evaluated in the subsequent evaluation.
[0031] The weighting coefficients α, β, and γ associated with the evaluation value represent the proportions of the three dimensions used in the evaluation. For example, α corresponds to the identity feature information, β corresponds to the behavioral feature information, and γ corresponds to the target density, with α + β + γ = 1. The values of α, β, and γ vary depending on the scenario. For example, in a security tracking scenario, α is increased, for example, to 0.5, β = 0.25, and γ = 0.25. In a sudden incident scenario, β is increased, for example, to 0.5, α = 0.25, and γ = 0.25. In a crowd control scenario, γ is increased, for example, to 0.5, β = 0.25, and α = 0.25.
[0032] The computer system sends mission parameters to each drone. Upon receiving the parameters, the drone performs an automatic self-check, including checks on battery level, GPS positioning, wireless communication, camera, obstacle avoidance sensors, and power system. The drone only initiates flight if the self-check is successful. If a fault is detected, the drone reports it to the computer system to alert the operator.
[0033] After the drone starts flying, it performs its flight mission according to the preset patrol route in the mission parameters. The edge computing nodes of the drone simultaneously start the video stream processing pipeline to process the video streams collected by the drone.
[0034] The drone continuously acquires video streams during flight, which are then input into edge computing nodes for processing at a rate of 30fps. The edge computing nodes perform multi-target detection and tracking on each frame, detecting targets in the current frame and determining the bounding box coordinates and tracking ID for each target. Specifically, for newly detected targets, a corresponding tracking ID is assigned; for previously detected targets, the corresponding trajectory state is updated to achieve continuous tracking of the same target across consecutive video frames.
[0035] After completing target detection and tracking, the edge computing nodes perform appearance feature extraction, face feature extraction, and human skeleton key point extraction for each identified target.
[0036] The apparent feature extraction includes: identifying the bounding box corresponding to the target and obtaining the color histogram feature vector F of the bounding box region corresponding to the target. color .
[0037] Facial feature extraction includes: locating the facial region within the bounding box corresponding to the target; performing facial recognition on the facial region to extract the facial feature vector; and then processing the extracted facial feature vector F. face The facial feature vectors of the target individuals are matched with those of the preset individuals to obtain the identity matching score of the target.
[0038] Human skeleton key point extraction includes: extracting the skeleton key points corresponding to the target, splicing the skeleton key points of T consecutive frames into a skeleton key point sequence according to the time sequence, inputting the skeleton key point sequence into the action recognition network for recognition, and outputting the action classification label and behavior abnormality score of the target. T is a preset number of frames, which can be 10, 20, 30, 50 or other values.
[0039] Based on the above processing results, the edge computing node generates target identification information and uploads it to the computer device. This target identification information includes multimodal information of at least one identified target, comprising appearance feature information, identity feature information, and behavioral feature information. The appearance feature information includes the color histogram feature vector F of the bounding box region corresponding to the identified target. color Identity feature information includes the facial feature vector F of the target being identified. face The score indicates the match between the identified target's identity and the behavioral characteristic information, including the target's action classification label L. act and behavioral abnormality score S act .
[0040] For example, assuming there are 3 drones, step S201 obtains target identification information from these 3 drones, such as target identification information 1 from drone 1, target identification information 2 from drone 2, and target identification information 3 from drone 3.
[0041] Target identification information 1 includes multimodal information about target 1, which is generated by the edge computing node of UAV 1. Target identification information 2 includes multimodal information about target 2 and target 3, both generated by the edge computing node of UAV 2. Target identification information 3 includes multimodal information about target 4 and target 5, both generated by the edge computing node of UAV 3.
[0042] Furthermore, the target identification information also includes the bounding box coordinates corresponding to each identified target and the position information of the UAV. For example, target identification information 1 also includes the bounding box coordinates corresponding to identified target 1 and the position information of the UAV that sent target identification information 1; target identification information 2 also includes the bounding box coordinates corresponding to identified target 2, the bounding box coordinates corresponding to identified target 3 and the position information of the UAV that sent target identification information 2; and target identification information 3 also includes the bounding box coordinates corresponding to identified target 4, the bounding box coordinates corresponding to identified target 5 and the position information of the UAV that sent target identification information 3.
[0043] S202, determine the description information for each of the identified targets, the description information being determined based on the multimodal information of the corresponding identified target.
[0044] Among them, the description information of the target is a comprehensive feature representation obtained by fusing its appearance feature information, identity feature information and behavioral feature information for the target, which is used to comprehensively depict the feature status of the target in the three dimensions of appearance, identity and behavior.
[0045] For example, suppose there are 5 targets to be identified: target 1, target 2, target 3, target 4, and target 5. Then, the description information of target 1 is determined based on the multimodal information of target 1, the description information of target 2 is determined based on the multimodal information of target 2, the description information of target 3 is determined based on the multimodal information of target 3, the description information of target 4 is determined based on the multimodal information of target 4, and the description information of target 5 is determined based on the multimodal information of target 5.
[0046] In one implementation, the multimodal information includes the appearance feature information, identity feature information, and behavioral feature information of the target being identified; For each of the identified targets, descriptive information of the identified target is determined, including: The appearance feature information, the identity feature information, and the behavioral feature information are fused to obtain the description information of the identified target.
[0047] Specifically, a multi-dimensional feature fusion engine is used to fuse the appearance, identity, and behavioral features of the target to obtain its descriptive information. Specifically, the color histogram feature vector F of the target is... color Facial feature vector F face Action category tag L act OneHot (L) encoding act ) and behavioral abnormality score S act Vector concatenation is performed to obtain the descriptive information D of the target being identified. target , .
[0048] OneHot (L act ) indicates the category label L for this action. act Perform one-hot encoding, that is, perform the action classification label L act Convert into a vector of length equal to the total number of preset action categories, where each vector contains only the action category label L. act The component belonging to the category is set to 1, and all other components are set to 0, in order to classify the discrete action category labels L. actConvert to a color histogram feature vector F color Facial feature vector F face A numerical representation of vector concatenation.
[0049] S203, based on preset filtering conditions and the description information, target filtering is performed to obtain a first target set.
[0050] Among them, the preset screening conditions are the judgment criteria that the operator sets in advance when the system is started, which are used to pre-filter the identification targets. Its function is to initially screen out the targets that meet specific requirements from all the current identification targets, form the first target set, avoid evaluating all the identification targets indiscriminately, and thus narrow the scope of the system's attention to the truly relevant target range in advance.
[0051] In one implementation, the step of identifying targets based on preset filtering conditions and the descriptive information to obtain a first target set includes: The preset filtering conditions are matched with the description information of each of the identified targets; The first target set is obtained by filtering the identification targets whose description information matches the preset filtering conditions.
[0052] Specifically, the preset filtering conditions are parsed into constraints on each component of the description information; for each target description, it is determined whether the description information satisfies the parsed constraints; if so, the description information of the target is determined to match the preset filtering conditions; the targets that match the preset filtering conditions are filtered out to obtain the first target set.
[0053] For example, when the preset filtering condition is "wearing a red shirt and running", the parsed constraint is: the response value of the red channel in the color histogram feature vector Fcolor exceeds a preset threshold θ. red And the action classification label L act The target is categorized as "running". The description information of the target is evaluated, and the target that meets the above constraints is identified as the target whose description information matches the preset screening conditions. The target is then added to the first target set.
[0054] S204, determine the evaluation value of each identified target in the first target set, and filter the identified targets in the first target set based on the evaluation values to obtain a second target set.
[0055] The evaluation value is a quantitative expression of the "attention value" or "tracking priority" of each identified target in the first target set. The higher the value, the more worthy the identified target is of priority attention and tracking.
[0056] In one implementation, for each identified target in the first target set, determining the evaluation value of the identified target includes: Acquire the identity feature information, behavioral feature information, and target density of the area where the target is located; A first evaluation value is determined based on the identity feature information, a second evaluation value is determined based on the behavioral feature information, and a third evaluation value is determined based on the target density. The evaluation value of the identified target is determined based on the first evaluation value, the second evaluation value, and the third evaluation value.
[0057] The target density in the area where the target is located is used to characterize the degree of aggregation of the identified targets in the surrounding area. The higher the target density, the more concentrated the identified targets are in the area where the target is located; the lower the target density, the more dispersed the identified targets are in the area where the target is located.
[0058] For each identified target in the first target set, a corresponding target region is determined with the location of the identified target as the center. The number of identified targets in the target region is counted, and the target density of the region where the identified target is located is determined based on the number of identified targets.
[0059] The size of the target region is preset, centered on the location of the identified target. Knowing the size of the region allows us to determine the specific location of the target region. Target density = number of identified targets / size of the target region. The location of the identified target is determined based on the bounding box coordinates corresponding to that target.
[0060] The determination of the first evaluation value based on the identity feature information includes: multiplying the identity matching score included in the identity feature information by the weight coefficient corresponding to the identity feature information, and using this product as the first evaluation value. The weight coefficient corresponding to the identity feature information is the aforementioned weight coefficient α, and the first evaluation value = identity matching score × α.
[0061] Determining a second evaluation value based on the behavioral feature information includes: multiplying the behavioral anomaly score included in the behavioral feature information by the weighting coefficient corresponding to the behavioral feature information, and using this product as the second evaluation value. The weighting coefficient corresponding to the behavioral feature information is the aforementioned weighting coefficient β, and the second evaluation value = behavioral anomaly score × β.
[0062] Determining a third evaluation value based on the target density includes: multiplying the target density by the weighting coefficient corresponding to the target density, and using this product as the third evaluation value. The weighting coefficient corresponding to the target density is the aforementioned weighting coefficient γ, and the third evaluation value = target density × γ.
[0063] Determining the evaluation value of the identified target based on the first evaluation value, the second evaluation value, and the third evaluation value includes: determining the sum of the first evaluation value, the second evaluation value, and the third evaluation value as the evaluation value of the identified target.
[0064] In one implementation, the step of identifying and filtering targets in the first target set based on the evaluation value to obtain a second target set includes: The second target set is obtained by filtering out the targets in the first target set whose evaluation value is greater than or equal to a preset value.
[0065] For example, assuming the preset value is 1, the first target set includes 3 targets, the evaluation value of target 1 is 1.2 > 1, the evaluation value of target 2 is 1.3 > 1, and the evaluation value of target 3 is 0.8 < 1. Then the second target set includes target 1 and target 2.
[0066] In another implementation, the step of identifying and filtering targets in the first target set based on the evaluation value to obtain a second target set includes: Each of the identified targets in the first target set is sorted in descending order according to the evaluation value; The first K targets are selected and arranged to obtain the second target set, where K is a positive integer.
[0067] For example, the first target set includes 3 targets: target 1 has an evaluation value of 1.2, target 2 has an evaluation value of 1.3, and target 3 has an evaluation value of 0.8. So, after sorting in descending order, the target set is target 2-target 1-target 3. Assuming K=2, the second target set includes target 1 and target 2.
[0068] S205, using the identified targets in the second target set as tracking targets, determining the control command for at least one of the multiple drones, and sending the corresponding control command to the at least one drone.
[0069] In one implementation, the step of using the identified targets in the second target set as tracking targets and determining the control commands for at least one of the multiple drones includes: Based on the location information of each of the drones and the location information of each of the tracking targets, at least one target drone is determined from the plurality of drones, and the tracking target to be tracked for each target drone is determined; For each target UAV, based on the position information of the target being tracked by the target UAV, the flight control command and gimbal control command of the target UAV are determined, and the flight control command and the gimbal control command are used as the control command of the target UAV.
[0070] Optionally, determining at least one target drone from the plurality of drones based on the location information of each of the drones and the location information of each tracking target, and determining the tracking target to be tracked for each target drone, includes: Based on the location information of each UAV and the location information of each tracked target, the distance between each UAV and each tracked target is determined; Based on the distance, at least one drone is assigned to each of the multiple drones to obtain the at least one target drone and the target drone to be tracked for each target drone; wherein the distance between the target drone and the corresponding target drone to be tracked meets a preset condition.
[0071] Among them, the preset conditions are used to determine whether the distance between the drone and the tracked target meets the allocation requirements. Its function is to ensure that the distance between the finally determined target drone and the assigned tracked target is within a reasonable range, rather than being arbitrarily allocated.
[0072] The Hungarian algorithm can be used to perform optimal task allocation based on the distance and the task status, taking into account the current task status of each UAV, so that each tracking target is assigned to the corresponding target UAV. At this time, the distance between the target UAV and the corresponding tracking target meets the preset condition, which means that the global optimal allocation is achieved by comprehensively considering the distance and the task status.
[0073] For example, suppose the tracking targets in the second target set include tracking target 1 and tracking target 2, and the drones participating in this task allocation include drone 1, drone 2 and drone 3.
[0074] Based on the position information of UAV 1, UAV 2, and UAV 3, as well as the position information of tracking target 1 and tracking target 2, the following distances are calculated: The distance between UAV 1 and target 1 being tracked is d11, and the distance between UAV 1 and target 2 being tracked is d12; The distance between UAV 2 and the tracking target 1 is d21, and the distance between UAV 2 and the tracking target 2 is d22; The distance between UAV 3 and target 1 is d31, and the distance between UAV 3 and target 2 is d32.
[0075] Assume that it is calculated that d21 < d11 < d31 and d12 < d32 < d22, that is, the unmanned aerial vehicle (UAV) 2 is closest to the tracking target 1, and the UAV 1 is closest to the tracking target 2. At this time, in combination with the current mission status of each UAV (for example, UAV 1, UAV 2, and UAV 3 are all in an idle and allocable state), the Hungarian algorithm is adopted, which takes the distance between each UAV and each tracking target as the allocation cost, and solves the optimal allocation scheme that minimizes the overall allocation cost, to obtain: UAV 2 is allocated to tracking target 1, and UAV 1 is allocated to tracking target 2.
[0076] At this time, UAV 1 and UAV 2 are the at least one target UAV; wherein, the tracking target to be tracked by UAV 1 is tracking target 2, and the tracking target to be tracked by UAV 2 is tracking target 1; the distance d12 between UAV 1 and tracking target 2 and the distance d21 between UAV 2 and tracking target 1 both satisfy the preset condition, that is, the global optimal allocation is achieved under the condition of comprehensively considering the distance and the mission status.
[0077] As for UAV 3, since the distances between it and tracking target 1 and tracking target 2 are both greater than those of the UAVs allocated to the corresponding tracking targets (that is, UAV 3 is not a member of the current optimal allocation result), it is not used as the target UAV for this time, and can continue to maintain the original patrol mission or serve as a standby UAV.
[0078] It should be noted that, if the number of tracking targets is greater than the number of currently allocable idle UAVs, or a certain tracking target has a high attention value (evaluation value) and requires multiple UAVs to track simultaneously to ensure that the target is not lost, the process of allocating at least one UAV to each tracking target based on the distance can also allocate two or more target UAVs to the same tracking target. The specific allocation quantity can be comprehensively determined in combination with factors such as the evaluation value corresponding to the tracking target and the pre-configured strategy of the system, which is not limited in the present application.
[0079] Optionally, said determining the flight control instruction and the pan-tilt control instruction of the target UAV based on the position information of the tracking target to be tracked by the target UAV comprises: determining a flight speed vector of the target UAV, a pitch angle adjustment amount of the pan-tilt of the target UAV, and a yaw angle adjustment amount of the pan-tilt of the target UAV based on the position information of the tracking target to be tracked by the target UAV and the position information of the target UAV; using the flight speed vector as the flight control instruction, and using the pitch angle adjustment amount of the pan-tilt and the yaw angle adjustment amount of the pan-tilt as the pan-tilt control instruction.
[0080] The determination of the flight speed vector of the target UAV may be achieved by: constructing a gravitational potential field using the current position and predicted future position of the target UAV to be tracked as the gravitational target; constructing a repulsive potential field using the positions of obstacles, geofence boundaries, and other UAVs sensed by the target UAV as the repulsive source; superimposing the gravitational potential field and the repulsive potential field to obtain a total potential field; and determining the flight speed vector of the target UAV based on the negative gradient direction of the total potential field.
[0081] The determination of the gimbal pitch angle adjustment and gimbal yaw angle adjustment of the target UAV can be achieved by: calculating the gimbal pitch angle adjustment and gimbal yaw angle adjustment through a PID controller based on the position deviation of the target UAV to be tracked in the image coordinate system, so as to drive the gimbal motor of the target UAV and make the tracked target in the center area of the image.
[0082] For example, taking the previously identified target UAV 2 and its tracking target 1 as an example, the specific process of determining flight control commands and gimbal control commands is explained.
[0083] Based on the position information of tracking target 1 (including current position coordinates and future position prediction output by the behavior prediction module) and the position information of target UAV 2, a gravitational potential field U is constructed, using the current position and future position prediction of tracking target 1 as the gravitational target. att The gravitational potential field causes the target drone 2 to exert an attractive force towards the tracking target 1; simultaneously, using the obstacle positions currently perceived by the target drone 2, the coordinates of the geofence boundary, and the positions of other drones such as drone 1 and drone 3 as repulsive force sources, a repulsive potential field U is constructed. rep The repulsive potential field causes the target drone 2 to generate a repulsive force away from obstacles, fence boundaries and other drones.
[0084] The gravitational potential field U att With repulsive potential field U rep By superposition, the total potential field U at the location of the target UAV 2 is obtained. total =U att +U rep Along the negative gradient direction of the total potential field, the flight velocity vector of the target UAV 2 is determined. The direction of the flight velocity vector is the direction in which the total potential field decreases the fastest, and its magnitude can be determined in combination with preset parameters such as the maximum flight speed limit.
[0085] For example, if target 1 is moving northeast, and there is a geofence boundary to the west of target drone 2 and drone 1 to the north, then the negative gradient direction after the superposition of the gravitational potential field and the repulsive potential field may yield a flight speed vector that is slightly biased to the east, avoiding the geofence boundary and drone 1, so that target drone 2 can avoid obstacles and other drones while tracking target 1.
[0086] Based on the position information of the tracking target 1 and the target UAV 2, the position deviation of the tracking target 1 in the coordinate system of the image captured by the target UAV 2 is determined (e.g., the tracking target 1 is currently located slightly to the left or lower than the center of the image). Based on this position deviation, the gimbal pitch angle adjustment Δ required to bring the tracking target 1 back to the center of the image is calculated by the PID controller. The yaw angle adjustment amount Δψ of the gimbal.
[0087] For example, if the tracked target 1 shifts to the left relative to the image center in the image coordinate system, the corresponding gimbal yaw angle adjustment Δψ is calculated to drive the gimbal to rotate to the left; if the tracked target 1 shifts downward relative to the image center, the corresponding gimbal pitch angle adjustment Δψ is calculated. It is used to drive the gimbal to rotate downwards. By adjusting the pitch angle and yaw angle together, the tracking target 1 is kept in the center of the screen, providing a stable and clear target image for the ground terminal.
[0088] The determined flight speed vector is used as the flight control command for the target UAV 2; the determined gimbal pitch angle adjustment Δ is used as the control command for the target UAV 2. The gimbal yaw angle adjustment Δψ serves as the gimbal control command for the target UAV 2. Together, they constitute the control command for the target UAV 2, which is sent to the target UAV 2 by the computer equipment for execution. This allows the target UAV 2 to adjust its flight position along the calculated flight speed vector to continuously track the target, and to continuously correct the shooting angle through the pitch and yaw rotation of the gimbal motors, ensuring that the tracked target 1 remains stably in the center of the frame.
[0089] In this embodiment, by aggregating target identification information from multiple UAVs, the overall perception range of ground-based group targets is expanded, providing a more comprehensive target information foundation for the rational control of UAVs. A first target set is obtained by matching and filtering identified targets based on preset screening conditions, and a second target set is obtained by prioritizing the first target set based on evaluation values. This achieves two-level filtering and quantitative assessment of the importance of group targets, ensuring that UAV control decisions are based on a reasonable judgment of target importance, avoiding the indiscriminate allocation of UAV resources to group targets. Furthermore, the identified targets in the second target set are used as tracking targets, and the control commands for at least one UAV are determined and executed by comprehensively considering the status of multiple UAVs, achieving a reasonable matching and dynamic scheduling between the tracking target and the executing UAV.
[0090] In one implementation, the method further includes: pushing early warning information to a safety management and monitoring terminal, the early warning information including the tracking ID, location information, evaluation value, action classification label of the tracked target, and real-time video stream collected by the target drone, so that the safety management and monitoring terminal can display the early warning information for operators to view; the operators can adjust the preset filtering conditions or the weight coefficients α, β, γ associated with the evaluation value through the safety management and monitoring terminal.
[0091] Furthermore, the method also includes: during the tracking of the target, continuously monitoring other identified targets within the field of view of the multiple drones, and periodically re-determining the evaluation values of all identified targets, so as to update and filter the identified targets based on the re-determined evaluation values, and avoid missing newly emerging identified targets.
[0092] Furthermore, the method also includes: after receiving the mission end instruction, controlling the target UAV to return along a preset return route, and saving the trajectory information and identification record of the identified target during this mission.
[0093] For example, continuing the previous example, after target UAV 2 determines the flight control commands and gimbal control commands and begins to execute the tracking task, the computer equipment generates a warning message for the tracked target 1. This warning message includes: the tracking ID of the tracked target 1 (e.g., "ID-1"), the current location coordinates (e.g., GPS coordinates "XX.XXXX°N, XX.XXXX°E"), the evaluation value (e.g., the comprehensive priority score P=0.82 calculated above), the identified action classification label (e.g., "running"), and the video stream collected in real time by target UAV 2. The computer equipment pushes this warning message to the safety management and monitoring terminal.
[0094] The security management monitoring terminal receives and displays the aforementioned warning information on its interface. Security personnel can view the real-time location trajectory, evaluation value changes, and real-time video of the tracked target 1 on the terminal interface. If, based on the actual situation, the personnel determine that the current preset screening criteria (e.g., "wearing a red shirt and running") are too narrow, potentially causing some targets to be overlooked, they can manually relax the preset screening criteria through the terminal interface (e.g., removing color constraints and retaining only action category constraints). Alternatively, if the current situation is an emergency, the personnel can adjust the weighting coefficients from the default values (α=0.25, β=0.5, γ=0.25) to settings that place greater emphasis on behavioral abnormality. The terminal then sends the adjusted screening criteria or weighting coefficients back to the computer for subsequent screening and evaluation value calculations.
[0095] Meanwhile, assuming that in addition to tracking target 1 and tracking target 2, there are also identification targets such as identification target 6 and identification target 7 within the current field of view, which previously had low evaluation values and were not selected into the second target set. While target drone 2 and target drone 1 are performing tracking tasks for tracking target 1 and tracking target 2 respectively, the system continues to monitor other identification targets such as identification target 6 and identification target 7 through other patrolling drones (e.g., drone 3), and recalculates the evaluation values of all identification targets, including identification target 6 and identification target 7, at a preset period (e.g., every 5 seconds). If identification target 6 exhibits abnormal behavior such as "falling over" within a certain period, its behavior abnormality score S... act An increase in the value causes the recalculated evaluation value to exceed the preset threshold θalert, or it may be added to a new list of high-priority targets L. high If the top K targets are identified, target 6 will be re-selected and added to the updated second target set. The system will then assign a new target drone to it and generate corresponding control commands to achieve timely response to newly emerging potential risk targets and avoid omissions.
[0096] After the mission is completed, target UAV 1 and target UAV 2 will return to their respective pre-set return routes. The computer equipment will save the complete trajectory information of target 1 and target 2 during the mission, as well as the identification records of other identified targets such as target 6 and target 7, for use in subsequent mission review, abnormal event tracing or evidence collection.
[0097] The method of this application has been described above; the apparatus of this application will be described below.
[0098] See Figure 3 , Figure 3 This is a schematic diagram of the structure of a drone control device provided in an embodiment of this application, as shown below. Figure 3 As shown, the 30 includes: The acquisition module 301 is used to acquire target identification information from multiple drones to obtain multiple target identification information, wherein the target identification information includes multimodal information of at least one identified target; The determining module 302 is used to determine the description information of each of the identified targets, the description information being determined based on the multimodal information of the corresponding identified target; The filtering module 303 is used to filter the identified targets based on preset filtering conditions and the description information to obtain a first target set; The determining module 302 is further configured to determine the evaluation value of each identified target in the first target set; The filtering module 303 is further configured to identify and filter targets in the first target set based on the evaluation value to obtain a second target set; The determining module 302 is further configured to use the identified targets in the second target set as tracking targets to determine the control command of at least one of the multiple drones; The communication module 304 is used to send corresponding control commands to the at least one drone. It should be noted that the drone control device 30 described above can execute the drone control method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the embodiments can be found in the drone control method provided in the embodiments of this application.
[0099] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device 40 provided in an embodiment of this application. The computer device 40 includes a processor 401 and a memory 402. The memory 402 is connected to the processor 401, for example, via a bus.
[0100] Processor 401 is configured to support the computer device 40 in performing the corresponding functions in the methods described in the above method embodiments. Processor 401 may be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0101] Memory 3402 is used to store program code, etc. Memory 402 may include volatile memory (VM), such as random access memory (RAM); memory 402 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 402 may also include combinations of the above types of memory.
[0102] The memory 402 is used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the UAV control method in the embodiments of this application. The processor executes various functional applications and data processing of the UAV control method by running the non-volatile software programs, instructions, and modules stored in the memory, thereby realizing the functions of the UAV control method provided in the above method embodiments.
[0103] The memory 402 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and application programs required for at least one function. The data storage area may store data created based on the use of the 3D simulation circuit display device. In some embodiments, the memory may include memory remotely located relative to the processor, which can be connected to the 3D simulation circuit display device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0104] The one or more modules are stored in the memory. When executed by the one or more processors, they perform the UAV control method in any of the above method embodiments. For example, they perform the method steps described in the above method embodiments to realize the functions of the modules described in the above device embodiments.
[0105] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in the foregoing embodiments.
[0106] This application also provides a computer program product that, when run on a computer device, causes any of the above methods to be executed.
[0107] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0108] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.
Claims
1. A method for controlling an unmanned aerial vehicle (UAV), characterized in that, include: Target identification information from multiple drones is acquired to obtain multiple target identification information, wherein the target identification information includes multimodal information of at least one identified target; Determine descriptive information for each of the identified targets, the descriptive information being determined based on the multimodal information of the corresponding identified target; Based on preset filtering conditions and the description information, the target is identified and filtered to obtain a first target set; Determine the evaluation value of each target in the first target set, and filter the targets in the first target set based on the evaluation values to obtain a second target set; The identified targets in the second target set are used as tracking targets. The control command for at least one of the multiple drones is determined, and the corresponding control command is sent to the at least one drone.
2. The method according to claim 1, characterized in that, The multimodal information includes the appearance features, identity features, and behavioral features of the target being identified; For each of the identified targets, descriptive information of the identified target is determined, including: The appearance feature information, the identity feature information, and the behavioral feature information are fused to obtain the description information of the identified target.
3. The method according to claim 1, characterized in that, The first target set is obtained by filtering targets based on preset filtering conditions and the description information, including: The preset filtering conditions are matched with the description information of each of the identified targets; The first target set is obtained by filtering the identification targets whose description information matches the preset filtering conditions.
4. The method according to claim 1, characterized in that, For each identified target in the first target set, determine the evaluation value of the identified target, including: Acquire the identity feature information, behavioral feature information, and target density of the area where the target is located; A first evaluation value is determined based on the identity feature information, a second evaluation value is determined based on the behavioral feature information, and a third evaluation value is determined based on the target density. The evaluation value of the identified target is determined based on the first evaluation value, the second evaluation value, and the third evaluation value.
5. The method according to claim 1, characterized in that, The step of identifying and filtering targets in the first target set based on the evaluation value to obtain a second target set includes: The second target set is obtained by filtering out targets in the first target set whose evaluation values are greater than or equal to preset values; Alternatively, each of the identified targets in the first target set can be sorted in descending order according to the evaluation value; The first K targets are selected and arranged to obtain the second target set, where K is a positive integer.
6. The method according to any one of claims 1-5, characterized in that, The step of using the identified targets in the second target set as tracking targets and determining the control commands for at least one of the multiple drones includes: Based on the location information of each of the drones and the location information of each of the tracking targets, at least one target drone is determined from the plurality of drones, and the tracking target to be tracked for each target drone is determined; For each target UAV, based on the position information of the target being tracked by the target UAV, the flight control command and gimbal control command of the target UAV are determined, and the flight control command and the gimbal control command are used as the control command of the target UAV.
7. The method according to claim 6, characterized in that, The step of determining at least one target drone from the plurality of drones based on the location information of each of the drones and the location information of each tracking target, and determining the tracking target to be tracked for each target drone, includes: Based on the location information of each UAV and the location information of each tracked target, the distance between each UAV and each tracked target is determined; Based on the distance, at least one drone is assigned to each of the multiple drones to obtain the at least one target drone and the target drone to be tracked for each target drone; wherein the distance between the target drone and the corresponding target drone to be tracked meets a preset condition.
8. The method according to claim 6, characterized in that, The step of determining the flight control commands and gimbal control commands of the target UAV based on the position information of the target UAV to be tracked includes: Based on the position information of the target drone to be tracked and the position information of the target drone, the flight speed vector of the target drone, the gimbal pitch angle adjustment amount of the target drone, and the gimbal yaw angle adjustment amount of the target drone are determined. The flight speed vector is used as the flight control command, and the gimbal pitch angle adjustment and the gimbal yaw angle adjustment are used as the gimbal control commands.
9. A computer device, characterized in that, The device includes a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor causing the computer device to perform the method as described in any one of claims 1-8 when executing the one or more computer programs.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-8.