Unmanned aerial vehicle cradle head control method, apparatus, and system, computer device, and medium

Abstract

An unmanned aerial vehicle cradle head control method, apparatus, and system, a computer device, and a medium, which relate to the technical field of unmanned aerial vehicle control. The method comprises: acquiring a set of image frames acquired by a cradle head camera (S102); sending to a server end a preset object category text and a specified image frame in the set of image frames, so that the server end, on the basis of the object category text and the specified image frame, determines a first object bounding box by means of an object detection model (S104); by means of an object tracking model and on the basis of the set of image frames and the first object bounding box, determining real-time tracking coordinates of an object (S106); and, on the basis of the real-time tracking coordinates of the object, controlling the cradle head camera, so that the object is located in a specified area in the image frames acquired by the cradle head camera (S108). Thus, the object can be tracked in real time, and the cradle head camera carried by the unmanned aerial vehicle end can be automatically controlled.

Description

UAV gimbal control method, device, system, computer device and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent disclosure No. 202311244240.1, filed with the State Intellectual Property Office of China on September 26, 2023, entitled “UAV gimbal control method, device, system, computer device and medium,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the field of drone control technology, and in particular to a drone gimbal control method, device, system, computer device, and medium. Background Art

[0004] During target inspection and target tracking, drones typically encounter a variety of scenarios and targets, which are usually detected using traditional closed-set target detection methods or large-model target detection methods. Traditional closed-set target detection methods, such as YOLOV5 and Faster-RCNN (Faster-region with CNN features), can only detect pre-defined target categories, and the models are trained based on datasets for specific fields and tasks, resulting in poor versatility. Large-model target detection methods, due to their large number of parameters, cannot meet the real-time application scenarios on end-side platforms with limited graphics memory resources, such as the Jetson Xavier NX of drones. In summary, both traditional closed-set target detection methods and large-model target detection methods cannot meet the real-time inspection and real-time tracking application scenarios of drones.

[0005] Summary of the Invention

[0006] In view of this, the purpose of the present disclosure is to provide a drone gimbal control method, device, system, computer device and medium, which can track the target object in real time and automatically control the gimbal camera carried by the drone.

[0007] In a first aspect, an embodiment of the present disclosure provides a method for controlling a gimbal of a drone. The method is applied to a drone, wherein the drone is equipped with a gimbal camera, the drone is in communication with a server, the drone is deployed with a target tracking model, and the server is deployed with a target detection model. The method includes:

[0008] Obtaining a set of image frames captured by the pan-tilt camera;

[0009] Sending a preset target object category text and a specified image frame in the image frame set to the server, so that the server determines a first target object detection frame based on the target object category text and the specified image frame through the target detection model;

[0010] Determining the real-time tracking coordinates of the target object based on the image frame set and the first target object detection frame using the target tracking model;

[0011] The pan-tilt camera is controlled based on the real-time tracking coordinates of the target object so that the target object is located in a specified area of ​​the image frame captured by the pan-tilt camera.

[0012] In one embodiment, the object detection model includes a large model object detection model;

[0013] The step of sending a preset target object category text and a specified image frame in the image frame set to the server, so that the server determines a first target object detection frame based on the target object category text and the specified image frame through the target detection model, includes:

[0014] Based on the preset target object category text and the specified image frame in the image frame set, a target detection request is sent to the server side, so that when the server side receives the target detection request, the large model target detection model is called, and a first target object detection frame is fed back for the target object category text and the specified image frame.

[0015] In one embodiment, the step of determining the real-time tracking coordinates of the target object based on the image frame set and the first target detection frame using the target tracking model includes:

[0016] Determining a template area based on the first object detection frame; and determining a search area based on the image frame set or the first object detection frame;

[0017] Determining, using the target tracking model, at least one candidate detection frame included in a current image frame in the set of image frames based on the template area and the search area, as well as position information and confidence information corresponding to each candidate detection frame; wherein the confidence information is used to represent a probability value of the candidate detection frame belonging to the target object;

[0018] Determining, from the candidate detection frames, a second object detection frame corresponding to the current image frame and target attribute information corresponding to the second object detection frame based on the confidence information; wherein the target attribute information includes a center point and target size information;

[0019] The real-time tracking coordinates of the target object are determined according to the target attribute information corresponding to the second target detection frame.

[0020] In one embodiment, the step of determining the template area based on the first object detection frame includes:

[0021] According to the preset first size information, cutting out a template area in the designated image frame with the designated point of the first object detection frame as the center;

[0022] The step of determining a search area based on the image frame set or the first object detection frame includes:

[0023] If the current image frame is the second image frame, intercepting a search area in the designated image frame with the designated point of the first object detection frame as the center according to preset second size information;

[0024] Alternatively, if the current image frame is an image frame other than the second image frame, a search area is intercepted in the current image frame based on the second size information and centered at the designated point of the second target object detection frame corresponding to the previous image frame.

[0025] In one embodiment, the target tracking model includes a feature extraction unit and a confidence prediction unit, and the confidence prediction unit includes a first branch and a second branch;

[0026] The step of determining, by the target tracking model, at least one candidate detection frame included in the current image frame in the image frame set, and position information and confidence information corresponding to each candidate detection frame based on the template area and the search area includes:

[0027] extracting, by the feature extraction unit, template feature information of the template area and search feature information of the search area;

[0028] Determining, by the first branch of the confidence prediction unit, position information corresponding to each candidate detection frame included in the current image frame based on the template feature information and the search feature information;

[0029] And through the second branch in the confidence prediction unit, based on the template feature information and the search feature information, the confidence information corresponding to each candidate detection box contained in the current image frame is determined.

[0030] In one embodiment, the step of determining, from the candidate detection frames based on the confidence information, a second target object detection frame corresponding to the current image frame and target attribute information corresponding to the second target object detection frame includes:

[0031] Adjusting the confidence information corresponding to each candidate detection frame according to the distance between each candidate detection frame and the center point of the current image frame to obtain target confidence information corresponding to each candidate detection frame;

[0032] Determining, from the candidate detection frames, a second target object detection frame corresponding to the current image frame and initial attribute information of the second target object detection frame in descending order of the target confidence information;

[0033] The initial attribute information of the second object detection frame corresponding to the current image frame is adjusted to obtain the target attribute information of the second object detection frame corresponding to the current image frame.

[0034] In one embodiment, the step of adjusting the initial attribute information of the second object detection frame corresponding to the current image frame to obtain the target attribute information of the second object detection frame corresponding to the current image frame includes:

[0035] Smoothing and weighted processing is performed on the target size information of the second target detection frame corresponding to the previous image frame and the initial size information of the target detection frame corresponding to the current image frame to obtain the target size information of the second target detection frame corresponding to the current image frame.

[0036] In a second aspect, an embodiment of the present disclosure further provides a drone gimbal control device, the device being applied to a drone end, the drone end being equipped with a gimbal camera, the drone end being communicatively connected to a server end, the drone end being deployed with a target tracking model, and the server end being deployed with a target detection model, the device comprising:

[0037] An image acquisition module, configured to acquire a set of image frames captured by the PTZ camera;

[0038] an object detection module, configured to send a preset object category text and a specified image frame in the image frame set to the server, so that the server determines a first object detection frame based on the object category text and the specified image frame through the object detection model;

[0039] A target tracking module, configured to determine the real-time tracking coordinates of the target object based on the image frame set and the first target object detection frame using the target tracking model;

[0040] The pan-tilt control module is used to control the pan-tilt camera based on the real-time tracking coordinates of the target object so that the target object is located in a specified area in the image frame captured by the pan-tilt camera.

[0041] In a third aspect, the embodiment of the present disclosure further provides a drone gimbal control system, comprising a drone end and a server end, wherein the drone end is in communication with the server end;

[0042] The drone is equipped with a pan-tilt camera and a target tracking model, and is configured to execute any one of the methods described in the first aspect;

[0043] The server is deployed with a target detection model.

[0044] In a fourth aspect, an embodiment of the present disclosure further provides a computer device comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0045] In a fifth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0046] The disclosed embodiments provide a method, device, system, computer device, and medium for controlling a drone gimbal. After acquiring a set of image frames captured by a gimbal camera, the method sends a preset target object category text and a specified image frame in the image frame set to a server, so that the server determines a first target object detection frame based on the target object category text and the specified image frame through a target detection model; then determines the target object's real-time tracking coordinates based on the image frame set and the first target object detection frame through a target tracking model; and finally controls the gimbal camera based on the real-time tracking coordinates of the target object so that the target object is located in a specified area in the image frame captured by the gimbal camera. The above method deploys a target detection model on the server and a target tracking model on the drone, respectively, and combines the target detection model and the target tracking model to determine a first target object detection frame with high accuracy using the target detection model, so that the target tracking model can track the target object in real time based on the first target object detection frame and the image frame set, obtain the target object's real-time tracking coordinates, and then automatically control the gimbal camera using the real-time tracking coordinates of the target object.

[0047] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or understood by practicing the present disclosure. The objectives and other advantages of the present disclosure are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0048] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] FIG1 is a flow chart of a method for controlling a gimbal of a drone according to an embodiment of the present disclosure;

[0051] FIG2 is a flow chart of a UAV gimbal control system according to an embodiment of the present disclosure;

[0052] FIG3 is a schematic structural diagram of a UAV gimbal control device provided by an embodiment of the present disclosure;

[0053] FIG4 is a schematic structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present disclosure without creative effort shall fall within the scope of protection of the present disclosure.

[0055] Currently, traditional closed-set object detection methods, such as YOLOV5 and FasterRCNN, can only detect predefined object categories. These models are trained on datasets specific to specific domains and tasks, resulting in poor generalizability. With the rise of large AI (artificial intelligence) models, transformer-based object detection algorithms have integrated multimodal information, such as images and language / text. These algorithms are then trained on massive amounts of data, resulting in open-set object detection methods that can understand text and have strong generalization capabilities. These large-model object detection algorithms, due to their large number of parameters, cannot meet the real-time application requirements of edge platforms with limited graphics memory resources, such as the Jetson Xavier NX used in drones.

[0056] Based on this, the present disclosure implements a method, device, system, computer device and medium for controlling a drone gimbal, which can track a target object in real time and automatically control the gimbal camera mounted on the drone.

[0057] To facilitate understanding of this embodiment, a method for controlling a pan-tilt platform of a drone disclosed in an embodiment of the present disclosure is first introduced in detail. The method is applied to a drone end, the drone end is equipped with a pan-tilt camera, the drone end is in communication with a server end, the drone end is deployed with a target tracking model, and the server end is deployed with a target detection model. Referring to the flow diagram of a method for controlling a pan-tilt platform of a drone shown in FIG1 , the method mainly includes the following steps S102 to S108:

[0058] Step S102: Acquire a set of image frames captured by the PTZ camera, wherein the set of image frames includes a plurality of continuous image frames.

[0059] Step S104 : sending the preset target object category text and the designated image frame in the image frame set to the server, so that the server determines a first target object detection frame based on the target object category text and the designated image frame through the target detection model.

[0060] Among them, the target detection model is also the large-model target detection model, and the large model refers to a machine learning model with a large number of parameters and a complex structure; the target category text is used to describe the category information of the target to be tracked, such as a vehicle; the specified image frame can be the initial image frame in the image frame set, or it can be other image frames, which can be set based on actual needs; the first target detection box is also the bounding box of the target output by the target detection model.

[0061] In one embodiment, a large model target detection model is deployed on the server side. By sending the target object category text and the specified image frame to the server side, the server side will call the large model target detection model and use the target object category text and the specified image frame as input of the large model target detection model. The large model target detection model will output the first target object detection box.

[0062] Step S106 : determining the real-time tracking coordinates of the target object based on the image frame set and the first target object detection frame through the target tracking model.

[0063] The target tracking model includes a feature extraction unit and a confidence prediction unit, and the confidence prediction unit includes a first (box) branch and a second (class) branch; the real-time tracking target of the target object is a two-dimensional coordinate box of the target object.

[0064] In one embodiment, a template area and a search area can be determined based on an image frame set and a first target object detection frame, and the template area and the search area are used as inputs of a feature extraction unit to extract template feature information and search feature information, respectively. The template feature information and the search feature information are then used as inputs of a first branch and a second branch. The first branch will output the position information corresponding to each candidate detection frame contained in the current image frame, and the second branch will output the confidence information corresponding to each candidate detection frame contained in the current image frame. Finally, the real-time tracking coordinates of the target object are determined based on the position information and confidence information corresponding to each candidate detection frame.

[0065] Step S108 : controlling the pan-tilt camera based on the real-time tracking coordinates of the target object so that the target object is located in a designated area of ​​the image frame captured by the pan-tilt camera.

[0066] The designated area may be the center of the pan-tilt camera's image. In one embodiment, the angle of the pan-tilt camera can be determined based on the real-time tracking coordinates of the target object, and the pan-tilt camera can be controlled according to the angle so that the target object is always located in the center of the pan-tilt camera's image.

[0067] The drone gimbal control method provided by the embodiment of the present disclosure deploys a target detection model on the server side and a target tracking model on the drone side, respectively, so as to combine the target detection model and the target tracking model, and use the target detection model to determine a first target detection frame with higher precision, so that the target tracking model can track the target in real time based on the first target detection frame and the image frame set, obtain the real-time tracking coordinates of the target, and then use the real-time tracking coordinates of the target to automatically control the gimbal camera.

[0068] For ease of understanding, the present disclosure provides a specific implementation of a drone gimbal control method.

[0069] Before executing step S102, the large-scale target detection model needs to be deployed on the server side in the form of a fastapi service. In addition, the target tracking model needs to be converted to an ONNX (Open Neural Network Exchange) model and lightweight accelerated using the tensorRT inference framework before being deployed on the Nvidia Jetson Xavier NX, an embedded platform on the drone side.

[0070] Regarding the aforementioned step S102 , in order to obtain the image frame set, the pan-tilt camera may be controlled to collect image frames in real time, and the collected continuous image frames are used as the image frame set.

[0071] For the aforementioned step S104, a target detection request can be sent to the server based on the preset target object category text and the specified image frame in the image frame set, so that when the server receives the target detection request, it calls the large model target detection model and feeds back a first target object detection frame for the target object category text and the specified image frame. In actual applications, the drone uses the target object category text to be tracked and an image frame (such as an initial image frame) captured by its own gimbal camera to request the server to provide a large model target detection service. The server inputs the received target object category text and image frame into the large model target detection model, and the large model target detection model outputs a first target object detection frame. The drone receives the first target object detection frame and uses the first target detection frame to initialize the target tracking model of the drone.

[0072] Regarding the aforementioned step S106, the embodiment of the present disclosure provides an implementation method for determining the real-time tracking coordinates of the target object based on the image frame set and the first target object detection frame through a target tracking model, as shown in the following steps 1 to 4:

[0073] Step 1: determine a template area based on a first object detection frame; and determine a search area based on an image frame set or the first object detection frame.

[0074] (1) Determining the Template Region: Based on the preset first size information, a template region can be captured in the specified image frame with the designated point of the first object detection frame as the center. The designated point can be a center point, and the first size information can be 127×127. For example, in the initial image frame, a 127×127 region is captured with the center point of the first object detection frame returned by the large model object detection model as the template.

[0075] (2) Determine the search area, which is divided into two situations:

[0076] In case 1, if the current image frame is the second image frame, the search area is intercepted in the specified image frame with the specified point of the first target object detection frame as the center according to the preset second size information. In actual applications, when the current image frame is the second image frame, the second target object detection frame corresponding to the previous image frame (that is, the initial image frame) is missing. At this time, a 255×255 area can be directly intercepted in the initial image frame with the center point of the first target object detection frame returned by the large model target detection model as the search area.

[0077] In the second case, if the current image frame is other than the second image frame, a search area is extracted from the current image frame based on the second size information, centered at the designated point of the second object detection frame corresponding to the previous image frame. In practical applications, for subsequent image frames, a 255×255 area can be extracted from the current image frame based on the center point of the second object detection frame corresponding to the previous image frame to serve as the search area.

[0078] In step 2, the target tracking model uses the template region and the search region to determine at least one candidate detection frame in the current image frame, as well as the position information and confidence information corresponding to each candidate detection frame. The position information describes the coordinate position of the candidate detection frame in the current image frame, and the confidence information represents the probability that the candidate detection frame belongs to the target object.

[0079] In one example, the target tracking model includes a feature extraction unit and a confidence prediction unit. The feature extraction unit, also known as the feature extraction backbone part, can adopt the Alexnet network. The confidence prediction unit, also known as the confidence prediction head part, can adopt the RPN (Recursive Pyramid Network) network. The confidence prediction unit includes a first (box) branch and a second (class) branch.

[0080] On this basis, the template area template and the search area search are fed into the target tracking model to detect the candidate bounding box and confidence information score (also known as score) of the target object. Specifically, the location information box and confidence information score of the candidate bounding box can be determined according to the following steps 2.1 to 2.2:

[0081] Step 2.1: Extract template feature information of the template area and search feature information of the search area through a feature extraction unit.

[0082] In one embodiment, feature extraction is performed separately for the template region (template) and the search region (search). Specifically, the feature extraction backbone is an AlexNet network, which undergoes Onnx model conversion and lightweight acceleration using the TensorRT inference framework. Based on this, AlexNet extracts features from the template region (template) to obtain template feature information of size (1,256,6,6). AlexNet extracts features from the search region (search) to obtain search feature information of size (1,256,22,22).

[0083] Step 2.2, through the first branch in the confidence prediction unit, based on the template feature information and the search feature information, determine the position information corresponding to each candidate detection box contained in the current image frame; and through the second branch in the confidence prediction unit, based on the template feature information and the search feature information, determine the confidence information corresponding to each candidate detection box contained in the current image frame.

[0084] In one embodiment, the template feature information and search feature information are fed into the class branch and box branch of the RPN network, respectively. The RPN network is then converted to an Onnx model and lightweight accelerated using the TensorRT inference framework. The class branch outputs a category prediction result (i.e., the confidence information) with a size of (1, 10, 17, 17), and the box branch outputs a position regression result (i.e., the position information) with a size of (1, 20, 17, 17). The output is 10 for each of the five anchors, with the probability of each anchor belonging to the background and target categories, and 20 for each of the five anchors, with the [x, y, w, h] offset, which is the position information.

[0085] Step 3: Based on the confidence information, determine the second object detection frame corresponding to the current image frame and the target attribute information corresponding to the second object detection frame from the candidate detection frame. The target attribute information includes the center point and target size information, which is also the target width and height information.

[0086] In a specific implementation, the second target object detection frame corresponding to the current image frame and the target attribute information corresponding to the second target object detection frame can be determined according to the following steps 3.1 to 3.2:

[0087] In step 3.1, the confidence information corresponding to each candidate detection frame is adjusted according to the distance between each candidate detection frame and the center point of the current image frame to obtain the target confidence information corresponding to each candidate detection frame.

[0088] In one embodiment, a window penalty can be performed on the confidence information score, that is, a window function (Hanning window, cosine window, etc.) is used to penalize the confidence information score of the edge candidate detection box that is farther away from the center point to lower its confidence information score, thereby obtaining the final target confidence information of each candidate detection box.

[0089] Step 3.2: Determine the second target detection frame corresponding to the current image frame and the initial attribute information of the second target detection frame from the candidate detection frames in descending order of target confidence information.

[0090] In one embodiment, the candidate detection frame corresponding to the highest target confidence information can be selected as the second target object detection frame, the center point of the selected candidate detection frame is also the center point of the second target object detection frame, and the width and height information of the selected candidate detection frame is also the initial width and height information of the second target object detection frame.

[0091] Step 3.3: Adjust the initial attribute information of the second target object detection frame corresponding to the current image frame to obtain the target attribute information of the second target object detection frame corresponding to the current image frame.

[0092] In one embodiment, the target size information of the second object detection frame corresponding to the previous image frame and the initial size information of the object detection frame corresponding to the current image frame can be smoothed and weighted to obtain the target size information of the second object detection frame corresponding to the current image frame. Specifically, the target width and height information of the second object detection frame in the previous image frame and the initial width and height information can be smoothed and weighted to obtain the target width and height information of the second object detection frame corresponding to the current image frame.

[0093] Step 4: Determine the real-time tracking coordinates of the target based on the target attribute information corresponding to the second target detection frame. Once the center point and target width and height information corresponding to the second target detection frame are known, a two-dimensional coordinate frame of the target can be obtained and used as the real-time tracking coordinates of the target.

[0094] In actual applications, the next image frame is processed according to the above steps 1 to 4, thereby achieving real-time tracking and photographing of the target object.

[0095] For the aforementioned step S108, the embodiment of the present disclosure provides an implementation method for controlling the gimbal camera based on the real-time tracking coordinates of the target object so that the target object is located in a specified area of ​​the image frame captured by the gimbal camera. The process of the gimbal camera can be controlled to receive the two-dimensional frame coordinates in real time, and the deviation between the target detection frame and the image center is obtained. The angle of the gimbal is controlled according to the deviation to keep the target object in the center area of ​​the picture.

[0096] In summary, the embodiments of the present disclosure have at least the following features:

[0097] (1) It integrates the advantages of strong generalization of large-scale target detection models and can track multiple scenes and multiple targets in real time on the terminal side.

[0098] (2) Optimize the target tracking algorithm to achieve real-time and accurate tracking;

[0099] (3) Provide a large model + UAV local target tracking method to achieve target detection and tracking for resource-constrained UAVs;

[0100] (4) Based on target recognition, the accuracy and speed of target tracking improve the accuracy of shooting targets.

[0101] For ease of understanding, the disclosed embodiment also provides another implementation method of the drone gimbal control method. Refer to the structural schematic diagram of a drone gimbal control system shown in Figure 2. Figure 2 illustrates that the system includes a drone end and a server end, and the drone end is communicatively connected to the server end; wherein, the drone end is equipped with a gimbal camera, the drone end is deployed with a target tracking model, and the drone end is used to execute the drone gimbal control method; the server end is deployed with a target detection model.

[0102] Specifically, the drone is equipped with a target tracking model, ONNX engine, and TensorRT engine, while the server is equipped with a large-scale target detection model and FastAPI services. The drone then sends the target category text and the initial image frame captured by the gimbal camera to the server in the form of a request. The server then responds with a first target detection frame to initialize the target tracking model. The gimbal camera also sends the captured continuous image frames to the drone. The drone then uses the target tracking model to determine a real-time second target detection frame based on the first target detection frame and the continuous image frames. This second target detection frame is then sent to the gimbal camera, enabling automated control of the gimbal camera.

[0103] The disclosed embodiment deploys a large-model target detection model on the server side, performs lightweight acceleration of the target tracking model using Onnx and TensorRT, and deploys it on the drone side. The target tracking model is initialized using the large-model target detection model, and the real-time target tracking model is used to track the target object. The drone's gimbal camera is automatically controlled to track and photograph the target object. In summary, the disclosed embodiment uses a large-model-based target detection model + target tracking model, while automatically controlling the onboard gimbal camera and tracking and photographing the target object in real time.

[0104] Based on the above embodiments, the present disclosure provides a drone gimbal control device. The device is applied to a drone, which is equipped with a gimbal camera. The drone is in communication with a server. The drone is equipped with a target tracking model, and the server is equipped with a target detection model. Referring to FIG3 , a schematic diagram of the structure of a drone gimbal control device is shown. The device mainly includes the following components:

[0105] An image acquisition module 302 is used to acquire a set of image frames captured by a PTZ camera;

[0106] The target detection module 304 is configured to send the preset target object category text and the specified image frame in the image frame set to the server, so that the server determines a first target object detection frame based on the target object category text and the specified image frame through the target detection model;

[0107] The target tracking module 306 is configured to determine the real-time tracking coordinates of the target object based on the image frame set and the first target object detection frame using a target tracking model;

[0108] The pan / tilt control module 308 is used to control the pan / tilt camera based on the real-time tracking coordinates of the target object so that the target object is located in a specified area of ​​the image frame captured by the pan / tilt camera.

[0109] The drone gimbal control device provided by the embodiment of the present disclosure deploys a target detection model on the server side and a target tracking model on the drone side, respectively, to combine the target detection model and the target tracking model, and use the target detection model to determine a first target detection frame with higher precision, so that the target tracking model can track the target in real time based on the first target detection frame and the image frame set, obtain the real-time tracking coordinates of the target, and then use the real-time tracking coordinates of the target to automatically control the gimbal camera.

[0110] In one embodiment, the target detection model includes a large model target detection model; the target detection module 304 is further configured to:

[0111] Based on the preset target object category text and the specified image frame in the image frame set, a target detection request is sent to the server side, so that when the server side receives the target detection request, the large model target detection model is called, and the first target object detection box is fed back for the target object category text and the specified image frame.

[0112] In one embodiment, the target tracking module 306 is further configured to:

[0113] Determining a template area based on the first object detection frame; and determining a search area based on the image frame set or the first object detection frame;

[0114] Using the target tracking model, based on the template area and the search area, determine at least one candidate detection frame contained in the current image frame in the image frame set, as well as the position information and confidence information corresponding to each candidate detection frame; the confidence information is used to represent the probability value of the candidate detection frame belonging to the target object;

[0115] Determining, from the candidate detection frames based on the confidence information, a second target detection frame corresponding to the current image frame and target attribute information corresponding to the second target detection frame; wherein the target attribute information includes a center point and target size information;

[0116] The real-time tracking coordinates of the target object are determined according to the target attribute information corresponding to the second target object detection frame.

[0117] In one embodiment, the target tracking module 306 is further configured to:

[0118] According to the preset first size information, taking the designated point of the first object detection frame as the center, intercepting the template area in the designated image frame;

[0119] The step of determining a search area based on the image frame set or the first object detection frame includes:

[0120] If the current image frame is the second image frame, intercepting a search area in the designated image frame with the designated point of the first object detection frame as the center according to the preset second size information;

[0121] Alternatively, if the current image frame is an image frame other than the second image frame, a search area is intercepted in the current image frame based on the second size information and centered at a designated point of the second target object detection frame corresponding to the previous image frame.

[0122] In one embodiment, the target tracking model includes a feature extraction unit and a confidence prediction unit, the confidence prediction unit includes a first branch and a second branch; the target tracking module 306 is further configured to:

[0123] Extracting template feature information of the template area and search feature information of the search area through a feature extraction unit;

[0124] Determine, by the first branch of the confidence prediction unit, the position information corresponding to each candidate detection frame contained in the current image frame based on the template feature information and the search feature information;

[0125] And through the second branch in the confidence prediction unit, based on the template feature information and the search feature information, the confidence information corresponding to each candidate detection box included in the current image frame is determined.

[0126] In one embodiment, the target tracking module 306 is further configured to:

[0127] According to the distance between each candidate detection frame and the center point of the current image frame, the confidence information corresponding to each candidate detection frame is adjusted to obtain the target confidence information corresponding to each candidate detection frame;

[0128] Determine, from the candidate detection frames, the second target object detection frame corresponding to the current image frame and the initial attribute information of the second target object detection frame in descending order of target confidence information;

[0129] The initial attribute information of the second target object detection frame corresponding to the current image frame is adjusted to obtain target attribute information of the second target object detection frame corresponding to the current image frame.

[0130] In one embodiment, the target tracking module 306 is further configured to:

[0131] The target size information of the second target object detection frame corresponding to the previous image frame and the initial size information of the target object detection frame corresponding to the current image frame are smoothed and weighted to obtain the target size information of the second target object detection frame corresponding to the current image frame.

[0132] The device provided in the embodiment of the present disclosure has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding content in the aforementioned method embodiment.

[0133] An embodiment of the present disclosure provides a drone gimbal control system, which includes a drone end and a server end, and the drone end is communicatively connected to the server end; wherein the drone end is equipped with a gimbal camera, a target tracking model is deployed on the drone end, and the drone end is used to execute a drone gimbal control method; and the server end is deployed with a target detection model.

[0134] An embodiment of the present disclosure provides a computer device. Specifically, the computer device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0135] Figure 4 is a structural schematic diagram of a computer device provided in an embodiment of the present disclosure. The computer device 100 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43. The processor 40, the communication interface 43 and the memory 41 are connected via the bus 42; the processor 40 is used to execute an executable module stored in the memory 41, such as a computer program.

[0136] The memory 41 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element communicates with at least one other network element via at least one communication interface 43 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0137] The bus 42 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG4 shows only one bidirectional arrow, but this does not mean that there is only one bus or one type of bus.

[0138] Among them, the memory 41 is used to store programs, and the processor 40 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the aforementioned embodiment of the present disclosure can be applied to the processor 40 or implemented by the processor 40.

[0139] The processor 40 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 40 or software instructions. The above processor 40 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 41, and processor 40 reads the information in memory 41 and, in conjunction with its hardware, completes the steps of the above method.

[0140] The computer program product of the readable storage medium provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the above method embodiments. The specific implementation can be found in the above method embodiments and will not be repeated here.

[0141] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0142] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims. Industrial Applicability

[0143] The disclosed embodiments provide a method, device, system, computer device, and medium for controlling a drone gimbal. After acquiring a set of image frames captured by a gimbal camera, the method sends a preset target object category text and a specified image frame in the image frame set to a server, so that the server determines a first target object detection frame based on the target object category text and the specified image frame through a target detection model; then determines the target object's real-time tracking coordinates based on the image frame set and the first target object detection frame through a target tracking model; and finally controls the gimbal camera based on the real-time tracking coordinates of the target object so that the target object is located in a specified area in the image frame captured by the gimbal camera. The above method deploys a target detection model on the server and a target tracking model on the drone, respectively, and combines the target detection model and the target tracking model to determine a first target object detection frame with high accuracy using the target detection model, so that the target tracking model can track the target object in real time based on the first target object detection frame and the image frame set, obtain the target object's real-time tracking coordinates, and then automatically control the gimbal camera using the real-time tracking coordinates of the target object.

[0144] Furthermore, it is understood that the drone gimbal control method, apparatus, system, computer device, and medium provided by the embodiments of the present disclosure are reproducible and can be used in a variety of industrial applications. For example, the drone gimbal control method, apparatus, system, computer device, and medium provided by the embodiments of the present disclosure can be used in the field of drone control technology.

Claims

1. A method for controlling a drone gimbal, characterized in that: The method is applied to a drone end, the drone end is equipped with a pan-tilt camera, the drone end is in communication connection with a server end, the drone end is deployed with a target tracking model, and the server end is deployed with a target detection model, and the method includes: Obtain a set of image frames captured by the PTZ camera; Sending a preset target object category text and a specified image frame in the image frame set to the server end, so that the server end determines a first target object detection frame based on the target object category text and the specified image frame through the target detection model; Determining the real-time tracking coordinates of the target object based on the image frame set and the first target object detection frame by using the target tracking model; The pan-tilt camera is controlled based on the real-time tracking coordinates of the target object so that the target object is located in a specified area in the image frame captured by the pan-tilt camera.

2. The drone gimbal control method according to claim 1, characterized in that: The target detection model includes a large model target detection model; The step of sending a preset target object category text and a specified image frame in the image frame set to the server end, so that the server end determines a first target object detection frame based on the target object category text and the specified image frame through the target detection model, comprises: Based on the preset target object category text and the specified image frame in the image frame set, a target detection request is sent to the server side, so that when the server side receives the target detection request, the large model target detection model is called, and a first target object detection box is fed back for the target object category text and the specified image frame.

3. The drone gimbal control method according to claim 1, characterized in that: The step of determining the real-time tracking coordinates of the target object based on the image frame set and the first target object detection frame by using the target tracking model includes: Determine a template area based on the first object detection frame; and determine a search area based on the image frame set or the first object detection frame; Determine, by means of the target tracking model, at least one candidate detection frame included in the current image frame in the image frame set, and position information and confidence information corresponding to each of the candidate detection frames based on the template area and the search area; wherein the confidence information is used to characterize a probability value that the candidate detection frame belongs to the target object; Based on the confidence information, determine a second target object detection frame corresponding to the current image frame and target attribute information corresponding to the second target object detection frame from the candidate detection frame; wherein the target attribute information includes a center point and target size information; The real-time tracking coordinates of the target object are determined according to the target attribute information corresponding to the second target object detection frame.

4. The method for controlling a UAV gimbal according to claim 3, characterized in that: The step of determining a template area based on the first object detection frame includes: According to the preset first size information, taking the designated point of the first target detection frame as the center, intercepting the template area in the designated image frame; The step of determining a search area based on the image frame set or the first target object detection frame includes: If the current image frame is the second image frame, based on the preset second size information, a search area is intercepted in the designated image frame with the designated point of the first target object detection frame as the center; Alternatively, if the current image frame is an image frame other than the second image frame, a search area is intercepted in the current image frame based on the second size information and centered on the designated point of the second target object detection frame corresponding to the previous image frame.

5. The drone gimbal control method according to claim 3, characterized in that: The target tracking model includes a feature extraction unit and a confidence prediction unit, and the confidence prediction unit includes a first branch and a second branch; The step of determining, by the target tracking model, at least one candidate detection frame included in the current image frame in the image frame set, and position information and confidence information corresponding to each candidate detection frame based on the template area and the search area, comprises: Extracting, by the feature extraction unit, template feature information of the template area and search feature information of the search area; Determine, by the first branch in the confidence prediction unit, position information corresponding to each of the candidate detection boxes included in the current image frame based on the template feature information and the search feature information; And through the second branch in the confidence prediction unit, based on the template feature information and the search feature information, the confidence information corresponding to each of the candidate detection boxes included in the current image frame is determined.

6. The method for controlling a UAV gimbal according to claim 3, characterized in that: The step of determining, from the candidate detection frame based on the confidence information, a second target object detection frame corresponding to the current image frame and target attribute information corresponding to the second target object detection frame comprises: According to the distance between each candidate detection frame and the center point of the current image frame, the confidence information corresponding to each candidate detection frame is adjusted to obtain the target confidence information corresponding to each candidate detection frame; Determining, from the candidate detection frames, a second target object detection frame corresponding to the current image frame and initial attribute information of the second target object detection frame in a descending order of the target confidence information; The initial attribute information of the second target object detection frame corresponding to the current image frame is adjusted to obtain the target attribute information of the second target object detection frame corresponding to the current image frame.

7. The method for controlling a UAV gimbal according to claim 6, characterized in that: The step of adjusting the initial attribute information of the second target object detection frame corresponding to the current image frame to obtain the target attribute information of the second target object detection frame corresponding to the current image frame includes: The target size information of the second target object detection frame corresponding to the previous image frame and the initial size information of the target object detection frame corresponding to the current image frame are smoothed and weighted to obtain the target size information of the second target object detection frame corresponding to the current image frame.

8. A drone gimbal control device, characterized in that: The device is applied to a drone end, the drone end is equipped with a pan-tilt camera, the drone end is connected to a server end in communication, the drone end is deployed with a target tracking model, and the server end is deployed with a target detection model. The device includes: An image acquisition module, used to acquire a set of image frames captured by the PTZ camera; A target detection module, used for sending a preset target object category text and a specified image frame in the image frame set to the server end, so that the server end determines a first target object detection frame based on the target object category text and the specified image frame through the target detection model; A target tracking module, configured to determine the real-time tracking coordinates of the target object based on the image frame set and the first target object detection frame through the target tracking model; The pan-tilt control module is used to control the pan-tilt camera based on the real-time tracking coordinates of the target object so that the target object is located in a specified area in the image frame captured by the pan-tilt camera.

9. A UAV gimbal control system, characterized in that: It includes a drone end and a server end, wherein the drone end is in communication connection with the server end; wherein, The drone end is equipped with a gimbal camera, and a target tracking model is deployed on the drone end, and the drone end is used to execute the method described in any one of claims 1 to 7; The server is deployed with a target detection model.

10. A computer device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 7.

Images