Unmanned aerial vehicle control method and device for aloft work

Abstract

The invention discloses an unmanned aerial vehicle control method and device for aloft work. The method comprises the following steps: when an unmanned aerial vehicle lifts to a preset height, acquiring an operation video of an operation site through a shooting device configured by the unmanned aerial vehicle, and marking a target object in the operation video by using a detection frame; tracking the target object based on a preset target tracking algorithm and the labeling box to obtain an object position of the target object; and determining a target angle of a pan-tilt of the unmanned aerial vehicle according to the object position through a proportional integral differential controller, and adjusting the pan-tilt to the target angle, so that the target object is located at the central position of the video frame of the target video, thereby solving the problem that the unmanned aerial vehicle cannot automatically and stably track the target in a working site, and improving the working efficiency of the unmanned aerial vehicle. The tracking precision and the video stability are improved, the manual intervention is reduced, and the working efficiency and the safety are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a method and device for controlling an unmanned aerial vehicle for high-altitude operations. Background Art

[0002] With the development of industries such as electricity and communications, the construction and maintenance of transmission lines and their supporting towers are increasing, among which aerial work has become an indispensable part. In order to ensure the safety and standardization of aerial work, real-time detection of aerial workers is particularly important.

[0003] The detection methods in related technologies, such as manual on-site inspections and fixed-point video detection, are often difficult to achieve full coverage when facing a wide operating area, and are inefficient in complex terrain conditions and severe weather conditions, and have large blind spots and potential safety hazards. The method of using drones for tracking detection requires the inspector to control the flight of the drone and adjust the shooting angle of the camera carried by the drone at the same time, which is inefficient and the video stability of the captured video is poor. Summary of the invention

[0004] The present invention provides a method and device for controlling a drone for high-altitude operations, so as to accurately track a target object at a work site, ensure that the target object is always centered, and improve detection efficiency and safety.

[0005] According to one aspect of the present invention, a method for controlling a drone for high-altitude operations is provided, which may include:

[0006] When the drone is lifted to a preset height, a video of the operation site is collected by a camera configured on the drone, and a target object in the video is marked with a detection frame;

[0007] Tracking the target object based on a preset target tracking algorithm and the annotation frame to obtain the object position of the target object;

[0008] The target angle of the gimbal of the UAV is determined according to the position of the object by a proportional-integral-differential controller, and the gimbal is adjusted to the target angle so that the target object is located at the center of a video frame of a target video.

[0009] According to another aspect of the present invention, there is provided a drone control device for high-altitude operations, which may include:

[0010] The operation video acquisition module is used to collect the operation video of the operation site through the shooting device configured by the drone when the drone is lifted to a preset height, and mark the target object in the operation video with a detection frame;

[0011] A target object tracking module, used to track the target object based on a preset target tracking algorithm and the annotation frame to obtain the object position of the target object;

[0012] The gimbal angle adjustment module is used to determine the target angle of the gimbal of the drone according to the position of the object through a proportional-integral-differential controller, and adjust the gimbal to the target angle so that the target object is located at the center of the video frame of the target video.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the drone control method for high-altitude operations described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the drone control method for high-altitude operations described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention, first, by collecting the operation video of the operation site through the shooting device configured by the drone when the drone is lifted to a preset height, and marking the target object in the operation video with a detection frame, the operation site video can be automatically collected, and by automatically identifying and marking the target object, the need for manual intervention is reduced, and the automation level of detection is improved; then, the target object is tracked based on the preset target tracking algorithm and the marking frame to obtain the object position of the target object, and the position change of the target object can be accurately tracked to provide support for the safety management and emergency response of the operation site; finally, the target angle of the pan-tilt of the drone is determined according to the object position through a proportional integral differential controller, and the pan-tilt is adjusted to the target angle, so that the target object is at the center of the video frame of the target video, and the position change of the target object can be quickly responded to, the shaking and delay are reduced, the stability of the video picture is improved, and the video detection personnel can clearly observe the target object. The technical solution of the embodiment of the present invention solves the problem that the drone cannot automatically and stably track the target at the operation site, improves the tracking accuracy and video stability, reduces manual intervention, and enhances the operation efficiency and safety.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flow chart of a method for controlling a drone for high-altitude operations provided according to Embodiment 1 of the present invention;

[0022] Figure 2 is a flow chart of a method for controlling a drone for high-altitude operations provided according to Embodiment 2 of the present invention;

[0023] Figure 3 is a flow chart of a method for controlling a drone for high-altitude operations provided according to Embodiment 3 of the present invention;

[0024] Figure 4 is a structural schematic diagram of a drone control device for high-altitude operations provided according to a fourth embodiment of the present invention;

[0025] Figure 5 It is a structural schematic diagram of an electronic device for implementing the drone control method for high-altitude operations according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0031] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0032] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0033] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0034] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0035] Embodiment 1

[0036] Figure 1 A flowchart of a drone control method for high-altitude operations is provided for the first embodiment of the present invention. This embodiment is applicable to the situation where a drone is controlled to stably track a target object at a work site. The method can be executed by a drone control device for high-altitude operations. The drone control device for high-altitude operations can be implemented in the form of hardware and / or software, and optionally, can be implemented by an electronic device, which can be a mobile terminal, a PC, a server, etc.

[0037] like Figure 1 As shown, the method may specifically include:

[0038] S110: When the drone is lifted to a preset height, a shooting device configured on the drone is used to collect a work video of the work site, and a target object in the work video is marked with a detection frame.

[0039] Among them, the preset height can be understood as the height value pre-set by the drone in the high-altitude operation task, so as to ensure that the drone maintains a suitable observation height when performing the task and avoids collision with obstacles. The work site can be understood as the location where the drone performs the task. The drone can use the configured shooting device at the work site to collect the work video of the work site. The work video can be understood as a video resource shot by the shooting device configured by the drone, which is used to obtain the situation at the work site. The target object can be understood as a specific object that needs to be identified or tracked in the work video. Exemplarily, the target object may include but is not limited to construction workers or maintenance personnel performing high-altitude operations.

[0040] On the basis of the above scheme, optionally, marking the target object in the operation video with a labeling box includes: using a target recognition model to identify the target object in the operation video, and obtaining the target object in the operation video marked with a labeling box, wherein the target recognition model includes a cross-stage partial network, a spatial pyramid pool, a path aggregation network and multiple output layers, each output layer is used to detect different target objects, and output the object labeling box, object category probability and confidence of the target object; the model loss of the target recognition model includes the regression loss corresponding to the object labeling box, the confidence loss used to characterize the existence and non-existence of the target object, and the classification loss corresponding to the object category probability.

[0041] Among them, the target recognition model can be understood as a machine learning model for identifying target objects in the operation video, and is used to automatically identify and locate multiple target objects in the operation video. The cross-stage partial network can be used to process input data in stages, focusing on processing information of different scales at different stages to improve the efficiency and accuracy of calculation. The spatial pyramid pool is used to process input images of different sizes, and extract spatial features of different levels by constructing a pyramid structure. The path aggregation network can enhance the representation ability of the entire feature hierarchy through bottom-up and top-down feature fusion, and shorten the information path between low-level and top-level features. The multiple output layers can be understood as each output layer is responsible for detecting different target objects. The output layer can be the last layer of the model, responsible for generating the final prediction result to output the object annotation box, object category probability and confidence, etc. The object annotation box can be used to mark the exact position of the target object. The object category probability can be understood as the prediction confidence of the category to which the target object in the identified annotation box belongs. The regression loss can be understood as the accuracy of evaluating the position of the object detection box. The confidence loss can be understood as the accuracy of evaluating the existence or non-existence of the target object by the target recognition model. The classification loss can be understood as the loss of accuracy of the target recognition model in predicting the category. Based on the above scheme, the model loss can be determined based on the following formula:

[0042] L=λ cls ·L cls +λ obj ·L obj +λ noobj ·L noobj +λ bbox ·L bbox ;

[0043] Among them, λ cls Represents the weight coefficient of the classification loss corresponding to the object category probability; L cls represents the classification loss corresponding to the object category probability; obj The weight coefficient representing the confidence loss of the target object; L obj Indicates the confidence loss of the target object; noobj The weight coefficient representing the confidence loss of the target object not existing; L noobj Indicates the confidence loss that the target object does not exist; λ bbox Represents the weight coefficient of the regression loss corresponding to the object annotation box; L bbox Represents the regression loss corresponding to the object annotation box.

[0044] S120: Track the target object based on a preset target tracking algorithm and the annotation frame to obtain an object position of the target object.

[0045] The target tracking algorithm may be understood as an algorithm for tracking a target object in an operation video, and may maintain continuous tracking of the target object in consecutive operation video frames.

[0046] On the basis of the above scheme, optionally, the target object is tracked based on a preset target tracking algorithm and the annotation box to obtain the object position of the target object, including: determining a prediction box of the target object in the latter video frame of two adjacent video frames according to the historical motion trajectory of the target object and the current motion model through a Kalman filter, and determining the intersection-over-union ratio between the detection box of the target object and the prediction box; extracting object appearance features of the target object by using a convolutional neural network, and determining the appearance similarity between the object appearance features in two adjacent video frames; determining a matching cost of a Hungarian algorithm according to the intersection-over-union ratio and the appearance similarity, determining the detection box that matches the prediction box using the Hungarian algorithm according to the matching cost, and determining the object position of the target object according to the position of the detection box that matches the prediction box.

[0047] Among them, the Kalman filter can be used as a probability statistical filter to estimate the state of the target object. The prediction frame can be understood as the position frame of the target object in the next frame predicted by the Kalman filter in the current frame, and the state of the filter is updated according to the result to correct the prediction error. The detection frame can be understood as the target position frame identified by the target tracking algorithm in the current frame. The detection frame can be compared with the prediction frame to determine the object position. The intersection-over-union ratio can be understood as a measure of the degree of matching between the prediction frame and the detection frame. The object appearance feature can be understood as an appearance feature extracted from the video frame by the convolutional neural network to represent the target object. The appearance similarity can be understood as an indicator to measure the similarity of the appearance features of the target object in two adjacent video frames, which is used to assist in determining whether the two detection frames belong to the same target object. The Hungarian algorithm can be understood as an algorithm for finding the matching cost with the minimum cost, which is used to determine the best match between the prediction frame and the detection frame in target tracking. The matching cost can be understood as an indicator used to measure the quality of the match between the prediction frame and the detection frame, so as to determine the best match.

[0048] Based on the above solution, in the Kalman filter, prediction can be made according to the following state prediction equation:

[0049]

[0050] Among them, F represents the state transfer matrix, k represents the time, represents the control input matrix, μ k represents the control vector, ω k is the process noise;

[0051] The equation for prediction update is:

[0052]

[0053] Among them, z k represents the predicted value, H represents the observation matrix, K represents the Kalman gain, represents the prediction residual.

[0054] Based on the above solution, the matching cost of the Hungarian algorithm can be calculated as follows:

[0055] d ij =α·(1-IOU ij )+(1-cos(f i ,f j ));

[0056] Among them, dij represents the matching cost between the i-th detection box and the j-th prediction box, α represents the weight factor that balances the distance between the two i-th detection boxes and the j-th prediction box, and IOU ij represents the intersection-over-union ratio of the i-th detection box and the j-th prediction box, cos(f i ,f j ) represents the similarity between the i-th detection box and the j-th prediction box.

[0057] S130. Determine a target angle of the gimbal of the drone according to the position of the object through a proportional-integral-differential controller, and adjust the gimbal to the target angle so that the target object is located at the center of a video frame of a target video.

[0058] Wherein, the proportional-integral-differential controller can be used to adjust the output to achieve a desired value. Exemplarily, the proportional-integral-differential controller may include a proportional term, an integral term, and a differential term. The proportional term is used to quickly respond to deviations. The integral term is used to eliminate steady-state errors. The differential term is used to reduce over-adjustment. The target angle can be understood as the angle to which the pan / tilt head needs to be adjusted based on the position of the target object to ensure that the pan / tilt head can keep the target object at the center of the video frame. The center position can be understood as the center point in the video frame, which can specifically be the middle position in the horizontal and vertical directions.

[0059] On the basis of the above scheme, optionally, the target angle includes a target pitch angle and a target yaw angle; the determining the target angle of the gimbal of the UAV according to the object position through a proportional-integral-differential controller includes: determining a first distance deviation in the horizontal direction and a second distance deviation in the vertical direction of the target object according to the object position and the center position; determining a first angular velocity change according to the first distance deviation through a proportional-integral-differential controller, determining a yaw angle change according to the first angular velocity change, and adjusting the gimbal of the UAV to the target pitch angle according to the yaw angle change; and, determining a second angular velocity change according to the second distance deviation through a proportional-integral-differential controller, determining a pitch angle change according to the second angle change, and adjusting the gimbal of the UAV to the target pitch angle according to the pitch angle change.

[0060] Wherein, the first distance deviation can be understood as the deviation of the target object in the horizontal direction relative to the center position of the video frame, so as to quantify the deviation degree of the target object in the horizontal direction. Correspondingly, the second distance deviation can be understood as the deviation of the target object in the vertical direction relative to the center position of the video frame, so as to quantify the deviation degree of the target object in the vertical direction. The first angular velocity change can be understood as the angular velocity change of the gimbal in the horizontal direction determined according to the first distance deviation, which is used to adjust the rotation speed of the gimbal in the horizontal direction. The yaw angle change can be understood as the rotation angle change of the gimbal in the horizontal direction determined according to the first angular velocity change, so as to adjust the horizontal rotation of the gimbal, so as to return the target object to the center of the video frame. The second angular velocity change can be understood as the angular velocity change of the gimbal in the vertical direction determined according to the second distance deviation, so as to adjust the rotation speed of the gimbal in the vertical direction. The pitch angle change can be understood as the rotation angle change of the gimbal in the vertical direction determined according to the second angular velocity change, so as to adjust the vertical rotation of the gimbal, so as to return the target object to the center of the video frame.

[0061] Exemplarily, first, the object position p of the target object in the current video frame can be obtained. a (x1,y1) and the center position p d (x2, y2); where x1 represents the horizontal position of the target object, y1 represents the vertical position of the target object, x2 represents the horizontal position of the center of the target object, and y2 represents the vertical position of the center of the target object;

[0062] Next, the first distance deviation Δx and the second distance deviation Δy are calculated based on the following formula:

[0063] Δx=x1-x2;

[0064] Δy=y1-y2

[0065] Then, the first angular velocity change u1 and the second angular velocity change u2 can be determined based on the following formula:

[0066]

[0067] Among them, K p Represents the proportional gain coefficient, K i Indicates the integral gain coefficient, K d represents the differential gain coefficient, and dt represents the sampling time interval.

[0068] The technical solution of the embodiment of the present invention, first, by collecting the operation video of the operation site through the shooting device configured by the drone when the drone is lifted to a preset height, and marking the target object in the operation video with a detection frame, the operation site video can be automatically collected, and by automatically identifying and marking the target object, the need for manual intervention is reduced, and the automation level of detection is improved; then, the target object is tracked based on the preset target tracking algorithm and the marking frame to obtain the object position of the target object, and the position change of the target object can be accurately tracked to provide support for the safety management and emergency response of the operation site; finally, the target angle of the pan-tilt of the drone is determined according to the object position through a proportional integral differential controller, and the pan-tilt is adjusted to the target angle, so that the target object is at the center of the video frame of the target video, and the position change of the target object can be quickly responded to, the shaking and delay are reduced, the stability of the video picture is improved, and the video detection personnel can clearly observe the target object. The technical solution of the embodiment of the present invention solves the problem that the drone cannot automatically and stably track the target at the operation site, improves the tracking accuracy and video stability, reduces manual intervention, and enhances the operation efficiency and safety.

[0069] Embodiment 2

[0070] Figure 2 A flowchart of a drone control method for high-altitude operations provided in Embodiment 2 of the present invention, this embodiment can be combined with the above embodiments. Optionally, after adjusting the gimbal to the target angle, it also includes: performing path planning for the drone so that the target object is within the shooting range of the shooting device of the drone. For specific implementation methods, please refer to the description of this embodiment. Among them, the technical features that are the same or similar to the above embodiments are not repeated here.

[0071] like Figure 2 As shown, the drone control method for high-altitude operations provided in this embodiment may specifically include:

[0072] S210, when the drone is lifted to a preset height, collecting a work video of the work site through a shooting device configured for the drone, and marking a target object in the work video with a detection frame;

[0073] S220, tracking the target object based on a preset target tracking algorithm and the annotation frame to obtain an object position of the target object;

[0074] S230, determining a target angle of the gimbal of the drone according to the position of the object through a proportional-integral-differential controller, and adjusting the gimbal to the target angle so that the target object is located at the center of a video frame of a target video.

[0075] S240: Perform path planning for the drone so that the target object is within the shooting range of the shooting device of the drone.

[0076] The path planning can be understood as determining the optimal or feasible path from the starting point to the end point, which is used to determine the flight of the drone and ensure that the target object is always within the shooting range of the shooting device. The shooting range can be understood as the maximum viewing angle range that the shooting device can capture.

[0077] On the basis of the above scheme, optionally, the path planning of the UAV so that the target object is within the shooting range of the shooting device of the UAV includes: predicting the position of the target object at a future moment according to the motion trajectory and motion speed of the target object to obtain the predicted position of the object; when the predicted position of the object is not within the shooting range of the shooting device, determining the flight path of the UAV based on the predicted position of the object and the current position of the UAV; and adjusting the flight parameters of the UAV so that the UAV flies according to the flight path.

[0078] Among them, the motion trajectory can be understood as the motion path of the target object that changes with time, which is used to determine the moving state of the target object and predict the object predicted position of the target object. The motion speed can be understood as the distance moved by the target object per unit time, and the object predicted position of the target object can be predicted in combination with the motion trajectory. The object predicted position can be understood as the position of the target object at a future moment predicted based on the motion trajectory and speed. The flight path can be understood as the predetermined path for the drone to reach the object predicted position from the current position. The flight parameters can be understood as parameters that control the flight state of the drone. Exemplarily, the flight parameters may at least include but are not limited to the flight direction, etc.

[0079] The technical solution of the embodiment of the present invention predicts the object predicted position of the target object and adjusts the flight path of the drone to ensure that the target is always within the shooting range, which not only improves the comprehensiveness and accuracy of detection, but also reduces the need for manual intervention, thereby improving operational efficiency and safety.

[0080] Embodiment 3

[0081] Figure 3 A flowchart of a drone control method for high-altitude operations provided in Embodiment 3 of the present invention, this embodiment can be combined with the above embodiments. Optionally, after the target angle of the gimbal of the drone is determined according to the position of the object through the proportional-integral-differential controller, it also includes: adjusting the focal length of the shooting device according to the distance between the target object and the drone and the size information of the target object; capturing the operation video based on the adjusted focal length, performing image enhancement on the video frames in the operation video, and displaying the enhanced video frames. Among them, the technical features that are the same or similar to those in the above embodiments are not repeated here.

[0082] like Figure 3 As shown, the drone control method for high-altitude operations provided in this embodiment may specifically include:

[0083] S310: When the drone is lifted to a preset height, a shooting device configured on the drone is used to collect a work video of the work site, and a target object in the work video is marked with a detection frame.

[0084] S320: Track the target object based on a preset target tracking algorithm and the annotation frame to obtain an object position of the target object.

[0085] S330. Determine the target angle of the gimbal of the drone according to the position of the object through a proportional-integral-differential controller, and adjust the gimbal to the target angle so that the target object is located at the center of a video frame of a target video.

[0086] S340: Adjust the focal length of the shooting device according to the distance between the target object and the drone and the size information of the target object.

[0087] The distance can be understood as the straight-line distance between the target object and the drone, which is used to determine the focal length that needs to be adjusted by the camera to obtain a clear image. The size information can be understood as the actual physical size of the target object, which may include but is not limited to width and height. The focal length can be understood as the optical properties of the camera lens to determine the magnification and viewing angle range of the target object in the video frame.

[0088] On the basis of the above scheme, optionally, adjusting the focal length of the shooting device according to the distance between the target object and the drone and the size information of the target object includes: measuring the shooting distance between the target object and the drone using a ranging sensor carried by the drone; determining the image size of the target object in the video frame by edge detection or feature matching algorithm, and determining the physical size of the target object according to the shooting parameters of the shooting device; determining the target focal length of the shooting device according to the physical size, the distance and the expected magnification of the target object, and adjusting the focal length of the shooting device to the target focal length.

[0089] Among them, the edge detection can be understood as a method for identifying the location where the pixel intensity in the image changes significantly. The edge detection can be used to determine the boundary of the target object in the video frame, and then determine the image size of the target object. The feature matching can be understood as a method for identifying visual features shared between different video frames. The feature matching is used to determine the feature points of the target object in the video frame, and then determine the image size of the target object. The physical size can be understood as the actual physical size of the target object, which may include width and height, etc. The physical size can be combined with the shooting distance and shooting parameters to determine the target focal length of the required shooting device to ensure the appropriate size of the target object in the video frame. The expected magnification can be understood as the magnification ratio of the target object in the video frame relative to its actual physical size, which can be used to determine the focal length required by the shooting device to achieve the desired image magnification effect.

[0090] S350: Capture the operation video based on the adjusted focal length, perform image enhancement on the video frames in the operation video, and display the enhanced video frames.

[0091] On the basis of the above scheme, optionally, image enhancement can be performed by adjusting the brightness of the video frame, performing noise reduction processing or contrast adjustment. Specifically, the image enhancement of the video frame in the operation video includes at least one of the following operations: adjusting the brightness of the video frame in the operation video; performing noise reduction processing on the video frame in the operation video using a preset noise reduction algorithm, wherein the preset noise reduction algorithm includes a median filter, a bilateral filter or a non-local mean noise reduction algorithm; using a histogram equalization or an adaptive contrast stretching algorithm to adjust the contrast of at least part of the picture area in the video frame in the operation video to distinguish the target object from background information other than the target object.

[0092] Among them, the median filter can be understood as taking the median of the pixel values ​​in the neighborhood of each pixel point in the video frame as the new value of the pixel. The bilateral filter can be understood as a filtering technology that can both smooth the video frame and retain the edge, and the filtering result can be calculated by using the weight of the spatial distance and the pixel value difference. The non-local mean denoising algorithm can be understood as an algorithm that uses the repetitive structure in the video frame to reduce noise. The non-local mean denoising algorithm can calculate the similarity between different regions of the video frame and then use the weighted average to estimate the noise-free pixel value, which can effectively remove noise while maintaining the texture and details of the video frame. The histogram equalization can be understood as being used to expand the dynamic range of the video frame, thereby enhancing the contrast. The histogram equalization can make the pixel intensity distribution more uniform by transforming the histogram distribution of the video frame. The adaptive contrast stretching can be understood as a method for adjusting the contrast of the local characteristics of the video frame. The adaptive contrast stretching can improve the contrast of the video frame picture area by stretching the grayscale of the video frame, especially when there are a large number of similar tones in the video frame.

[0093] The technical solution of the embodiment of the present invention, by adjusting the focal length of the drone shooting device and improving the video quality through image enhancement technology, not only makes video detection more accurate, but also maintains good visibility under various lighting and environmental conditions, and significantly improves the accuracy of target recognition through brightness adjustment and noise reduction processing. In addition, the optimization of contrast helps to better highlight the target object and reduce background interference, thereby achieving more efficient task execution and data analysis.

[0094] Embodiment 4

[0095] Figure 4 This is a schematic diagram of the structure of a drone control device for high-altitude operations provided by the third embodiment of the present invention. Figure 4 As shown, the device includes: an operation video acquisition module 410, a target object tracking module 420, and a pan / tilt angle adjustment module 430.

[0096] Among them, the operation video acquisition module 410 is used to collect the operation video of the operation site through the shooting device configured for the drone when the drone is lifted to a preset height, and mark the target object in the operation video with a detection frame; the target object tracking module 420 is used to track the target object based on a preset target tracking algorithm and the marking frame to obtain the object position of the target object; the gimbal angle adjustment module 430 is used to determine the target angle of the gimbal of the drone according to the object position through a proportional-integral-differential controller, and adjust the gimbal to the target angle so that the target object is at the center position of the video frame of the target video.

[0097] The technical solution of the embodiment of the present invention is as follows: first, when the drone is lifted to a preset height, the operation video of the operation site is collected by the shooting device configured by the drone, and the target object in the operation video is marked with a detection frame, so that the operation site video can be automatically collected, and the target object can be automatically identified and marked, which reduces the need for manual intervention and improves the automation level of detection; then, the target object is tracked by the target object tracking module based on the preset target tracking algorithm and the marking frame to obtain the object position of the target object, which can accurately track the position change of the target object and provide support for the safety management and emergency response of the operation site; finally, the pan-tilt angle adjustment module determines the target angle of the pan-tilt of the drone according to the object position through the proportional integral differential controller, and adjusts the pan-tilt to the target angle, so that the target object is at the center of the video frame of the target video, which can quickly respond to the position change of the target object, reduce shaking and delay, improve the stability of the video picture, and ensure that the video detection personnel can clearly observe the target object. The technical solution of the embodiment of the present invention solves the problem that the drone cannot automatically and stably track the target at the operation site, improves the tracking accuracy and video stability, reduces manual intervention, and enhances the operation efficiency and safety.

[0098] On the basis of the above scheme, optionally, the target object tracking module includes: an intersection-and-union ratio determination submodule, an appearance similarity determination submodule, and a position determination submodule. Among them, the intersection-and-union ratio determination submodule is used to determine the prediction frame of the target object in the latter of two adjacent video frames according to the historical motion trajectory of the target object and the current motion model through a Kalman filter, and determine the intersection-and-union ratio between the detection frame of the target object and the prediction frame; the appearance similarity determination submodule is used to extract the object appearance features of the target object using a convolutional neural network, and determine the appearance similarity between the object appearance features in two adjacent video frames; the position determination submodule is used to determine the matching cost of the Hungarian algorithm according to the intersection-and-union ratio and the appearance similarity, determine the detection frame that matches the prediction frame using the Hungarian algorithm according to the matching cost, and determine the object position of the target object according to the position of the detection frame that matches the prediction frame.

[0099] On the basis of the above scheme, the target angle includes a target pitch angle and a target yaw angle. Optionally, the gimbal angle adjustment module includes: a deviation determination submodule, a first pitch angle adjustment submodule, and a second pitch angle adjustment submodule. The deviation determination submodule is used to determine the first distance deviation of the target object in the horizontal direction and the second distance deviation in the vertical direction according to the object position and the center position; the first pitch angle adjustment submodule is used to determine the first angular velocity change according to the first distance deviation through a proportional integral differential controller, determine the yaw angle change according to the first angular velocity change, and adjust the gimbal of the drone to the target pitch angle according to the yaw angle change; and the second pitch angle adjustment submodule is used to determine the second angular velocity change according to the second distance deviation through a proportional integral differential controller, determine the pitch angle change according to the second angle change, and adjust the gimbal of the drone to the target pitch angle according to the pitch angle change.

[0100] On the basis of the above solution, optionally, the drone control device for high-altitude operation further includes: a path planning module. The path planning module is used to plan the path of the drone after adjusting the gimbal to the target angle, so that the target object is within the shooting range of the shooting device of the drone.

[0101] On the basis of the above scheme, optionally, the path planning module may include: a position prediction submodule, a flight path determination submodule and a flight parameter adjustment submodule. The position prediction submodule is used to predict the position of the target object at a future moment according to the motion trajectory and motion speed of the target object to obtain the predicted position of the object; the flight path determination submodule is used to determine the flight path of the drone based on the predicted position of the object and the current position of the drone when the predicted position of the object is not within the shooting range of the shooting device; the flight parameter adjustment submodule is used to adjust the flight parameters of the drone so that the drone flies according to the flight path, wherein the flight parameters may at least include the flight direction.

[0102] On the basis of the above scheme, optionally, the drone control device for high-altitude operation further includes: a focal length adjustment module and a video frame display module. The focal length adjustment module is used to adjust the focal length of the shooting device according to the distance between the target object and the drone and the size information of the target object after the target angle of the pan / tilt of the drone is determined according to the position of the object by the proportional integral differential controller; the video frame display module is used to capture the operation video based on the adjusted focal length, perform image enhancement on the video frames in the operation video, and display the enhanced video frames.

[0103] On the basis of the above scheme, optionally, the focus adjustment module may include: a shooting distance measurement submodule, a physical size determination module and a focus adjustment submodule. The shooting distance measurement submodule is used to measure the shooting distance between the target object and the drone using the distance measurement sensor carried by the drone; the physical size determination module is used to determine the image size of the target object in the video frame by edge detection or feature matching algorithm, and determine the physical size of the target object according to the shooting parameters of the shooting device; the focus adjustment submodule is used to determine the target focal length of the shooting device according to the physical size, the shooting distance and the expected magnification of the target object, and adjust the focal length of the shooting device to the target focal length.

[0104] On the basis of the above scheme, optionally, the video frame display module may include: a brightness adjustment submodule, a noise reduction submodule and a contrast adjustment module. The brightness adjustment submodule is used to adjust the brightness of the video frame in the operation video; the noise reduction submodule is used to use a preset noise reduction algorithm to perform noise reduction processing on the video frame in the operation video, wherein the preset noise reduction algorithm includes a median filter, a bilateral filter or a non-local mean noise reduction algorithm; the contrast adjustment module is used to use a histogram equalization or an adaptive contrast stretching algorithm to adjust the contrast of at least part of the picture area in the video frame in the operation video to distinguish the target object from the background information other than the target object.

[0105] On the basis of the above scheme, optionally, the operation video acquisition module can be used to: use a target recognition model to identify the target object in the operation video, and obtain the target object in the operation video marked with a labeling box, wherein the target recognition model includes a cross-stage partial network, a spatial pyramid pool, a path aggregation network and multiple output layers, each output layer is used to detect different target objects, and output the object labeling box, object category probability and confidence of the target object; the model loss of the target recognition model includes the regression loss corresponding to the object labeling box, the confidence loss used to characterize the existence and non-existence of the target object, and the classification loss corresponding to the object category probability.

[0106] The drone control device for high-altitude operations provided in the embodiment of the present invention can execute the drone control method for high-altitude operations provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0107] Example 5

[0108] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0109] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0110] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0111] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a drone control method for high-altitude operations.

[0112] In some embodiments, the drone control method for high-altitude operations may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the drone control method for high-altitude operations described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the drone control method for high-altitude operations in any other appropriate manner (e.g., by means of firmware).

[0113] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0115] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0116] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0117] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0118] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0119] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0120] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for controlling a drone for high-altitude operations, characterized in that: include: When the drone is lifted to a preset height, a video of the operation site is collected by a camera configured on the drone, and a target object in the video is marked with a detection frame; Tracking the target object based on a preset target tracking algorithm and the annotation frame to obtain the object position of the target object; The target angle of the gimbal of the UAV is determined according to the position of the object by a proportional-integral-differential controller, and the gimbal is adjusted to the target angle so that the target object is located at the center of a video frame of a target video.

2. The method according to claim 1, characterized in that The tracking of the target object based on a preset target tracking algorithm and the annotation frame to obtain the object position of the target object includes: Determine a prediction frame of the target object in a later video frame of two adjacent video frames according to the historical motion trajectory of the target object and the current motion model through a Kalman filter, and determine an intersection-over-union ratio between the detection frame of the target object and the prediction frame; Extracting object appearance features of the target object using a convolutional neural network, and determining appearance similarities between the object appearance features in two adjacent video frames; A matching cost of the Hungarian algorithm is determined according to the intersection-over-union ratio and the appearance similarity, a detection frame matching the prediction frame is determined using the Hungarian algorithm according to the matching cost, and an object position of the target object is determined according to a position of the detection frame matching the prediction frame.

3. The method according to claim 1, characterized in that The target angle includes a target pitch angle and a target yaw angle; The method of determining the target angle of the gimbal of the drone according to the position of the object by using a proportional-integral-differential controller comprises: Determine a first distance deviation amount in the horizontal direction and a second distance deviation amount in the vertical direction of the target object according to the object position and the center position; Determine, by a proportional-integral-differential controller, a first angular velocity change according to the first distance deviation, determine a yaw angle change according to the first angular velocity change, and adjust the gimbal of the UAV to a target pitch angle according to the yaw angle change; and Through a proportional-integral-differential controller, a second angular velocity change is determined according to the second distance deviation, a pitch angle change is determined according to the second angle change, and the gimbal of the UAV is adjusted to a target pitch angle according to the pitch angle change.

4. The method according to claim 1, characterized in that: After adjusting the pan / tilt platform to the target angle, the method further includes: A path planning is performed on the drone so that the target object is within the shooting range of the shooting device of the drone.

5. The method according to claim 4, characterized in that The performing path planning for the drone so that the target object is within the shooting range of the shooting device of the drone includes: Predicting the position of the target object at a future moment according to the motion trajectory and motion speed of the target object to obtain a predicted position of the object; In a case where the predicted position of the object is not within the shooting range of the shooting device, determining a flight path of the drone based on the predicted position of the object and the current position of the drone; The flight parameters of the UAV are adjusted so that the UAV flies according to the flight path, wherein the flight parameters at least include a flight direction.

6. The method according to claim 1, characterized in that After determining the target angle of the gimbal of the UAV according to the position of the object by using the proportional-integral-differential controller, the method further includes: Adjusting the focal length of the shooting device according to the distance between the target object and the drone and the size information of the target object; The operation video is captured based on the adjusted focal length, and image enhancement is performed on the video frames in the operation video, and the enhanced video frames are displayed.

7. The method according to claim 6, characterized in that The adjusting the focal length of the shooting device according to the distance between the target object and the drone and the size information of the target object includes: Using a distance measuring sensor carried by the drone to measure the shooting distance between the target object and the drone; Determine the image size of the target object in the video frame by edge detection or feature matching algorithm, and determine the physical size of the target object according to the shooting parameters of the shooting device; The target focal length of the photographing device is determined according to the physical size, the shooting distance and the expected magnification of the target object, and the focal length of the photographing device is adjusted to the target focal length.

8. The method according to claim 6, characterized in that The performing image enhancement on the video frames in the operation video includes at least one of the following operations: Adjusting the brightness of the video frames in the operation video; Using a preset noise reduction algorithm to perform noise reduction processing on the video frames in the operation video, wherein the preset noise reduction algorithm includes a median filter, a bilateral filter or a non-local mean noise reduction algorithm; A histogram equalization or adaptive contrast stretching algorithm is used to adjust the contrast of at least a portion of the picture area in the video frame in the operation video to distinguish the target object from background information other than the target object.

9. The method according to claim 1, characterized in that: The step of marking the target object in the operation video with a marking frame includes: A target recognition model is used to identify the target object in the operation video, and the target object in the operation video is annotated with a labeling box, wherein the target recognition model includes a cross-stage partial network, a spatial pyramid pool, a path aggregation network and multiple output layers, each output layer is used to detect a different target object, and output an object labeling box, an object category probability and a confidence of the target object; the model loss of the target recognition model includes a regression loss corresponding to the object labeling box, a confidence loss used to characterize the existence and non-existence of the target object, and a classification loss corresponding to the object category probability.

10. A drone control device for high-altitude operations, characterized in that: include: The operation video acquisition module is used to collect the operation video of the operation site through the shooting device configured by the drone when the drone is lifted to a preset height, and mark the target object in the operation video with a detection frame; A target object tracking module, used to track the target object based on a preset target tracking algorithm and the annotation frame to obtain the object position of the target object; The gimbal angle adjustment module is used to determine the target angle of the gimbal of the drone according to the position of the object through a proportional-integral-differential controller, and adjust the gimbal to the target angle so that the target object is located at the center of the video frame of the target video.

Images