Unmanned aerial vehicle adaptive adjustment method, device, equipment and medium

Abstract

The embodiment of the invention provides an unmanned aerial vehicle adaptive adjustment method and device, equipment and a medium. The method comprises the steps of determining a center coordinate of a monitoring target image, a pixel coordinate of a monitoring target and a relative position vector of the monitoring target relative to a holder of the unmanned aerial vehicle according to the monitoring target image acquired by image acquisition equipment of the unmanned aerial vehicle; determining whether the monitoring target is in the middle of the monitoring target image or not according to the center coordinate of the monitoring target image and the pixel coordinate of the monitoring target; if the monitoring target is not in the middle of the monitoring target image, determining a to-be-adjusted pitch angle and a to-be-adjusted horizontal angle of the unmanned aerial vehicle holder and a to-be-adjusted focal length of the image acquisition equipment according to the relative position vector; and finally, according to the to-be-adjusted pitch angle and horizontal angle of the unmanned aerial vehicle holder and the to-be-adjusted focal length of the image acquisition equipment, the angle of the unmanned aerial vehicle holder and the shooting focal length of the image acquisition equipment are adjusted, so that the target is always in the central area of the image, and the stability of the monitoring image is remarkably improved.

Description

Technical Field

[0001] This application relates to the field of automatic adjustment of unmanned aerial vehicles (UAVs), and particularly to a method, device, equipment and medium for adaptive adjustment of UAVs. Background Art

[0002] With the continuous expansion of the power grid scale and the continuous growth of the number of transmission lines, the traditional manual inspection method has been difficult to meet the requirements of efficient, accurate and frequent inspection of high-voltage transmission lines. In recent years, due to its flexibility in complex environments and the ability to provide perspectives and coverage that traditional monitoring devices cannot achieve, UAVs have gradually been widely used in the daily inspection and fault detection of high-voltage transmission lines. However, during the flight of a UAV, due to the continuous change of its relative position with respect to the ground monitoring target, the fixed gimbal of the UAV easily causes problems such as the target deviating from the center of the monitoring screen, video jitter, and image blurring in the monitoring screen, seriously affecting the quality of video data. Therefore, it is very important to improve the flexibility of the UAV with respect to the monitoring target and the quality of the monitoring screen.

[0003] In the prior art, the method for adjusting the monitoring target of a UAV usually relies on ground operators to manually control the flight attitude of the UAV and the orientation of the gimbal camera through a remote controller or a ground station system. Specifically, first, the operator manually identifies the monitoring target of interest through the monitoring screen; then, according to the position of the target, the operator manually operates the UAV to move and turn, so that its flight position is as close as possible to the target area; finally, the operator manually rotates the gimbal or adjusts the camera angle to keep the target in the center of the screen and ensure a better perspective is obtained.

[0004] However, the method for adjusting the monitoring target of a UAV in the prior art has a high labor cost and is difficult to meet the requirements of efficient, accurate and intelligent UAV monitoring. Summary of the Invention

[0005] Embodiments of this application provide a method, device, equipment and medium for adaptive adjustment of UAVs to solve the problem in the prior art that the method for adjusting the monitoring target of a UAV has a high labor cost and is difficult to meet the requirements of efficient, accurate and intelligent UAV monitoring.

[0006] In a first aspect, embodiments of this application provide a method for adaptive adjustment of a UAV, including:

[0007] Determine the center coordinates of the monitoring target image, the pixel coordinates of the monitoring target, and the relative position vector of the monitoring target with respect to the UAV gimbal according to the monitoring target image collected by the image acquisition device of the UAV;

[0008] Determine whether the monitoring target is in the middle of the monitoring target image according to the center coordinates of the monitoring target image and the pixel coordinates of the monitoring target;

[0009] If the monitored target is not in the middle of the monitored target image, determine the pitch angle, horizontal angle to be adjusted by the UAV gimbal, and the focal length to be adjusted by the image acquisition device according to the relative position vector;

[0010] Adjust the UAV gimbal angle and the shooting focal length of the image acquisition device according to the pitch angle, horizontal angle to be adjusted by the UAV gimbal, and the focal length to be adjusted by the image acquisition device.

[0011] In a possible implementation manner, the relative position vector includes a lateral position vector, a front-back position vector, and a vertical position vector;

[0012] The determining the pitch angle, horizontal angle to be adjusted by the UAV gimbal, and the focal length to be adjusted by the image acquisition device according to the relative position vector includes:

[0013] Calculate the horizontal distance and the relative distance between the monitored target and the UAV gimbal according to the lateral position vector, the front-back position vector, and the vertical position vector;

[0014] Determine the pitch angle to be adjusted by the UAV gimbal by using the arctangent function according to the horizontal distance and the vertical position vector;

[0015] Determine the horizontal angle to be adjusted by the UAV gimbal by using the arctangent function according to the lateral position vector and the front-back position vector;

[0016] Determine the focal length to be adjusted by the image acquisition device according to the relative distance and the preset imaging scale coefficient.

[0017] In a possible implementation manner, the determining the center coordinates of the monitored target image, the pixel coordinates of the monitored target, and the relative position vector of the monitored target relative to the UAV gimbal according to the monitored target image acquired by the image acquisition device of the UAV includes:

[0018] Determine the center coordinates of the monitored target image according to the pixels of the monitored target image;

[0019] Determine the pixel coordinates of the monitored target pixels by using a target detection algorithm according to the monitored target image;

[0020] Project the monitored target pixels into the global coordinate system according to the preset rotation matrix, preset translation vector, preset internal parameter matrix, and the pixel coordinates, and obtain the three-dimensional coordinates of the monitored target;

[0021] Determine the relative position vector of the monitored target relative to the UAV gimbal according to the three-dimensional coordinates of the monitored target and the three-dimensional coordinates of the UAV gimbal.

[0022] In a possible implementation, determining a relative position vector of the monitoring target relative to the drone pan-tilt according to the three-dimensional coordinates of the monitoring target and the three-dimensional coordinates of the drone pan-tilt includes:

[0023] Obtaining the three-dimensional coordinates of the drone pan-tilt according to the sensors of the drone pan-tilt;

[0024] Performing a subtraction process on the three-dimensional coordinates of the monitoring target and the three-dimensional coordinates of the drone pan-tilt to determine the relative position vector of the monitoring target relative to the drone pan-tilt.

[0025] In a possible implementation, determining whether the monitoring target is in the middle of the monitoring target image according to the center coordinates of the monitoring target image and the pixel coordinates of the monitoring target includes:

[0026] Performing a subtraction process on the center coordinates of the monitoring target image and the pixel coordinates of the monitoring target to obtain the deviation coordinates of the monitoring target and the monitoring target image;

[0027] When both the abscissa value and the ordinate value corresponding to the deviation coordinates are less than the preset coordinate values, it is determined that the monitoring target is in the middle of the monitoring target image.

[0028] In a possible implementation, the method further includes:

[0029] If the monitoring target is in the middle of the monitoring target image, determining the focal length to be adjusted by the image acquisition device according to the relative position vector.

[0030] In a possible implementation, after it is determined that the monitoring target is not in the middle of the monitoring target image, the method further includes:

[0031] Outputting an error signal for instructing the drone pan-tilt to adjust the pan-tilt angle.

[0032] In a second aspect, an embodiment of the present application provides a drone adaptive adjustment device, including:

[0033] A first determination module, configured to determine the center coordinates of the monitoring target image, the pixel coordinates of the monitoring target, and the relative position vector of the monitoring target relative to the drone pan-tilt according to the monitoring target image collected by the image acquisition device of the drone;

[0034] A second determination module, configured to determine whether the monitoring target is in the middle of the monitoring target image according to the center coordinates of the monitoring target image and the pixel coordinates of the monitoring target;

[0035] A third determination module, configured to, if the monitored target is not in the middle of the monitored target image, determine the pitch angle, horizontal angle to be adjusted by the drone gimbal, and the focal length to be adjusted by the image acquisition device according to the relative position vector;

[0036] An adjustment module, configured to: adjust the drone gimbal angle and the shooting focal length of the image acquisition device according to the pitch angle, horizontal angle to be adjusted by the drone gimbal, and the focal length to be adjusted by the image acquisition device.

[0037] In a possible implementation manner, the relative position vector includes a lateral position vector, a front-back position vector, and a vertical position vector;

[0038] In a possible implementation manner, the third determination module is specifically configured to:

[0039] Calculate the horizontal distance and the relative distance between the monitored target and the drone gimbal according to the lateral position vector, the front-back position vector, and the vertical position vector;

[0040] Determine the pitch angle to be adjusted by the drone gimbal by using the arctangent function according to the horizontal distance and the vertical position vector;

[0041] Determine the horizontal angle to be adjusted by the drone gimbal by using the arctangent function according to the lateral position vector and the front-back position vector;

[0042] Determine the focal length to be adjusted by the image acquisition device according to the relative distance and a preset imaging scale coefficient.

[0043] In a possible implementation manner, the first determination module is specifically configured to:

[0044] Determine the center coordinates of the monitored target image according to the pixels of the monitored target image;

[0045] Determine the pixel coordinates of the monitored target pixels according to the monitored target image by using a target detection algorithm;

[0046] Project the monitored target pixels into the global coordinate system according to a preset rotation matrix, a preset translation vector, a preset intrinsic parameter matrix, and the pixel coordinates, and obtain the three-dimensional coordinates of the monitored target;

[0047] Determine the relative position vector of the monitored target with respect to the drone gimbal according to the three-dimensional coordinates of the monitored target and the three-dimensional coordinates of the drone gimbal.

[0048] In a possible implementation manner, the first determination module is specifically configured to:

[0049] Obtain the three-dimensional coordinates of the drone gimbal according to the sensors of the drone gimbal;

[0050] Perform a difference operation on the three-dimensional coordinates of the monitoring target and the three-dimensional coordinates of the drone gimbal to determine the relative position vector of the monitoring target relative to the drone gimbal.

[0051] In a possible implementation manner, the second determination module is specifically configured to:

[0052] Perform a difference operation on the central coordinates of the monitoring target image and the pixel coordinates of the monitoring target to obtain the deviation coordinates of the monitoring target and the monitoring target image;

[0053] When both the abscissa value and the ordinate value corresponding to the deviation coordinates are less than the preset coordinate value, it is determined that the monitoring target is in the middle of the monitoring target image.

[0054] In a possible implementation manner, the drone adaptive adjustment device further includes a processing module, configured to:

[0055] If the monitoring target is in the middle of the monitoring target image, determine the focal length to be adjusted by the image acquisition device according to the relative position vector.

[0056] In a possible implementation manner, after it is determined that the monitoring target is not in the middle of the monitoring target image, the processing module is further configured to:

[0057] Output an error signal, where the error signal is used to instruct the drone gimbal to adjust the gimbal angle.

[0058] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0059] The memory stores computer execution instructions;

[0060] The processor executes the computer execution instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementation manners of the first aspect.

[0061] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored, and when the computer execution instructions are executed by a processor, they are used to implement the above first aspect and / or various possible implementation manners of the first aspect.

[0062] The method, device, equipment and medium for adaptive adjustment of an unmanned aerial vehicle (UAV) provided by an embodiment of the present application first determine the central coordinates of a monitored target image, the pixel coordinates of the monitored target, and the relative position vector of the monitored target relative to the UAV gimbal according to the monitored target image collected by the image acquisition device of the UAV gimbal; then, determine whether the monitored target is in the middle of the monitored target image according to the central coordinates of the monitored target image and the pixel coordinates of the monitored target; then, if the monitored target is not in the middle of the monitored target image, determine the pitch angle, horizontal angle to be adjusted by the UAV gimbal, and the focal length to be adjusted by the image acquisition device according to the relative position vector. Thus, by determining the pitch angle and horizontal angle to be adjusted by the UAV gimbal according to the relative position vector, it is ensured that the monitored target is always in the central area of the image, and the focal length of the image acquisition device is dynamically adjusted according to the target distance to ensure that the target still has good clarity at a long distance, avoiding image blurring or detail loss caused by a fixed focal length, and improving the monitoring quality and the accuracy of subsequent image recognition algorithms; finally, adjust the angle of the UAV gimbal and the shooting focal length of the image acquisition device according to the pitch angle, horizontal angle to be adjusted by the UAV gimbal, and the focal length to be adjusted by the image acquisition device. By this method, manual intervention is reduced, and the stability of the monitoring screen and the quality of the image composition are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with the present application and, together with the specification, are used to explain the principles of the present application.

[0064] Figure 1 is a schematic flow chart of the method for adaptive adjustment of an unmanned aerial vehicle provided by an embodiment of the present application Figure 1 ;

[0065] Figure 2 is a schematic flow chart of the method for adaptive adjustment of an unmanned aerial vehicle provided by an embodiment of the present application Figure 2 ;

[0066] Figure 3 is a schematic structural diagram of the device for adaptive adjustment of an unmanned aerial vehicle provided by an embodiment of the present application;

[0067] Figure 4 is a schematic structural diagram of the electronic device provided by an embodiment of the present application.

[0068] Through the above-mentioned accompanying drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and the textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0070] With the continuous expansion of the power grid scale and the continuous growth of the number of transmission lines, the traditional manual inspection method has been difficult to meet the requirements of efficient, accurate, and frequent inspections of high-voltage transmission lines. In recent years, unmanned aerial vehicles (UAVs) have been gradually widely used in the daily inspection and fault detection of high-voltage transmission lines because they can be flexible and mobile in complex environments and provide perspectives and coverage that cannot be achieved by traditional monitoring devices. However, during the flight of the UAV, due to the continuous change of its relative position with the ground monitoring target, the fixed gimbal of the UAV is likely to cause problems such as the target deviating from the center of the monitoring screen, video jitter, and image blurring, seriously affecting the quality of video data. Therefore, it is very important to improve the flexibility of the UAV with respect to the monitoring target and the quality of the monitoring screen.

[0071] In the prior art, the method of adjusting the UAV monitoring target usually relies on ground operators to manually control the flight attitude of the UAV and the orientation of the gimbal camera through a remote controller or a ground station system. Specifically, first, the operator manually identifies the monitoring target of interest through the monitoring screen; then, according to the position of the target, the operator manually operates the UAV to move and turn, making its flight position as close as possible to the target area; finally, the operator manually rotates the gimbal or adjusts the camera angle to keep the target in the center of the screen and ensure a better perspective.

[0072] However, the method of adjusting the UAV monitoring target in the prior art has a high labor cost. The operator needs to concentrate for a long time, and the manual adjustment has a slow reaction and a high fatigue level. Moreover, when the target moves quickly or the environment is complex, it is difficult to accurately and timely keep the target in the center of the screen.

[0073] Based on this, the present application proposes a method for adaptive adjustment of an unmanned aerial vehicle (UAV). Since traditional UAV gimbals generally adjust the direction manually, it is easy for the monitored target to deviate from the center of the screen during movement. Especially when the UAV itself is in motion, it is difficult to quickly and accurately keep the target centered by traditional methods that rely on manual operation or fixed control logic. That is, the traditional image capture method is more of a passive response. However, if the position offset of the target in the image and the relative spatial position vector can be used as the basis for active adjustment, an effect similar to "autofocus" can be achieved. Further, when the monitored target is at a long distance, if the focal length of the gimbal fails to be adjusted in time, the image will be blurred, affecting the accuracy of subsequent image recognition algorithms. Therefore, dynamically calculating the focal length can ensure that the image still has good clarity during long-distance monitoring.

[0074] The following uses specific embodiments to elaborate in detail on the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0075] Figure 1 Schematic flow of the method for adaptive adjustment of the UAV provided in the embodiments of the present application Figure 1 ; As Figure 1 shown, the method includes:

[0076] S101. Determine the center coordinates of the monitored target image, the pixel coordinates of the monitored target, and the relative position vector of the monitored target with respect to the UAV gimbal according to the monitored target image collected by the image acquisition device of the UAV.

[0077] Among them, the image acquisition device can generally be a camera, and can also include other sensor data such as lidar and infrared sensors. These sensors can provide supplementary information when the lighting conditions are poor or the target is partially blocked, so as to improve the target detection accuracy in complex environments; the relative position vector includes a horizontal position vector, a front-back position vector, and a vertical position vector.

[0078] It should be noted that after the monitoring target image is captured by the image acquisition device, it is necessary to use the target detection algorithm to determine the pixel coordinates of the monitoring target. The target detection algorithm includes deep learning models, such as the You Only Look Once (YOLO) algorithm, the Single Shot MultiBoxDetector (SSD) algorithm, and the Region-Based Convolutional Neural Network (Faster R-CNN). These algorithms can process the video stream captured by the UAV camera in real time, identify and locate the position of the target operator in the image. For example, the video stream captured by the camera is converted into static images at a set frequency (such as 5 frames per second); then, the image is scaled to the fixed size required by the model (such as 640×640 for YOLOv5), and the pixel values are normalized; finally, the deep learning model is used for inference to output the pixel coordinates of the monitoring target. The specific methods for determining the center coordinates of the monitoring target image, the pixel coordinates of the monitoring target, and the relative position vector of the monitoring target relative to the UAV gimbal are described in detail through Figure 2 the embodiments, and the embodiments of the present application will not be specifically described here.

[0079] It can be understood that accurately positioning the target in the image is achieved by determining the pixel coordinates of the monitoring target; the center coordinates of the monitoring target image can provide a quantitative basis for judging whether the monitoring target deviates from the center; finally, accurate mapping between the image and the spatial position is achieved by determining the relative position vector, which also provides accurate input parameters for subsequent gimbal angle control.

[0080] S102. Determine whether the monitoring target is in the middle of the monitoring target image according to the center coordinates of the monitoring target image and the pixel coordinates of the monitoring target.

[0081] In an implementable manner, first, the difference between the center coordinates of the monitoring target image and the pixel coordinates of the monitoring target is calculated to obtain the deviation coordinates of the monitoring target from the monitoring target image; then, when both the abscissa value and the ordinate value corresponding to the deviation coordinates are less than the preset coordinate value, it is determined that the monitoring target is in the middle of the monitoring target image.

[0082] For example, assume that (u c , v c ) are the center coordinates of the monitoring target image, (u t , v t ) are the pixel coordinates of the monitoring target, and the preset tolerance coordinate value is ; By subtracting the central coordinates of the monitored target image from the pixel coordinates of the monitored target, the deviation coordinates between the monitored target and the monitored target image are obtained as follows:

[0083]

[0084] Among them, Δu represents the deviation of the monitored target in the horizontal direction on the image plane; Δv represents the deviation of the monitored target in the vertical direction on the image plane.

[0085] When Δu < , and Δv < , it can be determined that the monitored target is in the middle of the monitored target image, and there is no need to control the pan-tilt to rotate.

[0086] It can be understood that in this way, it can be quickly determined whether the target is at the center of the camera's field of view, improving the response speed of the control system; at the same time, over-control is avoided, and no redundant adjustment is required when the monitored target is already at the center, reducing the computational complexity and energy consumption.

[0087] S103. If the monitored target is not in the middle of the monitored target image, then determine the pitch angle, horizontal angle to be adjusted by the UAV pan-tilt and the focal length to be adjusted by the image acquisition device according to the relative position vector.

[0088] In an implementable way, first, calculate the horizontal distance and relative distance between the monitored target and the UAV pan-tilt according to the lateral position vector, front-back position vector and vertical position vector; then, use the arctangent function to determine the pitch angle to be adjusted by the UAV pan-tilt according to the horizontal distance and the vertical position vector; then, use the arctangent function to determine the horizontal angle to be adjusted by the UAV pan-tilt according to the lateral position vector and the front-back position vector; finally, determine the focal length to be adjusted by the UAV image acquisition device according to the relative distance and the preset imaging scale factor.

[0089] It should be understood that if the three-dimensional coordinates of the monitored target are Pp=(X P ,Y P ,Z P ), and the three-dimensional coordinates of the UAV pan-tilt are Pu=(X u ,Y u ,Z u ), then the horizontal distance and relative distance between the monitored target and the UAV pan-tilt can be calculated as follows:

[0090]

[0091]

[0092] Among them, d h represents the horizontal distance between the monitored target and the UAV pan-tilt, Xp , Y p is represented as the horizontal position vector and the front-back position vector in the relative position vector; D h is represented as the relative distance between the monitoring target and the drone gimbal; where can be understood as the relative displacement in the east-west direction; can be understood as the relative displacement in the north-south direction; can be understood as the relative displacement in the height direction.

[0093] Furthermore, by according to the horizontal distance and the vertical position vector, using the arctangent function to determine the pitch angle of the drone gimbal to be adjusted, that is:

[0094]

[0095] where is the relative displacement in the height direction; D h is represented as the horizontal distance between the monitoring target and the drone gimbal; is the pitch angle of the drone gimbal to be adjusted.

[0096] The horizontal angle of the drone gimbal to be adjusted can be determined by using the arctangent function according to the horizontal position vector and the front-back position vector, that is:

[0097]

[0098] where is the horizontal angle of the drone gimbal to be adjusted; can be understood as the relative displacement in the east-west direction; can be understood as the relative displacement in the north-south direction.

[0099] Finally, according to the relative distance and the preset imaging scale coefficient, determine the focal length f of the drone image acquisition device to be adjusted, that is:

[0100] f = k * D h

[0101] where k is the preset imaging scale coefficient; D h is the relative distance.

[0102] It can be understood that in order to keep the target size appropriate at different distances, the drone gimbal can dynamically adjust the focal length f according to the relative distance of the monitoring target, keep the monitoring target clear and try to keep it in a suitable picture ratio. The focal length adjustment strategy can be based on the following criteria:

[0103] Target distance is far → Increase the focal length (magnify the target)

[0104] Target distance is near → Decrease the focal length (increase the field of view)

[0105] S104. Adjust the gimbal angle of the drone and the shooting focal length of the image acquisition device according to the pitch angle, horizontal angle to be adjusted of the drone gimbal and the focal length to be adjusted of the image acquisition device.

[0106] It can be understood that by adjusting the gimbal attitude and the imaging parameters of the image acquisition device in real time according to the pitch angle, horizontal angle to be adjusted of the drone gimbal and the focal length to be adjusted of the image acquisition device, it can ensure that the monitoring target is always located at the center of the image and maintain clear imaging, thereby improving the stability and accuracy of target tracking and realizing the efficient and continuous monitoring of the target by the drone in a dynamic environment.

[0107] The drone adaptive adjustment method provided by the embodiment of the present application first determines the center coordinates of the monitoring target image, the pixel coordinates of the monitoring target and the relative position vector of the monitoring target relative to the drone gimbal according to the monitoring target image collected by the image acquisition device of the drone gimbal; then, determines whether the monitoring target is in the middle of the monitoring target image according to the center coordinates of the monitoring target image and the pixel coordinates of the monitoring target; then, if the monitoring target is not in the middle of the monitoring target image, determines the pitch angle, horizontal angle to be adjusted of the drone gimbal and the focal length to be adjusted of the image acquisition device according to the relative position vector. At this point, by determining the pitch angle and horizontal angle to be adjusted of the drone gimbal according to the relative position vector, it ensures that the monitoring target is always in the central area of the image, and dynamically adjusts the focal length of the image acquisition device according to the target distance to ensure that the target still has good clarity at a long distance, avoiding image blurring or detail loss caused by a fixed focal length, and improving the monitoring quality and the accuracy of the subsequent image recognition algorithm; finally, adjusts the gimbal angle of the drone and the shooting focal length of the image acquisition device according to the pitch angle, horizontal angle to be adjusted of the drone gimbal and the focal length to be adjusted of the image acquisition device. By this method, manual intervention is reduced, and the stability of the monitoring screen and the quality of the image composition are significantly improved.

[0108] In an implementable manner, if it is determined that the monitoring target is in the middle of the monitoring target image, the focal length to be adjusted of the image acquisition device is determined according to the relative position vector.

[0109] It should be understood that if it is determined that the monitoring target is in the middle of the monitoring target image, the pitch angle and the horizontal angle are no longer adjusted, and only the focal length is adjusted according to the current relative position vector to keep the target in the clearest imaging all the time. By this means, the computational complexity is reduced, the unnecessary rotation of the drone gimbal is avoided, and the monitoring screen can be kept stable through the focal length adjustment of the image acquisition device, thereby improving the image processing efficiency.

[0110] Optionally, if the monitored target is not in the middle of the monitored target image, an error signal can be output, and the error signal is used to instruct the drone gimbal to determine the pitch angle, horizontal angle to be adjusted, and the focal length of the image acquisition device to adjust the drone gimbal and the shooting focal length of the image acquisition device.

[0111] It can be understood that after determining that the monitored target is not in the middle of the monitored target image according to the center coordinates of the monitored target image and the pixel coordinates of the monitored target, by outputting an error signal, the error signal is sent to a controller to generate a gimbal angle adjustment amount and an image acquisition device focal length adjustment amount to ensure that the monitored target is at the center of the image and ensure that the target image is always clear.

[0112] Figure 2 Flow schematic of the drone adaptive adjustment method provided by the embodiment of the present application Figure 2 , such as Figure 2 shown, on the basis of the Figure 1 embodiment, how to determine the center coordinates of the monitored target image, the pixel coordinates of the monitored target, and the relative position vector of the monitored target relative to the drone gimbal is described in detail. The method includes:

[0113] S201. Determine the center coordinates of the monitored target image according to the pixels of the monitored target image.

[0114] For example, if the pixels of the monitored target image are 1920×1080 pixels, the center coordinates of the monitored target image are:

[0115]

[0116] It should be understood that by calculating the center coordinates of the monitored target image, a fixed reference can be provided for subsequent judgment of whether the monitored target deviates from the center of the screen.

[0117] S202. Determine the pixel coordinates of the monitored target pixels according to the monitored target image by using a target detection algorithm.

[0118] It should be understood that according to the monitored target image, using a target detection algorithm (such as YOLOv8) to identify the object in the monitored image, and then the pixel coordinates of the target in the image can be obtained, such as (u, v).

[0119] It should be noted that in a monocular system, since there is only one camera, the complete three-dimensional coordinates cannot be directly obtained, and only the straight-line distance d of the monitored target in front of the image acquisition device can be estimated. The specific calculation method is:

[0120]

[0121] Among them, d represents the actual distance between the camera and the target (unit: meter); H represents the actual height of the target (such as the height of an operator, unit: meter); f represents the focal length of the camera (unit: pixel or millimeter); h represents the pixel height of the target in the vertical direction in the image (unit: pixel).

[0122] Furthermore, in order to convert this distance into three-dimensional coordinates, it is necessary to introduce the pixel coordinates in the target image and the camera pose (position and orientation) information, namely a preset rotation matrix, a preset translation vector, and a preset intrinsic matrix.

[0123] S203. Project the monitored target pixel into the global coordinate system according to the preset rotation matrix, preset translation vector, preset intrinsic matrix, and pixel coordinates, and obtain the three-dimensional coordinates of the monitored target.

[0124] For the convenience of better understanding the specific process of determining the three-dimensional coordinates of the monitored target, the embodiments of the present application are described in detail by way of examples.

[0125] Suppose the pixel coordinates are (u, v), the intrinsic matrix of the image acquisition device is K, the rotation matrix is R, the translation vector is t, the straight-line distance of the monitored target in front of the image acquisition device is d, and K is:

[0126]

[0127] Among them, (f x , f y ) is the focal length coordinate, (c x , c y ) is the principal point coordinate, and K is the intrinsic matrix.

[0128] Then the coordinates of the monitored target in the camera coordinate system can be calculated as:

[0129]

[0130] Among them, (x c , y c , z c ) represents the position of the monitored target in the camera coordinate system; d is the straight-line distance of the monitored target in front of the image acquisition device; (u, v) are the pixel coordinates; (f x , f y ) is the focal length coordinate; (c x , c y ) is the principal point coordinate.

[0131] Furthermore, through coordinate conversion, the monitored target pixel can be projected into the global coordinate system, and the three-dimensional coordinates of the monitored target can be obtained:

[0132]

[0133] wherein, R is a rotation matrix; t is a translation vector; (X P , Y P , Z P ) represents the three-dimensional coordinates of the monitoring target; (x c , y c , z c ) represents the position of the monitoring target in the camera coordinate system.

[0134] It should also be noted that if a binocular camera is used in the image acquisition device, that is, equipped with two cameras, a left camera and a right camera (the above example uses a monocular camera for illustration), the three-dimensional coordinates of the monitoring target can be determined through image parallax.

[0135] Specifically, first determine the position of the monitoring target in the camera coordinate system, that is:

[0136] ; ;

[0137] wherein, Z represents the distance between the target and the camera plane (front-back direction); X, Y represent the left-right and up-down positions of the target relative to the camera; f represents the camera focal length; B represents the baseline distance between the binocular cameras; D represents the parallax, that is, the difference in the target pixel positions in the left and right images ; x L , x R respectively represent the pixel abscissas of the target in the left and right images; y L , y R respectively represent the pixel ordinates of the target in the left and right images; W, H respectively represent the width and height of the image.

[0138] Furthermore, coordinate conversion is also required, that is, three-dimensional coordinate conversion is performed according to the internal parameter matrix, rotation matrix, and translation vector in the above example. The specific conversion formula is not elaborated in this embodiment of the present application.

[0139] S204. Determine the relative position vector of the monitoring target relative to the drone gimbal according to the three-dimensional coordinates of the monitoring target and the three-dimensional coordinates of the drone gimbal.

[0140] It should be noted that the position information of the drone gimbal is usually provided by sensors such as the Global Navigation Satellite System (GPS) and the Inertial Measurement Unit (IMU). The three-dimensional position after fusing GPS + IMU is expressed as: Pu = (X u , Y u , Z u ); wherein, Xu , Y u , Z u respectively represent the three-dimensional coordinates of the UAV in the global coordinate system.

[0141] In an implementable manner, first, according to the sensors of the UAV gimbal, obtain the three-dimensional coordinates of the UAV gimbal; then, perform a difference operation on the three-dimensional coordinates of the monitoring target and the three-dimensional coordinates of the UAV gimbal to determine the relative position vector of the monitoring target relative to the UAV gimbal.

[0142] It can be understood that from the above embodiments, the three-dimensional coordinates of the monitoring target Pp = (X P , Y P , Z P ), the three-dimensional coordinates of the UAV gimbal Pu = (X u , Y u , Z u ). By taking the difference, the relative position vector of the monitoring target relative to the UAV gimbal can be obtained, that is , and this relative position vector describes the direction and distance (spatial position offset) of the monitoring target relative to the UAV gimbal in three-dimensional space. Among them, can be understood as the relative displacement in the east-west direction; can be understood as the relative displacement in the north-south direction; can be understood as the relative displacement in the height direction.

[0143] It can be understood that determining the relative position vector provides a basis for calculating the angles for subsequent control of the gimbal rotation (pitch angle, horizontal angle), ensuring that the gimbal lens always faces the target.

[0144] Figure 3 is a schematic structural diagram of the UAV adaptive adjustment device provided by the embodiment of the present application. As Figure 3 shown, the device includes:

[0145] The first determination module 301 is used to determine the center coordinates of the monitoring target image, the pixel coordinates of the monitoring target, and the relative position vector of the monitoring target relative to the UAV gimbal according to the monitoring target image collected by the image acquisition device of the UAV;

[0146] The second determination module 302 is used to determine whether the monitoring target is in the middle of the monitoring target image according to the center coordinates of the monitoring target image and the pixel coordinates of the monitoring target;

[0147] The third determination module 303 is used to, if the monitoring target is not in the middle of the monitoring target image, determine the pitch angle, horizontal angle to be adjusted by the UAV gimbal, and the focal length to be adjusted by the image acquisition device according to the relative position vector;

[0148] An adjustment module 304 is configured to: adjust the angle of the drone gimbal and the shooting focal length of the image acquisition device according to the pitch angle, horizontal angle to be adjusted of the drone gimbal, and the focal length to be adjusted of the image acquisition device.

[0149] In a possible implementation, the relative position vector includes a lateral position vector, a front-back position vector, and a vertical position vector;

[0150] In a possible implementation, the third determination module 303 is specifically configured to:

[0151] Calculate the horizontal distance and the relative distance between the monitoring target and the drone gimbal according to the lateral position vector, the front-back position vector, and the vertical position vector;

[0152] Determine the pitch angle to be adjusted of the drone gimbal by using the arctangent function according to the horizontal distance and the vertical position vector;

[0153] Determine the horizontal angle to be adjusted of the drone gimbal by using the arctangent function according to the lateral position vector and the front-back position vector;

[0154] Determine the focal length to be adjusted of the image acquisition device according to the relative distance and the preset imaging scale coefficient.

[0155] In a possible implementation, the first determination module 301 is specifically configured to:

[0156] Determine the center coordinates of the monitoring target image according to the pixels of the monitoring target image;

[0157] Determine the pixel coordinates of the monitoring target pixels according to the monitoring target image by using the target detection algorithm;

[0158] Project the monitoring target pixels into the global coordinate system according to the preset rotation matrix, preset translation vector, preset internal parameter matrix, and pixel coordinates to obtain the three-dimensional coordinates of the monitoring target;

[0159] Determine the relative position vector of the monitoring target relative to the drone gimbal according to the three-dimensional coordinates of the monitoring target and the three-dimensional coordinates of the drone gimbal.

[0160] In a possible implementation, the first determination module 301 is specifically configured to:

[0161] Obtain the three-dimensional coordinates of the drone gimbal according to the sensors of the drone gimbal;

[0162] Perform a subtraction operation on the three-dimensional coordinates of the monitoring target and the three-dimensional coordinates of the drone gimbal to determine the relative position vector of the monitoring target relative to the drone gimbal.

[0163] In a possible implementation, the second determination module 302 is specifically configured to:

[0164] Subtract the central coordinates of the monitored target image from the pixel coordinates of the monitored target to obtain the deviation coordinates of the monitored target and the monitored target image;

[0165] When both the abscissa value and the ordinate value corresponding to the deviation coordinates are less than the preset coordinate values, it is determined that the monitored target is in the middle of the monitored target image.

[0166] In a possible implementation, the UAV adaptive adjustment device further includes a processing module for:

[0167] If the monitored target is in the middle of the monitored target image, determine the focal length to be adjusted by the image acquisition device according to the relative position vector.

[0168] In a possible implementation, after it is determined that the monitored target is not in the middle of the monitored target image, the processing module is further configured to:

[0169] Output an error signal, which is used to instruct the UAV gimbal to adjust the gimbal angle.

[0170] The UAV adaptive adjustment device provided in the embodiments of the present application can execute the method provided in the above method embodiments, and its implementation principle and technical effects are similar, which will not be elaborated here in this embodiment.

[0171] Figure 4 This is a schematic structural diagram of the electronic device provided in the embodiments of the present application. As Figure 4 shown, the electronic device 40 provided in this embodiment includes: at least one processor 401 and a memory 402. Optionally, the device 40 further includes a communication component 403. Among them, the processor 401, the memory 402, and the communication component 403 are connected through a bus 404.

[0172] In a specific implementation process, at least one processor 401 executes the computer execution instructions stored in the memory 402, so that at least one processor 401 executes the above method.

[0173] The specific implementation process of the processor 401 can be referred to the above method embodiments, and its implementation principle and technical effects are similar, which will not be elaborated here in this embodiment.

[0174] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU for short), or other general-purpose processors, digital signal processors (DSP for short), application specific integrated circuits (ASIC for short), etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly implemented by the execution of the hardware processor, or by the combination of hardware and software modules in the processor.

[0175] The memory may include a high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk memory.

[0176] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0177] This application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the processor executes the computer-executable instructions, the above method is implemented.

[0178] The above-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk or an optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0179] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in a device.

[0180] The division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Additionally, the couplings or direct couplings or communication connections shown or discussed between each other can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0181] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] Furthermore, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0183] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0184] Those of ordinary skill in the art will understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disk, or optical disk that can store program codes.

[0185] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention, which follow the general principles of the present invention and include common general knowledge or conventional technical means in the technical field not disclosed by the present invention. It is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. An adaptive adjustment method for an unmanned aerial vehicle, characterized in that Including: Determine the central coordinates of the monitored target image, the pixel coordinates of the monitored target, and the relative position vector of the monitored target with respect to the drone gimbal based on the monitored target image collected by the image acquisition device of the drone; Determine whether the monitored target is in the middle of the monitored target image according to the central coordinates of the monitored target image and the pixel coordinates of the monitored target; If the monitored target is not in the middle of the monitored target image, determine the pitch angle, horizontal angle to be adjusted for the drone gimbal, and the focal length to be adjusted for the image acquisition device according to the relative position vector; Adjust the gimbal angle of the drone and the shooting focal length of the image acquisition device according to the pitch angle, horizontal angle to be adjusted for the drone gimbal, and the focal length to be adjusted for the image acquisition device.

2. The method according to claim 1, wherein The relative position vector includes a lateral position vector, a front-back position vector, and a vertical position vector; The determining the pitch angle, horizontal angle to be adjusted for the drone gimbal, and the focal length to be adjusted for the image acquisition device according to the relative position vector includes: Calculate the horizontal distance and relative distance between the monitored target and the drone gimbal according to the lateral position vector, the front-back position vector, and the vertical position vector; Determine the pitch angle to be adjusted for the drone gimbal using the arctangent function according to the horizontal distance and the vertical position vector; Determine the horizontal angle to be adjusted for the drone gimbal using the arctangent function according to the lateral position vector and the front-back position vector; Determine the focal length to be adjusted for the image acquisition device according to the relative distance and a preset imaging scale factor.

3. The method according to claim 1, wherein The determining the central coordinates of the monitored target image, the pixel coordinates of the monitored target, and the relative position vector of the monitored target with respect to the drone gimbal based on the monitored target image collected by the image acquisition device of the drone includes: Determine the central coordinates of the monitored target image according to the pixels of the monitored target image; Determine the pixel coordinates of the monitored target pixels using a target detection algorithm according to the monitored target image; Project the monitored target pixels into the global coordinate system according to a preset rotation matrix, a preset translation vector, a preset intrinsic matrix, and the pixel coordinates to obtain the three-dimensional coordinates of the monitored target; Determine the relative position vector of the monitored target with respect to the drone gimbal according to the three-dimensional coordinates of the monitored target and the three-dimensional coordinates of the drone gimbal.

4. The method according to claim 3, wherein The determining the relative position vector of the monitored target with respect to the drone gimbal according to the three-dimensional coordinates of the monitored target and the three-dimensional coordinates of the drone gimbal includes: Obtain the three-dimensional coordinates of the drone gimbal according to the sensors of the drone gimbal; Perform a subtraction operation on the three-dimensional coordinates of the monitored target and the three-dimensional coordinates of the drone gimbal to determine the relative position vector of the monitored target with respect to the drone gimbal.

5. The method according to claim 1 or 3, characterized in that The determining whether the monitored target is in the middle of the monitored target image according to the central coordinates of the monitored target image and the pixel coordinates of the monitored target includes: Perform a difference operation on the central coordinates of the monitored target image and the pixel coordinates of the monitored target to obtain the deviation coordinates of the monitored target and the monitored target image; When both the abscissa value and the ordinate value corresponding to the deviation coordinates are less than the preset coordinate value, it is determined that the monitored target is in the middle of the monitored target image.

6. The method according to claim 1 or 2, characterized in that, The method further includes: If the monitored target is in the middle of the monitored target image, determine the focal length to be adjusted by the image acquisition device according to the relative position vector.

7. The method according to claim 1, characterized in that After it is determined that the monitored target is not in the middle of the monitored target image, the method further includes: Output an error signal, where the error signal is used to instruct the drone gimbal to adjust the gimbal angle.

8. An adaptive adjustment device for an unmanned aerial vehicle, characterized in that, It includes: A first determination module, configured to determine the central coordinates of the monitored target image, the pixel coordinates of the monitored target, and the relative position vector of the monitored target with respect to the drone gimbal according to the monitored target image collected by the image acquisition device of the drone; A second determination module, configured to determine whether the monitored target is in the middle of the monitored target image according to the central coordinates of the monitored target image and the pixel coordinates of the monitored target; A third determination module, configured to, if the monitored target is not in the middle of the monitored target image, determine the pitch angle, horizontal angle to be adjusted by the drone gimbal, and the focal length to be adjusted by the image acquisition device according to the relative position vector; An adjustment module, configured to: adjust the drone gimbal angle and the shooting focal length of the image acquisition device according to the pitch angle, horizontal angle to be adjusted by the drone gimbal, and the focal length to be adjusted by the image acquisition device.

9. An electronic device, characterized in that, It includes: A memory, a processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by the processor, they are used to implement the method according to any one of claims 1-7.

Images