A method for locating targets based on real-time position of unmanned aerial vehicle and linkage with camera

Abstract

The present invention provides a method for locating a target based on the real-time position of a drone and a camera, comprising the following steps: connecting a fixed-point camera on the drone route to a tower video convergence system and scheduling it through a drone platform; during the flight of the drone, taking a video frame with the highest confidence in identifying the target position; preprocessing the image of the video frame and converting it into coordinates in a world coordinate system; the system determines the fixed-point camera to be linked based on the drone coordinates and calculates the PTZ value to adjust the camera. Based on the real-time position and configuration information of the drone, the present invention links a fixed-point camera that can cover the target position, calculates the PTZ value of the camera, and synchronizes the drone and the camera to perform multi-directional monitoring of the located target.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) target positioning, and in particular to a method for positioning a target based on a real-time position of an unmanned aerial vehicle (UAV) linked to a camera. Background Art

[0002] After a camera at a fixed point or a manual report of a certain latitude and longitude sends an event, it is necessary to manually select nearby drones and cameras for multiple and manual operations, which wastes precious disposal and verification time in the cumbersome operation process, which is time-consuming and laborious. In addition, the visual range of a single fixed-point camera is single, and it is impossible to continuously detect the target in all directions, and the accuracy of the monitoring results obtained cannot be guaranteed.

[0003] In the existing technology, multiple drones are often used for coordinated monitoring. Each drone needs to stay on the same horizontal plane around the target to achieve multi-directional monitoring of the target. However, the endurance of drones is a great challenge in the observation of continuous events. If multiple drones are used for relay observation or coordinated observation, the cost is greatly increased. In remote operations, the flexibility and timeliness of continuous monitoring of target events are still insufficient. Summary of the invention

[0004] The present invention is made in view of the above-mentioned problems, and its purpose is to provide a method for locating a target based on the real-time position of a drone and a camera. The method processes the target image video frame with the highest confidence in the video shot by the drone, and based on the real-time position and configuration information of the drone, a fixed-point camera that can cover the target position is linked to calculate the PTZ value of the camera, so as to synchronize the drone and the camera to perform multi-directional monitoring of the located target.

[0005] Specifically, a first aspect of the present invention provides a method for locating a target based on a real-time position of a drone and a camera, comprising the following steps:

[0006] Step 1: Connect the fixed point cameras on the drone route to the tower video convergence system, and dispatch the drone through the drone platform when the drone dispatch conditions are met;

[0007] The tower video convergence system is a system that uses communication towers as infrastructure and realizes centralized management and application of video data through video surveillance technology. It not only supports multi-protocol access, such as RTSP, RTMP, GB28181, etc., but also has flexible expansion capabilities.

[0008] Step 2: During the flight of the drone, take the video frame with the highest confidence in target position recognition and obtain the positioning data of the drone and the drone gimbal at this time;

[0009] Step 3: Preprocess the image of the video frame, convert the pixel coordinates of the event target position in the image into coordinates in the camera coordinate system, and then convert them into coordinates in the world coordinate system. The formula is as follows:

[0010] ;

[0011] ;

[0012] ;

[0013] ;

[0014] in: is the pixel coordinate of the target in the image;

[0015] is the focal length of the camera;

[0016] is the pixel coordinate of the center point of the image;

[0017] is the distance between the target and the camera;

[0018] H is the height of the drone;

[0019] ( , , ) is the coordinate of the target in the camera coordinate system;

[0020] ( , , ) is the coordinate of the target in the world coordinate system;

[0021] ( , , ) is the GPS coordinates of the drone;

[0022] is the rotation matrix;

[0023] is the yaw angle of the drone gimbal;

[0024] is the pitch angle of the drone gimbal;

[0025] is the roll angle of the drone gimbal;

[0026] The pixel coordinate system is a rectangular coordinate system established with the upper left corner of the image as the origin. The pixel coordinate system is used to describe the position of a single pixel in an image and is usually used in image processing.

[0027] The camera coordinate system is a three-dimensional rectangular coordinate system with the optical center of the camera as the origin and the optical axis as the Z axis. The X axis and Y axis are parallel to the imaging plane, and the Z axis points to the front of the camera (that is, perpendicular to the imaging plane). The camera coordinate system is used to describe the position of an object as seen from the camera's perspective.

[0028] The world coordinate system is a three-dimensional rectangular coordinate system, which is usually used to describe the position of any object in the environment. Its origin can be freely set as needed, for example, it can be located on the base of the robot or the front end actuator. The world coordinate system is the reference frame of other coordinate systems, which is used to describe the position of cameras and other objects in the real world. The origin of the world coordinate system in the present invention is the same as GPS positioning, which is the center of mass of the earth.

[0029] Step 4: The system determines the fixed-point camera to be linked based on whether the visual range of the fixed-point cameras around the drone coordinates covers the event target location;

[0030] The fixed-point cameras on the tower have a visible range of 3 to 10 kilometers due to differences in model and focal length, and need to be adjusted according to the influence of weather. In foggy days and other weather conditions that block vision, the visible range will be reduced to varying degrees, so the impact of weather needs to be taken into account when making judgments.

[0031] Step 5: Calculate the PTZ value of the fixed-point camera and adjust the camera according to the PTZ value to synchronize the fixed-point camera with the target area of ​​the drone. The calculation formula is as follows:

[0032] ;

[0033] ;

[0034] ;

[0035] Where: P is the horizontal azimuth value;

[0036] T is the pitch azimuth value;

[0037] Z is the focal length magnification value of the camera lens;

[0038] arccos is the inverse cosine function;

[0039] arctan is the inverse tangent function;

[0040] is the distance from the camera to the target point;

[0041] b is the distance from the coordinate point with P value 0 to the target point;

[0042] c is the distance from the camera to the coordinate point where P value is 0;

[0043] is the height of the camera location;

[0044] is the height of the tower where the camera is located;

[0045] is the target point height;

[0046] distance is the distance from the camera to the target point;

[0047] is the pixel width of the image;

[0048] is the maximum focal length of the camera;

[0049] Min is the minimum value function;

[0050] The camera focal length reaches the maximum value The maximum actual distance of the target point that the camera can clearly capture when

[0051] Step 6: Stitch the videos shot by the drone and the videos shot by the fixed-point camera to conduct multi-directional continuous monitoring of the target location.

[0052] The videos shot by drones and fixed-point cameras are stitched together. Specifically, software such as FFmpegJoiner is used to merge videos shot at different angles of the same target location to generate a panoramic monitoring video image of the target location for multi-directional continuous monitoring.

[0053] Furthermore, the conditions for meeting the drone dispatching conditions specifically include:

[0054] The target event is detected in the video sent back to the tower video aggregation system by the fixed-point camera. The server collects and analyzes the event and sends an alarm to the person in charge. The person in charge confirms that the event needs to be tracked and a drone is dispatched for verification.

[0055] Or when the drone inspection time comes, the drone will be dispatched according to the configured fixed route.

[0056] Furthermore, the method of taking the video frame with the highest confidence in target position identification includes: for each frame image, the predicted position of the coordinates is calculated by a coordinate transformation method, and then the confidence of the frame image is evaluated according to the difference between the actual position of the target and the predicted position. The smaller the difference, the higher the confidence. Finally, the confidence of different video frames is compared, and the video frame with the highest confidence is selected as the final result.

[0057] Furthermore, the acquisition of the positioning data of the drone and the drone gimbal at this time specifically includes:

[0058] Obtain the pitch angle, roll angle, and yaw angle parameters of the drone gimbal according to the inspection route;

[0059] The pitch angle, roll angle, and yaw angle parameters of the drone gimbal are important parameters that describe the drone's attitude. They are usually measured by sensors such as gyroscopes and accelerometers, and are obtained based on the drone's inspection route settings and the drone's attitude data from the drone control system.

[0060] The pitch angle refers to the angle between the coordinate axis of the drone body and the horizontal plane. The pitch angle can be measured by integrating the gyroscope or by measuring the gravity component with an accelerometer without its own acceleration.

[0061] The roll angle is the angle between the coordinate axis of the drone body and the plumb plane. Similarly, the roll angle can be obtained by integrating the gyroscope or by measuring the gravity component with an accelerometer.

[0062] The yaw angle refers to the angle at which the drone rotates around the vertical axis, which is usually calculated by measuring the angular velocity with a gyroscope and integrating it. In addition, the yaw angle can also be calculated by combining a magnetometer with an accelerometer and fusing the data of the gyroscope and magnetometer through a Kalman filter algorithm.

[0063] Obtain the camera’s focal length, sensor size, image resolution and other parameter data based on the drone model;

[0064] Focal length refers to the distance from the optical center of the lens to the imaging plane (such as a CCD or CMOS sensor). Focal length determines the camera's viewing angle and imaging range.

[0065] Sensor size refers to the physical size of the image sensor, which affects the camera's imaging quality and image detail.

[0066] Image resolution refers to the amount of image detail that a camera can capture, usually expressed in pixels. The higher the resolution, the more detail that can be accommodated.

[0067] Get the GPS coordinates and flight altitude of the drone based on the built-in positioning module of the drone.

[0068] Furthermore, the preprocessing includes denoising and grayscale enhancement of the image.

[0069] Median filtering is performed on the image to smooth the noise and reduce the impact of noise. Then, the grayscale value of the original image is linearly transformed to expand the grayscale range, thereby enhancing the contrast of the image.

[0070] Furthermore, when calculating the pitch azimuth value, the relative altitude between the tower position and the target point is assumed to be 0 meters, and the height difference only calculates the hanging height of the camera.

[0071] The hanging height of the camera refers to the height of the camera from the ground when it is installed. In the present invention, the relative altitude between the tower position and the target point is ignored. The camera is installed on the top of the tower. Because of the visual range of the fixed-point camera, the distance between the tower position and the target point will not be too far, the relative altitude difference is small, and the impact on the camera is small compared to the tower height, so it is ignored.

[0072] Further, the adjusting of the camera according to the PTZ value is specifically: adjusting the horizontal rotation angle of the camera according to the P value; adjusting the pitch angle of the camera according to the T value; and adjusting the focal length magnification value of the camera lens according to the Z value.

[0073] In a second aspect, the present invention further provides a computing device, which has the function of implementing the method described in the first aspect above. The beneficial effects can be found in the description of the first aspect, which will not be repeated here. The functions can be implemented by hardware, or by executing corresponding software through hardware. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the device includes an acquisition module, a training module, and optionally, a construction module. These modules can implement the functions of the training node in the method example of the first aspect above. Please refer to the detailed description in the method example for details, which will not be repeated here.

[0074] In a third aspect, the present invention further provides a computing device, which is used to implement the functions of the method described in the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The structure of the computing device includes a processor and a memory, and the memory is used to store instructions and / or data. The memory is coupled to the processor, and when the processor executes the program instructions stored in the memory, the function of the training node in the example of the first aspect above can be implemented. The structure of the computing device also includes a communication interface for communicating with other devices.

[0075] In a fourth aspect, the present invention further provides a computer-readable storage medium, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer executes the method in the above-mentioned first aspect and various possible designs of the first aspect.

[0076] In a fifth aspect, the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the above-mentioned first aspect and various possible designs of the first aspect.

[0077] In a sixth aspect, the present invention further provides a computing chip, which is connected to a memory, and is used to read and execute a software program stored in the memory, and to execute the methods in the above-mentioned first aspect and various possible implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present drawings or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present drawings. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0079] Figure 1 is a flow chart of the steps of the present invention;

[0080] Figure 2 It is a schematic diagram of the scheduling result of the present invention.

[0081] The purpose, features and advantages of this figure will be further described in conjunction with the embodiments with reference to the accompanying drawings. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0083] Obviously, the drawings described below are only some examples or embodiments of the present invention. For ordinary technicians in this field, the present invention can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed by the present invention, some changes in design, manufacturing or production based on the technical content disclosed by the present invention are just conventional technical means, and should not be understood as insufficient content disclosed by the present invention.

[0084] If not otherwise specified, all embodiments and optional embodiments of the present invention can be combined with each other to form a new technical solution.

[0085] Unless otherwise specified, all technical features and optional technical features of the present invention can be combined with each other to form a new technical solution.

[0086] If not otherwise specified, all steps of the present invention may be performed sequentially or randomly, preferably sequentially. For example, the method includes steps (a) and (b), which means that the method may include steps (a) and (b) performed sequentially, or may include steps (b) and (a) performed sequentially. For example, the method may further include step (c), which means that step (c) may be added to the method in any order, for example, the method may include steps (a), (b) and (c), or may include steps (a), (c) and (b), or may include steps (c), (a) and (b), etc.

[0087] If there is no special explanation, the "include" and "comprising" mentioned in the present invention represent open-ended or closed-ended expressions. For example, the "include" and "comprising" may represent that other components not listed may also be included or only the listed components may be included or only the listed components may be included.

[0088] If not specifically stated, in the present invention, the term "or" is inclusive. For example, the phrase "A or B" means "A, B, or both A and B". More specifically, any of the following conditions satisfies the condition "A or B": A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); or both A and B are true (or exist).

[0089] In order to better understand the solutions of the embodiments of the present invention, some relevant terms and concepts that may be involved in the embodiments of the present invention are first introduced below.

[0090] (1) The tower video aggregation system is a system that uses communication towers as infrastructure and uses video surveillance technology to achieve centralized management and application of video data. It not only supports multi-protocol access, such as RTSP, RTMP, GB28181, etc., but also has flexible expansion capabilities.

[0091] (2) Pixel coordinate system is a rectangular coordinate system established with the upper left corner of the image as the origin. The pixel coordinate system is used to describe the position of a single pixel in the image and is usually used in image processing.

[0092] (3) The camera coordinate system is a three-dimensional rectangular coordinate system with the optical center of the camera as the origin and the optical axis as the Z axis. The X-axis and Y-axis are parallel to the imaging plane, and the Z axis points to the front of the camera (i.e., perpendicular to the imaging plane). The camera coordinate system is used to describe the position of an object as seen from the camera's perspective.

[0093] (4) The world coordinate system is a three-dimensional rectangular coordinate system that is usually used to describe the position of any object in the environment. Its origin can be freely set as needed, for example, it can be located at the base of the robot or the front end effector. The world coordinate system is the reference frame of other coordinate systems and is used to describe the position of cameras and other objects in the real world.

[0094] (5) The PTZ value of a camera refers to the parameter values ​​of Pan, Tilt, and Zoom. These parameters are used to control the movement and functions of a PTZ (Pan-Tilt-Zoom) camera. For example, horizontal pan (Pan) usually refers to the rotation angle of the camera in the horizontal direction, vertical tilt (Tilt) refers to the pitch angle of the camera lens in the vertical direction, and zoom (Zoom) refers to the adjustment of the camera focal length.

[0095] In this embodiment, Figure 1 As shown, a method for locating a target based on the real-time position of a drone and a camera linkage includes the following steps:

[0096] Step 1: Connect the fixed point cameras on the drone route to the tower video convergence system, and dispatch the drone through the drone platform when the drone dispatch conditions are met;

[0097] The tower video convergence system is a system that uses communication towers as infrastructure and realizes centralized management and application of video data through video surveillance technology. It not only supports multi-protocol access, such as RTSP, RTMP, GB28181, etc., but also has flexible expansion capabilities.

[0098] Step 2: During the flight of the drone, take the video frame with the highest confidence in target position recognition and obtain the positioning data of the drone and the drone gimbal at this time;

[0099] Step 3: Preprocess the image of the video frame, convert the pixel coordinates of the event target position in the image into coordinates in the camera coordinate system, and then convert them into coordinates in the world coordinate system. The formula is as follows:

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] The pixel coordinate system is a rectangular coordinate system established with the upper left corner of the image as the origin. The pixel coordinate system is used to describe the position of a single pixel in an image and is usually used in image processing.

[0105] The camera coordinate system is a three-dimensional rectangular coordinate system with the optical center of the camera as the origin and the optical axis as the Z axis. The X axis and Y axis are parallel to the imaging plane, and the Z axis points to the front of the camera (that is, perpendicular to the imaging plane). The camera coordinate system is used to describe the position of an object as seen from the camera's perspective.

[0106] The world coordinate system is a three-dimensional rectangular coordinate system, which is usually used to describe the position of any object in the environment. Its origin can be freely set as needed, for example, it can be located on the base of the robot or the front end actuator. The world coordinate system is the reference frame of other coordinate systems, which is used to describe the position of cameras and other objects in the real world. The origin of the world coordinate system in the present invention is the same as GPS positioning, which is the center of mass of the earth.

[0107] Step 4: The system determines the fixed-point camera to be linked based on whether the visual range of the fixed-point cameras around the drone coordinates covers the event target location;

[0108] The fixed-point cameras on the tower have a visible range of 3 to 10 kilometers due to differences in model and focal length, and need to be adjusted according to the influence of weather. In foggy days and other weather conditions that block vision, the visible range will be reduced to varying degrees, so the impact of weather needs to be taken into account when making judgments.

[0109] Step 5: Calculate the PTZ value of the fixed-point camera and adjust the camera according to the PTZ value to synchronize the fixed-point camera with the target area of ​​the drone. The calculation formula is as follows:

[0110] ;

[0111] ;

[0112] ;

[0113] Step 6: Stitch the videos shot by the drone and the videos shot by the fixed-point camera to conduct multi-directional continuous monitoring of the target location.

[0114] The videos shot by drones and fixed-point cameras are stitched together. Specifically, software such as FFmpegJoiner is used to merge videos shot at different angles of the same target location to generate a panoramic monitoring video image of the target location for multi-directional continuous monitoring.

[0115] Furthermore, the conditions for meeting the drone dispatching conditions specifically include:

[0116] The target event is detected in the video sent back to the tower video aggregation system by the fixed-point camera. The server collects and analyzes the event and sends an alarm to the person in charge. The person in charge confirms that the event needs to be tracked and a drone is dispatched for verification.

[0117] Or when the drone inspection time comes, the drone will be dispatched according to the configured fixed route.

[0118] Furthermore, the method of taking the video frame with the highest confidence in target position identification includes: for each frame image, the predicted position of the coordinates is calculated by a coordinate transformation method, and then the confidence of the frame image is evaluated according to the difference between the actual position of the target and the predicted position. The smaller the difference, the higher the confidence. Finally, the confidence of different video frames is compared, and the video frame with the highest confidence is selected as the final result.

[0119] Furthermore, the acquisition of the positioning data of the drone and the drone gimbal at this time specifically includes:

[0120] Obtain the pitch angle, roll angle, and yaw angle parameters of the drone gimbal according to the inspection route;

[0121] A drone gimbal is a device used to stabilize a camera or sensor, usually mounted on a drone to ensure the stability of the image during filming. The gimbal uses built-in motors and sensors to sense and offset the vibration and shaking of the drone during flight, thereby reducing image jitter and improving filming quality.

[0122] Gimbals usually have three-axis stabilization functions, namely pitch, roll and yaw, which can accurately control the angle and position of the camera. This stability is especially important for aerial photography, because drones are easily affected by wind and shake during flight, and the gimbal helps the camera stay stable.

[0123] The pitch angle, roll angle, and yaw angle parameters of the drone gimbal are important parameters for describing the drone's attitude. They are usually measured by sensors such as gyroscopes and accelerometers, and are obtained based on the drone's inspection route settings and the drone's attitude data from the drone control system.

[0124] The pitch angle refers to the angle between the coordinate axis of the drone body and the horizontal plane. The pitch angle can be measured by integrating the gyroscope or by measuring the gravity component with an accelerometer without its own acceleration.

[0125] The roll angle is the angle between the coordinate axis of the drone body and the plumb plane. Similarly, the roll angle can be obtained by integrating the gyroscope or by measuring the gravity component with an accelerometer.

[0126] The yaw angle refers to the angle at which the drone rotates around the vertical axis, which is usually calculated by measuring the angular velocity with a gyroscope and integrating it. In addition, the yaw angle can also be calculated by combining a magnetometer with an accelerometer and fusing the data of the gyroscope and magnetometer through a Kalman filter algorithm.

[0127] Obtain the camera’s focal length, sensor size, image resolution and other parameter data based on the drone model;

[0128] Focal length refers to the distance from the optical center of the lens to the imaging plane (such as a CCD or CMOS sensor). Focal length determines the camera's viewing angle and imaging range.

[0129] Sensor size refers to the physical size of the image sensor, which affects the camera's imaging quality and image detail.

[0130] Image resolution refers to the amount of image detail that a camera can capture, usually expressed in pixels. The higher the resolution, the more detail that can be accommodated.

[0131] Get the GPS coordinates and flight altitude of the drone based on the built-in positioning module of the drone.

[0132] Furthermore, the preprocessing includes denoising and grayscale enhancement of the image.

[0133] Median filtering is performed on the image to smooth the noise and reduce the impact of noise. Then, the grayscale value of the original image is linearly transformed to expand the grayscale range, thereby enhancing the contrast of the image.

[0134] Furthermore, when calculating the pitch azimuth value, the relative altitude between the tower position and the target point is assumed to be 0 meters, and the height difference only calculates the hanging height of the camera.

[0135] The hanging height of the camera refers to the height of the camera from the ground when it is installed. In the present invention, the relative altitude between the tower position and the target point is ignored. The camera is installed on the top of the tower. Because of the visual range of the fixed-point camera, the distance between the tower position and the target point will not be too far, the relative altitude difference is small, and the impact on the camera is small compared to the tower height, so it is ignored.

[0136] Furthermore, the camera is adjusted according to the PTZ value, specifically: the horizontal rotation angle of the camera is adjusted according to the P value; the pitch angle of the camera is adjusted according to the T value; and the focal length magnification value of the camera lens is adjusted according to the Z value.

[0137] In this embodiment, the fixed positioning cameras around the section from Shi'ao Lijia to Xianziping on the Beijing-Hong Kong-Aomen Expressway in Chenzhou found a fire target event in the video returned to the tower video convergence system.

[0138] That is, the longitude of the target point is 113.085580, and the latitude is 25.806906. The longitude of the surrounding camera A is 113.10107, and the latitude is 25.78735. The brand is Dahua. The longitude of camera B is 113.101267, and the latitude is 25.815051. The brand is Dahua. According to the calculation formula, the following is obtained:

[0139] The P value of camera A is 324.5 and the P value of camera B is 240.

[0140] The T value of camera A is 1.1 and the T value of camera B is 1.0.

[0141] Camera A has a Z value of 10.0 and Camera B has a Z value of 7.0.

[0142] After obtaining the PTZ values ​​of camera A and camera B respectively, they are converted using the PTZ conversion parameters of the corresponding camera brands and then a PTZ control command is sent to the camera to turn the camera to the target event position to obtain a video stitching interface.

[0143] The schematic diagram of the scheduling result of the present invention is as follows Figure 2 As shown, Figure 2 The oval circle in the middle represents the target event location. After the fixed-point camera discovers the fire event, the server initiates an alarm push to the responsible person. The responsible person confirms that the fire event needs to be tracked continuously and dispatches a drone to verify. The drone flies from the drone platform closest to the camera and automatically generates a flight route with the target as the camera. After the fire target is discovered during the flight, the server calculates the fire target location according to the calculation method of the present invention, dispatches the fixed-point camera whose visual range covers the target location to form multi-directional monitoring of the fire target, and splices the videos returned by the drone and the fixed-point camera to perform panoramic monitoring.

[0144] It should be noted that the present invention is not limited to the above-mentioned embodiments. The above-mentioned embodiments are only examples, and the embodiments having the same structure as the technical idea and exerting the same effect within the scope of the technical solution of the present invention are all included in the technical scope of the present invention. In addition, without departing from the scope of the main purpose of the present invention, various modifications that can be thought of by those skilled in the art to the embodiments and other methods of combining some of the constituent elements in the embodiments are also included in the scope of the present invention.

Claims

1. A method for locating a target based on the real-time position of a drone and a camera, characterized in that: The following steps are involved: Step 1: Connect the fixed point cameras on the drone route to the tower video convergence system, and dispatch the drone through the drone platform when the drone dispatch conditions are met; Step 2: During the flight of the drone, take the video frame with the highest confidence in target position recognition and obtain the positioning data of the drone and the drone gimbal at this time; Step 3: Preprocess the image of the video frame, convert the pixel coordinates of the event target position in the image into coordinates in the camera coordinate system, and then convert them into coordinates in the world coordinate system. The formula is as follows: ； ； ； ； in: is the pixel coordinate of the target in the image; is the focal length of the camera; is the pixel coordinate of the center point of the image; is the distance between the target and the camera; H is the height of the drone; ( , , ) is the coordinate of the target in the camera coordinate system; ( , , ) is the coordinate of the target in the world coordinate system; ( , , ) is the GPS coordinates of the drone; is the rotation matrix; is the yaw angle of the drone gimbal; is the pitch angle of the drone gimbal; is the roll angle of the drone gimbal; Step 4: The system determines the fixed-point camera to be linked based on whether the visual range of the fixed-point cameras around the drone coordinates covers the event target location; Step 5: Calculate the PTZ value of the fixed-point camera and adjust the camera according to the PTZ value to synchronize the fixed-point camera with the target area of ​​the drone. The calculation formula is as follows: ； ； ； Where: P is the horizontal azimuth value; T is the pitch azimuth value; Z is the focal length magnification value of the camera lens; arccos is the inverse cosine function; arctan is the inverse tangent function; is the distance from the camera to the target point; b is the distance from the coordinate point with P value 0 to the target point; c is the distance from the camera to the coordinate point where P value is 0; is the height of the camera location; is the height of the tower where the camera is located; is the target point height; distance is the distance from the camera to the target point; is the pixel width of the image; is the maximum focal length of the camera; Min is the minimum value function; The camera focal length reaches the maximum value The maximum actual distance of the target point that the camera can clearly capture when Step 6: Stitch the videos shot by the drone and the videos shot by the fixed-point camera to conduct multi-directional continuous monitoring of the target location.

2. The method for locating a target based on the real-time position of an unmanned aerial vehicle and a camera according to claim 1, characterized in that: The conditions for drone dispatching include: The target event is detected in the video sent back to the tower video aggregation system by the fixed-point camera. The server collects and analyzes the event and sends an alarm to the person in charge. The person in charge confirms that the event needs to be tracked and a drone is dispatched for verification. Or when the drone inspection time comes, the drone will be dispatched according to the configured fixed route.

3. The method for locating a target based on the real-time position of an unmanned aerial vehicle and a camera according to claim 1, characterized in that: The method of taking the video frame with the highest confidence in target position identification includes: for each frame of image, the predicted position of the coordinates is calculated by a coordinate transformation method, and then the confidence of the frame of image is evaluated according to the difference between the actual position of the target and the predicted position. The smaller the difference, the higher the confidence. Finally, the confidence of different video frames is compared, and the video frame with the highest confidence is selected as the final result.

4. The method for locating a target based on the real-time position of an unmanned aerial vehicle and a camera according to claim 1, characterized in that: The acquisition of the positioning data of the drone and the drone gimbal at this time specifically includes: Obtain the pitch angle, roll angle, and yaw angle parameters of the drone gimbal according to the inspection route; Get the camera's focal length, sensor size, and image resolution parameters based on the drone model; Get the GPS coordinates and flight altitude of the drone based on the built-in positioning module of the drone.

5. The method for locating a target based on the real-time position of an unmanned aerial vehicle and a camera according to claim 1, characterized in that: The preprocessing includes denoising and grayscale enhancement of the image.

6. The method for locating a target based on the real-time position of an unmanned aerial vehicle and a camera according to claim 1, characterized in that: When calculating the pitch azimuth value, the relative altitude between the tower position and the target point is assumed to be 0 meters, and the height difference only calculates the hanging height of the camera.

7. The method for locating a target based on the real-time position of an unmanned aerial vehicle and a camera according to claim 1, characterized in that: The camera is adjusted according to the PTZ value, specifically: the horizontal rotation angle of the camera is adjusted according to the P value; the pitch angle of the camera is adjusted according to the T value; and the focal length magnification value of the camera lens is adjusted according to the Z value.

Images