A visual-based unmanned aerial vehicle detection and tracking method and system

Abstract

This invention relates to the field of unmanned aerial vehicle (UAV) technology and discloses a vision-based UAV detection and tracking method and system. The method includes acquiring an image of an inspection area using a visual camera device, detecting UAVs within the image, and in response to UAV detection, acquiring UAV detection bounding box information and tracking mapping information. Subsequently, based on the UAV detection bounding box information and tracking mapping information, the scanning tracking information of the visual camera device is determined, and the visual camera device is controlled to track the UAV based on the scanning tracking information. This invention achieves continuous and stable detection and tracking of UAVs, significantly reducing the possibility of losing track of the UAV.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a vision-based UAV detection and tracking method and system. Background Technology

[0002] Current drone detection methods primarily rely on electromagnetic wave signals. However, due to the lack of visual information, effective tracking of detected drones is impossible. While common PTZ cameras used for ground security can perform area inspections, their main targets are people, vehicles, or other large moving objects, and they cannot identify aerial drones (especially at long distances). Furthermore, because drones possess high relative speed, rapid acceleration, and maneuverability, even if a drone is detected by a conventional security PTZ camera, continuous tracking is difficult. Summary of the Invention

[0003] In view of this, the present invention proposes a vision-based method and system for detecting and tracking unmanned aerial vehicles (UAVs), which solves the problem that UAVs are difficult to detect and track in traditional technical solutions.

[0004] On one hand, embodiments of the present invention provide a vision-based drone detection and tracking method, including:

[0005] Images of the inspection area are acquired using a visual camera device, and drone detection is performed on the images of the inspection area.

[0006] In response to the detection of a drone, obtain drone detection frame information and tracking mapping information;

[0007] Based on the drone detection frame information and the tracking mapping information, the scanning tracking information of the visual camera device is determined;

[0008] Based on the scanning and tracking information, the visual camera device is controlled to track the drone.

[0009] In some embodiments, the tracking mapping information includes position-velocity mapping information and position-direction mapping information. The position-velocity mapping information includes multiple velocities that spread concentrically outward from the center, and the position-direction mapping information includes multiple directions that spread radially outward from the center.

[0010] In some implementations, the scanning tracking information includes motion speed and motion direction; based on the UAV detection frame information and the tracking mapping information, the scanning tracking information of the visual camera device is determined as follows:

[0011] The location information of the drone in the image of the inspection area is determined based on the drone detection frame information;

[0012] Based on the location information and the tracking mapping information, the motion speed and direction of the visual camera device are determined.

[0013] In some implementations, determining the motion speed and direction of the visual camera device based on the location information and the tracking mapping information includes:

[0014] The tracking mapping information is mapped onto the inspection area image to obtain an inspection area image with tracking information.

[0015] Based on the inspection area image with tracking information and the location information, the movement speed and direction of the visual camera device are determined.

[0016] In some implementations, controlling the visual camera device to track the drone based on the scan tracking information includes:

[0017] According to a preset frequency, the visual camera device is controlled to track and detect the drone based on the scanning and tracking information;

[0018] If the drone is not detected during the tracking process, the process returns to the steps of acquiring an image of the inspection area using a visual camera device and performing drone detection on the image of the inspection area.

[0019] In some implementations, acquiring images of the inspection area using a visual camera device includes:

[0020] Determine the scanning information based on the distance range to be inspected;

[0021] Based on the scanning information, the visual camera device is controlled to scan the inspection area to obtain an image of the inspection area.

[0022] In some implementations, controlling the visual camera device to scan the inspection area includes:

[0023] Determine the scanning information based on the distance range to be inspected;

[0024] Based on the scanning information, the visual camera device is controlled to scan the inspection area.

[0025] In some implementations, the scanning information is determined based on the distance range to be inspected, including:

[0026] Based on the distance range, determine the lens magnification in the scan information;

[0027] The lens field of view in the scanning information is determined based on the lens magnification.

[0028] In some implementations, after determining the lens field of view in the scanning information, the method further includes: dividing the inspection area into grids according to the lens field of view to obtain a grid-shaped inspection area, wherein the field of view of each sub-region in the grid-shaped inspection area is less than or equal to the lens field of view;

[0029] Based on the scanning information, the visual camera device is controlled to scan the inspection area to obtain an image of the inspection area, including:

[0030] Based on the scanning information, the visual camera device is controlled to traverse all sub-regions in the grid-shaped inspection area to obtain images of the traversed sub-regions.

[0031] In some implementations, the vision-based drone detection and tracking method further includes: in response to receiving drone detection information sent by a drone detection device, determining a corresponding visual camera device based on the drone detection information; and determining the scanning information of the visual camera device based on the drone detection information.

[0032] Acquiring images of the inspection area using a visual camera device includes: controlling the visual camera device to scan the target airspace based on determined scanning information to acquire images of the inspection area.

[0033] On the other hand, embodiments of the present invention also provide a vision-based drone detection and tracking system, the system including a visual camera device and an electronic device communicatively connected to the visual camera device, the electronic device being configured to perform the steps of the method described in any of the above embodiments.

[0034] The present invention has at least the following beneficial effects:

[0035] This invention provides a vision-based method and system for detecting and tracking unmanned aerial vehicles (UAVs). The invention acquires images of an inspection area using a visual camera device, detects UAVs within these images, and, in response to UAV detection, obtains UAV detection bounding box information and tracking mapping information. Based on these information, the visual camera device's scanning and tracking information is determined, and the system controls the visual camera device to track the UAV. This approach achieves continuous and stable detection and tracking of UAVs. By detecting UAVs in the inspection area images, it is possible to initially determine whether a UAV requiring tracking exists in the inspection area. Using the UAV detection bounding box information and pre-configured tracking mapping information, the movement speed and direction of the visual camera device can be accurately determined. This determined movement direction and speed are then used as scanning and tracking information to control the visual camera device for continuous, stable, and accurate tracking of the UAV, thereby significantly reducing the possibility of losing track of the UAV. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0037] Figure 1 A flowchart illustrating a vision-based drone detection and tracking method provided in an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of an inspection route determined by a vision-based UAV detection and tracking method according to an embodiment of the present invention.

[0039] Figure 3 A flowchart illustrating yet another vision-based drone detection and tracking method provided in this embodiment of the invention;

[0040] Figure 4 This is a schematic diagram of the first inspection area image determined by the vision-based UAV detection and tracking method provided in an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of a second inspection area image determined by the vision-based UAV detection and tracking method provided in an embodiment of the present invention;

[0042] Figure 6 A flowchart illustrating another vision-based drone detection and tracking method provided in an embodiment of the present invention;

[0043] Figure 7 A flowchart illustrating another vision-based drone detection and tracking method provided in this embodiment of the invention;

[0044] Figure 8 A schematic diagram of a vision-based drone detection and tracking system provided in an embodiment of the present invention;

[0045] Figure 9 This is a schematic diagram of another vision-based drone detection and tracking system provided in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0047] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0048] The present invention will now be described in detail with reference to the embodiments and accompanying drawings.

[0049] The first aspect of this invention provides a vision-based method for detecting and tracking unmanned aerial vehicles (UAVs), such as... Figure 1 As shown, the method specifically includes steps S10 to S40.

[0050] S10. Obtain images of the inspection area using a visual camera device, and perform drone detection on the images of the inspection area.

[0051] Specifically, the inspection area image is an image of the area to be inspected. In this embodiment of the invention, the inspection area video can be acquired by a visual camera device, and the video can be decoded to obtain the inspection area image. The visual camera device may include a PTZ camera, a hemispherical camera, a panoramic camera, etc.

[0052] In this embodiment of the invention, drone detection in the inspection area image can be achieved based on a pre-trained drone detection model. This model uses the YOLO model as its base network and is trained using a drone image dataset. Drones are then identified in the inspection area image based on this model. If a drone is detected in the inspection area image, drone detection information is output, including drone detection bounding box information, category, confidence level, and other information.

[0053] In this embodiment of the invention, when detecting drones on an inspection area image, drone detection can be performed based on the inspection area image. When a suspected drone target is detected, the lens magnification of the visual camera device is increased to obtain a magnified image of the area where the suspected drone target is located (hereinafter referred to as the target area image). Drone detection is then performed based on the target area image to determine whether the suspected drone target is a real drone. In a specific embodiment, when performing drone detection on an inspection area image using the above scheme, it can be implemented based on two different drone detection models (e.g., a first drone detection model and a second drone detection model). The first and second drone detection models are used to identify drone targets of different pixel sizes. For example, the second drone detection model can identify targets with a larger pixel size than the first drone detection model. In one specific embodiment, drone detection can be performed on the inspection area image based on a first drone detection model. In response to the detection of a suspected drone target, the magnification of the vision camera lens is controlled, and the target area to be captured by the vision camera is determined based on the detection frame information of the suspected drone target. The vision camera is then controlled to capture the target area image according to the magnified lens magnification and the target area to be captured, and the captured target area image is obtained. Drone detection is then performed on the target area image based on a second drone detection model. This allows for confirmation of whether the suspected drone target is indeed a drone through the target area image with a larger pixel size, thereby improving the accuracy of drone detection. In one specific embodiment, the second drone detection model can identify targets with a pixel size greater than or equal to 20, while the first drone detection model can identify targets with a pixel size less than 20.

[0054] S20. In response to the detection of a drone, obtain drone detection frame information and tracking mapping information.

[0055] If a drone is detected in step S10, the drone detection frame information can be obtained to determine the drone's position in the inspection area image.

[0056] Tracking mapping information is used to determine the speed and direction of motion of the visual camera device based on the position of the UAV in the inspection area image. In some specific embodiments, the tracking mapping information may include position-velocity mapping information and position-direction mapping information. The position-velocity mapping information includes the mapping relationship between the UAV's position and the visual camera device's speed. The position-direction mapping information includes the mapping between the UAV's position and the visual camera device's direction of motion.

[0057] S30. Based on the UAV detection frame information and tracking mapping information, determine the scanning and tracking information of the visual camera device.

[0058] Specifically, if a drone is detected in step S10, its detection information, including the drone detection frame information, can be obtained. The scanning and tracking information of the visual camera device can include the motion information of the visual camera device, such as its speed and direction of motion. In one specific embodiment, the drone's position in the inspection area image can be determined based on the drone detection frame information. Based on the drone's position and position-velocity mapping information in the inspection area image, the speed of the visual camera device can be determined. Based on the drone's position and position-direction mapping information in the inspection area image, the direction of motion of the visual camera device can be determined. The determined direction and speed of motion of the visual camera device can be used as scanning and tracking information to control the visual camera device to continuously and stably track the drone.

[0059] In some embodiments, the scanning tracking information of the visual camera device may further include the lens magnification of the visual camera device. In a specific embodiment, the pixel size of the drone in the image can also be determined based on the drone detection box information. Based on the pixel size of the drone, the current lens magnification of the visual camera device can be determined to ensure that the visual camera device can accurately detect the drone, thereby further improving the tracking effect. More specifically, a pixel size threshold range can be set, and it can be determined whether the pixel size of the currently determined drone is within the set pixel size threshold range to determine whether the current lens magnification of the visual camera device needs to be adjusted. If it is within the set pixel size threshold range, no adjustment to the current lens magnification is required. If the pixel size of the currently determined drone is not within the set pixel size threshold range, the current lens magnification will be adjusted so that the visual camera device can scan the area where the drone is located using the adjusted lens magnification. Assuming the set pixel size threshold range is (width-min, width-max), the current lens magnification is s, and the currently determined drone pixel size is... l The specific method for adjusting the lens magnification can be as follows: When l When <width-min, increase s, when l When width - max, decrease s; when width - min ≤ l When the pixel size is ≤width-max, no adjustment is needed. This avoids both inaccurate detection due to excessively small drone pixel size and loss of tracking due to excessively large pixel size, thereby improving the continuity and stability of drone tracking and the accuracy of drone detection.

[0060] S40: Based on scanning tracking information, control the visual camera device to track the drone.

[0061] Specifically, a visual camera device can be controlled to scan and track a drone based on scanning and tracking information. In some specific embodiments, the scanning and tracking information may include movement speed and direction, allowing the visual camera device to be controlled to scan and track the drone based on this speed and direction, thereby achieving continuous and stable tracking of the drone. In some specific embodiments, the scanning and tracking information may include movement speed, direction, and lens magnification, allowing the visual camera device to be controlled to move to track the drone based on this speed and direction, and to acquire images of the scanned area based on the lens magnification. Subsequently, drone detection is performed on the scanned area images, thereby achieving continuous and stable tracking of the drone and obtaining accurate drone detection results.

[0062] This invention, through a technical solution involving acquiring an inspection area image using a visual camera device, detecting drones within that image, and in response to drone detection, acquiring drone detection frame information and tracking mapping information, then determining the visual camera device's scanning tracking information based on these information, and controlling the visual camera device to track the drone based on this scanning tracking information, achieves continuous and stable drone detection and tracking. In this embodiment, by detecting drones in the inspection area image, it is possible to initially determine whether a drone requiring tracking exists in the inspection area. Using the drone detection frame information and pre-configured tracking mapping information, the movement speed and direction of the visual camera device can be accurately determined. This determined movement direction and speed are then used as scanning tracking information to control the visual camera device to continuously, stably, and accurately track the drone, thereby significantly reducing the possibility of losing track of the drone.

[0063] In some embodiments of the present invention, the specific method for obtaining the inspection area image in step S10 may be: determining the inspection area according to the field of view of the visual camera device; controlling the visual camera device to scan the inspection area to obtain the inspection area image.

[0064] Specifically, the field of view of a visual camera device is usually determined by the manufacturer, and includes both horizontal and vertical angles. The area of ​​interest can be defined as the inspection zone based on the field of view and the actual application scenario. In one specific embodiment, a PTZ camera can be used as the visual camera device. Assuming the PTZ camera's field of view is 0°~360° horizontally and -15°~90° vertically, the area of ​​interest within this field of view is determined as the inspection zone. In some specific embodiments, for example, the horizontal angle range of the inspection zone can be 60°~120°, and the vertical angle range can be 30°~90°. For example, the horizontal angle range of the inspection zone can be 20°~100°, and the vertical angle range can be 15°~80°. For example, the horizontal angle range of the inspection zone can be 30°~180°, and the vertical angle range can be -10°~60°. Of course, depending on the actual application scenario, the field of view of the inspection area can be other angles, which are not specifically limited here.

[0065] In this embodiment of the invention, after determining the inspection area, the visual camera device can be controlled to scan the inspection area to obtain an image of the inspection area, thereby enabling the inspection of the inspection area to detect whether a drone is present.

[0066] In some embodiments of the present invention, when controlling the visual camera device to scan the inspection area, the visual camera device can be controlled to scan the inspection area according to the scanning information.

[0067] Specifically, the scanning information is used to guide the visual camera device to scan the inspection area, and may include information such as lens magnification and lens field of view.

[0068] In some specific embodiments, the visual camera device can be controlled to scan the inspection area according to the pre-configured lens magnification and lens field of view, thereby obtaining an image of the inspection area to achieve the detection of the drone.

[0069] In some specific embodiments, in order to improve the accuracy of drone detection, the lens magnification and field of view of the visual camera device can be determined according to the distance range to be inspected. This makes the determined lens magnification and field of view more accurate, so that the visual camera device can be controlled to scan the inspection area according to the lens magnification and field of view, thereby obtaining an image of the inspection area with appropriate size and angle, thus achieving accurate detection of drones and improving the accuracy of drone detection.

[0070] In some specific embodiments, the lens magnification in the scanning information can be determined based on the distance range to be inspected; and the lens field of view in the scanning information can be determined based on the lens magnification.

[0071] Specifically, the lens magnification can be determined through lens magnification configuration conditions. The lens magnification configuration conditions are that the pixel scale of the UAV in the inspection area image is not less than m pixels. The value of m can be set according to the actual situation, such as 8, 10, 12, 15 or 20, etc.

[0072] In practical applications, the greater the inspection distance, the higher the required lens magnification Z, resulting in a smaller field of view and a longer inspection time. Therefore, to improve inspection efficiency while ensuring accurate drone identification, the lens magnification configuration can be optimized by minimizing the lens magnification while maintaining a pixel size of at least m pixels in the inspection area image. In some specific embodiments, the lens magnification can be the minimum magnification that ensures the drone's pixel size in the inspection area image is at least m pixels. For example, with an inspection distance of 500m, a suitable lens magnification Z is 4.

[0073] In some embodiments of the present invention, after determining the lens field of view, the method further includes: dividing the inspection area into a grid according to the lens field of view to obtain a grid-shaped inspection area. Wherein, the field of view of each sub-region within the grid-shaped inspection area is less than or equal to the lens field of view.

[0074] After obtaining the grid-shaped inspection area, the vision camera device is controlled to scan the inspection area to obtain images of the inspection area. The vision camera device can be controlled to traverse all sub-regions in the grid-shaped inspection area to obtain images of each traversed sub-region.

[0075] In a grid-like inspection area, a sub-region corresponds to the area covered by a grid. By ensuring that the field of view of each sub-region is less than or equal to the field of view of the lens, the visual camera device can acquire a complete image of each sub-region when scanning it, thus avoiding missed scans and improving the scanning coverage and unmanned inspection coverage.

[0076] In one specific embodiment, after determining the lens magnification and lens field of view, the inspection airspace can be automatically divided into N×M grids based on the lens field of view, ensuring that the field of view of the sub-region covered by each grid is less than or equal to the lens field of view. Subsequently, the visual camera device is controlled to traverse each sub-region grid by grid along a preset inspection route according to the determined lens magnification and corresponding sub-region field of view, and stays in each sub-region for a preset time to detect whether a drone is present in that sub-region.

[0077] like Figure 2As shown, the preset inspection route can be a zigzag pattern, thereby improving traversal efficiency, but it is not limited to this and can also be other shapes. The preset time can be set based on the actual use scenario. In some specific embodiments, it can be 0.5s to 1s, but it is not limited to this and other times are also possible.

[0078] In one specific embodiment, taking a PTZ camera as an example, the process of dividing the inspection area into a grid is explained. The PTZ camera's field of view can be treated as a plane, for example, as... Figure 2 The plane shown should be understood as follows: Figure 2 This is merely a planar schematic of the PTZ camera's field of view, used to illustrate the grid-like division process of the inspection area, and is not intended to limit the invention. When the pitch angle t increases (i.e., the PTZ camera looks upwards), since the field of view is a spherical projection, the actual horizontal range narrows proportionally to cos(t). Therefore, the number of grids required to cover 360° horizontally decreases as the pitch angle t increases.

[0079] For the same model of PTZ camera, the actual field of view = maximum field of view / current magnification. Assume the horizontal field of view at the current lens magnification is FovH (degrees) and the vertical field of view is FovV (degrees). The horizontal angle range to be inspected is ScaleH (degrees), and the vertical angle range is ScaleV (degrees). When the gimbal pitch angle is t, the number of grids N(t) required to cover the horizontal direction of ScaleH and the number of grids M required to cover the vertical direction of ScaleV are respectively:

[0080] N(t) = ScaleH / FovH × cos(t);

[0081] M = ScaleV / FovV;

[0082] Where t is the pitch angle (unit: degrees), t=0° when horizontal and t=90° when vertically upward.

[0083] To ensure that the number of grid cells is an integer, the results N(t) and M can be rounded up to guarantee that the number of grid cells is an integer.

[0084] The embodiments of the present invention can accurately divide the inspection area image through the above solution, thereby enabling the visual camera device to accurately track the drone and improve the drone tracking effect.

[0085] In some embodiments of the present invention, the scanning tracking information includes motion speed and motion direction. Determining the scanning tracking information of the visual camera device based on the UAV detection box information and tracking mapping information in step S30 may include: determining the position information of the UAV in the inspection area image based on the UAV detection box information; and determining the motion speed and motion direction of the visual camera device based on the position information and tracking mapping information.

[0086] Specifically, the tracking mapping information includes position-velocity mapping information and position-direction mapping information. Based on the position and position-velocity mapping information of the UAV in the inspection area image, the movement speed of the visual camera device can be determined. Based on the position and position-direction mapping information of the UAV in the inspection area image, the movement direction of the visual camera device can be determined. The determined movement direction and movement speed of the visual camera device can be used as scanning tracking information to control the visual camera device to continuously and stably track the UAV.

[0087] In some specific embodiments, the position-velocity mapping information may include multiple velocities concentrically spreading outward from the center. More specifically, the position-velocity mapping information may be a two-dimensional array of m rows × n columns, where each position in the array corresponds to a specific velocity, the exact velocity depending on the partitioning strategy. In some specific embodiments, the array can be divided into regions with the center as the center, resulting in multiple concentric annular regions. Each annular region corresponds to a motion velocity used to control the movement of the visual camera device, with the annular region closer to the center of the inspected area image corresponding to a slower motion velocity. In some specific embodiments, the motion velocity corresponding to the annular region where the array center is located can be 0, and the motion velocities corresponding to other annular regions can be determined based on the motion level of the visual camera device. In some specific embodiments, the number of annular regions can be in the range of 2 to 8, for example, 3, 4, 5, 6, or 7.

[0088] In some specific embodiments, the position-direction mapping information may include multiple directions radiating outward from the center. More specifically, the position-direction mapping information may be a two-dimensional array of m rows × n columns, where each position in the array corresponds to a specific direction, the exact direction of which is determined based on a partitioning strategy. In some specific embodiments, the directions in the array are arranged radially based on the array center. In some specific embodiments, if the motion velocity corresponding to the array center is 0, then its corresponding motion direction is empty, meaning that there is no need to control the movement of the visual camera device at this time.

[0089] This invention provides a technical solution that determines the location information of a drone in an inspection area image based on drone detection frame information; and determines the movement speed and direction of a visual camera device based on the location information and tracking mapping information. This solution can accurately determine the movement speed and direction of the visual camera device, thereby improving the tracking capability of the visual camera device for drones, achieving continuous and stable detection and tracking of drones, and greatly reducing the possibility of losing track of the drone.

[0090] In some embodiments of the present invention, when determining the movement speed and direction of the visual camera device based on the position information and tracking mapping information of the UAV in the inspection area image, the position information of the UAV in the inspection area image can be converted to the coordinate system of the tracking mapping information, so as to obtain the corresponding movement speed and direction from the tracking mapping information based on the converted position information.

[0091] In some embodiments of the present invention, when determining the movement speed and direction of the visual camera device based on the position information and tracking mapping information of the UAV in the inspection area image, the tracking mapping information can be mapped to the inspection area image to obtain an inspection area image with tracking information; based on the inspection area image with tracking information and the position information, the movement speed and direction of the visual camera device are determined.

[0092] Specifically, the tracking mapping information can include position-velocity mapping information and position-direction mapping information. By transforming the tracking mapping information into the coordinate system of the inspection area image, tracking mapping information that matches the pixel size of the inspection area image can be obtained. Based on the transformed tracking mapping information, the motion velocity and motion direction corresponding to any position in the current frame inspection area image can be obtained.

[0093] In one specific embodiment, after converting the position-velocity mapping information to the coordinate system of the inspection area image, the resulting position-velocity mapping information corresponding to the inspection area image (i.e., the converted position-velocity mapping information) can be mapped onto the inspection area image to obtain an inspection area image with position-velocity mapping information. For example, Figure 4 The image shown here, containing position-velocity mapping information, is an inspection area image. Figure 4 In the process, the converted position-velocity mapping information divides the inspection area image into 5 concentric ring-shaped regions, namely L0~L4. L0 corresponds to the central region of the inspection area image, and L4 corresponds to the outer edge of the inspection area image.

[0094] In one specific embodiment, such as Figure 5As shown, this is the position-direction mapping information corresponding to the inspection area image after transforming the position-direction mapping information to the coordinate system of the inspection area image (i.e., the transformed position-direction mapping information). After mapping this transformed position-direction mapping information onto the inspection area image, an inspection area image with position-direction mapping information can be obtained. For example, Figure 5 The image shown has a location-direction mapping information for the inspected area. Figure 5 In the process, the converted position-direction mapping information divides the inspection area image into nine radially arranged regions (hereinafter referred to as direction regions), which correspond to nine directions: center, top, bottom, left, right, top left, top right, bottom left, and bottom right.

[0095] Through the above-described technical solution, the present invention can accurately determine the movement speed and direction of the visual camera device, thereby improving the tracking capability of the visual camera device for drones, achieving continuous and stable detection and tracking of drones, and greatly reducing the possibility of losing track of them.

[0096] In some embodiments of the present invention, step S40, which controls the visual camera device to track the drone based on the scanning tracking information, may include controlling the visual camera device to track the drone based on the scanning tracking information at a preset frequency; and, in response to the failure to detect the drone during the tracking process, returning to the step of acquiring an inspection area image based on the visual camera device and detecting the drone in the inspection area image.

[0097] Specifically, the preset frequency can be set based on actual usage needs, such as 5Hz, 10Hz, 15Hz or 20Hz. Based on the preset frequency, the drone can be tracked and detected simultaneously during the tracking process, so that changes in the drone's flight status can be detected in time, reducing the possibility of losing the drone due to its high speed and sudden changes in direction, and improving the continuity and stability of drone tracking.

[0098] In some specific embodiments, the scanning tracking information may include movement speed and direction. In this case, the visual camera device can be controlled to track the drone according to the movement speed and direction in the scanning tracking information at a preset frequency. During the tracking process, images of the tracking area are acquired at an image acquisition frequency (greater than or equal to a preset frequency), and the tracking area images are detected. If a drone is detected in the tracking area image, it means that the drone has been tracked. Tracking continues until the drone is no longer detected in the tracking area image, then the process returns to step S10 to scan the inspection area and obtain inspection area images. This embodiment achieves continuous and stable tracking of the drone through the above scheme, improving the continuity and stability of drone tracking and reducing the probability of losing the drone due to its high speed and sudden changes in direction.

[0099] In some specific embodiments, when controlling the visual camera device to track the drone based on the movement speed and direction in the scanning tracking information, a preset tracking duration corresponding to each tracking cycle can be determined based on a preset frequency. Based on this, when the actual tracking duration of the drone by the visual camera device reaches the preset tracking duration, an image of the tracking area can be acquired and detected. If the drone is detected in the tracking area image, it indicates that the drone has been tracked, and tracking continues at the preset frequency until the drone is no longer detected in the tracking area image. Then, the process returns to step S10 to scan the inspection area and obtain an image of the inspection area. This embodiment achieves continuous and stable tracking of the drone through the above scheme, improving the continuity and stability of drone tracking and reducing the probability of losing track of the drone due to its high speed and sudden changes in direction.

[0100] In one specific embodiment, the scanning tracking information may include movement speed, movement direction, and lens magnification. This allows the visual camera device to be controlled to move according to a preset frequency (e.g., 10Hz, 15Hz, or 20Hz) based on the movement speed and direction determined in step S30 to track the drone. During tracking, the visual camera device is controlled to acquire images of the tracking area at the lens magnification determined in step S30. Subsequently, drone detection is performed on the acquired tracking area images, thereby achieving continuous and stable tracking of the drone and accurate drone detection during the tracking process. This improves the continuity and stability of drone tracking, reduces the probability of losing track of the drone due to its high speed and sudden changes in direction, and improves the accuracy of drone detection.

[0101] like Figure 3 As shown, the vision-based drone detection and tracking method provided in this embodiment of the invention includes steps S310 to S360.

[0102] S310. Acquire images of the inspection area using a visual camera device, and perform drone detection on the images of the inspection area.

[0103] If no drone is detected, repeat step S310. If a drone is detected, execute steps S320 to S360.

[0104] S320: Acquire drone detection frame information and tracking mapping information.

[0105] S330. Map the tracking mapping information onto the inspection area image to obtain an inspection area image with tracking information.

[0106] S340. Determine the location information of the UAV in the inspection area image based on the UAV detection frame information.

[0107] S350: Based on the inspection area image and location information with tracking information, determine the movement speed and direction of the visual camera device.

[0108] S360, according to a preset frequency, controls the visual camera device to track and detect the drone based on the scanning tracking information.

[0109] In some specific embodiments, the visual camera device can be controlled to track and detect the drone according to the scanning tracking information at a preset frequency. If the drone is detected during the tracking process, step S360 can be executed repeatedly. If the drone is not detected during the tracking process, the process returns to step S310, thereby achieving continuous and stable detection and tracking of the drone, greatly reducing the possibility of losing track of it.

[0110] In some specific embodiments, if a drone is detected during tracking, the latest drone detection frame information can be obtained, and the process returns to step S340 to determine the final scan tracking information based on the latest drone detection frame information and tracking mapping information. This allows for drone tracking based on the newly determined scan tracking information, thereby improving the accuracy of drone tracking. If no drone is detected during tracking, the process returns to step S310, further improving the continuity and stability of drone tracking and significantly reducing the possibility of losing track of the drone.

[0111] This invention, through a technical solution involving acquiring images of an inspection area using a visual camera device and detecting drones within those images; acquiring drone detection frame information and tracking mapping information; determining the scanning and tracking information of the visual camera device based on the drone detection frame information and tracking mapping information; and controlling the visual camera device to track and detect drones at a preset frequency based on the scanning and tracking information, can accurately determine the movement speed and direction of the visual camera device. This improves the visual camera device's ability to track drones, enabling continuous and stable detection and tracking of drones, and significantly reducing the possibility of losing track of them.

[0112] The following specific examples illustrate... Figure 3 The following examples illustrate the drone detection and tracking process. It should be understood that the following embodiments are only for explaining the present invention and are not intended to limit the present invention.

[0113] The system acquires images of the inspection area using a vision camera and performs drone detection on these images. Upon detecting a drone in the inspection area image, it acquires drone detection bounding box information and tracking mapping information. Based on the drone detection bounding box information, it calculates the center point coordinates of the drone in the pixel coordinate system within the inspection area image; these center point coordinates represent the drone's position in the inspection image.

[0114] Tracking mapping information includes position-velocity mapping information and position-direction mapping information. Transforming this position-velocity mapping information and position-direction mapping information to the coordinate system of the inspection area image yields tracking mapping information that matches the pixel size of the inspection area image. Mapping this transformed tracking mapping information onto the inspection area image results in an inspection area image with tracking mapping information. For example, ... Figure 4 The image shows the inspection area with position-velocity mapping information, and as shown... Figure 5 The image shown contains a patrol area image with position-direction mapping information. Based on this, the corresponding speed and direction can be accurately obtained for each pixel region of the patrol area image.

[0115] Subsequently, the region where the detection frame is located can be determined based on the coordinates of its center point. Based on this region, its corresponding velocity and direction can be obtained. Specifically, when determining the motion speed of the visual camera device, such as... Figure 4As shown, the speed can be determined based on the annular region where the center point of the detection frame is located; this speed is the movement speed of the visual camera device. The closer the center point of the detection frame is to the central annular region (e.g., L0), the slower the corresponding movement speed; the closer the landing point is to the edge annular region (e.g., L4), the faster the corresponding movement speed. Taking a PTZ camera as an example, the PTZ camera includes a pan-tilt unit and a camera unit mounted on the pan-tilt unit. When controlling the movement of the PTZ camera, the movement of the pan-tilt unit is controlled, thereby driving the movement of the camera unit. The PTZ camera corresponds to seven movement levels, and L0 to L4 each correspond to one of the seven movement levels, with the movement speeds corresponding to the movement levels L0 to L4 increasing sequentially. At the same time, it is also necessary to determine which directional area the center point coordinates fall in. If it falls in the center direction area, no movement is required; if it falls in the direction of upward, the gimbal is controlled to move in the upward direction; if it falls in the direction of downward, the gimbal is controlled to move in the downward direction. The same applies to other launch areas, that is, the gimbal is controlled to move in the direction in which the center point representing the drone's position falls.

[0116] Based on the drone detection bounding box information obtained during the inspection of the inspection area image, the pixel size (i.e., width) of the drone in the pixel coordinate system of the inspection area image is calculated. The lens magnification is then adjusted using this pixel width and a set pixel size threshold range to obtain drone detection targets with suitable pixel sizes from the scanned image and to continuously and stably track the drone. Assuming the set pixel size threshold range is (width-min, width-max), the current lens magnification is s, and the currently determined drone pixel size is... l The specific method for adjusting the lens magnification can be as follows: When l When <width-min, increase s, when l When width - max, decrease s; when width - min ≤ l When the pixel size is ≤width-max, no adjustment is needed. This avoids both inaccurate detection due to excessively small drone pixel size and loss of tracking due to excessively large pixel size, thereby improving the continuity and stability of drone tracking and the accuracy of drone detection.

[0117] Based on the above conditions, the gimbal's movement direction and speed, as well as the lens magnification change direction, are determined. The API interface in the PTZ camera SDK is then called to control the gimbal's movement for a short period, such as 100ms; simultaneously, the lens magnification is controlled to change for a short period, such as 100ms. The above detection, calculation, and control process is then repeated. Repeating this process at a frequency of 10Hz enables real-time tracking of moving targets.

[0118] Objective: To control the PTZ camera to track a moving drone, keeping it as centered in the frame as possible while maintaining an appropriate size.

[0119] In some embodiments of the present invention, such as Figure 6 As shown, the drone detection and tracking method provided in this embodiment of the invention may include S01 in addition to steps S10 to S40.

[0120] S01. Pre-focus the visual camera device.

[0121] Specifically, this pre-focusing can be triggered when the interface of the vision camera device is invoked. The pre-focusing process involves selecting a marker (e.g., an unobstructed building) on ​​the ground before scanning the inspection area to obtain an image of the scanned area using the vision camera device. After selecting the marker, the vision camera device is focused on the marker, thereby ensuring that the vision camera device can obtain a clear image of the marker, improving the efficiency of subsequent inspections and drone tracking based on the vision camera device, and enhancing the accuracy of drone detection.

[0122] The pre-focusing process is described below through specific embodiments. It should be understood that the following embodiments are only used to explain the present invention and are not intended to limit the present invention.

[0123] Several landmarks at different distances are pre-selected as auxiliary focus points. For example, with a detection distance of 1 kilometer, two landmarks can be selected, one approximately 100m from the PTZ camera and the other approximately 500m from the camera. Using the landmark at 100m for focusing, the focal length range can roughly cover all targets within 300m; using the landmark at 500m for focusing, the focal length range can roughly cover all targets between 300m and 1000m. During inspection, based on the airspace distance to be detected, the corresponding landmark is selected for initial focusing to obtain an acceptable focus sharpness for the algorithm.

[0124] In this embodiment of the invention, by pre-focusing the PTZ camera, the efficiency of subsequent inspections and drone tracking based on the PTZ camera is improved, thereby increasing the accuracy of drone detection.

[0125] In some embodiments of the present invention, such as Figure 7 As shown, the vision-based drone detection and tracking method provided in this embodiment of the invention includes steps S710 to S770.

[0126] S710, in response to receiving drone detection information sent by the drone detection device, determines the target visual camera device based on the drone detection information.

[0127] Specifically, drone detection equipment can include RID receivers, phased array radars, 5G-A sensing base stations, and other devices capable of detecting drones through electromagnetic signals. Drone detection information can include the drone's latitude, longitude, altitude, size, and ID (or other identifier).

[0128] In this embodiment of the invention, the visual camera device that is closest to the drone and whose field of view can cover the airspace where the drone is located can be obtained from all visual camera devices based on the drone detection information, i.e., the target visual camera device.

[0129] S720 determines the scanning information of the target visual camera device based on the pose information of the target visual camera device and the detection information of the UAV.

[0130] Based on the pose information of the target visual camera device, such as extrinsic parameters, the location information of the UAV, such as latitude, longitude, and altitude, in the UAV detection information can be transformed into a local coordinate system. Then, based on the UAV's position information in the local coordinate system and the position information of the target visual camera device, the scanning information of the target visual camera device can be determined. This scanning information includes the field of view and lens magnification.

[0131] S730: Based on scanning information, control the target visual camera device to scan the target airspace to acquire target airspace images.

[0132] The spatial range to be scanned is determined centered on the field of view angle in the scanning information and designated as the target spatial range. The target visual camera is then controlled to scan the target spatial range according to the lens magnification specified in the scanning information, thereby acquiring an image of the target spatial range and transmitting it to the electronic device for processing. The following steps S740 to S770 are processed by the electronic device.

[0133] In some embodiments, since the position information of the UAV in the local coordinate system can be calculated, the corresponding landmark can be selected based on the position information of the UAV in the local coordinate system to perform fixed-point focusing on the target visual camera device, so that the target visual camera device can obtain a clear airspace image when performing airspace scanning.

[0134] In some embodiments, the target visual camera device has already been pre-focused before point-to-point focusing. In this case, it can be determined that the position information of the UAV in the local coordinate system is within the distance range corresponding to the pre-focus. If it is, point-to-point focusing is not required; otherwise, point-to-point focusing is performed.

[0135] S740: Acquire target airspace images using target vision camera devices and perform UAV detection on the target airspace images.

[0136] S750, in response to the detection of a drone, acquires drone detection bounding box information and tracking mapping information obtained by detecting the target airspace image.

[0137] S760 determines the scanning and tracking information of the visual camera device based on the drone detection frame information and tracking mapping information.

[0138] The S770 uses a visual camera device controlled by scanning tracking information to track drones.

[0139] The specific implementation methods of steps S740 to S770 are basically the same as those of steps S10 to S40. Therefore, the specific implementation methods of steps S740 to S770 will not be described in detail here.

[0140] The above process will be described below with reference to specific embodiments. It should be understood that the following embodiments are only used to explain the present invention and are not intended to limit the present invention.

[0141] Taking a phased array radar as the drone detection equipment and a PTZ camera as the visual imaging device as an example, when the phased array radar detects a suspected drone target, it sends the detected information (the latitude, longitude, altitude, size, ID, etc. of the suspected drone target) to the EMQ service via MQTT messages. The electronic device subscribes to relevant topics to obtain the latitude, longitude, altitude, size, ID, and other information of the suspected target; then, it searches for the nearest PTZ camera A among all PTZ cameras whose field of view covers the area where the suspected target is located; combining the pose (extrinsic parameters) information of A, through a series of coordinate transformations, it calculates the scanning information required by PTZ camera A, including the field of view (horizontal angle p and pitch angle t) and lens magnification z.

[0142] Because the target coordinates may have some errors and some lag, the PT value will be used as the center to search a surrounding airspace according to the lens magnification z. After finding the target, it will be magnified and photographed for evidence, and will continue to track it if necessary.

[0143] The electronic device can receive semantic information such as the detection results of all PTZ cameras and report it to the EMQ service in real time via MQTT messages. The video streams of each PTZ camera can be pushed to the electronic device in real time through the streaming media service. The electronic device can simultaneously save the captured images, video streams and semantic information locally.

[0144] The embodiments of the present invention achieve continuous and stable tracking of drones and accurate detection of drones during the tracking process through the above-described scheme, thereby improving the continuity and stability of drone tracking and the accuracy of drone detection.

[0145] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a vision-based unmanned aerial vehicle (UAV) detection and tracking system, such as... Figure 8As shown, the vision-based drone detection and tracking system 10 includes a vision camera device 11 and an electronic device 12 communicatively connected to the vision camera device 11. The electronic device is configured to perform the steps of the method described in any of the above embodiments.

[0146] In this embodiment of the invention, an electronic device can acquire images of an inspection area via a visual camera, detect drones within those images, and in response to drone detection, acquire drone detection frame information and tracking mapping information. Subsequently, based on the drone detection frame information and tracking mapping information, the scanning and tracking information of the visual camera is determined, and the visual camera is controlled to track the drone based on this information. Through this solution, the invention achieves continuous and stable detection and tracking of drones, significantly reducing the possibility of losing track of them.

[0147] In some embodiments of the present invention, such as Figure 8 and 9 As shown, the visual camera device 11 in the UAV detection and tracking system 10 is a PTZ camera. In addition to the PTZ camera and electronic device 12, the UAV detection and tracking system 10 also includes a switch. The electronic device 12 and the PTZ camera are connected via the switch. The electronic device 12 includes a PTZ camera control module, an inference detection module, an EQM service, and a streaming media service.

[0148] The PTZ camera control module is responsible for interacting with the PTZ camera, decoding the video captured by the PTZ camera to obtain images, controlling the PTZ camera's gimbal and zoom, and realizing functions such as inspection and tracking control. The detection and inference module is responsible for calling the UAV detection model to perform inference detection on the images, detecting UAVs, and outputting semantic results such as detection boxes. The main control module is the bridge between the internal and external systems. On the one hand, it collects internal detection results, working status and other information, and reports them to the EMQ (an open-source IoT message server) service through MQTT (Message Queuing Telemetry Transport, a lightweight message transmission protocol based on the publish / subscribe paradigm) for external platforms to subscribe to. On the other hand, it subscribes to messages such as linkage requests from external systems through the EMQ service, interacts with the PTZ camera control program, and completes tasks such as PTZ camera task scheduling, which involves coordinate transformations between pixel coordinate system, camera coordinate system, ENU coordinate system (a local rectangular coordinate system centered on the station) and geodetic coordinate system.

[0149] The workflow of the UAV detection and tracking system 10 is as follows: When performing an inspection task, the PTZ control module controls the PTZ camera's gimbal to move according to preset rules. Simultaneously, it acquires video streams from the PTZ camera, decodes them into images, and packages them along with relevant information such as the gimbal's PTZ value and timestamp. This image is then sent to the detection and inference module via shared memory. The detection and inference module performs inference detection on the received image and sends the detection results, such as target type, confidence level, detection box pixel coordinates, and timestamp, back to the PTZ control module via a predefined UDP message. The PTZ control module judges based on the detection results. If no UAV is detected, the inspection task continues; if a UAV is detected, it enters tracking mode. At this time, the PTZ control module controls the gimbal movement and lens zoom in real time based on the detection box position to track and detect the UAV, keeping it as centered in the frame as possible while maintaining an appropriate size.

[0150] In some embodiments of the present invention, the electronic device includes a processor and a memory, the memory storing a computer program that can run on the processor, and the processor executing the program performs the steps of the method described in any of the above embodiments.

[0151] The memory, as a non-volatile storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods described in the embodiments of this application. The processor executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in the memory, thereby implementing the methods described in the above embodiments.

[0152] The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the local module via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0153] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium for the program can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The above computer program embodiments can achieve the same or similar effects as any of the corresponding foregoing method embodiments.

[0154] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, blocks, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.

[0155] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. The sequence numbers of the disclosed embodiments of this invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0156] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0157] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A visual-based unmanned aerial vehicle detection and tracking method, characterized in that, include: Images of the inspection area are acquired using a visual camera device, and drone detection is performed on the images of the inspection area. In response to the detection of a drone, drone detection frame information and tracking mapping information are acquired. The tracking mapping information includes position-velocity mapping information and position-direction mapping information. The position-velocity mapping information includes multiple velocities that spread concentrically outward from the center, and the position-direction mapping information includes multiple directions that spread radially outward from the center. Based on the UAV detection frame information and the tracking mapping information, the scanning tracking information of the visual camera device is determined; Based on the scanning and tracking information, the visual camera device is controlled to track the drone.

2. The method of claim 1, wherein, The scanning and tracking information includes movement speed and direction; Based on the UAV detection frame information and the tracking mapping information, the scanning tracking information of the visual camera device is determined as follows: The location information of the drone in the image of the inspection area is determined based on the drone detection frame information; Based on the location information and the tracking mapping information, the motion speed and direction of the visual camera device are determined.

3. The method according to claim 2, characterized in that, Determining the motion speed and direction of the visual camera device based on the location information and the tracking mapping information includes: The tracking mapping information is mapped onto the inspection area image to obtain an inspection area image with tracking information. Based on the inspection area image with tracking information and the location information, the movement speed and direction of the visual camera device are determined.

4. The method according to any one of claims 1 to 3, characterized in that, Controlling the visual camera device to track the drone based on the scanned tracking information includes: According to a preset frequency, the visual camera device is controlled to track and detect the drone based on the scanning and tracking information; If the drone is not detected during the tracking process, the process returns to the steps of acquiring an image of the inspection area using a visual camera device and performing drone detection on the image of the inspection area.

5. The method according to any one of claims 1 to 3, characterized in that, Images of the inspection area acquired using a visual camera device include: Determine the scanning information based on the distance range to be inspected; Based on the scanning information, the visual camera device is controlled to scan the inspection area to obtain an image of the inspection area.

6. The method according to claim 5, characterized in that, Based on the distance range to be inspected, the scanning information includes: Based on the distance range, determine the lens magnification in the scan information; The lens field of view in the scanning information is determined based on the lens magnification.

7. The method according to claim 6, characterized in that, After determining the lens field of view in the scanning information, the method further includes: dividing the inspection area into grids according to the lens field of view to obtain a grid-shaped inspection area, wherein the field of view of each sub-region in the grid-shaped inspection area is less than or equal to the lens field of view; Controlling the visual camera device to scan the inspection area based on the scanning information to obtain an image of the inspection area includes: controlling the visual camera device to traverse all sub-regions in the grid-shaped inspection area based on the scanning information to obtain images of the traversed sub-regions.

8. The method according to any one of claims 1 to 3, characterized in that, Also includes: In response to receiving drone detection information sent by the drone detection device, the corresponding visual camera device is determined based on the drone detection information; Based on the drone detection information, the scanning information of the visual camera device is determined; Acquiring images of the inspection area using a visual camera device includes: controlling the visual camera device to scan the target airspace based on determined scanning information to acquire images of the inspection area.

9. A vision-based unmanned aerial vehicle (UAV) detection and tracking system, characterized in that, The method includes a visual camera device and an electronic device communicatively connected to the visual camera device, the electronic device being configured to perform the steps of the method as described in any one of claims 1 to 8.

Images