Flight obstacle detection method and device in flight environment, equipment and medium

Abstract

The invention provides a flight obstacle detection method in a flight environment, computer equipment and a computer readable storage medium. According to the method and the device, the area sub-images of the flight environment are acquired from the multiple view angles through the camera array, then the flight area panorama is obtained through splicing the area sub-images based on the multiple view angles, and flight obstacle monitoring is simultaneously performed from the multiple view field angles of the flight area to be monitored based on the flight area panorama. Therefore, radar and a zoom camera are replaced by the ultrahigh-resolution camera array, so that the overhead is saved, the time delay caused by transmission of detection signals is avoided, the detection efficiency of the flight obstacles is improved, and the simultaneous monitoring of the flight obstacles in multiple areas of an airport is realized by improving the frame rate of the camera and expanding the field angle; the problem of shielding of the monocular camera is solved, the monitoring blind area is reduced, and the detection accuracy of the flight obstacle is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for detecting flight obstacles in a flight environment, a computer device, and a computer-readable storage medium. Background Art

[0002] Obstacle detection in current flight environments typically uses a combination of radar and zoom cameras to identify flying obstacles such as birds or drones. The radar scans and identifies radar signals within a certain range. When a moving target is detected, the zoom camera is mobilized to capture a visible light image of the target's area. The camera then identifies any anomalies within the visible light image and determines whether they are flying obstacles such as birds. After detecting a moving target, the radar transmits the radar signal to the zoom camera, causing it to rotate. However, the radar signal transmission process has a certain delay, and the movement trajectory of flying obstacles is random. As a result, the zoom camera often fails to capture the moving target when receiving the radar signal for dynamic target detection, resulting in failure in obstacle detection and reduced accuracy. Summary of the Invention

[0003] The main purpose of the present invention is to provide a flight obstacle detection method, device, equipment and computer-readable storage medium in a flight environment, aiming to improve the accuracy of flight obstacle detection in a flight environment.

[0004] To achieve the above object, the present invention provides a flight obstacle detection method in a flight environment, the flight obstacle detection method comprising the following steps: Using a camera array corresponding to the flight area to be monitored, regional sub-images of the flight area to be monitored at various field of view angles are collected to expand the monitoring field of view angle based on the high camera frame rate of the camera array; Based on the target transformation matrix, each of the regional sub-images is stitched together to obtain a flight area panoramic image corresponding to the flight area to be monitored, wherein the flight area panoramic image covers the entire airspace of the flight area to be monitored; Based on the obstacle detection model, the flight obstacles in the panoramic view of the flight area are monitored in real time, and the flight obstacle monitoring is completed simultaneously in the corresponding areas of multiple field of view angles of the flight area to be monitored.

[0005] In addition, to achieve the above-mentioned objectives, the present invention also provides a computer device, comprising a processor, a memory, and a flight obstacle detection program in a flight environment stored in the memory and executable by the processor, wherein when the flight obstacle detection program in the flight environment is executed by the processor, the steps of the flight obstacle detection method in a flight environment as described above are implemented.

[0006] In addition, to achieve the above-mentioned objectives, the present invention also provides a computer-readable storage medium, on which a flight obstacle detection program in a flight environment is stored. When the flight obstacle detection program in a flight environment is executed by a processor, the steps of the flight obstacle detection method in a flight environment as described above are implemented.

[0007] The present application provides a method for detecting flight obstacles in a flight environment. The method uses a camera array corresponding to a flight area to be monitored to collect regional sub-images of the flight area to be monitored at various field of view angles, thereby expanding the monitoring field of view angle based on the high camera frame rate of the camera array; based on a target transformation matrix, each of the regional sub-images is spliced ​​to obtain a flight area panorama corresponding to the flight area to be monitored, wherein the flight area panorama covers the entire airspace of the flight area to be monitored; based on an obstacle detection model, flight obstacles in the flight area panorama are monitored in real time, and flight obstacle monitoring is completed simultaneously in areas corresponding to multiple field of view angles of the flight area to be monitored. In the above manner, the present application collects regional sub-images of the flight environment from multiple perspectives through a camera array, then obtains a flight area panorama based on the splicing of the regional sub-images from multiple perspectives, and based on the flight area panorama, simultaneously monitors flight obstacles from multiple field of view angles of the flight area to be monitored. Therefore, replacing radar and zoom cameras with ultra-high-resolution camera arrays not only saves costs and avoids the delay caused by transmitting detection signals, thereby improving the detection efficiency of flight obstacles, but also increases the camera frame rate and expands the field of view, thereby realizing simultaneous monitoring of flight obstacles in multiple areas such as airports, solving the occlusion problem of monocular cameras, reducing monitoring blind spots, and improving the detection accuracy of flight obstacles. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 1. A flow chart of a first embodiment of a method for detecting flight obstacles in a flight environment according to the present invention; Figure 2 1. This is a flow chart of a second embodiment of a method for detecting flight obstacles in a flight environment according to the present invention; Figure 3 The figure is a schematic diagram of the hardware structure of the computer device involved in the embodiment of the present invention.

[0009] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0010] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0011] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0012] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0013] It should be further understood that the term “and / or” used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0014] The flight obstacle detection method in a flight environment involved in the embodiments of the present invention is mainly applied to a computer device, which may be a device with display and processing functions, such as a PC, a portable computer, or a mobile terminal.

[0015] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0016] Reference Figure 1 , Figure 1 FIG. 4 is a flow chart of a first embodiment of a method for detecting flight obstacles in a flight environment according to the present invention.

[0017] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting flight obstacles in a flight environment. The method for detecting flight obstacles in a flight environment includes steps S10 to S30.

[0018] Step S10, using a camera array corresponding to the flight area to be monitored, collecting regional sub-images of the flight area to be monitored at various field of view angles, so as to expand the monitoring field of view angle based on the high camera frame rate of the camera array; To address the high cost of radar and the time delay of radar transmission detection signals, this embodiment provides a method for detecting flying obstacles in a flight environment. The method is applied to a multi-stage filtering-tracking-recognition super-resolution real-time bird detection system. Specifically, an array camera replaces the radar + zoom camera bird detection mode. This not only reduces costs and improves the efficiency of flying obstacle detection, but also allows for simultaneous monitoring of multiple viewing angles in the flight environment, avoiding failures in flying obstacle detection due to a single viewing angle and the randomness of flying obstacle flight paths, thereby improving the accuracy of flying obstacle detection.

[0019] In this embodiment, the camera array includes multiple synchronized, high-resolution visible light or infrared cameras, covering different viewing angles and altitude levels. Preset areas can include multiple camera groups (e.g., 6-8) deployed along both sides of the runway, covering a 360° horizontal field of view. The spacing between adjacent cameras is adjusted based on the detection distance (e.g., 10-50 meters for a baseline and 1-5 kilometers for a detection range). The wide-angle cameras in the array cover the near-field area, monitoring the low-altitude layer (0-100 meters), while the telephoto cameras monitor the takeoff and landing paths, monitoring the high-altitude layer (100-1000 meters).

[0020] One or more cameras capture a sub-image of a region from a single perspective, and then stitch together these sub-images from multiple perspectives to create a comprehensive monitoring image of the entire flight environment. This multi-perspective data fusion reduces both false and missed obstacle detection rates.

[0021] Specifically, any two cameras can cover the same area, ensuring positioning even when a single view is blocked. The cameras in this embodiment can use a visible light + infrared dual-mode array mode, that is, the visible light camera is used for image acquisition during the day, and the infrared mode can be switched to image acquisition at night or in foggy environments.

[0022] In a specific embodiment, the flight obstacle detection method provided in this embodiment can be applied to flight obstacle detection scenarios around airport runways, that is, a circular array camera group is deployed in the airport, covering an airspace with a radius of 5 kilometers, to monitor the bird threats during the takeoff / landing phase in real time; it can also be applied to wind power plants for bird detection, that is, an inclined array camera is installed on the top of the wind turbine to prevent birds from hitting the blades.

[0023] Step S20: splicing the sub-images of the region based on the target transformation matrix to obtain a panoramic image of the flight region corresponding to the flight region to be monitored, wherein the panoramic image of the flight region covers the entire airspace of the flight region to be monitored; The stitching operation is performed before any recognition or tracking calculations. The purpose is to generate an ultra-high-resolution (2 billion pixel) large-scene panoramic image in real time, covering the entire monitored airspace. Not only is the field of view wide (for example, a horizontal 140-degree field of view is created by stitching together 12 horizontal rows of cameras), but because each camera has a telephoto lens and high resolution, flying obstacles such as birds about 3 kilometers away can be detected in the stitched panoramic image. The above-mentioned equipment replaces the traditional obstacle detection system consisting of radar + a single zoom long-range camera. This provides an obstacle detection system that is low-cost (because radar is very expensive) and can simultaneously monitor large scene areas and track flying obstacles appearing in multiple areas. The transformation matrix used for stitching is not calculated in real time, but is pre-calibrated during the calibration process.

[0024] Before step S20, the method further includes: Based on the target matching point set, a transformation matrix of each of the area sub-images in the flight area panoramic image is calculated as the target transformation matrix.

[0025] In this embodiment, moving objects in the flight environment surveillance video are tracked, and corresponding points of the same moving object at different times in the overlapping portions of each regional sub-image are used as matching points. Based on these matching points, a homography matrix (i.e., a homography matrix) is calculated. This avoids the problem of finding a sufficient number of accurate matching feature points in static images, which often leads to errors in perspective transformation matrix calculation, and avoids feature matching deviations caused by excessive repeated textures in the image, thereby reducing stitching errors.

[0026] In a specific embodiment, for the monitoring scenario of bird detection during aircraft flight with high real-time requirements in this embodiment, SURF (Speeded Up Robust Features) is used as a feature detector to perform matching point detection on the panoramic image and regional sub-images.

[0027] A feature detector can also be used to perform feature detection on the flight area panorama and the area sub-image, respectively, to calculate the corresponding feature points. For example, feature matching can be performed on the feature points of the flight area panorama and the area sub-image using the k-nearest neighbor matching (knnmatch) algorithm to obtain the matched feature points. These matched feature points are then filtered to ensure that the distances between the feature points are within a preset range, thereby removing mismatched points and obtaining the target matching point set. After obtaining the target matching point set, a robust RANSAC-based method is used to calculate the homography matrix, resulting in the transformation matrix T (i.e., the scale transformation matrix).

[0028] Furthermore, the target matching point set includes a static matching point set and a dynamic matching point set, and the target matching point set corresponding to the flight area panorama based on the flight environment and each area sub-image further includes: The camera array is used to monitor the flight area to be monitored from various viewing angles to obtain a monitoring video corresponding to the flight area to be monitored; Based on the feature detector, feature point detection and feature point matching are performed on each pair of video frames in the surveillance video to obtain a static matching point set; Based on a preset moving object detection module, detecting moving objects in each frame of the surveillance video, wherein the moving object detection module includes a frame difference detection module, an optical flow detection module, and a YOLO detection module; Based on a preset tracking module, obtaining the motion trajectory of the moving object in the frame image sequence, wherein the tracking module includes an optical flow tracking module and a multi-target tracking module; Extracting feature points in overlapping areas of the flight area panorama and each of the area sub-images based on the motion trajectory of the moving object as a dynamic matching point set; Based on the static matching point set and the dynamic matching point set, a transformation matrix of each of the regional sub-images in the flight area panoramic image is calculated as the target transformation matrix.

[0029] Based on the static feature matching strategy, feature point detection and matching algorithms such as SIFT, SURF, or RANSAC (Random Sample Consensus) are used for each pair of adjacent frames in the flight area panorama and each area sub-image to extract the initial matching point set as the static matching point set.

[0030] In order to avoid the problem of not being able to find enough accurate matching feature points in static images, which often leads to errors in the calculation of the perspective transformation matrix, we further use moving object detection and tracking to find more accurate matching points in the panorama of the flight area and each area sub-image to calculate the Homography matrix.

[0031] Specifically, motion detection methods, such as frame difference and optical flow, or object detection algorithms like YOLO, are used to detect moving objects in each frame. Tracking algorithms, such as optical flow tracking or SORT (Simple Online and Realtime Tracking), are then used to obtain the trajectories of the moving objects in the frame sequence. Feature points in overlapping regions are extracted based on the trajectories of the moving objects to construct a dynamic matching point set.

[0032] In a specific embodiment, erroneous matching points in the target matching point set composed of the static matching point set and the dynamic matching point set may be removed based on RANSAC. Erroneous matching points may also be removed based on the distances between the matching points in the target matching point set.

[0033] By combining the static and dynamic matching point sets, the Homography matrices corresponding to the flight area panorama and each sub-image are calculated. Using these calculated Homography matrices, each sub-image is transformed into a unified coordinate system, such as the flight area panorama, facilitating stitching of the sub-images. The stitched boundaries are then optimized (e.g., for seamless blending or gradient transitions). The video sequence is processed frame by frame to generate a complete stitched video or image. Finally, the stitched image can be further modified (e.g., for color balancing and ghosting removal) to enhance the stitched image quality.

[0034] Finally, based on each transformation matrix, each sub-image of the region is spliced ​​together to obtain a panoramic image of the flight area.

[0035] Step S30 , based on the obstacle detection model, real-time monitoring of flight obstacles in the panoramic view of the flight area is performed, and flight obstacle monitoring is performed simultaneously in areas corresponding to multiple field of view angles of the flight area to be monitored.

[0036] In this embodiment, the flight obstacles in the panoramic view of the flight area are identified based on a preset obstacle detection model, so as to complete the flight obstacle detection from multiple perspectives in the flight environment based on the panoramic view of the flight area.

[0037] In this embodiment, a perspective transformation is performed on each of the regional sub-images based on the corresponding scaling matrix T, yielding a perspective-transformed image. Each transformed regional sub-image is then cropped to remove surrounding pixels (i.e., the black edges caused by the transformation are removed). The transformed regional sub-images are sequentially placed onto corresponding areas of the flight area panorama, and a multi-perspective panoramic view of the flight area is obtained by stitching them together. Thus, to meet the needs of bird detection around aircraft, a multi-perspective collaborative detection system is implemented using an array camera. Furthermore, 3D positioning and spatiotemporal information fusion can be combined to determine the bird's trajectory and real-time position, significantly improving the accuracy and robustness of bird detection.

[0038] The preset obstacle detection models include a target tracking model, a bird motion model, a flight object classification model specifically for detecting aerial objects, and a target vision model specifically for monitoring small objects. These obstacle detection models are used to classify and track flight obstacles within the panoramic image of the flight area.

[0039] This embodiment provides a method for detecting flight obstacles in a flight environment. The method uses a camera array corresponding to a flight area to be monitored to collect regional sub-images of the flight area to be monitored at various field of view angles, thereby expanding the monitoring field of view angle based on the high camera frame rate of the camera array; based on a target transformation matrix, each of the regional sub-images is spliced ​​to obtain a flight area panorama corresponding to the flight area to be monitored, wherein the flight area panorama covers the entire airspace of the flight area to be monitored; based on an obstacle detection model, flight obstacles in the flight area panorama are monitored in real time, and flight obstacle monitoring is completed simultaneously in areas corresponding to multiple field of view angles of the flight area to be monitored. In the above manner, the present application collects regional sub-images of the flight environment from multiple perspectives through a camera array, then splices the regional sub-images from multiple perspectives to obtain a flight area panorama, and based on the flight area panorama, flight obstacle monitoring is performed simultaneously from multiple field of view angles of the flight area to be monitored. Therefore, replacing radar and zoom cameras with ultra-high-resolution camera arrays not only saves costs and avoids the delay caused by transmitting detection signals, thereby improving the detection efficiency of flight obstacles, but also increases the camera frame rate and expands the field of view, thereby realizing simultaneous monitoring of flight obstacles in multiple areas such as airports, solving the occlusion problem of monocular cameras, reducing monitoring blind spots, and improving the detection accuracy of flight obstacles.

[0040] Reference Figure 2 , Figure 2 FIG2 is a flow chart of a second embodiment of a method for detecting flight obstacles in a flight environment according to the present invention.

[0041] like Figure 2 As shown, step S30 specifically includes.

[0042] Step S31, performing target detection on the flight obstacle in the panoramic image of the flight area based on the target visual model, and determining a target detection frame containing the flight obstacle; Before performing target detection on the flight obstacles in the panoramic view of the flight area based on the target visual model, the method further includes: The resolution of the panoramic image of the flight area is scaled to a preset pixel value, and the frame rate of the panoramic image of the flight area is reduced to a preset value to save computing power.

[0043] Step S32: Tracking the target detection frame based on the target tracking model to obtain an obstacle motion trajectory corresponding to the flying obstacle in the target detection frame; Step S33: identifying the obstacle motion trajectory based on the bird motion model, and determining whether the obstacle motion trajectory in the target detection frame conforms to the bird motion trajectory; Step S34: If the trajectory matches the bird's motion trajectory, the flying obstacle in the target detection frame is identified based on the flying object category recognition model to obtain the target obstacle category. Step S35 , based on the target tracking model and the target obstacle category, the flight obstacle is tracked in a first local tracking area magnified by a first factor at a first frame rate to monitor the flight obstacle in real time.

[0044] After tracking the flight obstacle in a first local tracking area magnified by a first factor of the target detection frame at a first frame rate based on the target tracking model and the target obstacle category, the method further includes: Step S36: performing secondary target detection on the flying obstacle in the first local tracking area at a second frame rate based on the flying object category recognition model, wherein the second frame rate is lower than the first frame rate; Step S37 : When the detection result of the secondary target detection is different from the target obstacle category, the target obstacle category is used as the standard, and the flight obstacle is tracked in the local tracking area based on the target obstacle category.

[0045] Furthermore, based on the above embodiment, the method further includes: Step S38: When the tracking information of the flight obstacle corresponding to the target obstacle category is lost and the flight obstacle cannot be detected in the local tracking area, flight obstacle detection is performed in a second local tracking area magnified by a second magnification of the target detection frame, where the second magnification is smaller than the first magnification. Step S39: If the flight obstacle detection fails in the second local tracking area, the flight area panoramic image corresponding to the flight area to be monitored is re-acquired to re-detect the flight obstacle.

[0046] In this embodiment, in order to achieve real-time detection of flight obstacles in super-resolution videos (i.e., videos composed of panoramic images of the flight area with 2 billion pixels and a frame rate of 20 fps), the following specific steps are used: a. Scale the 8K footage from the 36-camera array to 1080p, then use a vision model trained specifically for small object detection to simultaneously detect targets. To conserve computing power, detection can be performed at a frame rate of 5 fps. b. Run the tracking algorithm on each detected small target detection frame to track the target. Only when the target frame matches the motion model of a flying bird will it proceed to the next step; otherwise, the target frame will be discarded. c. Magnify the target box filtered in the previous two steps by 3 times to restore the area to 8K resolution and use the dedicated model trained for detecting aerial objects to identify the flying object category; d. Further local tracking of the identified flying object at 8K resolution at a frame rate of 20 fps. The local tracking area here is the same as the target area magnified 3 times in the previous step; e. Run the model in step c at a frame rate of 5 fps in the same local area to perform target detection and compare it with the target tracking result frame. If there is any difference, the target detection result frame shall prevail and tracking shall continue; f. When tracking is lost and no flying object is detected in the current local area, the area is expanded by 2 times for flying object detection; g. If failed, return to the first step and restart the test.

[0047] Specifically, to further improve obstacle detection efficiency, the camera array's image is scaled from 8K to 1080p. Then, combined with a visual model specifically trained for small object detection, the system simultaneously detects objects within the flight area by stitching together sub-images of corresponding areas from multiple viewpoints. To conserve computing power, detection can be performed at a lower frame rate (e.g., 5 fps).

[0048] In a specific embodiment, because the stitched image itself is ultra-high resolution, the visual model in this embodiment uses a method of sampling at different resolutions to monitor flying obstacles such as birds in real time on ultra-high resolution images, from coarse to fine. This eliminates the need to reconstruct resolution through algorithms to restore details, further saving computing resources. Furthermore, in order to train a visual model for detecting small objects, the following are specifically included: a. Use a multi-scale feature fusion (feature pyramic network) network structure as the network structure of the visual model; b. Incorporating an attention mechanism into the visual model and introducing a channel / spatial attention module to further highlight the features of small objects while maintaining real-time performance; c. During the training process of the visual model, collect and label more small object graphics consistent with the application scenario to obtain a training data set to improve the recognition rate of the model's equivalent network for small objects in various scenarios.

[0049] First, based on the target vision model, obstacles in the panoramic image of the flight area are detected, and a target detection frame containing the flying obstacles is determined. Then, based on the target tracking module, a tracking algorithm is run on each detected small target detection frame to track the obstacle's motion trajectory. Based on the bird motion model, the obstacle's motion trajectory is identified, and optical flow can be used to distinguish between flying birds (i.e., irregular motion trajectories) and drones or aircraft (i.e., regular motion trajectories).

[0050] If the target within the target detection frame conforms to the motion rules of a flying bird, the bird is further classified. Otherwise, the target detection frame is discarded. For the filtered target detection frame containing a flying bird, the image region of the target detection frame is magnified 3 times and image processing is performed, such as restoring the 8K resolution to improve image quality. Then, using a dedicated model trained for detecting flying objects in the air, the bird in the processed image is classified as a flying object.

[0051] Specifically, the motion trajectory of the flying bird can be further used to calculate the predicted motion path of the flying bird, and based on the predicted motion path, determine whether the flying bird will have an impact on the flight path of the aircraft.

[0052] Further, based on the above embodiment, after identifying the obstacle motion trajectory based on the bird motion model and determining whether the obstacle motion trajectory in the target detection frame conforms to the bird motion trajectory, the method further includes: If the trajectory of the bird is consistent, the target spatial position of the bird is calculated based on the multi-camera parallax algorithm, the regional sub-images of each field of view angle, and the obstacle trajectory; calculating the shortest distance between the bird and each aircraft path based on the target spatial position, and determining a bird warning level based on the shortest distance; Based on the bird warning level, a target warning method for the bird is determined, and a bird warning is performed based on the target warning method.

[0053] The step of calculating the shortest distance between the bird and each aircraft flight path based on the target spatial position and determining the bird warning level based on the shortest distance includes: obtaining a flight speed of the bird, and determining that the bird warning level is low risk when the flight speed is lower than a minimum speed threshold and the shortest distance is greater than a preset distance threshold; When the flight speed is higher than a minimum speed threshold and lower than a maximum speed threshold, and the shortest distance is greater than a preset distance threshold, determining that the bird warning level is medium risk; When the flight speed is higher than a maximum speed threshold and the shortest distance is less than a preset distance threshold, the bird warning level is determined to be high risk.

[0054] In this embodiment, multi-camera parallax is obtained from regional sub-images corresponding to multiple perspectives, and the spatial position (including but not limited to distance, altitude, and speed) of the bird is calculated using this multi-camera parallax. The bird's flight speed and the minimum distance from each aircraft path according to the bird's motion pattern are then used to determine the bird's warning level. The bird's flight speed is the bird's approach speed relative to the aircraft path, which is the relative speed between the bird and the aircraft path (i.e., vector composite). The minimum distance is the distance between the bird's position predicted within a preset time period (e.g., 10 seconds or 1 minute) based on the bird's motion trajectory and the aircraft path. The smaller the minimum distance, the greater the risk; the higher the aircraft speed, the greater the risk.

[0055] Specifically, bird warning levels include low, medium, and high risk. For high risk, laser bird repellent or drones can be used for emergency repelling. For medium risk, an acoustic warning or a tower notification can be issued. For low risk, no action is required, but the bird's movement data is recorded for subsequent model training.

[0056] In a specific embodiment, the bird detection frequency may be further dynamically adjusted according to the bird warning level. For example, the higher the bird warning level, the higher the detection frequency.

[0057] Furthermore, if the target in the target detection frame is a flying bird, after generating the flying bird warning information, the method includes: If the target in the target detection frame is a flying bird, performing image processing on the local area image where the flying bird is located using preset image parameters; In the processed local area image, locally tracking the flying bird to obtain a local tracking result frame; When the tracking target of the target detection frame is lost, the flying bird is tracked based on the local tracking result frame.

[0058] In this embodiment, after determining the target detection frame containing the flying bird, the telephoto camera is used to partially zoom in on the area where the target detection frame is located to facilitate better target tracking. Specifically, the identified flying object is further locally tracked at an 8K resolution at a frame rate of 20fps. The local tracking area is consistent with the magnified area (such as 2x or 3x) of the target detection frame. The bird motion model and tracking model are run in this local area at a frame rate of 5fps to perform target detection and tracking, and compared with the target tracking result frame. When the tracking of the target detection frame is lost, the flying bird can be tracked based on the local tracking result frame.

[0059] In a specific embodiment, when no flying object is detected in the target detection frame or the local tracking result frame, flying object detection is performed on an area that is doubled in size from the target detection frame. If the detection fails, the process returns to the first step and starts detection again.

[0060] Furthermore, the step of stitching the sub-images of the region based on the transformation matrices to obtain a panoramic view of the flight region includes: Based on the transformation matrices corresponding to the regional sub-images, a rectangular area of ​​each regional sub-image on the panoramic image of the flight area is calculated; Determining a maximum enclosing rectangular area among the rectangular areas on the flight area panoramic image, and cropping the flight area panoramic image based on the maximum enclosing rectangular area; Enlarging the cropped flight area panoramic image, and performing feature point matching on each of the region sub-images and the cropped and enlarged flight area panoramic image to obtain a transformation matrix of each of the region sub-images on the cropped and enlarged flight area panoramic image; Based on the transformation matrix of each of the area sub-images on the cropped and enlarged flight area panoramic image, each of the area sub-images is spliced ​​to obtain the flight area panoramic image.

[0061] In this embodiment, after obtaining the transformation matrix T corresponding to the flight area panoramic map and each regional sub-image, a perspective transformation is performed on the four points of each regional sub-image according to the transformation matrix T to obtain the four points of each regional sub-image on the flight area panoramic map, and a circumscribed rectangle is calculated for these four points in the flight area panoramic map to obtain a mapping frame of each regional sub-image on the flight area panoramic map.

[0062] Specifically, the circumscribed enclosing matrix is ​​calculated based on the area of ​​each regional sub-image on the panorama (such as the first mapping frame), and the boundRect area of ​​all regional sub-images on the panorama is obtained (such as the second mapping frame including all first mapping frames). The part of the area corresponding to the second mapping frame in the panorama is cropped and enlarged. Among them, the enlargement formula is (3840*a+200, 2160*b+200); the value corresponding to +200 can be appropriately adjusted according to the size of the black edge after the actual sub-image transformation, so as to reserve it for subsequent black edge cropping, so that the resolution of the spliced ​​result image reaches (3840*a, 2160*b). After enlarging the panorama, perform feature detection and matching on each sub-image again to calculate the corresponding transformation matrix. The obtained transformation matrix is ​​the enlarged one, and the transformed sub-image also has the enlarged resolution. That is: First, perform the first feature matching between the flight area panorama and each area sub-image: Perform feature point matching on each sub-image (i.e., n sub-images) with the panoramic image. Calculate the panorama / sub-image transformation matrix T for each sub-image on the flight area panorama. Then, use the transformation matrix T to calculate the rectangular area of ​​each sub-image on the panorama. Determine the largest enclosing rectangular area based on the rectangular areas of the n sub-images. This largest enclosing rectangular area includes all the rectangular areas of the n sub-images. Record the panorama / sub-image transformation matrix T, the rectangular area rect, and the largest enclosing rectangular area for each sub-image.

[0063] Then the rectangular area in the panorama containing each region sub-image is enlarged: Crop the panorama based on the largest enclosing rectangular area to obtain a cropped image. Enlarge the cropped image to (e.g., 1920*b+200, 1080*a+200). By enlarging the cropped flight area panorama (+200), we prevent the stitched image from exceeding the edges when subsequently matching the flight area panorama with the individual regional sub-images. This preserves the cropped edges of the subsequent stitching result to increase the resolution of the final stitched image (e.g., 1920*b, 1080*a).

[0064] Then perform a second feature matching on the cropped flight area panorama and each area sub-image: Match feature points of each of the n sub-images with the cropped and enlarged panorama. Calculate the scale transformation matrix T_scale for each sub-image on the panorama of the flight area. Use the scale transformation matrix T_scale to calculate the scaled rectangular area rect_scale of each sub-image on the panorama. Record the scale transformation matrix T_scale and the scaled rectangular area rect_scale for each sub-image and the panorama.

[0065] In more embodiments, in order to avoid enlarging the panoramic image and performing feature detection, matching and transformation matrix calculation again, especially at a high resolution, the feature detection matching and transformation matrix calculation is very time-consuming, and in order to further reduce the calculation time of the transformation matrix, the enlargement scale factor is calculated according to the area of ​​each sub-image on the panoramic image (the first mapping frame). The enlargement scale factor is calculated by finding the minimum width and height in all area frames (to ensure that the minimum can be scaled to the specified resolution (3840, 2160), and the large ones can be cropped), and the enlargement scale factor is obtained by calculating the scale ratio based on the minimum width and height and (target width and height + edge increment). The edge increment is used for black edge cropping after subsequent transformation, so that each sub-image can reach the target resolution (3840, 2160) after cropping. The calculation formula of the enlargement scale factor is: Sfactor=( min(rect.witdh) / (Wtarget+△W), min(rect.height) / (Htarget+△H) ).

[0066] After calculating the scale factor, the scaling factor parameter in the transformation matrix is ​​replaced with the scale factor, and the sub-image is transformed using the transformation matrix, which is the enlarged transformation result.

[0067] Through the above method, the preparation for image stitching is completed by processing the feature points of the image, calculating the homography matrix (Homography), matching the rectangular frame, and scaling the image. The stitching parameters generated and adjusted in the above process are saved in the calibration file. Among them: Feature point extraction and matching: For the reference image (i.e., the flight area panorama) and each sub-image (i.e., each sub-image of the area), feature points and descriptors are extracted and then matched. Successfully matched feature points are used to calculate the homography matrix.

[0068] Compute homography: Calculate the homography between the reference image (e.g., a panorama of the flight area) and the sub-images using feature matching.

[0069] Rectangle processing: Calculate the matching rectangle for each image. Calculate the enclosing rectangle of all rectangles as the overall layout of the stitched image.

[0070] Image scaling: Adjusts the images so that they fit into the stitched panorama by calculating a scaling factor. The scaling factor is calculated based on the minimum size of the matching rectangles.

[0071] Display rectangle: Calculate the final display position based on the scaled rectangle and calculate the appropriate display area for the stitched image.

[0072] Saving stitching parameters: The resulting stitching parameters (including homography, rectangle, scaling factor, etc.) are saved to a calibration file for subsequent image stitching. For example, the panorama / sub-image transformation matrix T, the rectangular region rect, and the largest enclosing rectangular region recorded from the first feature matching, and the scale transformation matrix T_scale and scale rectangular region rect_scale recorded from the second feature matching are recorded as calibration parameters in the calibration file. These calibration parameters can then be applied to obtain the transformed sub-images. Specifically, each sub-image is perspective-transformed according to its scale transformation matrix T to obtain the transformed image. Each transformed sub-image is then cropped to remove surrounding pixels (to remove any black edges caused by the transformation), resulting in the final transformed n sub-images.

[0073] See also Figure 3 , Figure 3 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0074] See Figure 3 The computer device includes a processor, a memory, and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0075] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, which, when executed, can enable the processor to execute a flight obstacle detection method in any flight environment.

[0076] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0077] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute the flight obstacle detection method in any flight environment.

[0078] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0079] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0080] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps: By deploying array cameras in a preset area, a regional sub-image of the flight environment in at least one viewing angle is collected; Based on the flight area panoramic map of the flight environment and the target matching point set corresponding to each area sub-image, calculating the transformation matrix of each area sub-image on the flight area panoramic map; Based on each transformation matrix, each of the sub-images of the area is spliced ​​together to obtain a panoramic image of the flight area, and flight obstacles in the panoramic image of the flight area are identified based on a preset obstacle detection model, so as to complete flight obstacle detection from multiple perspectives in the flight environment based on the panoramic image of the flight area.

[0081] In one embodiment, the processor is configured to run a computer program stored in the memory and further configured to implement: Based on the object recognition module in the obstacle detection model, perform object detection on the panoramic image of the flight area to determine a target detection frame containing the object; Tracking the target detection frame based on the target tracking module in the obstacle detection model to obtain the obstacle motion trajectory; Based on the bird motion module in the obstacle detection model, the obstacle motion trajectory is identified to determine whether the target in the target detection frame is a bird; If the target in the target detection frame is a flying bird, bird warning information is generated.

[0082] In one embodiment, the processor is configured to run a computer program stored in the memory and further configured to implement: Calculating the target spatial position of the bird based on a multi-camera parallax algorithm, the regional sub-images captured by the array camera, and the obstacle motion trajectory; calculating the shortest distance between the bird and each aircraft path based on the target spatial position, and determining a bird warning level based on the shortest distance; Based on the bird warning level, a target warning method for the bird is determined, and the bird warning information is generated based on the target warning method.

[0083] In one embodiment, the processor is configured to run a computer program stored in the memory and further configured to implement: obtaining a flight speed of the bird, and determining that the bird warning level is low risk when the flight speed is lower than a minimum speed threshold and the shortest distance is greater than a preset distance threshold; When the flight speed is higher than a minimum speed threshold and lower than a maximum speed threshold, and the shortest distance is greater than a preset distance threshold, determining that the bird warning level is medium risk; When the flight speed is higher than a maximum speed threshold and the shortest distance is less than a preset distance threshold, the bird warning level is determined to be high risk.

[0084] In one embodiment, the processor is configured to run a computer program stored in the memory and further configured to implement: If the target in the target detection frame is a flying bird, performing image processing on the local area image where the flying bird is located using preset image parameters; In the processed local area image, locally tracking the flying bird to obtain a local tracking result frame; When the tracking target of the target detection frame is lost, the flying bird is tracked based on the local tracking result frame.

[0085] In one embodiment, the processor is configured to run a computer program stored in the memory and further configured to implement: Based on the transformation matrices corresponding to the regional sub-images, a rectangular area of ​​each regional sub-image on the panoramic image of the flight area is calculated; Determining a maximum enclosing rectangular area among the rectangular areas on the flight area panoramic image, and cropping the flight area panoramic image based on the maximum enclosing rectangular area; Enlarging the cropped flight area panoramic image, and performing feature point matching on each of the region sub-images and the cropped and enlarged flight area panoramic image to obtain a transformation matrix of each of the region sub-images on the cropped and enlarged flight area panoramic image; Based on the transformation matrix of each of the area sub-images on the cropped and enlarged flight area panoramic image, each of the area sub-images is spliced ​​to obtain the flight area panoramic image.

[0086] In one embodiment, the processor is configured to run a computer program stored in the memory and further configured to implement: The flight environment is monitored from various viewing angles by the array camera to obtain a monitoring video corresponding to the flight environment; Based on the feature detector, feature point detection and feature point matching are performed on each pair of video frames in the surveillance video to obtain the static matching point set; Based on a preset moving object detection module, detecting moving objects in each frame of the surveillance video, wherein the moving object detection module includes a frame difference detection module, an optical flow detection module, and a YOLO detection module; Based on a preset tracking module, obtaining the motion trajectory of the moving object in the frame image sequence, wherein the tracking module includes an optical flow tracking module and a multi-target tracking module; Based on the motion trajectory of the moving object, feature points in the overlapping area of ​​the flight area panorama and each area sub-image are extracted as the dynamic matching point set.

[0087] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program includes program instructions. The processor executes the program instructions to implement any flight obstacle detection method in a flight environment provided in any embodiment of the present application.

[0088] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a flash memory card, etc., equipped on the computer device.

[0089] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for detecting flight obstacles in a flight environment, characterized in that: The flight obstacle detection method comprises the following steps: Using a camera array corresponding to the flight area to be monitored, regional sub-images of the flight area to be monitored at various field of view angles are captured, thereby expanding the monitoring field of view angle based on the high camera frame rate of the camera array; Based on the target transformation matrix, each of the regional sub-images is stitched together to obtain a flight area panoramic image corresponding to the flight area to be monitored, wherein the flight area panoramic image covers the entire airspace of the flight area to be monitored; Based on the obstacle detection model, the flight obstacles in the panoramic view of the flight area are monitored in real time, and flight obstacle monitoring is completed simultaneously in the corresponding areas of multiple field of view angles of the flight area to be monitored.

2. The flight obstacle detection method in a flight environment according to claim 1, wherein: The obstacle detection model includes a target tracking model, a bird motion model, a flight object classification model for specifically detecting aerial objects, and a target vision model for specifically monitoring small objects. The obstacle detection model is used to perform real-time monitoring of flight obstacles in the panoramic view of the flight area, including: performing target detection on the flight obstacle in the panoramic image of the flight area based on the target visual model, and determining a target detection frame containing the flight obstacle; Tracking the target detection frame based on the target tracking model to obtain an obstacle motion trajectory corresponding to the flying obstacle in the target detection frame; Based on the bird motion model, the obstacle motion trajectory is identified to determine whether the obstacle motion trajectory in the target detection frame conforms to the bird motion trajectory; If the target obstacle meets the bird's motion trajectory, the target obstacle category is identified based on the flying object category recognition model to obtain the target obstacle category. Based on the target tracking model and the target obstacle category, the flight obstacle is tracked in a first local tracking area that is magnified by a first factor in the target detection frame at a first frame rate to monitor the flight obstacle in real time.

3. The flight obstacle detection method in a flight environment according to claim 2, wherein: After tracking the flight obstacle in a first local tracking area magnified by a first factor by the target detection frame at a first frame rate based on the target tracking model and the target obstacle category, the method further includes: performing secondary target detection on the flying obstacle in the first local tracking area at a second frame rate based on the flying object category recognition model, wherein the second frame rate is lower than the first frame rate; When the detection result of the secondary target detection is different from the target obstacle category, the target obstacle category is taken as the standard, and the flight obstacle is tracked in the local tracking area based on the target obstacle category.

4. The flight obstacle detection method in a flight environment according to claim 3, wherein: The method further comprises: When the tracking information of the flight obstacle corresponding to the target obstacle category is lost and the flight obstacle cannot be detected in the local tracking area, performing flight obstacle detection in a second local tracking area that is magnified by a second magnification of the target detection frame, where the second magnification is smaller than the first magnification; If the flight obstacle detection fails in the second local tracking area, the flight area panoramic image corresponding to the flight area to be monitored is re-acquired to re-detect the flight obstacle.

5. The flight obstacle detection method in a flight environment according to claim 2, wherein: Before performing target detection on the flight obstacles in the panoramic view of the flight area based on the target visual model, the method further includes: The resolution of the panoramic image of the flight area is scaled to a preset pixel value, and the frame rate of the panoramic image of the flight area is reduced to a preset value to save computing power.

6. The flight obstacle detection method in a flight environment according to claim 2, wherein: After identifying the obstacle motion trajectory based on the bird motion model and determining whether the obstacle motion trajectory in the target detection frame conforms to the bird motion trajectory, the method further includes: If the trajectory of the bird is consistent, the target spatial position of the bird is calculated based on the multi-camera parallax algorithm, the regional sub-images of each field of view angle, and the obstacle trajectory; calculating the shortest distance between the bird and each aircraft path based on the target spatial position, and determining a bird warning level based on the shortest distance; Based on the bird warning level, a target warning method for the bird is determined, and a bird warning is performed based on the target warning method.

7. The flight obstacle detection method in a flight environment according to claim 6, wherein: The step of calculating the shortest distance between the bird and each aircraft flight path based on the target spatial position, and determining the bird warning level based on the shortest distance, includes: obtaining a flight speed of the bird, and determining that the bird warning level is low risk when the flight speed is lower than a minimum speed threshold and the shortest distance is greater than a preset distance threshold; When the flight speed is higher than a minimum speed threshold and lower than a maximum speed threshold, and the shortest distance is greater than a preset distance threshold, determining that the bird warning level is medium risk; When the flight speed is higher than a maximum speed threshold and the shortest distance is less than a preset distance threshold, the bird warning level is determined to be high risk.

8. The flight obstacle detection method in a flight environment according to any one of claims 1 to 7, characterized in that: The step of splicing the sub-images of the region based on the target transformation matrix to obtain a panoramic view of the flight region corresponding to the flight region to be monitored includes: The camera array is used to monitor the flight area to be monitored from various viewing angles to obtain a monitoring video corresponding to the flight area to be monitored; Based on the feature detector, feature point detection and feature point matching are performed on each pair of video frames in the surveillance video to obtain a static matching point set; Based on a preset moving object detection module, detecting moving objects in each frame of the surveillance video, wherein the moving object detection module includes a frame difference detection module, an optical flow detection module, and a YOLO detection module; Based on a preset tracking module, obtaining the motion trajectory of the moving object in the frame image sequence, wherein the tracking module includes an optical flow tracking module and a multi-target tracking module; Extracting feature points in overlapping areas of the flight area panorama and each of the area sub-images based on the motion trajectory of the moving object as a dynamic matching point set; Based on the static matching point set and the dynamic matching point set, a transformation matrix of each of the regional sub-images in the flight area panoramic image is calculated as the target transformation matrix.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a flight obstacle detection program in a flight environment stored in the memory and executable by the processor, wherein when the flight obstacle detection program in the flight environment is executed by the processor, the steps of the flight obstacle detection method in a flight environment described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a flight obstacle detection program in a flight environment, wherein when the flight obstacle detection program in a flight environment is executed by a processor, the steps of the flight obstacle detection method in a flight environment according to any one of claims 1 to 7 are implemented.

Images