Obstacle detection method and apparatus therefor, storage medium

Abstract

The application relates to an obstacle detection method and device and a storage medium, which comprises the following steps: receiving m groups of small field of view images; performing image splicing processing on multiple frames of small field of view images in each group of small field of view images respectively to obtain m frames of large field of view images corresponding to m array cameras; performing small target detection on the m frames of large field of view images to obtain m groups of first obstacle target information, wherein each group of first obstacle target information comprises first spatial positions, speed vectors and obstacle types of h obstacle targets; calculating m groups of second obstacle target information according to the m frames of large field of view images, wherein each group of second obstacle information comprises second spatial positions of the h obstacle targets; and performing fusion processing on the speed vectors and the obstacle types in the m groups of first obstacle target information and the second spatial positions in the m groups of second obstacle target information respectively to obtain the speed vectors, the obstacle types and the spatial positions of the obstacle targets; and the application can improve small target detection precision.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle technology, specifically to an obstacle detection method, device, and storage medium. Background Technology

[0002] In the future urban air traffic, autonomous driving is the most important technology for achieving efficient and safe operation of unmanned aircraft. However, the development of urban low-altitude transportation in China is slow and the management of low-altitude airspace is lagging behind. The autonomous driving of unmanned aircraft is threatened by flocks of birds, drones, ground obstacles and weather conditions. It lacks cost-effective and reliable perception methods, which seriously hinders its development and application.

[0003] Currently, obstacle detection in the field of unmanned aerial vehicles (UAVs) mainly relies on two types of perception methods: First, environmental perception is achieved through cameras, represented by consumer / professional UAVs. These devices use single or dual cameras to identify and locate targets, but they suffer from drawbacks such as short detection range and low reliability, making them unsuitable for manned UAVs. Second, environmental perception is achieved through millimeter-wave radar, represented by small airborne phased array radars. However, this technology is currently mainly used in the military field, lacking suitable products for small civilian manned aircraft. Its high price makes it difficult to promote its application in the short term. Summary of the Invention

[0004] The purpose of this invention is to propose an obstacle detection method, device, and storage medium to solve the technical problem of small target detection for unmanned aerial vehicles, improve detection accuracy, and reduce detection costs.

[0005] To achieve the above objectives, a first aspect of the present invention provides an obstacle detection method, the method comprising:

[0006] Receive m sets of small field-of-view images captured by m array cameras; wherein, the m array cameras are set at the front of the aircraft, each array camera includes multiple cameras arranged in an array, the multiple cameras are used to capture the environment in front of the aircraft to obtain corresponding multi-frame small field-of-view images, m>1;

[0007] Each group of small field-of-view images is processed by image stitching of multiple frames of small field-of-view images to obtain m frames of large field-of-view images corresponding to the m array cameras.

[0008] The m frames of large field-of-view images are subjected to small target detection according to a pre-set small target detection algorithm to obtain m sets of first obstacle target information. Each set of first obstacle target information includes the first spatial position, velocity vector and obstacle type of h obstacle targets; where h≥0;

[0009] Based on the pre-set dual-camera positioning algorithm and the m frames of large field-of-view images, m sets of second obstacle target information are calculated, and each set of second obstacle information includes the second spatial positions of h obstacle targets;

[0010] The velocity vector, obstacle type, and spatial position of the obstacle target are obtained by fusing the velocity vector and obstacle type in the first set of m sets of first obstacle target information and the second spatial position in the second set of m sets of second obstacle target information.

[0011] Preferably, the step of performing small target detection on the m frames of the large field-of-view image according to a pre-set small target detection algorithm to obtain m sets of first obstacle target information includes:

[0012] The pixel clusters in the m-frame large field-of-view images are detected to obtain corresponding m sets of suspected obstacle information, and each set of suspected obstacle information includes the first spatial position, velocity vector and obstacle type of n suspected obstacles; wherein, the pixel clusters include multiple adjacent pixels, n≥h≥0;

[0013] Obtain aircraft flight route information;

[0014] Based on the flight route information and the m sets of suspected obstacle information, the corresponding m sets of first obstacle target information are obtained; wherein, if any suspected obstacle interferes with the flight of the aircraft, then any suspected obstacle is an obstacle target, and n≥h≥0.

[0015] Preferably, the step of detecting pixel clusters in the m-frame large field-of-view images to obtain corresponding m sets of suspected obstacle information includes:

[0016] Each of the m frames of the large field-of-view image is detected to obtain m corresponding groups of pixel clusters, and each group of pixel clusters includes f pixel clusters; where f≥n;

[0017] Each group of pixels is filtered to obtain m groups of suspected obstacle information; if the size of any pixel group meets the preset requirements, then the pixel group is determined to be a suspected obstacle, and the first spatial position, velocity vector and obstacle type of the suspected obstacle are calculated.

[0018] Preferably, the first spatial position, velocity vector, and obstacle type of the suspected obstacle are calculated as follows:

[0019] For any suspected obstacle in any set of suspected obstacle information, acquire the current time and the two frames before and after the previous time of any array camera corresponding to that suspected obstacle;

[0020] The first spatial position and velocity vector of the aircraft are obtained. Based on the position of any suspected obstacle in the two preceding and following large field-of-view images, and the first spatial position and velocity vector of the aircraft, the first spatial position and velocity vector of the suspected obstacle are obtained. The obstacle type of the suspected obstacle is determined based on the velocity vector of the suspected obstacle.

[0021] Preferably, each group of first obstacle target information further includes h existence probabilities of obstacle targets;

[0022] The step of performing small target detection on the m frames of large field-of-view images according to a pre-set small target detection algorithm to obtain m groups of first obstacle target information also includes:

[0023] The similarity between any two obstacle targets in each set of first obstacle target information is calculated based on their first spatial position and velocity vectors, thereby obtaining the similarity between any obstacle target in each set of first obstacle target information and other obstacle targets.

[0024] The probability of the existence of any obstacle target in each group of first obstacle target information is determined based on the similarity between any obstacle target and other obstacle targets, thereby obtaining the probability of the existence of h obstacle targets in m groups of first obstacle target information.

[0025] Preferably, the similarity between any two obstacle targets is calculated based on their first spatial positions and velocity vectors in each group of first obstacle target information, specifically calculated in the following manner:

[0026] X 1-2 = (P1 / P2)*a + (V1 / V2)*b

[0027] Among them, X 1-2 Let P1 be the first spatial position of the first obstacle target and P2 be the second spatial position of the second obstacle target, V1 be the velocity vector of the first obstacle target and V2 be the velocity vector of the second obstacle target, and a and b be pre-set weighting coefficients.

[0028] Preferably, the step of fusing the velocity vector, obstacle type, and spatial position of the obstacle target based on the velocity vector and obstacle type in the m sets of first obstacle target information and the second spatial position in the m sets of second obstacle target information further includes:

[0029] The existence probabilities of h obstacle targets in the m groups of first obstacle target information are fused to obtain the existence probabilities of h obstacle targets.

[0030] Preferably, receiving m sets of small field-of-view images captured by m array cameras includes:

[0031] Receive status signals from k array cameras output by the camera monitoring system; wherein the status signals are normal or abnormal;

[0032] Based on the status signals of the k array cameras, determine the m array cameras whose status signals are normal, and receive m sets of small field-of-view images captured by the m array cameras; where k≥m.

[0033] A second aspect of the present invention also provides an obstacle detection device for implementing the obstacle detection method described in the first aspect, the device comprising:

[0034] An image receiving module is used to receive m sets of small field-of-view images captured by m array cameras; wherein, the m array cameras are set at the front of the aircraft, each array camera includes multiple cameras arranged in an array, the multiple cameras are used to capture the environment in front of the aircraft to obtain corresponding multi-frame small field-of-view images, m>1;

[0035] The image processing module is used to perform image stitching processing on multiple frames of small field-of-view images in each group of small field-of-view images to obtain m frames of large field-of-view images corresponding to the m array cameras.

[0036] The target detection module is used to perform small target detection on the m frames of large field of view images according to a pre-set small target detection algorithm to obtain the first spatial position, velocity vector and obstacle type of m groups of obstacle targets;

[0037] The dual-camera positioning module is used to calculate the second spatial position of m groups of obstacle targets based on a pre-set dual-camera positioning algorithm and the m frames of large field-of-view images;

[0038] The fusion processing module is used to perform fusion processing based on the velocity vectors and obstacle types of the m groups of obstacle targets and the second spatial positions of the m groups of obstacle targets to obtain the spatial position, velocity vector, and obstacle type of the obstacle targets.

[0039] A third aspect of the invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the obstacle detection method as described in the first aspect above.

[0040] Compared with the prior art, the present invention has at least the following beneficial effects:

[0041] The aircraft is equipped with m array cameras at the front. During flight, the aircraft automatically receives m sets of small field-of-view images captured by these m array cameras. Multiple frames of small field-of-view images in each set are stitched together to obtain m corresponding large field-of-view images. A pre-set small target detection algorithm is used to detect small targets in the m large field-of-view images to obtain m sets of first obstacle target information. Each set of first obstacle target information includes the first spatial position, velocity vector, and obstacle type of h obstacle targets. A pre-set dual-camera positioning algorithm and the m large field-of-view images are used to calculate m sets of second obstacle target information. Each set of second obstacle target information includes the second spatial position of h obstacle targets. Finally, the velocity vectors, obstacle types, and second spatial positions of the m sets are fused to obtain the velocity vectors, obstacle types, and spatial positions of the obstacle targets. Based on the above description, this invention employs a redundant array of m cameras and proposes methods for small target detection and result fusion using images from each of the m array cameras. Furthermore, it proposes two localization methods combining camera-based image localization, achieving reliable obstacle detection and localization for aircraft based on cameras. This method offers higher reliability than traditional camera-based perception methods and lower cost compared to millimeter-wave radar perception methods. Therefore, this invention overcomes the limitation of existing cameras where resolution and detection range cannot be simultaneously maximized, providing a high-performance, highly reliable solution for the perception of autonomous flight systems in unmanned aerial vehicles.

[0042] It is easy to promote and has very high application value, which can effectively promote the development of manned unmanned aircraft.

[0043] Other features and advantages of the present invention will be set forth in the following detailed description. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of an obstacle detection method according to one embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram showing the distance relationship between three identical array cameras in one embodiment of the present invention.

[0047] Figure 3 This is a frame structure diagram of an obstacle detection device according to one embodiment of the present invention. Detailed Implementation

[0048] The various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. Furthermore, numerous specific details are set forth in the following detailed embodiments to better illustrate the invention. Those skilled in the art will understand that the invention can be practiced without certain specific details. In some instances, means well known to those skilled in the art have not been described in detail in order to highlight the spirit of the invention.

[0049] One embodiment of the present invention provides an obstacle detection method that can be applied to unmanned aerial vehicles. (See attached document.) Figure 1 The method in this embodiment includes the following steps:

[0050] Step S1: Receive m sets of small field-of-view images captured by m array cameras; wherein, the m array cameras are set at the front of the aircraft, each array camera includes multiple cameras arranged in an array, the multiple cameras are used to capture the environment in front of the aircraft to obtain corresponding multi-frame small field-of-view images, m>1;

[0051] Specifically, the value of m is not limited to a single value and can be set according to specific needs; this embodiment uses m=3 as an example for illustration to explain the content of the invention; when m=3, as... Figure 2 As shown, the three identical array cameras, namely camera 1, camera 2, and camera 3, satisfy the following geometric relationship: with the front of the aircraft as the projection direction, after the three array cameras are projected onto this direction, the distances between camera 1, camera 2, and camera 3 are L1, L2, and L3, respectively; distances L1, L2, and L3 are used for subsequent second spatial positioning.

[0052] For example, the number of cameras included in each array camera is not limited to a certain value. Preferably, in this embodiment, each array camera may include more than 10 cameras. This embodiment uses 10 cameras per array camera as an example to illustrate the content of the present invention. Each camera can capture a frame of a small field of view image. The field of view represents the maximum range that the camera can observe, usually expressed in angle. The larger the field of view, the larger the observation range.

[0053] Step S2: Perform image stitching on multiple frames of small field-of-view images in each group of small field-of-view images to obtain m frames of large field-of-view images corresponding to the m array cameras.

[0054] As mentioned above, m=3, and an array camera includes 10 cameras. Each frame of a large field-of-view image is obtained by stitching together 10 frames of small field-of-view images from a corresponding array camera. The image stitching method is not limited to a particular method. The purpose of this embodiment is simply to obtain a large field-of-view image.

[0055] Step S3: Perform small target detection on the m frames of the large field of view image according to the pre-set small target detection algorithm to obtain m sets of first obstacle target information. Each set of first obstacle target information includes the first spatial position, velocity vector and obstacle type of h obstacle targets; where h≥0;

[0056] Specifically, in this embodiment, the small target detection algorithm is not limited to one type. It can be implemented based on a deep learning neural network model. Small target detection is achieved by pre-training a deep learning-based neural network model. Theoretically, each frame of a large field of view image will obtain h obstacle targets after small target detection. That is to say, each group of first obstacle target information will include h obstacle targets, but their first spatial position, velocity vector and obstacle type may be slightly different.

[0057] Step S4: Calculate m sets of second obstacle target information based on the pre-set dual-camera positioning algorithm and the m frames of large field of view images. Each set of second obstacle information includes the second spatial positions of h obstacle targets.

[0058] Specifically, the large field-of-view images of any two array cameras and the dual-camera positioning algorithm can obtain a second spatial position. As mentioned above, when m=3, dual-camera positioning is performed between each pair of the three large field-of-view images to obtain three second spatial positions for each obstacle target.

[0059] It is understandable that the position of the obstacle target obtained based on the dual-camera positioning algorithm is actually the relative position of the suspected obstacle and the aircraft. Combined with the spatial position of the aircraft itself given by the aircraft's positioning system, the second spatial position of the suspected obstacle can be further obtained.

[0060] Step S5: Based on the velocity vector and obstacle type in the m sets of first obstacle target information, and the second spatial position in the m sets of second obstacle target information, perform fusion processing to obtain the velocity vector, obstacle type, and spatial position of the obstacle target;

[0061] Specifically, there are many algorithms for data fusion. In this embodiment, the preferred but not limited to Kalman filter fusion algorithm is used to fuse multiple types of data. That is, m velocity vectors of the same obstacle target are fused to obtain one velocity vector, m obstacle types of the same obstacle target are fused to obtain one obstacle type, and m second spatial positions of the same obstacle target are fused to obtain one spatial position. Finally, after obtaining the velocity vectors, obstacle types, and spatial positions of one or more obstacle targets, the data is output to the aircraft's flight decision system to provide a reference for the aircraft's flight decision.

[0062] It should be noted that this embodiment employs m array cameras with redundant design, proposes to perform small target detection and detection result fusion on the detection images of the m array cameras respectively, and also proposes two localization methods that combine camera image-based localization. This achieves higher reliability in detecting and locating aircraft obstacles based on cameras. Compared with traditional camera perception methods, this method is more reliable and lower in cost than millimeter-wave radar perception methods. Therefore, the method in this embodiment is easy to promote and has very high application value.

[0063] It should also be noted that traditional array cameras are commonly used in the security field. They generally have a resolution of hundreds of millions of pixels and are capable of detecting human-sized objects at a distance of 1km with a large detection range. However, such array cameras used for security have deficiencies in resolution and reliability. They cannot detect objects the size of consumer-grade drones, and their reliability does not meet the requirements of aviation safety. In this embodiment, through redundant design and the combination of multiple algorithms, the requirements of aircraft applications can be met.

[0064] In some embodiments, in step S3, small target detection is performed on the m frames of large field-of-view images according to a pre-set small target detection algorithm to obtain m sets of first obstacle target information, including the following sub-steps:

[0065] Step S31: Detect pixel clusters in the m-frame large field-of-view images to obtain corresponding m sets of suspected obstacle information, and each set of suspected obstacle information includes the first spatial position, velocity vector and obstacle type of n suspected obstacles; wherein, the pixel cluster includes multiple adjacent pixels, n≥h≥0;

[0066] Step S32: Obtain the aircraft's flight route information;

[0067] Specifically, the flight route information can be obtained through the aircraft's navigation system;

[0068] Step S33: Obtain the corresponding m sets of first obstacle target information based on the flight route information and the m sets of suspected obstacle information; wherein, if any suspected obstacle interferes with the flight of the aircraft, then any suspected obstacle is an obstacle target, and n≥h≥0;

[0069] Specifically, suspected obstacle interference with aircraft flight refers to a suspected obstacle posing a collision risk to the aircraft. For example, the suspected obstacle may appear in or be close to the aircraft's flight path. The existence of a collision risk can be determined based on the speed vector of the suspected obstacle and the aircraft's flight path.

[0070] In some embodiments, step S31 involves detecting pixel clusters in the m-frame wide-field-of-view images to obtain corresponding m sets of suspected obstacle information, including:

[0071] Step S311: Detect pixel clusters in the m frames of the large field of view image to obtain corresponding m groups of pixel clusters, and each group of pixel clusters includes f pixel clusters; where f≥n;

[0072] Specifically, this embodiment can use edge detection, grayscale detection and other technologies to preprocess the original image to obtain a pre-processed image. It mainly identifies points with obvious brightness changes in the image, removes irrelevant information, and retains the important structural attributes of the image, which can greatly reduce the amount of data that needs to be stored.

[0073] Specifically, this embodiment utilizes edge detection, grayscale detection, and other techniques to preprocess the original large field-of-view image to obtain a pre-processed image. This primarily involves identifying points with significant brightness changes in the image, removing irrelevant information, and preserving important structural attributes, which significantly reduces the amount of data that needs to be stored. Further, noise filtering is applied to the pre-processed image to remove interference from the imaging process. The effective detection region in the image is segmented to obtain its geometric location and clarify the effective detection range. This region segmentation can be achieved through edge detection, detecting locations with abrupt changes in grayscale levels or structure, indicating the end of one region and the beginning of another; this discontinuity allows for segmentation. Based on the processed image, pixel clusters can be detected within the effective detection range of the image. Correspondingly, three frames of large field-of-view images can yield three groups of pixel clusters.

[0074] Step S312: Filter each group of pixel clusters to obtain the corresponding m groups of suspected obstacle information; wherein, if the size of any pixel cluster meets the preset requirements, then the pixel cluster is determined to be a suspected obstacle, and the first spatial position, velocity vector and obstacle type of the suspected obstacle are calculated.

[0075] Specifically, the preset requirement is, for example, a pixel cluster larger than 5 pixels * 5 pixels, that is, a length and width both greater than 5 pixels.

[0076] In some embodiments, in step S312, the first spatial position, velocity vector, and obstacle type of the suspected obstacle can be calculated in the following manner:

[0077] (1.1) For any suspected obstacle in any set of suspected obstacle information, acquire the current time and the two frames before and after the previous time of any array camera corresponding to the suspected obstacle.

[0078] (1.2) Obtain the first spatial position and velocity vector of the aircraft, obtain the first spatial position and velocity vector of the suspected obstacle based on the position of the suspected obstacle in the two frames of the large field of view image before and after, and the first spatial position and velocity vector of the aircraft, and determine the obstacle type of the suspected obstacle based on the velocity vector of the suspected obstacle.

[0079] Specifically, the velocity vector can be estimated based on the displacement of a pixel cluster in the two large-field-of-view images before and after the corresponding time. It can be understood that the position of the suspected obstacle obtained based on the two large-field-of-view images before and after the actual position of the suspected obstacle relative to the aircraft. Combined with the spatial position of the aircraft itself given by the aircraft's positioning system, the first spatial position of the suspected obstacle can be further obtained.

[0080] It should be noted that conventional deep learning-based small object detection algorithms generally require pixel clusters of 28 pixels * 28 pixels or larger to identify obstacle types. However, the small object detection algorithm in this embodiment does not need to identify obstacle types from the image itself. Instead, it uses conventional image detection algorithms to confirm that an obstacle does exist in the image and estimates its velocity vector. The velocity vector includes the speed and direction of movement, which can reflect the motion characteristics of the obstacle. In this embodiment, the obstacle type is mainly determined based on the motion characteristics of the obstacle, thus reducing the requirement for image pixel clusters to 5 pixels * 5 pixels or larger.

[0081] In some embodiments, each group of first obstacle target information further includes h existence probabilities of obstacle targets; the existence probability is similar to confidence level, representing the reliability of the detection result, and is used to provide a reference for the aircraft's flight decision.

[0082] In step S3, the process of performing small target detection on the m frames of the large field-of-view image according to a pre-set small target detection algorithm to obtain m sets of first obstacle target information also includes:

[0083] Step S34: Calculate the similarity between any two obstacle targets in each group of first obstacle target information based on their first spatial position and velocity vectors, thereby obtaining the similarity between any obstacle target in each group of first obstacle target information and other obstacle targets;

[0084] Specifically, assuming there are 10 targets in each group of first obstacle target information, i.e., h = 10; then the similarity between any obstacle target in each group of first obstacle target information and other obstacle targets has h-1 = 9 similarities. Finally, the similarity corresponding to any obstacle target can be calculated based on these 9 similarities, for example, by taking the average of the 9 similarities.

[0085] Step S35: Determine the existence probability of any obstacle target in each group of first obstacle target information based on the similarity between any obstacle target and other obstacle targets, thereby obtaining the existence probabilities of h obstacle targets in m groups of first obstacle target information;

[0086] Specifically, in this embodiment, the probability of existence is obtained by querying a preset table based on the calculated similarity to the obstacle target.

[0087] The preset table can be obtained in the following way:

[0088] Step 1. Based on the small target detection results of m array cameras, obtain the first spatial position, motion direction, velocity, and classification data of m groups of obstacle targets, and number each obstacle target according to a unified rule;

[0089] Step 2. Calculate the relative relationship of the first spatial position, direction of motion and velocity of each obstacle in the three sets of data to obtain the relative relationship of every two obstacles in each set of data, thus forming a relative relationship database, with each relative relationship corresponding to a number pair.

[0090] Step 3. Calculate the similarity of the pairwise relationships of each number pair in the three sets of data. The similarity calculation method is as follows: First, divide the spatial parameter and velocity pairwise to obtain the pairwise ratio of each parameter. Then, weight them with specific weights to obtain the similarity of each number pair.

[0091] Step 4. Based on the test results, statistical analysis is performed to derive the functional relationship between similarity and the probability of the presence of the obstacle target, and the result is stored in the system as a table.

[0092] It should be noted that due to factors such as manufacturing deviations, aging during use, or dirty lenses, a single array camera will inevitably have unreliable obstacle detection during small target detection, such as misidentifying a water droplet on the camera as a drone or a bird. However, it is almost impossible for multiple array cameras to have the same false detection at the same time. Based on this idea, the above method is used to calculate the probability of the presence of obstacles in front of the aircraft, providing more dimensions of reference for aircraft piloting decisions.

[0093] In some embodiments, the similarity between any two obstacle targets is calculated based on the first spatial position and velocity vector of any two obstacle targets in each set of first obstacle target information. Specifically, this can be calculated in the following manner:

[0094] X 1-2 = (P1 / P2)*a + (V1 / V2)*b

[0095] Among them, X 1-2 Let P1 be the first spatial position of the first obstacle target and P2 be the second spatial position of the second obstacle target, V1 be the velocity vector of the first obstacle target and V2 be the velocity vector of the second obstacle target, and a and b be pre-set weighting coefficients.

[0096] In some embodiments, the step of fusing the velocity vector, obstacle type, and spatial position of the obstacle target based on the velocity vector and obstacle type in the m sets of first obstacle target information and the second spatial position in the m sets of second obstacle target information further includes:

[0097] The existence probabilities of h obstacle targets in the m groups of first obstacle target information are fused to obtain the existence probabilities of h obstacle targets;

[0098] Specifically, the fusion processing of the existence probabilities of h obstacle targets preferably adopts, but is not limited to, the Kalman filter fusion algorithm.

[0099] In some embodiments, receiving m sets of small field-of-view images captured by m array cameras includes:

[0100] Step S6: Receive the status signals of k array cameras output by the camera monitoring system; wherein the status signals are normal or abnormal;

[0101] Step S7: Determine the m array cameras whose status signals are normal based on the status signals of the k array cameras, and receive m sets of small field-of-view images captured by the m array cameras; where k≥m.

[0102] Specifically, assuming the aircraft is equipped with 5 array cameras, if 2 of the array cameras malfunction, the images captured by the 3 normal array cameras will be used for small target detection.

[0103] See Figure 3 Another embodiment of the present invention provides an obstacle detection device for implementing the obstacle detection method described in the above embodiments, the device comprising:

[0104] Image receiving module 1 is used to receive m sets of small field-of-view images captured by m array cameras; wherein, the m array cameras are set at the front of the aircraft, each array camera includes multiple cameras arranged in an array, the multiple cameras are used to capture the environment in front of the aircraft to obtain corresponding multi-frame small field-of-view images, m>1;

[0105] Image processing module 2 is used to perform image stitching processing on multiple frames of small field-of-view images in each group of small field-of-view images to obtain m frames of large field-of-view images corresponding to the m array cameras;

[0106] Target detection module 3 is used to perform small target detection on the m frames of large field of view images according to a pre-set small target detection algorithm to obtain the first spatial position, velocity vector and obstacle type of m groups of obstacle targets;

[0107] Dual-camera positioning module 4 is used to calculate the second spatial position of m groups of obstacle targets based on a pre-set dual-camera positioning algorithm and the m frames of large field-of-view images;

[0108] The fusion processing module 5 is used to perform fusion processing based on the velocity vectors and obstacle types of the m groups of obstacle targets and the second spatial positions of the m groups of obstacle targets to obtain the spatial position, velocity vectors and obstacle types of the obstacle targets.

[0109] It should be noted that if the device in the above embodiments is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0110] Another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the obstacle detection method described in the above embodiments.

[0111] Specifically, the computer-readable storage medium may include any entity or recording medium capable of carrying the computer program instructions, such as a USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (RON), random access memory (RAN), electrical carrier signals, telecommunication signals, and software distribution media.

[0112] As can be seen from the above description of the embodiments, the embodiments of the present invention have the following advantages:

[0113] The embodiments of this invention employ a redundant design of m array cameras, proposing to perform small target detection and detection result fusion on the detection images from the m array cameras respectively. Simultaneously, it also proposes two localization methods combining camera-based image localization, achieving reliable detection and localization of aircraft obstacles based on cameras. This method offers higher reliability compared to traditional camera-based perception methods and lower cost compared to millimeter-wave radar-based perception methods. Therefore, this invention overcomes the limitation of existing cameras in achieving both high resolution and high detection range, providing a high-performance, highly reliable solution for the perception of autonomous driving in unmanned aerial vehicles. It is easy to promote and has significant application value, effectively promoting the development of manned unmanned aerial vehicles.

[0114] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and substitutions will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An obstacle detection method, characterized in that, The method includes: Receive m sets of small field-of-view images captured by m array cameras; wherein, the m array cameras are set at the front of the aircraft, each array camera includes multiple cameras arranged in an array, the multiple cameras are used to capture the environment in front of the aircraft to obtain corresponding multi-frame small field-of-view images, m>1; Each group of small field-of-view images is processed by image stitching of multiple frames of small field-of-view images to obtain m frames of large field-of-view images corresponding to the m array cameras. The m frames of large field-of-view images are subjected to small target detection according to a pre-set small target detection algorithm to obtain m sets of first obstacle target information. Each set of first obstacle target information includes the first spatial position, velocity vector and obstacle type of h obstacle targets; where h≥0; Based on the pre-set dual-camera positioning algorithm and the m frames of large field-of-view images, m sets of second obstacle target information are calculated, and each set of second obstacle target information includes the second spatial positions of h obstacle targets; The velocity vector, obstacle type, and spatial position of the obstacle target are obtained by fusing the velocity vector and obstacle type in the m sets of first obstacle target information and the second spatial position in the m sets of second obstacle target information, respectively. The step of performing small target detection on the m frames of the large field-of-view image according to a pre-set small target detection algorithm to obtain m sets of first obstacle target information includes: The pixel clusters in the m-frame large field-of-view images are detected to obtain corresponding m sets of suspected obstacle information, and each set of suspected obstacle information includes the first spatial position, velocity vector and obstacle type of n suspected obstacles; wherein, the pixel clusters include multiple adjacent pixels, n≥h≥0; Obtain aircraft flight route information; Based on the flight route information and the m sets of suspected obstacle information, the corresponding m sets of first obstacle target information are obtained; wherein, if any suspected obstacle interferes with the flight of the aircraft, then any suspected obstacle is an obstacle target.

2. The obstacle detection method according to claim 1, characterized in that, The step of detecting pixel clusters in the m-frame large field-of-view images to obtain corresponding m sets of suspected obstacle information includes: Each of the m frames of the large field-of-view image is detected to obtain m corresponding groups of pixel clusters, and each group of pixel clusters includes f pixel clusters; where f≥n; Each group of pixels is filtered to obtain m groups of suspected obstacle information; if the size of any pixel group meets the preset requirements, then the pixel group is determined to be a suspected obstacle, and the first spatial position, velocity vector and obstacle type of the suspected obstacle are calculated.

3. The obstacle detection method according to claim 2, characterized in that, The calculation methods for the first spatial position, velocity vector, and obstacle type of a suspected obstacle are as follows: For any suspected obstacle in any set of suspected obstacle information, acquire the current time and the two frames before and after the previous time of any array camera corresponding to that suspected obstacle; The first spatial position and velocity vector of the aircraft are obtained. Based on the position of any suspected obstacle in the two preceding and following large field-of-view images, and the first spatial position and velocity vector of the aircraft, the first spatial position and velocity vector of the suspected obstacle are obtained. The obstacle type of the suspected obstacle is determined based on the velocity vector of the suspected obstacle.

4. The obstacle detection method according to claim 1, characterized in that, in, Each set of first obstacle target information also includes h existence probabilities of obstacle targets; The step of performing small target detection on the m frames of large field-of-view images according to a pre-set small target detection algorithm to obtain m groups of first obstacle target information also includes: The similarity between any two obstacle targets in each set of first obstacle target information is calculated based on their first spatial position and velocity vectors, thereby obtaining the similarity between any obstacle target in each set of first obstacle target information and other obstacle targets. The probability of the existence of any obstacle target in each group of first obstacle target information is determined based on the similarity between any obstacle target and other obstacle targets, thereby obtaining the probability of the existence of h obstacle targets in m groups of first obstacle target information.

5. The obstacle detection method according to claim 4, characterized in that, The similarity between any two obstacle targets in each group of first obstacle target information is calculated based on their first spatial position and velocity vectors. Specifically, the similarity is calculated in the following manner: ； in, The similarity between the first obstacle target and the second obstacle target. The first spatial position of the first obstacle target. The first spatial position of the second obstacle target. Let be the velocity vector of the first obstacle target. Let a be the velocity vector of the second obstacle target, and a and b are pre-set weighting coefficients.

6. The obstacle detection method according to claim 4, characterized in that, The step of fusing the velocity vector, obstacle type, and spatial position of the obstacle target based on the velocity vector and obstacle type in the m sets of first obstacle target information and the second spatial position in the m sets of second obstacle target information, respectively, further includes: The existence probabilities of h obstacle targets in the m groups of first obstacle target information are fused to obtain the existence probabilities of h obstacle targets.

7. The obstacle detection method according to any one of claims 1-6, characterized in that, The receiving of m sets of small field-of-view images captured by m array cameras includes: Receive status signals from k array cameras output by the camera monitoring system; wherein the status signals are normal or abnormal; Based on the status signals of the k array cameras, determine the m array cameras whose status signals are normal, and receive m sets of small field-of-view images captured by the m array cameras; where k≥m.

8. An obstacle detection device, characterized in that, The apparatus for implementing the obstacle detection method according to any one of claims 1-7 comprises: An image receiving module is used to receive m sets of small field-of-view images captured by m array cameras; wherein, the m array cameras are set at the front of the aircraft, each array camera includes multiple cameras arranged in an array, the multiple cameras are used to capture the environment in front of the aircraft to obtain corresponding multi-frame small field-of-view images, m>1; The image processing module is used to perform image stitching processing on multiple frames of small field-of-view images in each group of small field-of-view images to obtain m frames of large field-of-view images corresponding to the m array cameras. The target detection module is used to perform small target detection on the m frames of the large field of view image according to a pre-set small target detection algorithm to obtain m sets of first obstacle target information. Each set of first obstacle target information includes the first spatial position, velocity vector and obstacle type of h obstacle targets; where h≥0; The dual-camera positioning module is used to calculate m sets of second obstacle target information based on the pre-set dual-camera positioning algorithm and the m frames of large field of view images. Each set of second obstacle target information includes the second spatial positions of h obstacle targets. The fusion processing module is used to perform fusion processing on the velocity vector and obstacle type in the m sets of first obstacle target information and the second spatial position in the m sets of second obstacle target information to obtain the spatial position, velocity vector and obstacle type of the obstacle target; Specifically, the target detection module is used to detect pixel clusters in the m-frame large field-of-view images to obtain corresponding m sets of suspected obstacle information, and each set of suspected obstacle information includes the first spatial position, velocity vector, and obstacle type of n suspected obstacles; wherein, the pixel cluster includes multiple adjacent pixels, n≥h≥0; acquire the aircraft's flight path information; and obtain corresponding m sets of first obstacle target information based on the flight path information and the m sets of suspected obstacle information; wherein, if any suspected obstacle interferes with the aircraft's flight, then any suspected obstacle is an obstacle target.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it performs the obstacle detection method as described in any one of claims 1-7.

Images