Depth estimation method and system based on unmanned aerial vehicle cluster

Abstract

The invention relates to a depth estimation method and system based on an unmanned aerial vehicle cluster, and is applied to the technical field of unmanned aerial vehicles, and the method comprises the steps: carrying out the correction and binocular stereo matching of each fisheye image, obtaining the parallax data, and converting the parallax data into a depth value; determining point cloud data according to the position coordinate and the depth value of each fisheye image under the camera coordinate system; performing coordinate alignment on the point cloud data of the fisheye image and the point cloud data of other fisheye images to obtain a first overlapping region and a first non-overlapping region; splicing the point cloud data of the first overlapping region and the point cloud data of the first non-overlapping region to obtain first target point cloud data; performing coordinate alignment on the first target point cloud data corresponding to the unmanned aerial vehicle and first target point cloud data corresponding to other unmanned aerial vehicles to obtain a second overlapping region and a second non-overlapping region; and splicing the point cloud data of the second overlapping region and the point cloud data of the second non-overlapping region to obtain second target point cloud data. The precision of depth estimation can be improved.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a depth estimation method and system based on UAV swarms. Background Technology

[0002] With the rapid development of drone technology, drones are increasingly being used in agriculture, logistics, security, disaster relief, and other fields. However, drones face limited obstacle avoidance capabilities when performing tasks, especially in complex environments. For example, traditional monocular or binocular vision systems have limited depth estimation and 3D reconstruction capabilities in wide-angle fields of view, making it difficult to meet the needs of drones in complex environments. Summary of the Invention

[0003] To address the aforementioned technical problems, this application provides a depth estimation method, apparatus, electronic device, and storage medium based on unmanned aerial vehicle (UAV) swarms.

[0004] According to a first aspect of this application, a depth estimation method based on a drone swarm is provided, applied to a single drone in a drone swarm, wherein the drone is equipped with four sets of cameras covering a 360° field of view, each set of cameras including a fisheye camera and an RGB camera, the method comprising: Four sets of RGB images and fisheye images around the drone are acquired in real time. Each fisheye image is corrected and stereo matched with binoculars to obtain the corresponding disparity data, and the disparity data is converted into depth values. The point cloud data of each fisheye image is determined based on its position coordinates in the camera coordinate system and the depth value. Semantic segmentation is performed on each RGB image to obtain semantic segmentation information; For each set of RGB images and fisheye images, based on the semantic segmentation information in the RGB images and other RGB images, and the position information of the point cloud data of the fisheye images and other fisheye images, the point cloud data of the fisheye images are aligned with the point cloud data of other fisheye images to obtain a first overlapping region and a first non-overlapping region. The point cloud data of the first overlapping region and the first non-overlapping region are stitched together to obtain the first target point cloud data corresponding to the UAV. Using the four semantic segmentation information corresponding to the UAV and other UAVs, as well as the position information of the first target point cloud data corresponding to the UAV and other UAVs, the first target point cloud data corresponding to the UAV and other UAVs are aligned with coordinates to obtain the second overlapping region and the second non-overlapping region. The point cloud data of the second overlapping region and the second non-overlapping region are stitched together to obtain the second target point cloud data corresponding to the UAV.

[0005] Optionally, aligning the point cloud data of the fisheye image with the point cloud data of other fisheye images based on the semantic segmentation information in the RGB image and other RGB images, and the positional information of the point cloud data of the fisheye image and other fisheye images, includes: The point cloud data of the fisheye image is divided into feature-rich regions and feature-sparse regions. Based on the position information of the point cloud data of the feature-rich region in the fisheye image and the position information of the point cloud data of the corresponding region in other fisheye images, the point cloud data of the feature-rich region is aligned with the point cloud data of other fisheye images. Based on the semantic segmentation information of the sparse feature region in the fisheye image and the semantic segmentation information of the corresponding region in other RGB images, as well as the position information of the point cloud data of the sparse feature region and the position information of the point cloud data of the corresponding region in other fisheye images, the point cloud data of the sparse feature region is aligned with the point cloud data of other fisheye images.

[0006] Optionally, the semantic segmentation information includes object category information and object boundary information; The step of aligning the point cloud data of the sparse region with the point cloud data of the other fisheye image based on the semantic segmentation information of the sparse region in the fisheye image and the semantic segmentation information of the corresponding region in other RGB images, as well as the position information of the point cloud data of the sparse region with the position information of the point cloud data of the corresponding region in other fisheye images, includes: Based on the object category information and object boundary information of the sparse feature region in the fisheye image, construct the first semantic feature vector of each point cloud data in the sparse feature region; Based on the object category information and object boundary information of the corresponding regions in other fisheye images, construct the second semantic feature vector of each point cloud data; Calculate the similarity between the first semantic feature vector and the second semantic feature vector of each point cloud data within the sparse feature region; The distance between the point cloud data is calculated based on the position information of each point cloud data in the sparse feature region and the position information of the corresponding point cloud data in other fisheye images. The similarity and distance are weighted to obtain a weighted value, and the sum of the weighted values ​​corresponding to all point cloud data in the sparse feature region is minimized to align the point cloud data in the sparse feature region with the point cloud data of other fisheye images.

[0007] Optionally, the point cloud data of the first overlapping region / second overlapping region are stitched together, including: Obtain the depth value of each point cloud data in the first overlapping region / second overlapping region from each viewpoint, and calculate the variance of the depth values ​​from each viewpoint; If the variance is less than or equal to a preset threshold, the depth values ​​of each viewpoint are weighted and averaged to obtain a weighted average value, and the depth values ​​of the point cloud data of the first overlapping region / second overlapping region are updated to the weighted average value. If the variance is greater than a preset threshold, the stability of the depth values ​​of the first overlapping region / second overlapping region at multiple times within the first historical time period is judged, and the depth values ​​of the point cloud data of the first overlapping region / second overlapping region are updated to the depth values ​​with the best stability.

[0008] Optionally, the point cloud data of the first non-overlapping region / second non-overlapping region are stitched together, including: For each of the multiple moments within the second historical time period, calculate the variance of the depth values ​​of all point cloud data in the first non-overlapping region / second non-overlapping region at that moment; Calculate the RGB gradient values ​​of all point cloud data in the first non-overlapping region / second non-overlapping region at the stated time; Calculate the motion distance of the point cloud data of the first non-overlapping region / second non-overlapping region at the time; The variance, RGB gradient values, and motion distance are weighted to obtain the confidence level; The depth values ​​of the point cloud data in the first non-overlapping region / second non-overlapping region are then updated to the depth values ​​of the point cloud data at the time corresponding to the maximum confidence value.

[0009] Optionally, binocular stereo matching is performed on the corrected fisheye image to obtain the corresponding disparity data, including: The corrected fisheye image is epipolar aligned to obtain the grayscale images of the left and right eyes; The Census transform is used to calculate the features of each pixel in the left and right grayscale images to obtain the first left-eye feature and the first right-eye feature. The Sobel operator is used to calculate the features of each pixel in the left and right grayscale images to obtain the second left-eye feature and the second right-eye feature. Calculate the first gradient BT map based on the first left-eye feature and the first right-eye feature; calculate the second gradient BT map based on the second left-eye feature and the second right-eye feature; weight the first gradient BT map and the second gradient BT map to obtain the target gradient BT map; Calculate the grayscale BT image based on the left and right eye grayscale images; Based on the grayscale BT image and the target gradient BT image, calculate the cost of each pixel in the left-eye grayscale image and the right-eye grayscale image; Cost aggregation and optimization are performed on the cost of each pixel in the left and right grayscale images to obtain disparity data.

[0010] According to a second aspect of this application, a depth estimation system based on a drone swarm is provided, the system comprising: a drone swarm including multiple drones, each drone being equipped with four sets of cameras covering a 360° field of view, each set of cameras including a fisheye camera and an RGB camera; each drone including a computing module and a communication module; The fisheye camera and the RGB camera are used to acquire fisheye images and RGB images of the area around the drone in real time, respectively. The calculation module is used to acquire the RGB image and the fisheye image, perform correction and binocular stereo matching on each fisheye image to obtain corresponding disparity data, and convert the disparity data into depth values; determine the point cloud data of each fisheye image based on its position coordinates in the camera coordinate system and the depth value; perform semantic segmentation on each RGB image to obtain semantic segmentation information; for each set of RGB images and fisheye images, based on the semantic segmentation information in the RGB image and other RGB images, and the position information of the point cloud data of the fisheye image and other fisheye images, align the point cloud data of the fisheye image with the point cloud data of other fisheye images to obtain a first overlapping region and a first non-overlapping region; and stitch the point cloud data of the first overlapping region and the first non-overlapping region together to obtain the first target point cloud data corresponding to the UAV. The communication module is used to communicate with other drones and obtain the first target point cloud data and semantic segmentation information corresponding to other drones; The calculation module is further configured to utilize the four semantic segmentation information corresponding to the UAV and other UAVs, as well as the position information of the first target point cloud data corresponding to the UAV and other UAVs, to perform coordinate alignment of the first target point cloud data corresponding to the UAV and the first target point cloud data corresponding to other UAVs to obtain a second overlapping region and a second non-overlapping region; and to stitch together the point cloud data of the second overlapping region and the second non-overlapping region to obtain the second target point cloud data corresponding to the UAV.

[0011] Optionally, the calculation module is specifically configured to perform coordinate alignment of the point cloud data of the fisheye image with the point cloud data of other fisheye images based on the semantic segmentation information in the RGB image and other RGB images, and the positional information of the point cloud data of the fisheye image and other fisheye images, through the following steps: The point cloud data of the fisheye image is divided into feature-rich regions and feature-sparse regions. Based on the position information of the point cloud data of the feature-rich region in the fisheye image and the position information of the point cloud data of the corresponding region in other fisheye images, the point cloud data of the feature-rich region is aligned with the point cloud data of other fisheye images. Based on the semantic segmentation information of the sparse feature region in the fisheye image and the semantic segmentation information of the corresponding region in other RGB images, as well as the position information of the point cloud data of the sparse feature region and the position information of the point cloud data of the corresponding region in other fisheye images, the point cloud data of the sparse feature region is aligned with the point cloud data of other fisheye images.

[0012] Optionally, the semantic segmentation information includes object category information and object boundary information; The calculation module is specifically used to perform coordinate alignment of the point cloud data of the sparse feature region with the point cloud data of the other fisheye image based on the semantic segmentation information of the sparse feature region in the fisheye image and the semantic segmentation information of the corresponding region in the other RGB image, as well as the position information of the point cloud data of the sparse feature region and the position information of the point cloud data of the corresponding region in the other fisheye image, through the following steps: Based on the object category information and object boundary information of the sparse feature region in the fisheye image, construct the first semantic feature vector of each point cloud data in the sparse feature region; Based on the object category information and object boundary information of the corresponding regions in other fisheye images, construct the second semantic feature vector of each point cloud data; Calculate the similarity between the first semantic feature vector and the second semantic feature vector of each point cloud data within the sparse feature region; The distance between the point cloud data is calculated based on the position information of each point cloud data in the sparse feature region and the position information of the corresponding point cloud data in other fisheye images. The similarity and distance are weighted to obtain a weighted value, and the sum of the weighted values ​​corresponding to all point cloud data in the sparse feature region is minimized to align the point cloud data in the sparse feature region with the point cloud data of other fisheye images.

[0013] Optionally, the calculation module is specifically used to stitch together the point cloud data of the first overlapping region / second overlapping region through the following steps: Obtain the depth value of each point cloud data in the first overlapping region / second overlapping region from each viewpoint, and calculate the variance of the depth values ​​from each viewpoint; If the variance is less than or equal to a preset threshold, the depth values ​​of each viewpoint are weighted and averaged to obtain a weighted average value, and the depth values ​​of the point cloud data of the first overlapping region / second overlapping region are updated to the weighted average value. If the variance is greater than a preset threshold, the stability of the depth values ​​of the first overlapping region / second overlapping region at multiple times within the first historical time period is judged, and the depth values ​​of the point cloud data of the first overlapping region / second overlapping region are updated to the depth values ​​with the best stability.

[0014] Optionally, the calculation module is specifically used to stitch together the point cloud data of the first non-overlapping region / second non-overlapping region through the following steps: For each of the multiple moments within the second historical time period, calculate the variance of the depth values ​​of all point cloud data in the first non-overlapping region / second non-overlapping region at that moment; Calculate the RGB gradient values ​​of all point cloud data in the first non-overlapping region / second non-overlapping region at the stated time; Calculate the motion distance of the point cloud data of the first non-overlapping region / second non-overlapping region at the time; The variance, RGB gradient values, and motion distance are weighted to obtain the confidence level; The depth values ​​of the point cloud data in the first non-overlapping region / second non-overlapping region are then updated to the depth values ​​of the point cloud data at the time corresponding to the maximum confidence value.

[0015] Optionally, the calculation module is specifically used to perform binocular stereo matching on the corrected fisheye image through the following steps to obtain the corresponding disparity data: The corrected fisheye image is epipolar aligned to obtain the grayscale images of the left and right eyes; The Census transform is used to calculate the features of each pixel in the left and right grayscale images to obtain the first left-eye feature and the first right-eye feature. The Sobel operator is used to calculate the features of each pixel in the left and right grayscale images to obtain the second left-eye feature and the second right-eye feature. Calculate the first gradient BT map based on the first left-eye feature and the first right-eye feature; calculate the second gradient BT map based on the second left-eye feature and the second right-eye feature; weight the first gradient BT map and the second gradient BT map to obtain the target gradient BT map; Calculate the grayscale BT image based on the left and right eye grayscale images; Based on the grayscale BT image and the target gradient BT image, calculate the cost of each pixel in the left-eye grayscale image and the right-eye grayscale image; Cost aggregation and optimization are performed on the cost of each pixel in the left and right grayscale images to obtain disparity data.

[0016] According to a third aspect of this application, an electronic device is provided, comprising: a processor configured to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the method described in the first aspect.

[0017] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0018] According to a fifth aspect of this application, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform the method described in the first aspect.

[0019] The technical solution provided in this application has the following advantages compared with the prior art: Fisheye cameras have a large field of view (typically exceeding 180°), providing an all-around view. Four camera groups are deployed around each drone, each group including a fisheye camera and an RGB camera, enabling 360° 3D environment modeling and reducing blind spots by 80%. Within a single drone, a high-precision depth map of a single fisheye image is generated through binocular stereo matching. The point cloud data from the four fisheye images are aligned using semantic segmentation information from the RGB images and positional information from the fisheye image point cloud data—a multimodal feature-based (geometric and semantic) alignment that improves the robustness of feature matching. This alignment is then stitched and fused to obtain the depth information surrounding each drone, thus improving the accuracy of depth estimation for that individual drone. Multiple drones within a cluster share depth information via wireless communication. The point cloud data from multiple drones are further aligned using semantic segmentation information from the RGB images and stitched together to obtain the final depth information. By combining the depth information from multiple drones for depth estimation, comprehensive depth information can be obtained, further improving the accuracy of depth estimation. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a configuration diagram of the four fisheye lenses in the UAV in the embodiments of this application; Figure 2 This is a flowchart of a depth estimation method based on a drone swarm in an embodiment of this application; Figure 3 This is a schematic diagram of refraction in a three-sphere camera model; Figure 4 This is a schematic diagram of epipolar alignment of a corrected fisheye image in an embodiment of this application; Figure 5 This is a schematic diagram of a binocular stereo matching algorithm in an embodiment of this application; Figure 6 This is a schematic diagram of disparity subpixel fitting in an embodiment of this application; Figure 7 This is a schematic diagram of a depth estimation system based on a drone swarm in an embodiment of this application. Detailed Implementation

[0023] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0024] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0025] The depth estimation method based on drone swarms in this application is applied to a single drone in a drone swarm. The drone is equipped with four sets of cameras covering a 360° field of view, each set including a fisheye camera and an RGB camera. Figure 1 As shown, each drone has four sets of cameras installed at 90° intervals, located at the front left, rear left, front right, and rear right of the drone, covering a 360° field of view. The overlap rate of adjacent camera fields of view is ≥70%. A high-precision depth map is generated within a single drone through fisheye binocular matching. Multiple drones in the cluster share depth information through wireless communication to construct a global 3D environment model, thereby obtaining omnidirectional 360° depth obstacle information for the drone.

[0026] See Figure 2 , Figure 2 This is a flowchart of a depth estimation method based on a drone swarm in this application embodiment, which may include the following steps: Step S202: Acquire four sets of RGB images and fisheye images around the drone in real time, perform correction and binocular stereo matching on each fisheye image to obtain the corresponding disparity data, and convert the disparity data into depth values.

[0027] Since fisheye images cannot be directly used for depth estimation, they need to be distorted first to generate distortion-free image data. For example, the Triple Sphere Camera Model can be used for calibration, with a reprojection error ≤ 1 pixel. The Triple Sphere Camera Model considers that light undergoes three refractions before entering the imaging plane. This multi-refractive model can more accurately simulate the imaging characteristics of a fisheye lens and has a closed-form backprojection solution. Figure 3 As shown, this projection model considers the incident light to be refracted three times, and the displacements of the three unit centers of the sphere are respectively... , and After three refractions, the incident light is finally projected onto the image plane according to the pinhole camera model. The optical center of the pinhole camera model is displaced from the third unit sphere by... O1 represents the center of the first layer of spheres, O2 represents the center of the second layer of spheres, O3 represents the center of the third layer of spheres, and O4 represents the center of the three-sphere model.

[0028] These displacement parameters enable the model to more accurately describe the actual path of light rays. They include seven intrinsic parameters: focal length fx, fy, optical center cx, cy, and three parameters describing the offset of the spherical center. , and Its projection model defines the mapping relationship from world coordinates to image coordinates, and the specific formula is as follows: (1); (2); (3); (4); (5); Where i is the intrinsic parameter vector. Let X, Y, and Z be the reprojection function, where X, Y, and Z represent the coordinates of a point on the sphere. , and These represent the distances from a pixel on the camera plane to the centers of the three spheres, respectively.

[0029] The 3D coordinates of a set of effectively projected checkerboard corner points are calculated using the following formula 6-8. By using the calculated 3D coordinates, the pixel coordinates of the 2D checkerboard corner points, and the transformation relationship of the four coordinate systems, the intrinsic and extrinsic parameters of the camera are solved.

[0030] (6); (7); (8); and Indicates weight, Indicates greater than A collection of pixels.

[0031] After distortion correction, four distortion-free lens data points are obtained. However, depth matching is required. These four distortion-free lens data points are vertically divided into two equal parts. The right half of each lens and the left half of the adjacent lens form a left-right image pair, thus creating a binocular system. This involves epipolar alignment of the corrected fisheye images to obtain the left-eye and right-eye grayscale images. The four fisheye lenses can then form four pairs of left-right images. Figure 4 As shown, each fisheye camera is abstracted as two pinhole cameras at a 90° angle, thus transforming the four fisheye camera systems into four independent binocular systems. Epipolar correction is then performed on each binocular system to align the left and right images horizontally. Fusiello can be used for epipolar correction to achieve this alignment.

[0032] After obtaining four pairs of aligned stereo images, stereo matching can be performed to generate depth maps. These four depth maps provide the positions of obstacles relative to the drone in a 360° radius. The stereo matching algorithm consists of three parts: cost calculation, cost aggregation, and disparity optimization.

[0033] Cost calculation is based on a feature point in the left image, and then finding feature points with the same characteristics in the same row of the right image. Since drones mostly operate outdoors in complex environments, this embodiment uses a combination of Census transform and Sobel operator to calculate the cost in order to adapt to different environments.

[0034] See Figure 5 , Figure 5This is a schematic diagram of a binocular stereo matching algorithm in an embodiment of this application. The Census transform is used to calculate the features of each pixel in the left and right grayscale images, respectively, to obtain the first left-eye feature and the first right-eye feature. The Sobel operator is used to calculate the features of each pixel in the left and right grayscale images, respectively, to obtain the second left-eye feature and the second right-eye feature. Based on the first left-eye feature and the first right-eye feature, a first gradient BT image is calculated; based on the second left-eye feature and the second right-eye feature, a second gradient BT image is calculated; the first gradient BT image and the second gradient BT image are weighted to obtain the target gradient BT image. The weights of the first gradient BT image and the second gradient BT image can be adjusted according to different scenarios, combining the advantages of both the Census transform and the Sobel operator, to improve the accuracy of cost calculation. For example, when the image texture is weak, the weight of the first gradient BT image can be increased; when the image edge contour is obvious, the weight of the second gradient BT image can be increased.

[0035] Based on the left and right grayscale images, a grayscale BT image is calculated. Then, based on the grayscale BT image and the target gradient BT image, the cost value of each pixel in both the left and right grayscale images is calculated. Finally, cost aggregation and optimization are performed based on the cost values ​​of each pixel in the left and right grayscale images to obtain disparity data.

[0036] The cost aggregation and optimization process is as follows: After calculating the cost, the left and right cost maps can be matched to select the optimal feature points. Since the cost calculation step only considers local correlations, it is very sensitive to noise and cannot be directly used to calculate the optimal disparity. Referring to the cost aggregation strategy of semi-global matching (SGBM), a 5-path strategy is adopted, which ensures robust performance while maintaining frame rate. Dynamically adjustable penalty parameters P1 / P2 are added to the cost aggregation, filtering out most invalid point matches, allowing the aggregated cost value to more accurately reflect the correlation between pixels. The specific principle is as follows: the new aggregated cost value is stored in an aggregation cost space S of the same size as the matching cost space, with each element's position corresponding one-to-one. To obtain better matching results, the semi-global matching algorithm still adopts the idea of ​​the global stereo matching algorithm, namely, a global energy optimization strategy, which finds the optimal disparity of each pixel to minimize the global energy function of the entire image. The definition of the global energy function is shown in Equation 9: (9); Where d represents the disparity value of each pixel. For data items, it is a measure of the overall matching cost corresponding to the disparity map; The smoothing term, to ensure the disparity map satisfies certain conditional assumptions, such as the continuity of the scene surface, penalizes cases where the disparity change between adjacent pixels exceeds a certain pixel limit (resulting in a larger energy function value). Pixels at image edges are an exception, as they are more likely to be considered as disparity discontinuities. Since the energy function is two-dimensional, minimizing it is a two-dimensional optimization problem, an NP-complete problem. Many approximate, relatively efficient energy-optimization strategies, such as graph cut, confidence propagation, and cooperative optimization, have been used to solve this problem, but their efficiency still needs further improvement. To solve this two-dimensional optimization problem more efficiently, a semi-global matching algorithm uses a scan-line-like approach, employing one-dimensional path aggregation to approximate the two-dimensional optimum. Compared to other solutions, this method is more efficient and achieves comparable results. The cost aggregation formula is shown in Equation 10.

[0037] (10); Where C is the matching cost of pixel p. This is a data item, indicating when the disparity map is... The sum of matching costs for all pixels is given, where D is the disparity set and q represents the pixel in the right-eye grayscale image corresponding to pixel p in the left-eye grayscale image. This represents the disparity value between pixel q and pixel p. and This is a smoothing term, representing a penalty applied to all pixels q within the Np neighborhood of pixel p. The penalty is relatively small (the penalty coefficient P1 is small), and it is used to penalize cases where the disparity between adjacent pixels changes very little (1 pixel). T represents the set of search strategies. A stronger penalty (penalty coefficient P2 > penalty coefficient P1) is applied to cases where the disparity between adjacent pixels changes significantly (greater than one pixel). A smaller penalty allows the algorithm to adapt to situations with small disparity changes, such as tilted planes or continuous curved surfaces. A larger penalty allows the algorithm to correctly handle disparity discontinuities. Since disparity discontinuities are more likely at image grayscale edges, the penalty coefficient P2 is often dynamically adjusted based on the grayscale difference between adjacent pixels to protect against disparity discontinuities in real-world scenes.

[0038] Semi-global matching proposes a path cost aggregation approach, which involves a one-dimensional aggregation of the matching costs under all disparities of a pixel across all paths surrounding the pixel to obtain the path cost value for each path. Then, all path cost values ​​are summed to obtain the aggregated matching cost value for that pixel. The path cost calculation method for pixel p along a certain path r is shown in Formula 11 below: (11); in, This indicates the matching cost value and belongs to the data item; It is a smoothing term, and the value accumulated on the path cost is the minimum value among the three cases of no penalty, P1 penalty, and P2 penalty. To ensure that the cost Lr of the new path does not exceed a certain upper limit, multiple paths are aggregated for cost, and the minimum value is selected as the matching aggregated path. The minimum value is the disparity data.

[0039] The depth distance at each location can be obtained by transforming the parallax data using the camera's intrinsic parameters, and can be calculated using formula 12: (12); The camera's focal length f and baseline B can be obtained through calibration. Therefore, as long as X is known... R – X L The distance information of a pixel can be obtained by taking the value of (disparity d). The depth map obtained by stereo matching already provides the disparity value d for each pixel, so it can be used directly to calculate the distance to each pixel.

[0040] Optionally, disparity optimization can be performed before converting the disparity data into depth values. The purpose of disparity optimization is to further improve the disparity obtained in the previous step and improve the quality of the disparity map. This includes steps such as removing erroneous disparities, smoothing the disparities, and optimizing subpixel precision. Generally, the left-right check algorithm is used to remove erroneous disparities caused by occlusion and noise; the small connected region removal algorithm is used to remove isolated outliers; and smoothing algorithms such as median filtering and bilateral filtering are used to smooth the disparity map. In addition, other effective methods to improve the quality of disparity maps, such as robust plane fitting, intensity consistency constraints, and locally consistency constraints, are also commonly used. Since the disparity value obtained by the WTA algorithm is integer pixel precision, further sub-pixel refinement is needed to obtain higher sub-pixel precision. A commonly used sub-pixel refinement method is the quadratic curve fitting method. This method fits a quadratic curve using the cost value under the optimal disparity and the cost values ​​under the left and right disparities. The disparity value represented by the minimum point of the quadratic curve is taken as the sub-pixel disparity value. Figure 6 As shown, the obtained parallax accuracy can reach the micrometer level.

[0041] Step S204: Determine the point cloud data of each fisheye image based on its position coordinates and depth values ​​in the camera coordinate system.

[0042] After obtaining the depth values, they can be combined with fisheye images to convert the two-dimensional fisheye images into three-dimensional point cloud data. Each drone corresponds to four sets of point cloud data, which can be aligned and stitched together to obtain the point cloud data around each drone. Furthermore, the point cloud data around multiple drones can be aligned and stitched together to obtain a more comprehensive and complete picture of obstacles around the drones.

[0043] Step S206: Perform semantic segmentation on each RGB image to obtain semantic segmentation information.

[0044] In related technologies, the ICP (Iterative Closest Point) algorithm or its variants are commonly used for point cloud coordinate alignment. However, these algorithms rely on the initial position and are prone to getting trapped in local optima, leading to a high registration failure rate at large viewpoints. In the embodiments of this application, an adaptive feature selection mechanism combining geometric and semantic features can robustly perform feature matching.

[0045] Semantic segmentation is performed on each RGB image, which involves classifying each pixel and assigning it a corresponding semantic label. For example, in a city street view image, pixels representing people, vehicles, and roads are labeled into different categories, forming semantic region divisions. Semantic segmentation information includes object category information and object boundary information. Thus, by combining the semantic information of the RGB images with the location information of the point cloud data, multimodal features are utilized to improve the accuracy of coordinate alignment.

[0046] Step S208: For each group of RGB images and fisheye images, based on the semantic segmentation information in the RGB image and other RGB images, and the position information of the point cloud data of the fisheye image and other fisheye images, the point cloud data of the fisheye image is aligned with the point cloud data of other fisheye images to obtain the first overlapping region and the first non-overlapping region.

[0047] In this embodiment, all regions of the fisheye image can be aligned using semantic segmentation information and point cloud data location information. Alternatively, the fisheye image can be divided into regions based on its features, with different features used for coordinate alignment in different regions.

[0048] Optionally, the point cloud data of the fisheye image can be divided into feature-rich regions and feature-sparse regions. Feature-rich regions can be accurately aligned using only the positional information of the point cloud data. Therefore, based on the positional information of the point cloud data of the feature-rich regions in the fisheye image and the positional information of the point cloud data of corresponding regions in other fisheye images, the point cloud data of the feature-rich regions are aligned with the point cloud data of other fisheye images. Feature-rich regions can be matched using the geometric feature FPFH (Fast Point Feature Histogram) descriptor.

[0049] Semantic segmentation information can be used to improve the accuracy of coordinate alignment for sparse regions. Based on the semantic segmentation information of sparse regions in the fisheye image and corresponding regions in other RGB images, as well as the positional information of the point cloud data of the sparse regions and corresponding regions in other fisheye images, the point cloud data of the sparse regions is coordinate-aligned with the point cloud data of other fisheye images. This reduces computational cost while maintaining coordinate alignment accuracy.

[0050] Optionally, a first semantic feature vector can be constructed for each point cloud data within the sparse feature region based on the object category and boundary information of the sparse feature region in the fisheye image. A second semantic feature vector is constructed for each point cloud data based on the object category and boundary information of the corresponding regions in other fisheye images. Different weights can be assigned to the object category and boundary information when constructing the first and second semantic feature vectors. The similarity between the first and second semantic feature vectors of each point cloud data within the sparse feature region is calculated. The distance between the point cloud data is calculated based on the positional information of each point cloud data within the sparse feature region and the positional information of the corresponding point cloud data in other fisheye images. The similarity and distance are weighted to obtain a weighted value, and the sum of the weighted values ​​corresponding to all point cloud data within the sparse feature region is minimized to align the point cloud data in the sparse feature region with the point cloud data in other fisheye images.

[0051] The first overlapping region and the first non-overlapping region refer to the overlapping and non-overlapping regions of point cloud data from various perspectives within each UAV. The second overlapping region and the second non-overlapping region mentioned below refer to the overlapping and non-overlapping regions of point cloud data between different UAVs. The method for stitching the point cloud data in the first overlapping region and the second overlapping region is the same, as is the method for stitching the point cloud data in the first non-overlapping region and the second non-overlapping region.

[0052] Step S210: The point cloud data of the first overlapping region and the first non-overlapping region are stitched together to obtain the first target point cloud data corresponding to the UAV.

[0053] In related technologies, point cloud data is directly fused based on its location information. However, when point cloud data from multiple perspectives overlap, inconsistencies often arise due to noise and occlusion, resulting in ghosting or blurring. To address this issue, the depth values ​​of the overlapping areas are checked for consistency before stitching, and then stitching is performed based on the consistency results. Specifically:

[0054] The depth values ​​of each point cloud data point in the first overlapping region are obtained from various viewpoints, and the variance of the depth values ​​from each viewpoint is calculated. If the variance is less than or equal to a preset threshold, it indicates that the depth values ​​from each viewpoint fluctuate relatively little. The depth values ​​from each viewpoint are then weighted and averaged to obtain a weighted average, and the depth values ​​of the point cloud data in the first overlapping region are updated to this weighted average. If the variance is greater than the preset threshold, it indicates that the depth values ​​from each viewpoint fluctuate significantly. The stability of the depth values ​​of the first overlapping region at multiple times within the first historical time period is assessed, and the depth values ​​of the point cloud data in the first overlapping region are updated to the depth values ​​with the best stability.

[0055] For non-overlapping regions, a smooth transition method based on Poisson reconstruction can be used to avoid boundary cracks. Specifically, for each time point within the second historical time period, the variance of the depth values ​​of all point cloud data in the first non-overlapping region at that time point is calculated. The RGB gradient values ​​of all point cloud data in the first non-overlapping region at that time point are calculated. The motion distance of the point cloud data in the first non-overlapping region at that time point is calculated. The variance, RGB gradient values, and motion distance are weighted to obtain the confidence score. The depth values ​​of the point cloud data in the first non-overlapping region are then updated to the depth values ​​of the point cloud data at the time point corresponding to the maximum confidence score. This can be expressed as the following formula: (13); (14); Where m represents the point cloud data corresponding to the i-th pixel. This represents the confidence level of the depth value of the i-th pixel at time t. Let represent the variance of the depth value of the i-th pixel in the time interval (th~t). This represents the RGB gradient value of the i-th pixel at time t. , and Indicates the weighting coefficient. This represents the distance traveled by point cloud data m over the time interval (th~t). Let represent the depth value of the i-th pixel at time t, and N represent the total number of pixels in the first non-overlapping region. This represents the final depth value of the i-th pixel at time t.

[0056] Using the above stitching method, the PSNR (Peak Signal-to-Noise Ratio) of dynamic scene reconstruction was improved by 12.7dB, and the reconstruction integrity of motion-blurred regions was improved by 89%.

[0057] Step S212: Using the four semantic segmentation information corresponding to the UAV and other UAVs, as well as the position information of the first target point cloud data corresponding to the UAV and other UAVs, the coordinates of the first target point cloud data corresponding to the UAV and the first target point cloud data corresponding to other UAVs are aligned to obtain the second overlapping region and the second non-overlapping region.

[0058] As mentioned earlier, the method for stitching the second overlapping region is the same as that for stitching the first overlapping region. Specifically, the depth value of each point cloud data point in the second overlapping region is obtained from each viewpoint, and the variance of the depth values ​​from each viewpoint is calculated. If the variance is less than or equal to a preset threshold, the depth values ​​from each viewpoint are weighted and averaged to obtain a weighted average, and the depth values ​​of the point cloud data in the second overlapping region are updated to this weighted average. If the variance is greater than the preset threshold, the stability of the depth values ​​of the second overlapping region at multiple times within the first historical time period is assessed, and the depth values ​​of the point cloud data in the second overlapping region are updated to the depth values ​​with the best stability.

[0059] Step S214: The point cloud data of the second overlapping region and the second non-overlapping region are stitched together to obtain the second target point cloud data corresponding to the UAV.

[0060] Similarly, the method for stitching the second non-overlapping region is the same as that for the first non-overlapping region. That is, for each of the multiple moments within the second historical time period, the variance of the depth values ​​of all point cloud data in the second non-overlapping region at that moment is calculated. The RGB gradient values ​​of all point cloud data in the second non-overlapping region at that moment are calculated. The motion distance of the point cloud data in the second non-overlapping region at that moment is calculated. The variance, RGB gradient values, and motion distance are weighted to obtain the confidence score. The depth values ​​of the point cloud data in the second non-overlapping region are then updated to the depth values ​​of the point cloud data at the moment corresponding to the maximum confidence score.

[0061] The depth estimation method based on a drone swarm in this application embodiment sets four sets of cameras around each drone, each set including a fisheye camera and an RGB camera, enabling 360° 3D environment modeling and reducing blind spots by 80%. Within a single drone, a high-precision depth map of a single fisheye image is generated through binocular stereo matching. The point cloud data of the four fisheye images are aligned using the semantic segmentation information of the RGB images and the positional information of the point cloud data of the fisheye images, i.e., coordinate alignment based on multimodal features (geometric and semantic features), which improves the robustness of feature matching. Then, the data is stitched and fused to obtain the depth information around a single drone, thereby improving the accuracy of depth estimation for that single drone. Multiple drones within the swarm share depth information via wireless communication. The point cloud data of multiple drones are further aligned using the semantic segmentation information of the RGB images and the stitched point cloud data, and then stitched and fused to obtain the final depth information. By combining the depth information of multiple drones for depth estimation, comprehensive depth information can be obtained, and the accuracy of depth estimation can be improved, for example, the ranging error within 5 meters is ≤1.5%. Robustness is enhanced, supporting dynamic lighting and weak texture scenes, and the fault tolerance rate of multi-machine collaboration is improved by 40%.

[0062] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0063] This application also provides a depth estimation system based on a drone swarm. The depth estimation system includes a drone swarm comprising multiple drones, where the number of drones can be two, three, or more. See also... Figure 7 The number of drones is 3, and each drone 700 is equipped with four sets of cameras 710 covering a 360° field of view. Each set of cameras 710 includes a fisheye camera 711 and an RGB camera 712; each drone 700 includes a computing module 720 and a communication module 730. Fisheye camera 711 and RGB camera 712 are used to acquire fisheye and RGB images of the area around the UAV 700 in real time, respectively. The calculation module 720 is used to acquire RGB images and fisheye images, perform correction and binocular stereo matching on each fisheye image to obtain the corresponding disparity data, and convert the disparity data into depth values; determine the point cloud data of each fisheye image based on its position coordinates and depth values ​​in the camera coordinate system; perform semantic segmentation on each RGB image to obtain semantic segmentation information; for each set of RGB images and fisheye images, based on the semantic segmentation information in the RGB images and other RGB images, and the position information of the point cloud data of the fisheye images and other fisheye images, align the point cloud data of the fisheye images with the point cloud data of other fisheye images to obtain a first overlapping region and a first non-overlapping region; and stitch the point cloud data of the first overlapping region and the first non-overlapping region together to obtain the first target point cloud data corresponding to the UAV. The communication module 730 is used to communicate with other drones and obtain the first target point cloud data and semantic segmentation information corresponding to other drones; The calculation module 720 is also used to utilize the four semantic segmentation information corresponding to the UAV and other UAVs, as well as the position information of the first target point cloud data corresponding to the UAV and other UAVs, to coordinate the first target point cloud data corresponding to the UAV and the first target point cloud data corresponding to other UAVs to obtain a second overlapping region and a second non-overlapping region; and to stitch the point cloud data of the second overlapping region and the second non-overlapping region respectively to obtain the second target point cloud data corresponding to the UAV.

[0064] Optionally, the calculation module 720 is specifically used to perform coordinate alignment of the point cloud data of the fisheye image with the point cloud data of other fisheye images based on semantic segmentation information in the RGB image and other RGB images, and the positional information of the point cloud data of the fisheye image and other fisheye images, through the following steps: The point cloud data of the fisheye image is divided into feature-rich regions and feature-sparse regions. Based on the positional information of the point cloud data of the feature-rich region in the fisheye image and the positional information of the point cloud data of the corresponding region in other fisheye images, the point cloud data of the feature-rich region is aligned with the point cloud data of other fisheye images. Based on the semantic segmentation information of the sparse feature region in the fisheye image and the semantic segmentation information of the corresponding region in other RGB images, as well as the position information of the point cloud data of the sparse feature region and the position information of the point cloud data of the corresponding region in other fisheye images, the point cloud data of the sparse feature region is aligned with the point cloud data of other fisheye images.

[0065] Optionally, the semantic segmentation information includes object category information and object boundary information; The calculation module 720 is specifically used to perform coordinate alignment of the point cloud data of the sparse feature region with the point cloud data of the other fisheye image based on the semantic segmentation information of the sparse feature region in the fisheye image and the semantic segmentation information of the corresponding region in the other RGB image, as well as the position information of the point cloud data of the sparse feature region with the position information of the point cloud data of the corresponding region in the other fisheye image, through the following steps: Based on the object category information and object boundary information of the sparse feature region in the fisheye image, construct the first semantic feature vector of each point cloud data in the sparse feature region; Based on the object category information and object boundary information of the corresponding regions in other fisheye images, construct the second semantic feature vector of each point cloud data; Calculate the similarity between the first semantic feature vector and the second semantic feature vector of each point cloud data within the sparse feature region; The distance between the point cloud data is calculated based on the location information of each point cloud data in the sparse feature region and the location information of the corresponding point cloud data in other fisheye images. Similarity and distance are weighted to obtain a weighted value, and the sum of the weighted values ​​of all point cloud data in the sparse feature region is minimized to align the point cloud data in the sparse feature region with the point cloud data of other fisheye images.

[0066] Optionally, the calculation module 720 is specifically used to stitch together the point cloud data of the first overlapping region / second overlapping region through the following steps: Obtain the depth value of each point cloud data in the first overlapping region / second overlapping region from each viewpoint, and calculate the variance of the depth value from each viewpoint; If the variance is less than or equal to the preset threshold, the depth values ​​of each viewpoint are weighted and averaged to obtain the weighted average value, and the depth values ​​of the point cloud data of the first overlapping region / second overlapping region are updated to the weighted average value. If the variance is greater than a preset threshold, the stability of the depth values ​​of the first overlapping region / second overlapping region at multiple times within the first historical time period is judged, and the depth values ​​of the point cloud data of the first overlapping region / second overlapping region are updated to the depth values ​​with the best stability.

[0067] Optionally, the calculation module 720 is specifically used to stitch together the point cloud data of the first non-overlapping region / second non-overlapping region through the following steps: For each of the multiple moments within the second historical time period, calculate the variance of the depth values ​​of all point cloud data in the first non-overlapping region / second non-overlapping region at that moment; Calculate the RGB gradient values ​​of all point cloud data in the first non-overlapping region / second non-overlapping region at time t; Calculate the distance traveled by the point cloud data of the first non-overlapping region / second non-overlapping region at time 1; The confidence level is obtained by weighting the variance, RGB gradient values, and motion distance. The depth values ​​of the point cloud data in the first non-overlapping region / second non-overlapping region are then updated to the depth values ​​of the point cloud data at the time corresponding to the maximum confidence value.

[0068] Optionally, the calculation module 720 is specifically used to perform binocular stereo matching on the corrected fisheye image through the following steps to obtain the corresponding disparity data: The corrected fisheye image is epipolar aligned to obtain the grayscale images of the left and right eyes; The Census transform is used to calculate the features of each pixel in the left and right grayscale images respectively, so as to obtain the first left-eye feature and the first right-eye feature. The Sobel operator is used to calculate the features of each pixel in the left and right grayscale images respectively, to obtain the second left-eye feature and the second right-eye feature. Calculate the first gradient BT map based on the first left-eye feature and the first right-eye feature; calculate the second gradient BT map based on the second left-eye feature and the second right-eye feature; weight the first gradient BT map and the second gradient BT map to obtain the target gradient BT map; Calculate the grayscale BT image based on the left and right eye grayscale images; Based on the grayscale BT image and the target gradient BT image, calculate the cost of each pixel in the left and right grayscale images; Disparity data is obtained by cost aggregation and optimization based on the cost value of each pixel in the left and right grayscale images.

[0069] The specific details of each module in the above system have been described in detail in the corresponding methods, so they will not be repeated here.

[0070] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0071] This application also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the depth estimation method based on UAV swarm described in this example embodiment.

[0072] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned depth estimation method based on a drone swarm.

[0073] It should be noted that the computer-readable storage medium shown in this application can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0074] In this embodiment of the application, a computer program product is also provided, which, when run on a computer, causes the computer to execute the above-described depth estimation method based on a drone swarm.

[0075] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0076] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A depth estimation method based on UAV swarms, characterized in that, The method, applied to a single drone in a drone swarm, wherein the drone is equipped with four sets of cameras covering a 360° field of view, each set of cameras including a fisheye camera and an RGB camera, comprises: Four sets of RGB images and fisheye images around the drone are acquired in real time. Each fisheye image is corrected and stereo matched with binoculars to obtain the corresponding disparity data, and the disparity data is converted into depth values. The point cloud data of each fisheye image is determined based on its position coordinates in the camera coordinate system and the depth value. Semantic segmentation is performed on each RGB image to obtain semantic segmentation information; For each set of RGB images and fisheye images, based on the semantic segmentation information in the RGB images and other RGB images, and the position information of the point cloud data of the fisheye images and other fisheye images, the point cloud data of the fisheye images are aligned with the point cloud data of other fisheye images to obtain a first overlapping region and a first non-overlapping region. The point cloud data of the first overlapping region and the first non-overlapping region are stitched together to obtain the first target point cloud data corresponding to the UAV. Using the four semantic segmentation information corresponding to the UAV and other UAVs, as well as the position information of the first target point cloud data corresponding to the UAV and other UAVs, the first target point cloud data corresponding to the UAV and other UAVs are aligned with coordinates to obtain the second overlapping region and the second non-overlapping region. The point cloud data of the second overlapping region and the second non-overlapping region are stitched together to obtain the second target point cloud data corresponding to the UAV.

2. The method according to claim 1, characterized in that, The step of aligning the point cloud data of the fisheye image with the point cloud data of other fisheye images based on the semantic segmentation information in the RGB image and other RGB images, and the position information of the point cloud data of the fisheye image and other fisheye images, includes: The point cloud data of the fisheye image is divided into feature-rich regions and feature-sparse regions. Based on the position information of the point cloud data of the feature-rich region in the fisheye image and the position information of the point cloud data of the corresponding region in other fisheye images, the point cloud data of the feature-rich region is aligned with the point cloud data of other fisheye images. Based on the semantic segmentation information of the sparse feature region in the fisheye image and the semantic segmentation information of the corresponding region in other RGB images, as well as the position information of the point cloud data of the sparse feature region and the position information of the point cloud data of the corresponding region in other fisheye images, the point cloud data of the sparse feature region is aligned with the point cloud data of other fisheye images.

3. The method according to claim 2, characterized in that, The semantic segmentation information includes object category information and object boundary information; The step of aligning the point cloud data of the sparse region with the point cloud data of the other fisheye image based on the semantic segmentation information of the sparse region in the fisheye image and the semantic segmentation information of the corresponding region in other RGB images, as well as the position information of the point cloud data of the sparse region with the position information of the point cloud data of the corresponding region in other fisheye images, includes: Based on the object category information and object boundary information of the sparse feature region in the fisheye image, construct the first semantic feature vector of each point cloud data in the sparse feature region; Based on the object category information and object boundary information of the corresponding regions in other fisheye images, construct the second semantic feature vector of each point cloud data; Calculate the similarity between the first semantic feature vector and the second semantic feature vector of each point cloud data within the sparse feature region; The distance between the point cloud data is calculated based on the position information of each point cloud data in the sparse feature region and the position information of the corresponding point cloud data in other fisheye images. The similarity and distance are weighted to obtain a weighted value, and the sum of the weighted values ​​corresponding to all point cloud data in the sparse feature region is minimized to align the point cloud data in the sparse feature region with the point cloud data of other fisheye images.

4. The method according to claim 1, characterized in that, The point cloud data of the first overlapping region / second overlapping region are stitched together, including: Obtain the depth value of each point cloud data in the first overlapping region / second overlapping region from each viewpoint, and calculate the variance of the depth values ​​from each viewpoint; If the variance is less than or equal to a preset threshold, the depth values ​​of each viewpoint are weighted and averaged to obtain a weighted average value, and the depth values ​​of the point cloud data of the first overlapping region / second overlapping region are updated to the weighted average value. If the variance is greater than a preset threshold, the stability of the depth values ​​of the first overlapping region / second overlapping region at multiple times within the first historical time period is judged, and the depth values ​​of the point cloud data of the first overlapping region / second overlapping region are updated to the depth values ​​with the best stability.

5. The method according to claim 1, characterized in that, The point cloud data of the first non-overlapping region / second non-overlapping region are stitched together, including: For each of the multiple moments within the second historical time period, calculate the variance of the depth values ​​of all point cloud data in the first non-overlapping region / second non-overlapping region at that moment; Calculate the RGB gradient values ​​of all point cloud data in the first non-overlapping region / second non-overlapping region at the stated time; Calculate the motion distance of the point cloud data of the first non-overlapping region / second non-overlapping region at the time; The variance, RGB gradient values, and motion distance are weighted to obtain the confidence level; The depth values ​​of the point cloud data in the first non-overlapping region / second non-overlapping region are then updated to the depth values ​​of the point cloud data at the time corresponding to the maximum confidence value.

6. The method according to claim 1, characterized in that, Binocular stereo matching was performed on the corrected fisheye image to obtain the corresponding disparity data, including: The corrected fisheye image is epipolar aligned to obtain the grayscale images of the left and right eyes; The Census transform is used to calculate the features of each pixel in the left and right grayscale images to obtain the first left-eye feature and the first right-eye feature. The Sobel operator is used to calculate the features of each pixel in the left and right grayscale images to obtain the second left-eye feature and the second right-eye feature. Calculate the first gradient BT map based on the first left-eye feature and the first right-eye feature; calculate the second gradient BT map based on the second left-eye feature and the second right-eye feature; weight the first gradient BT map and the second gradient BT map to obtain the target gradient BT map; Calculate the grayscale BT image based on the left and right eye grayscale images; Based on the grayscale BT image and the target gradient BT image, calculate the cost of each pixel in the left-eye grayscale image and the right-eye grayscale image; Cost aggregation and optimization are performed on the cost of each pixel in the left and right grayscale images to obtain disparity data.

7. A depth estimation system based on unmanned aerial vehicle (UAV) swarms, characterized in that, The system includes: a drone swarm comprising multiple drones, each drone equipped with four sets of cameras covering a 360° field of view, each set of cameras including a fisheye camera and an RGB camera; each drone includes a computing module and a communication module; The fisheye camera and the RGB camera are used to acquire fisheye images and RGB images of the area around the drone in real time, respectively. The calculation module is used to acquire the RGB image and the fisheye image, perform correction and binocular stereo matching on each fisheye image to obtain corresponding disparity data, and convert the disparity data into depth values; determine the point cloud data of each fisheye image based on its position coordinates in the camera coordinate system and the depth value; perform semantic segmentation on each RGB image to obtain semantic segmentation information; for each set of RGB images and fisheye images, based on the semantic segmentation information in the RGB image and other RGB images, and the position information of the point cloud data of the fisheye image and other fisheye images, align the point cloud data of the fisheye image with the point cloud data of other fisheye images to obtain a first overlapping region and a first non-overlapping region; and stitch the point cloud data of the first overlapping region and the first non-overlapping region together to obtain the first target point cloud data corresponding to the UAV. The communication module is used to communicate with other drones and obtain the first target point cloud data and semantic segmentation information corresponding to other drones; The calculation module is further configured to utilize the four semantic segmentation information corresponding to the UAV and other UAVs, as well as the position information of the first target point cloud data corresponding to the UAV and other UAVs, to perform coordinate alignment of the first target point cloud data corresponding to the UAV and the first target point cloud data corresponding to other UAVs to obtain a second overlapping region and a second non-overlapping region; and to stitch together the point cloud data of the second overlapping region and the second non-overlapping region to obtain the second target point cloud data corresponding to the UAV.

8. The system according to claim 7, characterized in that, The calculation module is specifically used to perform coordinate alignment of the point cloud data of the fisheye image with the point cloud data of other fisheye images based on the semantic segmentation information in the RGB image and other RGB images, and the position information of the point cloud data of the fisheye image and other fisheye images, through the following steps: The point cloud data of the fisheye image is divided into feature-rich regions and feature-sparse regions. Based on the position information of the point cloud data of the feature-rich region in the fisheye image and the position information of the point cloud data of the corresponding region in other fisheye images, the point cloud data of the feature-rich region is aligned with the point cloud data of other fisheye images. Based on the semantic segmentation information of the sparse feature region in the fisheye image and the semantic segmentation information of the corresponding region in other RGB images, as well as the position information of the point cloud data of the sparse feature region and the position information of the point cloud data of the corresponding region in other fisheye images, the point cloud data of the sparse feature region is aligned with the point cloud data of other fisheye images.

9. The system according to claim 8, characterized in that, The semantic segmentation information includes object category information and object boundary information; The calculation module is specifically used to perform coordinate alignment of the point cloud data of the sparse feature region with the point cloud data of the other fisheye image based on the semantic segmentation information of the sparse feature region in the fisheye image and the semantic segmentation information of the corresponding region in the other RGB image, as well as the position information of the point cloud data of the sparse feature region with the position information of the point cloud data of the corresponding region in the other fisheye image, through the following steps: Based on the object category information and object boundary information of the sparse feature region in the fisheye image, construct the first semantic feature vector of each point cloud data in the sparse feature region; Based on the object category information and object boundary information of the corresponding regions in other fisheye images, construct the second semantic feature vector of each point cloud data; Calculate the similarity between the first semantic feature vector and the second semantic feature vector of each point cloud data within the sparse feature region; The distance between the point cloud data is calculated based on the position information of each point cloud data in the sparse feature region and the position information of the corresponding point cloud data in other fisheye images. The similarity and distance are weighted to obtain a weighted value, and the sum of the weighted values ​​corresponding to all point cloud data in the sparse feature region is minimized to align the point cloud data in the sparse feature region with the point cloud data of other fisheye images.

10. The system according to claim 7, characterized in that, The calculation module is specifically used to stitch together the point cloud data of the first overlapping region / second overlapping region through the following steps: Obtain the depth value of each point cloud data in the first overlapping region / second overlapping region from each viewpoint, and calculate the variance of the depth values ​​from each viewpoint; If the variance is less than or equal to a preset threshold, the depth values ​​of each viewpoint are weighted and averaged to obtain a weighted average value, and the depth values ​​of the point cloud data of the first overlapping region / second overlapping region are updated to the weighted average value. If the variance is greater than a preset threshold, the stability of the depth values ​​of the first overlapping region / second overlapping region at multiple times within the first historical time period is judged, and the depth values ​​of the point cloud data of the first overlapping region / second overlapping region are updated to the depth values ​​with the best stability.

Images