Unmanned aerial vehicle surveying and mapping method and device, unmanned aerial vehicle, medium and product

Abstract

The embodiment of the invention discloses an unmanned aerial vehicle surveying and mapping method and device, an unmanned aerial vehicle, a medium and a product, and the method comprises the steps: employing a camera to shoot a plurality of environment images in a flight process of the unmanned aerial vehicle according to a preset track, enabling each environment image to serve as a to-be-matched image, enabling the adjacent to-be-matched images to have a coincident shooting region, and enabling the to-be-matched image to be matched with the shooting region; the overlapped shooting area comprises key points; determining initial camera external parameters according to key points in each group of adjacent to-be-matched images and body internal parameters and height information of the camera; optimizing the initial camera extrinsic parameters according to a global structure motion reconstruction algorithm to obtain target camera extrinsic parameters; determining a coverage range of an orthograph based on a bounding box of each to-be-matched image, wherein the bounding box is determined based on a target camera extrinsic parameter and a field angle; determining an initial orthograph according to pixel values in the coverage area; and correction and multi-band fusion are carried out on the initial orthograph to obtain the target orthograph, so that the efficiency and the accuracy of generating the orthographs in various geographical environments are improved.

Description

Technical Field

[0001] The disclosed embodiments relate to the technical field of drone mapping, and more particularly to a drone surveying and mapping method, device, drone, medium and product. Background Art

[0002] Many existing drone mapping technologies rely on a single type of sensor, such as optical cameras or lidar. When faced with complex environments, single sensor systems cannot obtain complete ground images or lack color and texture information due to problems such as line of sight occlusion and reflection.

[0003] Traditional ortho mapping algorithms, such as image-based ortho projection methods, may encounter performance bottlenecks when dealing with large-scale or complex terrains. These algorithms are computationally intensive and have poor real-time performance when processing large amounts of data, making it difficult to meet the needs of rapidly changing environments. In addition, high-precision sensors and powerful computing platforms are often costly, limiting their popularity in commercial and industrial applications. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device, drone, medium and product for surveying and mapping by drones, which improve the efficiency and quality of orthophoto generation.

[0005] In a first aspect, a method for surveying and mapping by an unmanned aerial vehicle is provided, which is applied to an unmanned aerial vehicle, wherein the unmanned aerial vehicle includes a camera; the method includes:

[0006] When the drone is flying along a preset trajectory, the camera is used to capture multiple environment images, wherein each environment image is used as an image to be matched, and adjacent images to be matched have overlapping capture areas, and the overlapping capture areas include key points;

[0007] Determine the initial camera external parameters according to the key points in each group of adjacent images to be matched, the camera's internal parameters and height information; wherein the height information includes the height of the key points in the topographic map;

[0008] The initial camera extrinsic parameters are optimized according to the global structure motion reconstruction GLOMAP algorithm to obtain the target camera extrinsic parameters;

[0009] Determine the coverage of the orthophoto based on the bounding boxes of each image to be matched, wherein the bounding boxes are determined based on the target camera extrinsics and the field of view; the field of view corresponds to the image to be matched; and determine the initial orthophoto according to the pixel values ​​in the coverage;

[0010] The initial orthophoto is corrected and multi-band fused to obtain a target orthophoto.

[0011] In a second aspect, a UAV surveying and mapping device is provided, comprising:

[0012] An image acquisition module, used to use the camera to capture multiple environment images when the drone is flying along a preset trajectory, wherein each environment image is used as an image to be matched, and adjacent images to be matched have overlapping shooting areas, and the overlapping shooting areas include key points;

[0013] An initial camera extrinsic parameter determination module, used to determine the initial camera extrinsic parameters according to the key points in each group of adjacent images to be matched, the camera's body internal parameters and height information; wherein the height information includes the height of the key point in the topographic map;

[0014] A target camera extrinsic parameter determination module is used to optimize the initial camera extrinsic parameters according to the global structure motion reconstruction GLOMAP algorithm to obtain the target camera extrinsic parameters;

[0015] An initial orthophoto determination module is used to determine the coverage of the orthophoto based on the bounding boxes of each image to be matched, wherein the bounding boxes are determined based on the target camera extrinsics and the field of view; the field of view corresponds to the image to be matched; and determine the initial orthophoto according to the pixel values ​​in the coverage;

[0016] The target orthophoto determination module is used to correct and fuse the multi-bands of the initial orthophoto to obtain the target orthophoto.

[0017] In a third aspect, a drone is provided, comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the drone surveying and mapping method as described in the first aspect above.

[0021] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the drone mapping method as described in the first aspect is implemented.

[0022] In a fifth aspect, a computer program product is provided, the computer program product comprising a computer program, and when the computer program is executed by a processor, the drone surveying and mapping method as described in the first aspect is implemented.

[0023] The disclosed embodiments disclose a method, device, drone, medium and product for surveying and mapping by unmanned aerial vehicle, including: when the drone is flying along a preset trajectory, using the camera to shoot a plurality of environment images, wherein each environment image is used as an image to be matched, and adjacent images to be matched have overlapping shooting areas, and the overlapping shooting areas include key points; determining initial camera extrinsics according to key points in each group of adjacent images to be matched, the camera's body internal parameters and height information; wherein the height information includes the height of the key point in a topographic map; optimizing the initial camera extrinsics according to a global structure motion reconstruction GLO MAP algorithm to obtain target camera extrinsics; determining the coverage of an orthophoto based on the bounding boxes of each image to be matched, the bounding box being determined based on the target camera extrinsics and a field of view; the field of view corresponds to the image to be matched; and determining an initial orthophoto according to pixel values ​​in the coverage; correcting and multi-band fusion the initial orthophoto to obtain a target orthophoto. This technical solution determines the initial camera extrinsics based on the image to be matched, the camera's body internal parameters and height information, and corrects the initial camera extrinsics to obtain the target camera extrinsics, determines the bounding box based on the target camera extrinsics and the field of view, and determines the coverage of the orthophoto based on the bounding box, and determines the target orthophoto by the pixel values ​​of each image to be matched within the coverage range, thereby improving the efficiency and accuracy of orthophoto generation in a variety of geographical environments, making it possible to directly conduct land surveying and mapping on high-definition maps, improving the efficiency and quality of orthophoto generation, and meeting the requirements of industrial-level practical applications, providing an efficient and reliable solution for the fields of geographic information systems, urban planning, environmental monitoring, and disaster assessment.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the embodiments of the present disclosure. Other features of the embodiments of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 It is a flow chart of a method for surveying and mapping by an unmanned aerial vehicle provided in the first embodiment of the present disclosure;

[0027] Figure 2 This is an application example of a drone surveying and mapping method provided in the first embodiment of the present disclosure;

[0028] Figure 3 It is a structural schematic diagram of a drone surveying and mapping device provided in the second embodiment of the present disclosure;

[0029] Figure 4 It is a structural schematic diagram of a drone provided in Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the embodiments of the present disclosure.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0032] Embodiment 1

[0033] Figure 1 This is a flow chart of a method for surveying and mapping by a drone provided in the first embodiment of the present disclosure. This embodiment is applicable to the case of mapping by using a drone. The method can be executed by a drone surveying and mapping device. The drone surveying and mapping device can be implemented in the form of hardware and / or software. The drone surveying and mapping device can be configured in a drone. Figure 1 As shown, the method includes:

[0034] S110. When the drone is flying along a preset trajectory, a plurality of environment images are captured by a camera, wherein each environment image is used as an image to be matched, and adjacent images to be matched have overlapping capture areas, and the overlapping capture areas include key points.

[0035] In this embodiment, the preset trajectory may be a pre-set trajectory, and the drone may fly according to the preset trajectory. When the drone flies according to the preset trajectory, a camera installed on the drone may be used to capture multiple environmental images of the preset trajectory, and each environmental image may be used as an image to be matched. Adjacent images to be matched have overlapping shooting areas, and the overlapping shooting areas include key points.

[0036] Continuing with the above description, key points can be feature points with significance and stability in the image to be matched, which are usually points that will not disappear due to changes in lighting, scale, rotation, etc., such as corner points, edge points, bright spots in dark areas, and dark spots in bright areas. Key points can be extracted from the image to be matched by a preset algorithm, and the preset algorithm can be a Scale-Invariant Feature Transform (SIFT). The SIFT algorithm is a feature extraction algorithm widely used in image processing and computer vision.

[0037] S120, determining initial camera extrinsic parameters according to key points in each group of adjacent images to be matched, camera body internal parameters and height information, wherein the height information includes the height of the key points in the topographic map.

[0038] Specifically, the key points in each group of adjacent images to be matched can be extracted by the SIFT algorithm, and the feature descriptors corresponding to each key point can be generated. The initial camera extrinsic parameters can be determined by the descriptors corresponding to the key points in each group of adjacent images to be matched, the camera's body internal parameters and height information. Among them, the initial camera extrinsic parameters can be the camera extrinsic parameters of each image to be matched relative to a common reference frame (usually a world coordinate system). In this embodiment, the height information can be the height of the key point in the topographic map; the topographic map can be used to represent the actual environment of the preset trajectory. The camera's body internal parameters can be obtained by calibrating the camera, and the camera's body internal parameters may include: focal length, stationary point position and distortion coefficient.

[0039] It should be noted that when using the SIFT algorithm to extract key points and generate corresponding feature descriptors, OpenCL can be used for hardware acceleration on a low-cost graphics processing unit (GPU), which can process more data in the same time and avoid the purchase of expensive high-performance GPUs, thereby reducing overall hardware costs. Among them, OpenCL (Open Computing Language) can be an open standard parallel programming framework.

[0040] S130, optimizing the initial camera extrinsic parameters according to the global structure motion reconstruction GLOMAP algorithm to obtain the target camera extrinsic parameters.

[0041] In this embodiment, after the initial camera extrinsic parameters are obtained, the initial camera extrinsic parameters may be optimized according to a global structure-from-motion revisited (GLOMAP) algorithm to obtain target camera extrinsic parameters.

[0042] For example, the Structure from Motion (SFM) method can be used to correct the initial camera extrinsics using two key steps: rotation averaging and global pose estimation in GLOMAP, so that the rotation and position of the camera are aligned with the global reference frame. The rotation average estimates the globally consistent camera rotation. The global pose estimation jointly estimates the global position of all cameras by directly optimizing the positions of camera rays and 3D points.

[0043] The optimization of the initial camera extrinsic parameters using global rotation average can be expressed as follows: given the relative rotation matrix between multiple pairs of cameras, the objective function is constructed based on the relative rotation matrix between multiple pairs of cameras and the global rotation matrix of each camera relative to the global reference frame; wherein the relative rotation matrix between multiple pairs of cameras refers to a matrix that describes the rotation relationship between two camera coordinate systems, and the global reference frame can be a fixed reference system used to describe the position of an object in the real world. For example, i and j can represent the i-th and j-th cameras, and the global rotation matrix of the i-th camera relative to the global reference frame can be expressed as R i , the global rotation matrix of the jth camera relative to the global reference frame can be expressed as R j , the relative rotation matrix between the i-th and j-th cameras can be expressed as R ij , then the objective function can be expressed as:

[0044]

[0045] Choose the right R i , so that the value of the objective function is minimized. Among them, E is the set of adjacent image pairs to be matched, and log represents the logarithmic mapping of the matrix, which is used to convert the difference of the rotation matrix into a vector form. This function is essentially the least squares method, which minimizes the cumulative error of all relative measurements.

[0046] Global pose estimation can mean that in the SFM process, not only the rotation of the camera is estimated, but also the global position of the camera is estimated. The optimization of the initial camera extrinsic parameters using global pose estimation can be expressed as follows: given a set of relative poses of cameras and point-to-camera observations, the relative pose refers to the relative position and direction relationship between the two cameras, which is implicit in the observation model. Among them, the observation model can be obtained based on the correspondence between each key point in the adjacent images to be matched. The observation model uses these correspondences to estimate the camera pose and the position of the 3D point, reflecting the mapping relationship from the camera pose and the 3D point position to the 2D projection on the image. Point-to-camera observation refers to the projection of the 3D point in the camera coordinate system, that is, the 2D image coordinates. By jointly estimating the global pose of all cameras and the position of the 3D points. Construct the following objective function:

[0047]

[0048] Among them, x i is the global pose of the ith camera, p j is the position of the jth 3D point, r ij is the observation of the jth 3D point in the i-th camera, that is, the position of the point on the image, and f can be the observation model. The purpose of this objective function is to find x that minimizes the sum of squares of all observation errors. i and p j The objective function can be used to find a set of camera poses x i and the 3D point position p j , so that these poses and positions can best explain all the observation data. It should be noted that r ij The value of can be obtained by converting the key points in the first image to be matched to the second image to be matched. i and p j An initial estimate of can be obtained by using feature matching and triangulation methods to get a rough estimate. During the optimization process, x i and p j The initial estimate of is continuously updated through an iterative method. Compared with the traditional method of performing translation averaging followed by triangulation, this method improves the accuracy and robustness of positioning.

[0049] S140, determining the coverage of the orthophoto based on the bounding boxes of each image to be matched, where the bounding boxes are determined based on the extrinsic parameters of the target camera and the field of view; the field of view corresponds to the image to be matched; and determining the initial orthophoto according to the pixel values ​​in the coverage.

[0050] In this embodiment, the orthophoto may be an image produced by photogrammetry, which converts the tilted aerial or satellite image into a perspective perpendicular to the ground after geometric correction, just like it is directly photographed from a direction perpendicular to the ground surface. This image processing method eliminates the perspective distortion caused by the undulating terrain, so that the orthophoto has the characteristics of a map, that is, the position and size of the objects are consistent with the actual terrain.

[0051] Continuing with the above description, the bounding box can be the boundary of the image to be matched, such as: if the image to be matched occupies a part of the orthographic image, the bounding box can be understood as the boundary of this part. The bounding box can be determined based on the external parameters and field of view of the target camera. Among them, the field of view can correspond to the image to be matched, that is, the field of view can be the field of view when the camera shoots the image to be matched. The field of view can be calculated by the intrinsic parameters of the camera. The field of view can include the horizontal field of view and the vertical field of view. The horizontal field of view can be expressed as: FOVx, and the vertical field of view can be expressed as FOVy, then:

[0052]

[0053] Where f can be the focal length of the camera, W can be the width of the image sensor, and H can be the height of the image sensor. The bounding box of each image to be matched can be calculated based on the field of view (FOV) of the camera and the real-world position coordinates of the camera. This position coordinate is determined by the target camera extrinsic parameters of the camera, because the target camera extrinsic parameters provide the position information of the camera coordinate system relative to the world coordinate system.

[0054] Continuing with the above description, after the bounding boxes of the images to be matched are determined, the coverage of the orthophoto can be determined based on the bounding boxes of the images to be matched, wherein the coverage can be the width and height of the orthophoto.

[0055] Exemplarily, the bounding boxes of each image to be matched can be traversed, and the largest bounding box and the smallest bounding box can be selected from each bounding box: the smallest horizontal coordinate and the smallest vertical coordinate of the smallest bounding box can be used as the coordinates of the lower left corner of the orthophoto; the maximum horizontal coordinate and the maximum vertical coordinate of the largest bounding box. These values ​​will be used as the coordinates of the upper right corner of the orthophoto. The width and height of the orthophoto can be determined based on the two coordinates. It should be noted that the width and height of the orthophoto can also be adjusted according to the ground distance.

[0056] S150, correcting and multi-band fusion of the initial orthophoto to obtain a target orthophoto.

[0057] Specifically, after obtaining the initial orthophoto, the initial orthophoto can be corrected and band fused, and the visual effect and quality of the image can be optimized through multiple steps to obtain the target orthophoto.

[0058] For example, the initial orthophoto can be subjected to Gamma correction. Gamma correction, also known as gamma correction, is an image processing technology that is mainly used to adjust the brightness and contrast of an image to make it more consistent with the visual perception of the human eye. Through Gamma correction, the brightness and contrast of the resulting orthophoto are adjusted. The principle of Gamma correction can be expressed by the following formula:

[0059]

[0060] Among them, I in is the pixel value of the original initial orthophoto, I out is the pixel value of the corrected initial orthophoto, ranging from 0 to 255. γ is the gamma value, which is used to control brightness and contrast. A value less than 1 will make the initial orthophoto darker, and a value of 1 will make the initial orthophoto brighter. Since the initial orthophoto is composed of multiple images, the exposure of each image may be different. If some areas of the stitched image are overexposed, the overall image will look whitish. Using gamma correction can improve the visual effect of the image and make the orthophoto result closer to reality.

[0061] Exemplarily, multiband blending can also be performed on the initial orthophoto, where the initial orthophoto is decomposed into pyramid levels of different scales, starting from the coarsest level, and each level is gradually fused upwards, and weighted averaging is used to enhance details on the fused image to retain the high-frequency information of the image and obtain a smooth transition between pixels from different images.

[0062] The present embodiment provides a method for surveying and mapping by a drone, comprising: when the drone is flying along a preset trajectory, using the camera to shoot a plurality of environment images, wherein each environment image is used as an image to be matched, adjacent images to be matched have overlapping shooting areas, and the overlapping shooting areas include key points; determining initial camera extrinsics according to the key points in each group of adjacent images to be matched, the camera's body internal parameters and height information; optimizing the initial camera extrinsics according to a global structure motion reconstruction GLOMAP algorithm to obtain target camera extrinsics; wherein the height information includes the height of the key point in a topographic map; determining the coverage of an orthophoto based on a bounding box of each image to be matched, the bounding box being determined based on the target camera extrinsics and a field of view; the field of view corresponding to the image to be matched; and determining an initial orthophoto based on pixel values ​​in the coverage; correcting and multi-band fusion of the initial orthophoto to obtain a target orthophoto, thereby improving the efficiency and accuracy of generating orthophotos in various geographical environments, making it possible to directly conduct land surveying and mapping on a high-definition map, and improving the efficiency and quality of orthophoto generation.

[0063] As an optional implementation of this embodiment, the drone also includes a global positioning system GPS and an inertial measurement unit IMU; the drone surveying and mapping method provided in this embodiment may also include:

[0064] 1) When the UAV is flying along a preset trajectory, the GPS and the IMU are used to collect the current flight information of the UAV in real time; the flight information includes speed information, altitude information, position information and heading angle information.

[0065] In this embodiment, the Global Positioning System (GPS) may be a satellite-based navigation system that provides geographic location and time information worldwide, and determines the precise location of a receiver by receiving signals transmitted from a group of satellites in Earth orbit. An Inertial Measurement Unit (IMU) may be a device that measures and reports an object's specific force, angular velocity, and in some cases, the magnetic field of an object surrounding it. An IMU may include an accelerometer (measuring linear acceleration), a gyroscope (measuring angular velocity), and a magnetometer (measuring magnetic field), among others.

[0066] Specifically, when the drone is flying along a preset trajectory, GPS and IMU can be used to collect the drone's current flight information in real time. The flight information includes speed information, altitude information, location information, and heading angle information.

[0067] Among them, the speed information may include the ground speed provided by GPS and the acceleration and angular velocity of the drone on the x, y, and z axes provided by IMU; the altitude information may include the altitude information provided by GPS; the position information may include the latitude and longitude information of the drone in the earth coordinate system provided by GPS; and the heading angle information may include the heading angle information provided by GPS.

[0068] 2) Based on the flight information at the current moment, a state prediction model is constructed using a Kalman filter to determine the predicted position and predicted speed of the UAV at the next moment using the state prediction model;

[0069] Specifically, after obtaining the flight information, the Kalman filter can be used to link the data provided by GPS and IMU with the state variables to build a state prediction model; the state variables can be variables for estimating the state of the drone. This process can specifically include: based on the acceleration and angular velocity data provided by the IMU, the Kalman filter predicts the position and speed of the drone at the next moment according to the state equation.

[0070] 3) adjusting the flight parameters of the UAV according to the predicted position and the predicted speed so that the UAV flies according to the preset trajectory.

[0071] Specifically, after obtaining the preset position and predicted speed, the precise position and speed information provided by GPS can be used to adjust the flight parameters of the drone so that the drone flies according to the preset trajectory, thereby reducing the error caused by IMU integration drift.

[0072] As an optional implementation of this embodiment, the drone further includes: a 4D millimeter wave radar; the drone surveying and mapping method provided in this embodiment may also include:

[0073] 1) Before the UAV takes off, the GPS and the IMU are synchronized in time and aligned in coordinates, and the external parameters of the UAV are determined;

[0074] It is known that before the drone takes off, the GPS and IMU need to be calibrated to ensure that the data they provide are synchronized in time and correctly aligned with the drone's coordinate system. The drone's external parameters are determined through the calibration operation. The drone's external parameters can be parameters that describe the spatial relationship between the drone's body coordinate system and the world coordinate system. For example, the drone's external parameters can be expressed as Among them, B can represent the drone body coordinate system, and W can represent the world coordinate system.

[0075] 2) determining the external parameters of the 4D millimeter wave radar based on the external parameters of the drone and the external parameters of the 4D millimeter wave radar relative to the drone body coordinate system;

[0076] In this embodiment, the 4D millimeter wave radar can be an advanced radar technology that can measure the distance, direction and speed (3D) of the target, and can also measure the height of the target. The body external parameters of the 4D millimeter wave radar can be determined based on the external parameters of the drone and the external parameters of the 4D millimeter wave radar relative to the drone body coordinate system.

[0077] For example, the external parameters of the 4D millimeter wave radar relative to the drone body coordinate system can be expressed as R can represent the 4D millimeter wave radar, and B can represent the coordinate system of the drone body. The external parameters of the 4D millimeter wave radar can be expressed as R can represent the 4D millimeter-wave radar, and W can represent the world coordinate system. Based on the external parameters of the drone and the external parameters of the 4D millimeter-wave radar relative to the drone body coordinate system, the external parameters of the 4D millimeter-wave radar can be determined as follows:

[0078]

[0079] 3) constructing an initial topographic map according to the radar information acquired by the 4D millimeter wave radar; the initial topographic map includes first coordinates; the first coordinates are the coordinates of each topographic point in the coordinate system of the 4D millimeter wave radar;

[0080] Specifically, radar information can be obtained based on the 4D millimeter wave radar, and the radar information may include distance, speed, azimuth, and altitude information. An initial terrain map can be constructed based on the radar information. Specifically, points on the ground can be detected and tracked based on the radar information to generate a terrain map. The radar measures the distance and speed to surrounding objects by transmitting and receiving millimeter wave signals. These measurement results are recorded in the radar's coordinate system to obtain the first coordinate detected in the radar coordinate system. The first coordinate may be the coordinate of each terrain point in the coordinate system of the 4D millimeter wave radar. Exemplarily, the first coordinate may be expressed as P R .

[0081] 4) Based on the external parameters of the 4D millimeter-wave radar, the first coordinates are converted into second coordinates to obtain the topographic map, where the second coordinates are the position coordinates of each topographic point in the world coordinate system.

[0082] Specifically, the first coordinate can be converted into a second coordinate based on the external parameters of the 4D millimeter wave radar, and the second coordinate can be the position coordinates of each terrain point in the world coordinate system. The terrain point can be the spatial position information of each point on the ground or obstacle detected by the radar. The first coordinate is converted into the world coordinate system of the drone. For example, the external parameters of the 4D millimeter-wave radar can be expressed as The second coordinate can be expressed as P W , then:

[0083]

[0084] Following the above description, the terrain point P in the obtained world coordinate system W After that, B-spline can be used to represent the terrain map. It realizes the partition fitting in 2D space (X, Y plane). By giving x, y coordinates, the terrain height of the corresponding position can be queried. Among them, B-spline can be based on a series of control points to define curves or surfaces, which can be combined with these control points through a specific set of basis functions to generate smooth curves or surface shapes.

[0085] As an optional implementation of this embodiment, for any group of adjacent images to be matched, the adjacent images to be matched include a first image to be matched and a second image to be matched;

[0086] The initial camera external parameters are determined according to the key points in each group of adjacent images to be matched, the camera's body internal parameters and height information, including:

[0087] 1) For any group of adjacent images to be matched, the first pixel coordinates of each key point in the first image to be matched are converted into corresponding second pixel coordinates according to the estimated body external parameters, body internal parameters and height information of the camera; the estimated body external parameters are determined based on the body external parameters of the drone and the external parameters of the camera relative to the drone body coordinate system; the first pixel coordinates correspond to the pixel coordinate system of the first image to be matched; the second pixel coordinates correspond to the pixel coordinate system of the second image to be matched.

[0088] In this embodiment, for any group of adjacent images to be matched, the estimated external parameters of the drone corresponding to two of the adjacent images to be matched can be obtained. The estimated external parameters of the drone can be determined based on the external parameters of the drone and the external parameters of the camera relative to the drone body coordinate system. and the external parameters of the camera relative to the drone body coordinate system Determine, the estimated external parameters of the machine can be expressed as: Then we have:

[0089]

[0090] Continuing with the above description, the first pixel coordinates of each key point in the first image to be matched can be converted into corresponding second pixel coordinates based on the camera's estimated external parameters, internal parameters and height information; the first pixel coordinates correspond to the pixel coordinate system of the first image to be matched; the second pixel coordinates correspond to the pixel coordinate system of the second image to be matched.

[0091] 2) dividing the pixel coordinate system of the second to-be-matched image into a plurality of division areas according to a preset division distance;

[0092] It can be known that after the first pixel coordinate is converted into the second pixel coordinate, the pixel coordinate system of the second image to be matched can be divided into multiple division areas according to a preset division distance. The preset division distance can be a pre-set division distance. For example, the preset division distance can be the distance of a single pixel coordinate.

[0093] 3) For each divided area, determine the target matching key point according to the distance between the feature sub-points corresponding to each key point in the divided area; the target matching key point includes two key points, and the two key points are key points representing the same geographical location in adjacent images to be matched;

[0094] Specifically, after converting the first pixel coordinates into the second pixel coordinates, the key points in the first image to be matched and the second image to be matched are both located in the pixel coordinate system of the second image to be matched. After dividing the pixel coordinate system of the second image to be matched, multiple divided regions can be obtained. In each divided region, the target matching key point can be determined according to the distance between the feature subs corresponding to each key point in the divided region. For example, the two key points with the smallest distance between the feature subs corresponding to the key points in the divided region can be used as the target matching key point. It should be noted that the target matching key points in the divided region can also be determined by matching algorithms such as brute force matching and matching based on FLANN (Fast Library for Approximate Nearest Neighbors). Brute-Force Matching is a simple and direct feature point matching method. For a feature point in an image, brute force matching compares its feature descriptor with the feature descriptors of all feature points in another image, and finds the point with the closest distance as a match; brute force matching can use metrics such as Euclidean distance or Hamming distance to calculate the distance between feature descriptors; FLANN-based matching can be a library for approximate nearest neighbor search, which can quickly find the nearest neighbor of a data point in a high-dimensional space. FLANN-based matching takes advantage of this efficient search capability and accelerates the matching process between feature points by building an index, which is particularly suitable for large-scale data sets.

[0095] It should be explained that in order to make up for the lack of accuracy of the real-time external parameters of the camera and improve the robustness of the overall matching, this technical solution expands the divided area during the matching process. This expansion strategy ensures that even if there is a certain error in the camera pose estimation, the correct matching point can be found in the adjacent blocks, thereby effectively improving the accuracy and reliability of the matching process.

[0096] 4) Determine the initial camera extrinsic parameters based on the target matching key points and the camera intrinsic parameters.

[0097] Specifically, after determining the target matching key points of each divided area, the matching relationship between the key points can be obtained, and the observation model can be constructed according to the matching relationship between the key points. The initial camera extrinsic parameters are determined using the observation model and the camera's intrinsic parameters.

[0098] Optionally, converting the first pixel coordinates of each key point in the first to-be-matched image into the second pixel coordinates according to the estimated body external parameters, body internal parameters and height information of the camera includes:

[0099] 1) For each key point in the first image to be matched, convert the original pixel coordinates of the key point in the pixel coordinate system of the first image to be matched into homogeneous pixel coordinates, and normalize the homogeneous pixel coordinates to obtain normalized pixel coordinates.

[0100] Specifically, the adjacent images to be matched include a first image to be matched and a second image to be matched. After extracting the key points on each image to be matched using the SIFT algorithm and generating feature descriptors corresponding to the key points, for each key point in the first image to be matched, the original pixel coordinates of the key point in the pixel coordinate system of the first image to be matched can be converted into homogeneous pixel coordinates. The coordinate system of the original pixel coordinates can be the coordinate system of the first image to be matched, and the homogeneous coordinates can be a coordinate method for representing a point in a two-dimensional or three-dimensional space, which simplifies the representation and calculation of geometric transformations by adding an additional coordinate. In homogeneous coordinates, the coordinates of a point are represented as a vector.

[0101] For example, the original pixel coordinates can be expressed as P p1 =(x p1 ,y p1 ), the camera’s intrinsic parameters can be expressed as K, and the homogeneous pixel coordinates can be expressed as P ch1 , the original pixel coordinates can be converted to homogeneous pixel coordinates in the camera coordinate system through the camera's intrinsic parameters:

[0102]

[0103] Continuing with the above description, after obtaining the homogeneous pixel coordinates, the aligned sub-pixel coordinates may be normalized to obtain normalized pixel coordinates.

[0104] 2) Determine the first pixel coordinate of the key point based on the normalized pixel coordinate and the corresponding height information.

[0105] Specifically, after determining the normalized pixel coordinates, the first pixel coordinates of the key point can be determined based on the normalized pixel coordinates and the height information corresponding to the key point. The height information can be the height information obtained from the topographic map, and the first pixel coordinates can be the actual coordinates of the key point in the camera coordinate system. Exemplarily, the first pixel coordinates can be expressed as P c1 .

[0106] 3) transforming the first pixel coordinates according to the first estimated external parameter of the camera to obtain transformed pixel coordinates; the transformed pixel coordinates correspond to a world coordinate system; the first estimated external parameter of the camera is the estimated external parameter of the camera when the first image to be matched is captured;

[0107] In this embodiment, the first estimated external parameters of the body may be estimated external parameters of the body when the camera takes the first image to be matched. Exemplarily, the first estimated external parameters of the body may be expressed as: The first estimated camera extrinsic parameter may be used to transform the first pixel coordinates to obtain transformed pixel coordinates; the transformed pixel coordinates correspond to a world coordinate system, that is, the coordinate system of the transformed pixel coordinates is the world coordinate system.

[0108] For example, the converted pixel coordinates can be expressed as P w1 , then:

[0109]

[0110] 4) According to a second estimated external parameter of the camera, the converted pixel coordinates are converted to obtain a second pixel coordinate; the second estimated external parameter is the estimated external parameter of the camera when the second image to be matched is captured.

[0111] In this embodiment, the second estimated external parameters of the body are estimated external parameters of the body when the second image to be matched is captured. Exemplarily, the second estimated external parameters of the body can be expressed as: The converted pixel coordinates may be transformed using the second estimated camera extrinsic parameters to obtain second pixel coordinates; a coordinate system of the second pixel coordinates is a coordinate system of the second image to be matched.

[0112] Exemplarily, the second pixel coordinate can be expressed as P p1to2 , then:

[0113]

[0114] Among them, K can be the intrinsic parameter of the camera, Can be expressed The component in the z direction, P w1 It can be the converted pixel coordinates, It can be a second estimated external parameter of the body.

[0115] As an optional implementation of this embodiment, determining the initial orthophoto according to the pixels in the coverage range includes:

[0116] 1) Determine a set of pixel values ​​corresponding to each pixel position in the coverage range in each image to be matched.

[0117] Specifically, a set of pixel values ​​corresponding to each pixel position in the coverage range in each image to be matched can be determined. The orthophoto contains multiple pixels, and each pixel in the orthophoto is the best pixel selected from the pixels at the corresponding position in all images to be matched as the pixel in the orthophoto.

[0118] 2) For each of the pixel value sets, a target pixel value is determined from the pixel value set using a viewing angle score.

[0119] It can be known that after obtaining each pixel value set, the target pixel value can be determined from each pixel value set using the viewing angle score, and the target pixel value can be determined based on the viewing angle score. It needs to be explained that a scoring layer can be created in advance, and the scoring layer can be used to store the viewing angle score of each pixel.

[0120] For example, the viewing angle score is calculated based on the angle between the viewing direction from the camera to the ground point and the vertical direction, and the dot product of the normalized viewing direction and the vertical unit vector is calculated. The score ranges from -1 (looking completely downward) to 1 (looking directly upward). A score close to 1 indicates a good viewing angle. Specifically:

[0121]

[0122] Among them, (x, y, z) is the coordinate of the actual physical point in the world coordinate system (ENU coordinate system) corresponding to the pixel in the orthophoto image, (x c ,y c ,z c ) are the ENU coordinates of the corresponding camera. Through this process, the projection of each image is accurately mapped to the orthomap, while taking into account the viewpoint score to select the optimal pixels. This approach ensures that the generated orthomaps are of high quality and correctly geographically aligned.

[0123] 3) The target pixel values ​​of the pixel value set are used to form the initial orthophoto; and the viewing angle score is used to evaluate the shooting viewing angle of the camera.

[0124] Specifically, after obtaining the target pixel values ​​of each pixel value set, an initial orthophoto can be constructed based on each target pixel value. The viewing angle score is used to evaluate the shooting angle of the camera. Exemplarily, for each pixel in the orthophoto, the pixel with the highest viewing angle score is selected as the final pixel.

[0125] As an optional implementation of this embodiment, the drone surveying and mapping method provided in this embodiment, before determining each set of pixel values ​​corresponding to each pixel position in the orthophoto in each image to be matched, the method further includes:

[0126] For each image to be matched, Gaussian blur and Nyquist sampling theorem are used to pre-process the image to be matched.

[0127] It should be explained that after the image to be matched is obtained, Gaussian blur and Nyquist sampling theorem can be used to pre-process the image to be matched.

[0128] Specifically, for each image to be matched, according to the Nyquist sampling theorem, the highest frequency component f in the image to be matched is determined. max , calculate the Nyquist frequency f Nyquist =2·f max Before downsampling, you can also apply Gaussian blur to the image to be matched. When selecting pixels during orthophoto generation, the pixel values ​​after Gaussian blur are selected to reduce seams and jaggedness in the resulting image.

[0129] Figure 2 An example of an application of a drone surveying and mapping method provided in this embodiment is as follows: Figure 2 As shown, GPS, IMU, 4D millimeter-wave radar and camera can be installed on the drone. When operating, the drone is flown according to a predetermined trajectory to collect the output information of GPS, IMU and 4D millimeter-wave radar, and continuous images to be matched are continuously taken to ensure that there is a certain overlap area between adjacent images. The external parameters of the installed GPS and IMU are calibrated, and a topographic map is constructed through the terrain data provided by the 4D millimeter-wave radar. The process of feature extraction (key point extraction) and matching is performed by hardware acceleration through OpenCL to obtain the initial camera external parameters, and the camera external parameters (initial camera external parameters) are optimized by the SFM method to obtain the target camera external parameters; the bounding box of each image to be matched is determined by the target camera external parameters, so as to generate an initial orthophoto through the bounding box and each image to be matched, and the initial orthophoto is optimized to obtain the target orthophoto.

[0130] This technical solution uses drones to fly and shoot along a predetermined route, achieving the ability to quickly generate large-scene orthophoto maps, making it possible to directly conduct land surveying and mapping work on high-definition maps. By fusing data from multiple sensors to adapt to complex and changing terrain conditions, efficiency and accuracy in a variety of geographical environments are ensured. Using OpenCL for parallel computing acceleration solves the accuracy and robustness problems faced by traditional drones when generating orthophotos in complex terrain and changing environments. Through multi-sensor fusion technology, combined with 4D millimeter-wave radar and camera data, the efficiency and accuracy of obtaining geographic information are significantly improved, and large amounts of image data can be quickly processed on low-cost GPUs. It also significantly improves the image processing speed, while making full use of prior position information, thereby significantly reducing the number of feature points that need to be extracted during feature matching.

[0131] Furthermore, this technical solution not only improves the efficiency and quality of orthophoto map generation, but also meets the requirements of industrial-level practical applications, providing an efficient and reliable solution for fields such as geographic information systems, urban planning, environmental monitoring and disaster assessment.

[0132] Embodiment 2

[0133] Figure 3 is a structural schematic diagram of an unmanned aerial vehicle surveying and mapping device provided in the second embodiment of the present disclosure; Figure 3 As shown, the device includes: an image acquisition module 210, an initial camera extrinsic parameter determination module 220, a target camera extrinsic parameter determination module 230, an initial orthophoto determination module 240, and a target orthophoto determination module 250.

[0134] The image acquisition module 210 is used to capture multiple environment images using the camera when the drone is flying along a preset trajectory, wherein each environment image is used as an image to be matched, and adjacent images to be matched have overlapping shooting areas, and the overlapping shooting areas include key points;

[0135] An initial camera extrinsic parameter determination module 220 is used to determine the initial camera extrinsic parameters according to the key points in each group of adjacent images to be matched, the camera's body internal parameters and height information; wherein the height information includes the height of the key point in the topographic map;

[0136] A target camera extrinsic parameter determination module 230 is used to optimize the initial camera extrinsic parameters according to the global structure motion reconstruction GLOMAP algorithm to obtain target camera extrinsic parameters;

[0137] An initial orthophoto determination module 240 is used to determine the coverage of the orthophoto based on the bounding boxes of each image to be matched, wherein the bounding boxes are determined based on the target camera extrinsics and the field of view; the field of view corresponds to the image to be matched; and determine the initial orthophoto according to the pixel values ​​in the coverage;

[0138] The target orthophoto determination module 250 is used to correct and fuse multiple bands of the initial orthophoto to obtain a target orthophoto.

[0139] Embodiment 2 of the present disclosure provides a drone mapping device, which improves the efficiency and accuracy of generating orthophotos in various geographical environments, makes it possible to directly perform land surveying and mapping work on high-definition maps, and improves the efficiency and quality of orthophoto generation.

[0140] Furthermore, the drone also includes a global positioning system GPS and an inertial measurement unit IMU;

[0141] The device also includes:

[0142] An information collection module, used to collect the current flight information of the UAV in real time by using the GPS and the IMU when the UAV is flying along a preset trajectory; the flight information includes speed information, altitude information, position information and heading angle information;

[0143] A prediction module, configured to construct a state prediction model using a Kalman filter based on the flight information at the current moment, so as to determine the predicted position and predicted speed of the UAV at the next moment using the state prediction model;

[0144] The adjustment module is used to adjust the flight parameters of the UAV according to the predicted position and the predicted speed so that the UAV flies according to the preset trajectory.

[0145] Furthermore, the drone also includes: a 4D millimeter wave radar; the device also includes:

[0146] The UAV body external parameter determination module is used to perform time synchronization and coordinate alignment on the GPS and the IMU before the UAV takes off, and determine the UAV body external parameters;

[0147] A 4D millimeter-wave radar body external parameter determination module, used to determine the body external parameters of the 4D millimeter-wave radar based on the body external parameters of the drone and the external parameters of the 4D millimeter-wave radar relative to the drone body coordinate system;

[0148] An initial topographic map construction module, used to construct an initial topographic map according to the radar information obtained by the 4D millimeter wave radar; the initial topographic map includes first coordinates; the first coordinates are coordinates of each topographic point in the coordinate system of the 4D millimeter wave radar;

[0149] A topographic map determination module is used to convert the first coordinate into a second coordinate based on the external parameters of the 4D millimeter wave radar to obtain the topographic map, where the second coordinate is the position coordinate of each topographic point in the world coordinate system.

[0150] Further, for any group of adjacent images to be matched, the adjacent images to be matched include a first image to be matched and a second image to be matched; the initial camera extrinsic parameter determination module 220 further includes:

[0151] A pixel coordinate conversion unit, for any group of adjacent images to be matched, converting first pixel coordinates of each key point in a first image to be matched into corresponding second pixel coordinates according to estimated body external parameters, body internal parameters and height information of the camera; the estimated body external parameters are determined based on the body external parameters of the drone and the external parameters of the camera relative to the drone body coordinate system; the first pixel coordinates correspond to the pixel coordinate system of the first image to be matched; the second pixel coordinates correspond to the pixel coordinate system of the second image to be matched;

[0152] A division unit, configured to divide the pixel coordinate system of the second to-be-matched image into a plurality of division areas according to a preset division distance;

[0153] A target matching key point determination unit is used to determine the target matching key point for each divided area according to the distance between the feature sub-points corresponding to each key point in the divided area; the target matching key point includes two key points, and the two key points are key points representing the same geographical location in adjacent images to be matched;

[0154] A determination unit is used to determine the initial camera extrinsic parameters according to the target matching key points and the intrinsic parameters of the camera.

[0155] Furthermore, the pixel coordinate conversion unit is also used for:

[0156] For each key point in the first image to be matched, converting the original pixel coordinates of the key point in the pixel coordinate system of the first image to be matched into homogeneous pixel coordinates, and normalizing the homogeneous pixel coordinates to obtain normalized pixel coordinates;

[0157] Determining a first pixel coordinate of the key point based on the normalized pixel coordinate and the corresponding height information;

[0158] According to the first estimated external parameter of the camera, the first pixel coordinate is converted to obtain a converted pixel coordinate; the converted pixel coordinate corresponds to a world coordinate system; the first estimated external parameter of the camera is the estimated external parameter of the camera when the first image to be matched is taken;

[0159] The converted pixel coordinates are converted according to a second estimated external parameter of the camera to obtain a second pixel coordinate; the second estimated external parameter of the camera is an estimated external parameter of the camera when the second image to be matched is captured.

[0160] Furthermore, the initial orthophoto determination module 240 is further configured to:

[0161] Determine a set of pixel values ​​corresponding to each pixel position in the coverage range in each to-be-matched image;

[0162] For each of the pixel value sets, determining a target pixel value from the pixel value set using a viewing angle score;

[0163] The initial orthophoto is formed based on the target pixel values ​​of each pixel value set; and the viewing angle score is used to evaluate the shooting viewing angle of the camera.

[0164] The drone surveying and mapping device provided in the embodiments of the present disclosure can execute the drone surveying and mapping method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0165] Embodiment 3

[0166] Figure 4 A schematic diagram of a drone 10 that can be used to implement embodiments of the present disclosure is shown. The drone is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present disclosure described and / or claimed herein.

[0167] like Figure 4 As shown, the drone 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the drone 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0168] A number of components in the drone 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the drone 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0169] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microprocessors, etc. The processor 11 executes the various methods and processes described above, such as the drone mapping method.

[0170] In some embodiments, the drone mapping method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the drone 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the drone mapping method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the drone mapping method in any other appropriate manner (e.g., by means of firmware).

[0171] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] The computer programs for implementing the methods of the embodiments of the present disclosure may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0173] In the context of the disclosed embodiments, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] To provide interaction with a user, the systems and techniques described herein may be implemented on a drone having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the drone. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0175] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0176] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0177] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the embodiments of the present disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions of the embodiments of the present disclosure can be achieved, and this document does not limit this.

[0178] The above specific implementations do not constitute a limitation on the protection scope of the embodiments of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the embodiments of the present disclosure shall be included in the protection scope of the embodiments of the present disclosure.

[0179] The embodiments of the present disclosure also provide a computer program product, including a computer program and / or instructions, which, when executed by a processor, implements the drone surveying and mapping method provided in any embodiment of the present application.

[0180] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to write computer program codes for performing the operation of the disclosed embodiments, the programming languages ​​including object-oriented programming languages, such as Java, Smalltalk, C++, and also including conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0181] Note that the above are only preferred embodiments of the embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the embodiments of the present disclosure are not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the embodiments of the present disclosure. Therefore, although the embodiments of the present disclosure are described in more detail through the above embodiments, the embodiments of the present disclosure are not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the embodiments of the present disclosure, and the scope of the embodiments of the present disclosure is determined by the scope of the attached claims.

Claims

1. A method for surveying and mapping using an unmanned aerial vehicle, characterized in that: Applied to a drone, the drone includes a camera; the method includes: When the drone is flying along a preset trajectory, the camera is used to capture multiple environment images, wherein each environment image is used as an image to be matched, and adjacent images to be matched have overlapping capture areas, and the overlapping capture areas include key points; Determine the initial camera external parameters according to the key points in each group of adjacent images to be matched, the camera's internal parameters and height information; wherein the height information includes the height of the key points in the topographic map; The initial camera extrinsic parameters are optimized according to the global structure motion reconstruction GLOMAP algorithm to obtain the target camera extrinsic parameters; Determine the coverage of the orthophoto based on the bounding boxes of each image to be matched, wherein the bounding boxes are determined based on the target camera extrinsics and the field of view; the field of view corresponds to the image to be matched; and determine the initial orthophoto according to the pixel values ​​in the coverage; The initial orthophoto is corrected and multi-band fused to obtain a target orthophoto.

2. The method according to claim 1, characterized in that The drone also includes a global positioning system GPS and an inertial measurement unit IMU; The method further comprises: When the UAV is flying along a preset trajectory, the GPS and the IMU are used to collect the current flight information of the UAV in real time; the flight information includes speed information, altitude information, position information and heading angle information; Based on the flight information at the current moment, a state prediction model is constructed using a Kalman filter, so as to determine the predicted position and predicted speed of the UAV at the next moment using the state prediction model; The flight parameters of the UAV are adjusted according to the predicted position and the predicted speed so that the UAV flies according to the preset trajectory.

3. The method according to claim 1, characterized in that The drone also includes: a 4D millimeter wave radar; The method further comprises: Before the UAV takes off, the GPS and the IMU are synchronized in time and aligned in coordinates, and the external parameters of the UAV are determined; Determine the external parameters of the 4D millimeter wave radar based on the external parameters of the drone and the external parameters of the 4D millimeter wave radar relative to the drone body coordinate system; An initial topographic map is constructed according to radar information acquired by the 4D millimeter-wave radar; the initial topographic map includes first coordinates; the first coordinates are coordinates of each topographic point in the coordinate system of the 4D millimeter-wave radar; Based on the external parameters of the 4D millimeter-wave radar, the first coordinates are converted into second coordinates to obtain the topographic map, where the second coordinates are the position coordinates of each topographic point in the world coordinate system.

4. The method according to claim 3, characterized in that For any group of adjacent images to be matched, the adjacent images to be matched include a first image to be matched and a second image to be matched; The initial camera external parameters are determined according to the key points in each group of adjacent images to be matched, the camera's body internal parameters and height information, including: For any group of adjacent images to be matched, the first pixel coordinates of each key point in the first image to be matched are converted into corresponding second pixel coordinates according to the estimated body external parameters, body internal parameters and height information of the camera; the estimated body external parameters are determined based on the body external parameters of the drone and the external parameters of the camera relative to the drone body coordinate system; the first pixel coordinates correspond to the pixel coordinate system of the first image to be matched; the second pixel coordinates correspond to the pixel coordinate system of the second image to be matched; Dividing the pixel coordinate system of the second to-be-matched image into a plurality of division areas according to a preset division distance; For each divided area, a target matching key point is determined according to the distance between the feature sub-points corresponding to each key point in the divided area; the target matching key point includes two key points, and the two key points are key points representing the same geographical location in adjacent images to be matched; The initial camera extrinsic parameters are determined according to the target matching key points and the intrinsic parameters of the camera.

5. The method according to claim 4, characterized in that The converting the first pixel coordinates of each key point in the first to-be-matched image into the second pixel coordinates according to the estimated body external parameters, body internal parameters and height information of the camera includes: For each key point in the first image to be matched, converting the original pixel coordinates of the key point in the pixel coordinate system of the first image to be matched into homogeneous pixel coordinates, and normalizing the homogeneous pixel coordinates to obtain normalized pixel coordinates; Determining a first pixel coordinate of the key point based on the normalized pixel coordinate and the corresponding height information; According to the first estimated external parameter of the camera, the first pixel coordinate is converted to obtain a converted pixel coordinate; the converted pixel coordinate corresponds to a world coordinate system; the first estimated external parameter of the camera is the estimated external parameter of the camera when the first image to be matched is taken; The converted pixel coordinates are converted according to a second estimated external parameter of the camera to obtain a second pixel coordinate; the second estimated external parameter of the camera is an estimated external parameter of the camera when the second image to be matched is captured.

6. The method according to claim 1, characterized in that Determining an initial orthophoto according to pixels in the coverage area includes: Determine a set of pixel values ​​corresponding to each pixel position in the coverage range in each to-be-matched image; For each of the pixel value sets, determining a target pixel value from the pixel value set using a viewing angle score; The initial orthophoto is formed based on the target pixel values ​​of each pixel value set; and the viewing angle score is used to evaluate the shooting viewing angle of the camera.

7. An unmanned aerial vehicle surveying and mapping device, characterized in that: include: An image acquisition module, used to use the camera to capture multiple environment images when the drone is flying along a preset trajectory, wherein each environment image is used as an image to be matched, and adjacent images to be matched have overlapping shooting areas, and the overlapping shooting areas include key points; An initial camera extrinsic parameter determination module, used to determine the initial camera extrinsic parameters according to the key points in each group of adjacent images to be matched, the camera's body internal parameters and height information; wherein the height information includes the height of the key point in the topographic map; A target camera extrinsic parameter determination module is used to optimize the initial camera extrinsic parameters according to the global structure motion reconstruction GLOMAP algorithm to obtain the target camera extrinsic parameters; An initial orthophoto determination module is used to determine the coverage of the orthophoto based on the bounding boxes of each image to be matched, wherein the bounding boxes are determined based on the target camera extrinsics and the field of view; the field of view corresponds to the image to be matched; and determine the initial orthophoto according to the pixel values ​​in the coverage; The target orthophoto determination module is used to correct and fuse the multi-bands of the initial orthophoto to obtain the target orthophoto.

8. A drone, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the drone surveying and mapping method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the UAV surveying and mapping method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the drone surveying and mapping method as described in any one of claims 1 to 6.

Images