Orthographic image generation method and device, orthographic index map generation method and device

Abstract

This application provides a method and apparatus for generating orthorectified images and an orthorectified index map, relating to the field of image processing technology. The orthorectified image generation method includes: acquiring M frames of images corresponding to the M cameras at the current moment using M cameras at different wavelengths of an aircraft, where M is a positive integer greater than 1; determining the imaging pose information corresponding to each of the M frames based on the imaging pose information; and determining the orthorectified image corresponding to each of the M frames at the current moment based on the imaging pose information. This application can determine the imaging pose information of each frame in the M frames in real time online, solving the problem of not being able to determine the orthorectified image at the current moment in real time. This allows for the provision of real-time, accurate, and rich geographic coordinate information for subsequent crop analysis, thereby better enabling tasks such as environmental monitoring.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a method and apparatus for generating orthophoto images and an orthophoto index diagram. Background Technology

[0002] In agricultural settings, images captured by multiple cameras at different wavelengths on an aircraft typically need to be converted into orthophotos at different wavelengths in order to further analyze information such as crop growth.

[0003] However, due to the real-time changes in the positions of the multiple cameras on the aircraft and the complex data processing procedures, there is a problem that orthophotos cannot be generated in real time, making it difficult to analyze information such as crop growth in real time. Summary of the Invention

[0004] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a method and apparatus for generating orthophoto images, and a method and apparatus for generating orthophoto index maps.

[0005] In a first aspect, one embodiment of this application provides an orthorectified image generation method, the method comprising: acquiring M frames of images corresponding to the M cameras at the current time based on M cameras of different bands of an aircraft, where M is a positive integer greater than 1; determining the imaging pose information corresponding to each of the M frames based on the imaging pose information corresponding to each of the M frames; and determining the orthorectified image corresponding to each of the M frames at the current time based on the imaging pose information corresponding to each of the M frames.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, determining the orthorectified image of each of the M frames at the current moment based on the imaging pose information corresponding to each of the M frames includes: dividing the M frames into N reference images and MN non-reference images, where N is a positive integer less than M; determining the dense point cloud data corresponding to each of the N reference images based on the N reference images; determining the elevation information corresponding to each of the N reference images based on the dense point cloud data corresponding to each of the N reference images; and determining the orthorectified image of each of the M frames at the current moment based on the imaging pose information corresponding to each of the MN non-reference images and the elevation information corresponding to each of the N reference images.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, determining the orthorectified image of each of the M frames at the current moment based on the imaging pose information corresponding to each of the M non-reference images and the elevation information corresponding to each of the N reference images includes: determining the orthorectified image of each of the N reference images based on the elevation information corresponding to each of the N reference images; and determining the orthorectified image of each of the M non-reference images at the current moment based on the imaging pose information corresponding to each of the M non-reference images, the camera parameter information corresponding to each of the M non-reference images, and the elevation information corresponding to each of the N reference images.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, determining the orthorectified image corresponding to each of the N reference images based on the elevation information corresponding to each of the N reference images includes: determining the image matrix template corresponding to each of the N reference images based on the elevation information corresponding to each of the N reference images; determining the image pixel data corresponding to each of the N reference images based on the elevation information corresponding to each of the N reference images; and determining the orthorectified image corresponding to each of the N reference images based on the image matrix template and the image pixel data corresponding to each of the N reference images.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, determining the dense point cloud data corresponding to each of the N reference images based on the N reference images includes: determining the camera pose and sparse point cloud data corresponding to each of the N reference images based on the N reference images; and determining the dense point cloud data corresponding to each of the N reference images based on the camera pose and sparse point cloud data corresponding to each of the N reference images.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, determining the dense point cloud data corresponding to each of the N reference images based on the camera pose and sparse point cloud data corresponding to each of the N reference images includes: for each reference image in the N reference images, determining the next frame image corresponding to each reference image, wherein the next frame image is the image acquired at the next time corresponding to the current time; determining the depth map corresponding to each reference image based on the camera pose and sparse point cloud data corresponding to each reference image and the camera pose and sparse point cloud data corresponding to the next frame image; and determining the dense point cloud data corresponding to each reference image based on the depth map and camera matrix information corresponding to each reference image.

[0011] Secondly, one embodiment of this application provides an orthorectified index diagram generation method, the method comprising: determining the orthorectified image at the current moment corresponding to each of the M frame images, wherein the M frame images are images at the current moment captured by M cameras of different bands of the aircraft, and the orthorectified image at the current moment corresponding to each of the M frame images is determined based on the orthorectified image generation method mentioned in the first aspect above; and determining the orthorectified index diagram at the current moment based on the orthorectified image at the current moment corresponding to each of the M frame images.

[0012] Thirdly, one embodiment of this application provides an orthorectified image generation apparatus, which includes: an acquisition module configured to acquire M frames of images corresponding to the M cameras at the current time based on M cameras of different bands of the aircraft, where M is a positive integer greater than 1; a first determination module configured to determine the imaging pose information corresponding to each of the M frames based on the M frames; and a second determination module configured to determine the orthorectified image corresponding to each of the M frames at the current time based on the imaging pose information corresponding to each of the M frames.

[0013] Fourthly, one embodiment of this application provides an orthorectified image generation apparatus, the apparatus comprising: a third determining module configured to determine the orthorectified image at the current moment corresponding to each of M frame images, wherein the M frame images are images at the current moment acquired by M cameras of different bands of an aircraft, and the orthorectified image at the current moment corresponding to each of the M frame images is determined based on the orthorectified image generation method mentioned in the first aspect above; and a fourth determining module configured to determine the orthorectified image at the current moment based on the orthorectified image at the current moment corresponding to each of the M frame images.

[0014] Fifthly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the methods mentioned in the first and / or second aspects above.

[0015] In a sixth aspect, one embodiment of this application provides an electronic device comprising: a processor; a memory for storing processor-executable instructions; and the processor for performing the methods mentioned in the first and / or second aspects above.

[0016] The orthorectified image generation method provided in this application first acquires M frames of images corresponding to the M cameras at the current moment, based on M cameras in different bands of the aircraft; then, it determines the imaging pose information corresponding to each of the M frames based on the M frames; finally, it determines the orthorectified image corresponding to each of the M frames at the current moment based on the imaging pose information corresponding to each of the M frames. This application can determine the imaging pose information of each frame in the M frames in real time online, solving the problem of not being able to determine the orthorectified image at the current moment in real time. This allows for the provision of real-time, accurate, and rich geographic coordinate information for subsequent crop analysis, thereby better enabling tasks such as environmental monitoring. Attached Figure Description

[0017] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 The diagram shown is a scenario applicable to an embodiment of this application.

[0019] Figure 2 The diagram shown illustrates another scenario applicable to the embodiments of this application.

[0020] Figure 3 The diagram shown is a flowchart illustrating an orthophoto generation method provided in an exemplary embodiment of this application.

[0021] Figure 4 The diagram shown is a flowchart illustrating an orthophoto generation method provided in another exemplary embodiment of this application.

[0022] Figure 5 The diagram shown is a flowchart illustrating an orthophoto generation method provided in another exemplary embodiment of this application.

[0023] Figure 6 The diagram shown is a flowchart illustrating the process of determining the orthophoto image corresponding to each of the N reference images provided in an exemplary embodiment of this application.

[0024] Figure 7 The diagram shown is a flowchart illustrating an orthophoto generation method provided in another exemplary embodiment of this application.

[0025] Figure 8 The diagram shown is a flowchart illustrating the process of determining the dense point cloud data corresponding to each of the N reference images provided in an exemplary embodiment of this application.

[0026] Figure 9The diagram shown is a flowchart illustrating an orthophoto graph generation method provided in an exemplary embodiment of this application.

[0027] Figure 10 The diagram shown is a schematic diagram of the structure of an orthophoto image generation apparatus provided in an exemplary embodiment of this application.

[0028] Figure 11 The diagram shown is a schematic representation of the structure of the second determining module provided in an exemplary embodiment of this application.

[0029] Figure 12 The diagram shown is a schematic diagram of the structure of an orthophoto determination unit provided in an exemplary embodiment of this application.

[0030] Figure 13 The diagram shown is a schematic diagram of the structure of the first determining subunit provided in an exemplary embodiment of this application.

[0031] Figure 14 The diagram shown is a structural schematic of a dense point cloud data determination unit provided in an exemplary embodiment of this application.

[0032] Figure 15 The diagram shown is a structural schematic of the fifth determining subunit provided in an exemplary embodiment of this application.

[0033] Figure 16 The diagram shown is a schematic diagram of the orthophoto graph generation apparatus provided in an exemplary embodiment of this application.

[0034] Figure 17 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] Figure 1 The diagram shown illustrates a scenario applicable to an embodiment of this application. Figure 1 As shown, the scenario applicable to this application embodiment is an agricultural drone operation scenario. Specifically, this scenario includes an aircraft 2 equipped with an image acquisition device 20 and a server 1 connected to the image acquisition device 20. The aircraft can be a drone.

[0037] Image acquisition device 20 may include M cameras in different spectral bands, such as multispectral cameras. In practical applications, the M cameras in image acquisition device 20 are used to acquire M frames of images at the current moment. Each of the M frames corresponds one-to-one with one of the M cameras, where M is a positive integer greater than 1. Server 1 is used to determine the imaging pose information corresponding to each of the M frames based on the imaging pose information. Based on the imaging pose information corresponding to each of the M frames, server 1 determines the orthorectified image corresponding to each of the M frames at the current moment. That is, this scenario implements an orthorectified image generation method. For example, multiple aircraft can share a single server, meaning the server can receive data uploaded by different aircraft. Therefore, updating the server can update multiple aircraft, which helps save resources.

[0038] In addition, based on the orthophotos corresponding to each of the M frames at the current time, the orthophoto index map at the current time can be further determined, providing real-time, accurate, and rich geographic coordinate information for crop analysis.

[0039] It should be noted that this application also applies to another scenario. Figure 2 The diagram shows another scenario applicable to the embodiments of this application. Specifically, this scenario includes an aircraft 2, which includes an image acquisition module 201 and a computing module 202, and the image acquisition module 201 and the computing module 202 are connected by communication.

[0040] Specifically, the image acquisition module 201 onboard aircraft 2 includes M cameras in different spectral bands, such as multispectral cameras. In practical applications, the M cameras in the image acquisition module 201 are used to acquire M frames of images at the current moment. Each of the M frames corresponds one-to-one with one of the M cameras, where M is a positive integer greater than 1. The calculation module 202 in aircraft 2 is used to determine the imaging pose information corresponding to each of the M frames based on the M frames; and to determine the orthorectified image corresponding to each of the M frames at the current moment based on the imaging pose information. That is, this scenario implements an orthorectified image generation method. Figure 1 Compared to the scenario shown, this scenario does not require data transmission with servers or other related devices. Therefore, this scenario can guarantee the real-time performance of the orthophoto generation method.

[0041] In addition, the orthophoto generation method provided in this application embodiment is also applicable to fields such as dynamic monitoring of land resources, desertification monitoring, forest monitoring, flood monitoring, river changes, and drought monitoring.

[0042] Exemplary methods

[0043] Figure 3 The diagram shown is a schematic flowchart of an orthophoto generation method provided in an exemplary embodiment of this application. Figure 3 As shown in the embodiments of this application, the orthophoto generation method includes the following steps.

[0044] Step 100: Based on the M cameras of different bands of the aircraft, acquire M frames of images corresponding to the M cameras at the current time, where M is a positive integer greater than 1.

[0045] For example, the M cameras of different bands mentioned in step 100 can be airborne multi-view cameras or airborne multispectral cameras. The aircraft can be a drone or other flying equipment, and this application does not specifically limit it.

[0046] For example, the M frames of images corresponding to the M cameras mentioned in step 100 can be multiple frames of images of the target scene simultaneously captured by the aircraft's onboard multi-view camera, or multiple frames of images of different spectral bands simultaneously captured by the aircraft's onboard multispectral camera of the target scene. This application embodiment does not specifically limit this.

[0047] Step 200: Based on the M-frame images, determine the imaging pose information corresponding to each of the M-frame images.

[0048] For example, the imaging pose information mentioned in step 200 includes camera pose and sparse point cloud data.

[0049] Step 300: Based on the imaging pose information corresponding to each of the M frames, determine the orthophoto image of each of the M frames at the current moment.

[0050] In one embodiment, the multispectral camera mounted on the UAV takes pictures of the target scene during flight, acquiring M frames of images in different bands at the current moment. These M frames are then transmitted to the attitude estimation module to calculate the image attitude corresponding to each of the M frames at the current moment. The M frames can be of at least two types: RGB images, near-infrared images, red band images, or green band images.

[0051] For example, the pose estimation module can be a real-time localization algorithm based on Visual Simultaneous Localization and Mapping (VSLAM) or a real-time localization algorithm based on Simultaneous Localization and Mapping (SLAM). This application does not specifically limit this, as long as the algorithm can provide real-time localization. In one embodiment, the specific calculation process of the pose estimation module includes: first, extracting feature points and descriptors for each frame of images in different bands; matching each frame of images with its adjacent frames (which can be adjacent frames in different bands); and eliminating mismatched points through epipolar geometric constraints to determine the matching relationship between two frames of images. Then, based on the matching relationship between two frames of images, performing frame-by-frame image tracking in different bands and calculating the initial pose corresponding to each frame of images. Finally, optimizing each frame of images to determine the pose information and sparse point cloud data of each frame of images.

[0052] For example, the optimization method can be implemented using a bundle adjustment (BA) algorithm and / or a graph optimization algorithm. Bundle adjustment (BA) algorithms can be divided into global bundle adjustment (BA) algorithms and local bundle adjustment (BA) algorithms. This application does not specifically limit the optimization method.

[0053] Specifically, during the photo-taking process, the drone's onboard real-time kinematic (RTK) module generates map data including elevation and altitude information, which constitutes geographic coordinate reference information. When optimizing a frame of image using the Bundle Adjustment (BA) algorithm, it is necessary to determine whether the geographic coordinate reference information has been initialized.

[0054] If the geographic coordinate reference information is not initialized, an optimization problem is constructed using the reprojection error of all image frames in the map data and the error between the camera pose and the geographic coordinate position corresponding to the image frame. Global optimization is then performed to solve the pose of each image frame in the geographic coordinate system.

[0055] If the geographic coordinate reference information has been initialized, an optimization problem is constructed using the reprojection error of the image frame in the local map and the error between the camera pose and its corresponding geographic coordinates. Local optimization is then performed, and the pose information and sparse point cloud data of the image frame are output to the next stage. During the acquisition of each image frame, the corresponding band channel is recorded simultaneously. The images can then be classified based on the band channel information, obtaining the pose information and sparse point cloud data for each frame in each band channel. Based on the pose information and sparse point cloud data for each frame in each band channel, the orthorectified image of each band image is calculated.

[0056] The orthorectified image generation method provided in this application involves acquiring M frames of images corresponding to the M cameras at the current moment using M cameras in different bands of an aircraft. Then, by determining the imaging pose information corresponding to each of the M frames, the method aims to determine the orthorectified image of each of the M frames at the current moment based on the imaging pose information of each frame. This application embodiment can recover the pose information of each frame in real time online, solving the problem in the prior art that it is impossible to obtain the orthorectified image at the current moment in real time, thus providing a prerequisite for subsequent analysis of crop growth. Furthermore, this application embodiment has the advantage of wide applicability.

[0057] It is understandable that multispectral cameras are used to provide multi-band spectral data for agricultural remote sensing. A multispectral camera consists of multiple independent imagers, each equipped with a specially designed filter, allowing each imager to receive spectral data across different wavelength ranges. By taking pictures of farmland scenes using a multispectral camera, images of farmland in different spectral bands such as red, green, blue, red-edge, and near-infrared can be acquired at the current moment. Then, the pose of each frame at each moment is calculated in real time, and finally, an orthophoto image corresponding to each band is obtained.

[0058] Figure 4 The diagram shown is a flowchart illustrating an orthophoto generation method provided in another exemplary embodiment of this application. Figure 3 This application extends from the embodiments shown. Figure 4 The illustrated embodiment will be described in detail below. Figure 4 The illustrated embodiments and Figure 3 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0059] like Figure 4 As shown, in the orthophoto generation method provided in this application embodiment, the orthophoto of each of the M frames at the current time is determined based on the imaging pose information corresponding to each of the M frames (step 300), including the following steps.

[0060] Step 310: Divide the M images into N reference images and MN non-reference images. Here, N is a positive integer less than M.

[0061] For example, M frames of images are divided according to spectral bands to obtain N reference images. The N reference images can be a single frame of the same band or multiple frames of the same band.

[0062] Step 320: Based on the N reference images, determine the dense point cloud data corresponding to each of the N reference images.

[0063] For example, the dense point cloud data corresponding to each of the N reference images is used to characterize the dense point cloud data corresponding to each image in the same band. Because the image pose and point cloud coordinates in the scene are real geographic coordinates and belong to the same world reference coordinate system, it is only necessary to calculate the dense point cloud of one band image, and its dense point cloud can represent the dense point cloud of other bands at that moment.

[0064] Step 330: Based on the dense point cloud data corresponding to each of the N reference images, determine the elevation information corresponding to each of the N reference images.

[0065] Specifically, the dense point cloud calculation module receives the dense point cloud data corresponding to each of the aforementioned N reference images, performs interpolation calculations on the dense point cloud of each frame, and thus determines the elevation information corresponding to each frame. Taking the green band channel as an example, the elevation information corresponding to each frame of the green band channel can be determined based on the dense point cloud corresponding to each frame of the green band channel.

[0066] For example, the interpolation algorithm may include, but is not limited to, the inverse-distance-weighted interpolation algorithm (IDW), the nearest neighbor interpolation algorithm, and the Delaunay triangulation interpolation algorithm.

[0067] For example, elevation information can be a digital surface model (DSM). A digital surface model is a ground elevation model showing the heights of surface buildings, bridges, and trees.

[0068] Step 340: Based on the imaging pose information corresponding to each of the MN frame non-reference images and the elevation information corresponding to each of the N frame reference images, determine the orthophoto image corresponding to each of the M frame images at the current time.

[0069] Specifically, since the image poses and dense point clouds of all band channels are represented based on the same world reference coordinate system, when the elevation information of a band at the current moment is calculated, the images of all band channels at that moment are visible. For example, after determining the elevation information corresponding to each frame of the green band channel, the geographically informative digital orthophoto image corresponding to each band image can be determined based on the poses of multiple frames of images from different bands and the elevation information corresponding to the green band channel image.

[0070] The orthorectified image generation method provided in this application divides M frames of images into N reference images and MN non-reference images. Then, based on the dense point cloud data corresponding to each of the N reference images, the elevation information corresponding to each of the N reference images is determined. Finally, based on the imaging pose information corresponding to each of the MN non-reference images and the elevation information corresponding to each of the N reference images, the orthorectified image of each of the M frames at the current time is determined. By using the elevation information corresponding to a single band image, a digital orthorectified image with geographic information corresponding to each band image can be determined, reducing the computational load for acquiring orthorectified images and facilitating real-time orthorectified image generation.

[0071] Figure 5 The diagram shown is a flowchart illustrating an orthophoto generation method provided in another exemplary embodiment of this application. Figure 4 This application extends from the embodiments shown. Figure 5 The illustrated embodiment will be described in detail below. Figure 5 The illustrated embodiments and Figure 4 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0072] like Figure 5 As shown, in the orthophoto generation method provided in this application embodiment, the orthophoto of each of the M frames corresponding to the current time is determined based on the imaging pose information of each of the MN frames of non-reference images and the elevation information of each of the N frames of reference images (step 340), including the following steps.

[0073] Step 341: Based on the elevation information corresponding to each of the N reference images, determine the orthophoto corresponding to each of the N reference images.

[0074] For example, based on the elevation information corresponding to each frame of the green band channel, a digital orthophoto with geographic information corresponding to each frame of the green band channel is determined.

[0075] Step 342: Based on the imaging pose information corresponding to each of the MN non-reference images, the camera parameter information corresponding to each of the MN non-reference images, and the elevation information corresponding to each of the N reference images, determine the orthophoto image corresponding to each of the MN non-reference images at the current time.

[0076] For example, given the elevation information of the green band at the same time, the corresponding geographically-informed digital orthorectified images of other bands can be calculated using the imaging pose information of those other bands. Because the same elevation information is used, the orthorectified images of each band at the same time can achieve pixel-level alignment.

[0077] The orthorectified image generation method provided in this application determines the orthorectified image corresponding to each of the N reference images based on the elevation information corresponding to each of the N reference images; and determines the orthorectified image of each of the MN non-reference images at the current moment based on the imaging pose information, camera parameter information, and elevation information of each of the N reference images. Because the orthorectified image of each band at the same moment is determined based on the elevation information of the same band, pixel-level alignment can be achieved for the orthorectified image of each band.

[0078] Figure 6 The diagram shown is a schematic flowchart illustrating the process of determining the orthophoto image corresponding to each of N reference images, provided in an exemplary embodiment of this application. Figure 5 This application extends from the embodiments shown. Figure 6 The illustrated embodiment will be described in detail below. Figure 6 The illustrated embodiments and Figure 5 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0079] like Figure 6 As shown, in the orthophoto generation method provided in this application embodiment, the orthophoto corresponding to each of the N reference images is determined based on the elevation information corresponding to each of the N reference images (step 341), including the following steps.

[0080] Step 3410: Based on the elevation information corresponding to each of the N reference images, determine the image matrix template corresponding to each of the N reference images.

[0081] Step 3411: Based on the elevation information corresponding to each of the N reference images, determine the image pixel data corresponding to each of the N reference images.

[0082] Step 3412: Based on the image matrix template and image pixel data corresponding to each of the N frame reference images, determine the orthophoto image corresponding to each of the N frame reference images.

[0083] Specifically, when all N reference images are green band images, taking one of the reference images as an example, firstly, a matrix of the same size as the elevation information corresponding to that reference image is created. Then, each grid point (x, y, z) corresponding to the elevation information is traversed. Then, the pixel value corresponding to each grid point in the reference image is obtained by projecting it onto the reference image. The pixel value is assigned to the (x, y) coordinates of the matrix, and the digital orthophoto with geographic information corresponding to the green band reference image can be calculated.

[0084] The orthorectified image generation method provided in this application determines the image matrix template corresponding to each of the N reference images based on the elevation information corresponding to each of the N reference images; then, based on the elevation information corresponding to each of the N reference images, it determines the image pixel data corresponding to each of the N reference images; finally, based on the image matrix template and image pixel data corresponding to each of the N reference images, it determines the orthorectified image corresponding to each of the N reference images. The orthorectified image generation method provided in this application has a fast processing speed and good implementation effect, effectively improving the accuracy of orthorectified images and providing a prerequisite foundation for subsequent application in the analysis of farmland operation information.

[0085] Figure 7 The diagram shown is a flowchart illustrating an orthophoto generation method provided in another exemplary embodiment of this application. Figure 4 This application extends from the embodiments shown. Figure 7 The illustrated embodiment will be described in detail below. Figure 7 The illustrated embodiments and Figure 4 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0086] like Figure 7 As shown, in the orthophoto generation method provided in this application embodiment, based on N reference images, the dense point cloud data corresponding to each of the N reference images is determined (step 320), which includes the following steps.

[0087] Step 3210: Based on the N reference images, determine the camera pose and sparse point cloud data corresponding to each of the N reference images.

[0088] For example, the imaging pose information corresponding to each of the N reference images is calculated based on the pose estimation module. The imaging pose information includes the camera pose and sparse point cloud data corresponding to each of the N reference images.

[0089] Step 3211: Based on the camera pose and sparse point cloud data corresponding to each of the N reference images, determine the dense point cloud data corresponding to each of the N reference images.

[0090] Specifically, calculations are performed based on the camera pose and sparse point cloud data corresponding to each of the N reference images to determine the denser point cloud data corresponding to each of the N reference images.

[0091] The orthophoto generation method provided in this application determines the camera pose and sparse point cloud data corresponding to each of the N reference images based on the N reference images. By using the camera pose and sparse point cloud data corresponding to each of the N reference images, the method aims to determine the dense point cloud data corresponding to each of the N reference images, which is beneficial for further determining the elevation information corresponding to each of the N reference images.

[0092] Figure 8 The diagram illustrates a flowchart of an exemplary embodiment of this application for determining dense point cloud data corresponding to each of N reference images. Figure 7 This application extends from the embodiments shown. Figure 8 The illustrated embodiment will be described in detail below. Figure 8 The illustrated embodiments and Figure 7 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0093] like Figure 8 As shown, in the orthophoto generation method provided in this application embodiment, based on the camera pose and sparse point cloud data corresponding to each of the N reference images, the dense point cloud data corresponding to each of the N reference images is determined (step 3211), including the following steps.

[0094] Step 3212: For each reference image in the N frames of reference images, determine the next frame image corresponding to each reference image, where the next frame image is the image acquired at the next time corresponding to the current time.

[0095] Step 3213: Based on the camera pose and sparse point cloud data corresponding to each frame of the reference image and the camera pose and sparse point cloud data corresponding to the next frame of the image, determine the depth map corresponding to each frame of the reference image.

[0096] Step 3214: Based on the depth map and camera matrix information corresponding to each frame of reference image, determine the dense point cloud data corresponding to each frame of reference image.

[0097] For example, let's define one of the N reference images as the current frame. The poses of the current and next frames are represented as pose1(R1|C1) and pose2(R2|C2), respectively, where R1 is the rotation matrix of the current frame, C1 is the image position of the current frame, R2 is the corresponding rotation matrix of the current frame, and C2 is the image position of the current frame. Based on the sparse point cloud of the current frame, calculate the depth values ​​depth_min and depth_max, estimating the depth map range corresponding to the current frame to be 0.75*depth_min and 1.45*depth_max. Calculate the disparity ranges depth_min and depth_max of the image, where f is the camera focal length and the baseline is baseline = |C2 - C1|. Based on the rotation matrix R1|R2 and the epipolar geometry principle, calculate the rotation matrix R. n This is achieved by rotating the two images until they are coplanar and parallel to the baseline. The new camera projection matrices corresponding to the two images are P and P, respectively. n1 =K[R n |-R n C1]、P n2 =K[R n |-R n C2], the correction transformation matrices are respectively T1=(P n1 (1:3,1:3)*(K*R1′)′)′, T2=(P n2 (1:3,1:3)*(K*R2′)′)′. Based on the correction transformation matrix T1|T2, a resampling mapping transformation is performed on the two frames of images to obtain two new frames. n1 and images n2 , where K is a known camera intrinsic parameter.

[0098] The two new known images n1 and images n2 And the disparity ranges disparity_min and disparity_max, calculated using the SGM algorithm for images. n1 Corresponding disparity map image disparity1 .according to Calculate disparity map image disparity1 Corresponding depth map image depth1 Where f is the focal length value of the camera intrinsic parameters, and disparity is the disparity value. Finally, based on the known depth map image... depth1 By combining the camera projection matrix, the dense point cloud corresponding to the current frame image can be calculated.

[0099] It should be understood that the stereo matching algorithm can be a semi-global matching (SGM) algorithm, a local stereo matching algorithm, or a global stereo matching algorithm, etc. This application does not specifically limit it in this regard.

[0100] The orthophoto generation method provided in this application, for each reference image in N reference images, determines the next frame image corresponding to each reference image; based on the camera pose and sparse point cloud data corresponding to each reference image and the camera pose and sparse point cloud data corresponding to the next frame image, determines the depth map corresponding to each reference image; based on the depth map and camera matrix information corresponding to each reference image, the method achieves the purpose of determining the dense point cloud data corresponding to each reference image, providing a prerequisite for subsequently determining the elevation information corresponding to each of the N reference images.

[0101] Figure 9 The diagram shown is a flowchart illustrating an orthophoto map generation method provided in an exemplary embodiment of this application. Figure 9 As shown, the orthophoto graph generation method provided in this application embodiment includes the following steps.

[0102] Step 400: Determine the orthophoto of each of the M frames at the current moment. The M frames are images captured by M cameras of different bands of the aircraft at the current moment. The orthophoto of each of the M frames at the current moment is determined based on the orthophoto generation method mentioned in any of the above embodiments.

[0103] Step 401: Determine the orthophoto map at the current moment based on the orthophoto map corresponding to each of the M frames at the current moment.

[0104] For example, the orthophoto map mentioned in step 401 is a normalized vegetation index map with geographic information coordinates.

[0105] For example, the Normalized Vegetation Index (NDVI) can be calculated using formula (1).

[0106]

[0107] Wherein, NIR represents near-infrared orthophoto, and RED represents infrared orthophoto.

[0108] It should be understood that other different index maps can also be obtained by calculating using orthophotos of other different bands.

[0109] The orthophoto generation method provided in this application provides a method for generating orthophoto maps at the current time corresponding to each of the M frame images, wherein the M frame images are images collected at the current time by M cameras of different bands of the aircraft; based on the orthophoto maps at the current time corresponding to each of the M frame images, the orthophoto map at the current time is determined. The orthophoto map can be used to accurately locate vegetation growth and facilitate the acquisition of farmland operation information.

[0110] In one embodiment, based on the orthorectified images corresponding to multiple time points of each of the M frames, the orthorectification index map for each time point can be determined in real time. This enables the aircraft to output the orthorectification index map in real time during aerial photography, facilitating real-time monitoring and analysis of crop growth.

[0111] Exemplary device

[0112] The above text combined Figures 1 to 9 The method embodiments of this application are described in detail below, in conjunction with... Figures 10 to 17 The present application provides a detailed description of the apparatus embodiments. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0113] Figure 10 The diagram shown is a structural schematic of an orthophoto image generation apparatus provided in an exemplary embodiment of this application. Figure 10 As shown, the orthophoto generation apparatus provided in this application embodiment includes an acquisition module 500, a first determination module 600, and a second determination module 700.

[0114] The acquisition module 500 is configured to acquire M frames of images corresponding to the M cameras at the current moment, based on M cameras in different bands of the aircraft, where M is a positive integer greater than 1. The first determination module 600 is configured to determine the imaging pose information corresponding to each of the M frames of images based on the imaging pose information corresponding to each of the M frames of images. The second determination module 700 is configured to determine the orthophoto image corresponding to each of the M frames of images at the current moment based on the imaging pose information corresponding to each of the M frames of images.

[0115] Figure 11 The diagram shown is a structural schematic of the second determining module provided in an exemplary embodiment of this application. Figure 11 As shown, in the orthophoto image generation apparatus provided in this application embodiment, the second determining module 700 includes a division unit 701, a dense point cloud data determining unit 702, an elevation information determining unit 703, and an orthophoto image determining unit 704.

[0116] The segmentation unit 701 is configured to segment M frames of images to obtain N frames of reference images and MN frames of non-reference images, where N is a positive integer less than M. The dense point cloud data determination unit 702 is configured to determine the dense point cloud data corresponding to each of the N frames of reference images. The elevation information determination unit 703 is configured to determine the elevation information corresponding to each of the N frames of reference images based on the dense point cloud data corresponding to each of the N frames of reference images. The orthorectified image determination unit 704 is configured to determine the orthorectified image of each of the M frames of images at the current time based on the imaging pose information corresponding to each of the MN frames of non-reference images and the elevation information corresponding to each of the N frames of reference images.

[0117] Figure 12 The diagram shown is a schematic representation of the structure of an orthophoto image determination unit provided in an exemplary embodiment of this application. Figure 12 As shown, in the orthophoto image generation apparatus provided in this application embodiment, the orthophoto image determination unit 704 includes a first determination subunit 7040 and a second determination subunit 7041.

[0118] The first determining subunit 7040 is configured to determine the orthorectified image corresponding to each of the N reference images based on the elevation information corresponding to each of the N reference images. The second determining subunit 7041 is configured to determine the orthorectified image corresponding to the current time of each of the N non-reference images based on the imaging pose information, camera parameter information, and elevation information of each of the N reference images.

[0119] Figure 13 The diagram shown is a schematic representation of the structure of a first defined subunit provided in an exemplary embodiment of this application. Figure 13 As shown, in the orthophoto image generation apparatus provided in this application embodiment, the first determining subunit 7040 includes an image matrix template determining subunit 7140, an image pixel data determining subunit 7240, and a third determining subunit 7340.

[0120] The image matrix template determining subunit 7140 is configured to determine the image matrix template corresponding to each of the N reference images based on the elevation information corresponding to each of the N reference images. The image pixel data determining subunit 7240 is configured to determine the image pixel data corresponding to each of the N reference images based on the elevation information corresponding to each of the N reference images. The third determining subunit 7340 is configured to determine the orthophoto corresponding to each of the N reference images based on the image matrix template and the image pixel data corresponding to each of the N reference images.

[0121] Figure 14 The diagram shown is a structural schematic of a dense point cloud data determination unit provided in an exemplary embodiment of this application. Figure 14As shown, in the orthophoto image generation apparatus provided in this application embodiment, the dense point cloud data determination unit 702 includes a fourth determination subunit 7021 and a fifth determination subunit 7022.

[0122] The fourth determining subunit 7021 is configured to determine the camera pose and sparse point cloud data corresponding to each of the N reference images, based on the N reference images. The fifth determining subunit 7022 is configured to determine the dense point cloud data corresponding to each of the N reference images, based on the camera pose and sparse point cloud data corresponding to each of the N reference images.

[0123] Figure 15 The diagram shown is a structural schematic of the fifth determining subunit provided in an exemplary embodiment of this application. Figure 15 As shown, in the orthophoto image generation apparatus provided in this application embodiment, the fifth determining subunit 7022 includes a sixth determining subunit 7122, a seventh determining subunit 7222, and an eighth determining subunit 7322.

[0124] The sixth determining subunit 7122 is configured to determine the next frame image corresponding to each reference image in the N frames of reference images, wherein the next frame image is the image acquired at the next time step corresponding to the current time step. The seventh determining subunit 7222 is configured to determine the depth map corresponding to each reference image based on the camera pose and sparse point cloud data corresponding to each reference image and the camera pose and sparse point cloud data corresponding to the next frame image. The eighth determining subunit 7322 is configured to determine the dense point cloud data corresponding to each reference image based on the depth map and camera matrix information corresponding to each reference image.

[0125] Figure 16 The diagram shown is a schematic representation of an orthophoto graph generation apparatus provided in an exemplary embodiment of this application. Figure 16 As shown in the embodiment of this application, the orthophoto graph generation device includes a third determining module 800 and a fourth determining module 900.

[0126] The third determining module 800 is configured to determine the orthorectified image at the current moment corresponding to each of the M frame images, wherein the M frame images are images acquired at the current moment by M cameras of different bands of the aircraft, and the orthorectified image at the current moment corresponding to each of the M frame images is determined based on the orthorectified image generation method mentioned in any of the above embodiments. The fourth determining module 900 is configured to determine the orthorectified index map at the current moment based on the orthorectified image at the current moment corresponding to each of the M frame images.

[0127] Exemplary electronic devices

[0128] Below, for reference Figure 17 This describes an electronic device according to embodiments of the present application. Figure 17 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application.

[0129] like Figure 17 As shown, the electronic device 1000 includes one or more processors 1001 and memory 1002.

[0130] The processor 1001 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1000 to perform desired functions.

[0131] The memory 1002 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1001 may execute the program instructions to implement the orthorectified image generation method and / or orthorectified index map generation method of the various embodiments of this application described above, and / or other desired functions. The computer-readable storage medium may also store various contents such as M frames of images corresponding to M cameras at the current time.

[0132] In one example, the electronic device 1000 may also include an input device 1003 and an output device 1004, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0133] The input device 1003 may include, for example, a keyboard, a mouse, etc.

[0134] The output device 1004 can output various information to the outside, including an orthophoto of the current moment. The output device 1004 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0135] Of course, for the sake of simplicity, Figure 17 Only some of the components of the electronic device 1000 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 1000 may include any other suitable components depending on the specific application.

[0136] Exemplary computer-readable storage media

[0137] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the orthophoto generation method and / or orthophoto index map generation method according to various embodiments of this application described above.

[0138] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0139] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the orthophoto generation method and / or orthophoto index map generation method according to various embodiments of this application described above.

[0140] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0141] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0142] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0143] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0144] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0145] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for generating orthophotos, characterized in that, include: Based on M cameras of different frequency bands of the aircraft, M frames of images corresponding to the M cameras at the current moment are acquired, where M is a positive integer greater than 1; Based on the M frames of images, determine the imaging pose information corresponding to each of the M frames; The M frames of images are divided into N reference frames and MN non-reference frames, where N is a positive integer less than M. Based on the N reference images, determine the dense point cloud data corresponding to each of the N reference images; Based on the dense point cloud data corresponding to each of the N reference images, determine the elevation information corresponding to each of the N reference images. Based on the elevation information corresponding to each of the N reference images, determine the image matrix template corresponding to each of the N reference images; Based on the elevation information corresponding to each of the N reference images, the image pixel data corresponding to each of the N reference images is determined. Based on the image matrix template and image pixel data corresponding to each of the N reference images, the orthophoto image corresponding to each of the N reference images is determined; Based on the imaging pose information corresponding to each of the MN non-reference images, the camera parameter information corresponding to each of the MN non-reference images, and the elevation information corresponding to each of the N reference images, the orthophoto image corresponding to each of the MN non-reference images at the current time is determined.

2. The orthophoto generation method according to claim 1, characterized in that, The step of determining the dense point cloud data corresponding to each of the N reference images based on the N reference images includes: Based on the N reference images, determine the camera pose and sparse point cloud data corresponding to each of the N reference images; Based on the camera pose and sparse point cloud data corresponding to each of the N reference images, the dense point cloud data corresponding to each of the N reference images is determined.

3. The orthophoto generation method according to claim 2, characterized in that, The step of determining the dense point cloud data corresponding to each of the N reference images based on the camera pose and sparse point cloud data of each of the N reference images includes: For each of the N reference frames, Determine the next frame image corresponding to each reference image, wherein the next frame image is the image acquired at the next time corresponding to the current time. Based on the camera pose and sparse point cloud data corresponding to each frame of reference image and the camera pose and sparse point cloud data corresponding to the next frame of image, the depth map corresponding to each frame of reference image is determined. Based on the depth map and camera matrix information corresponding to each frame of the reference image, the dense point cloud data corresponding to each frame of the reference image is determined.

4. A method for generating an orthophoto graph, characterized in that, include: Determine the orthorectified image of each of the M frames at the current time, wherein the M frames are images of the current time captured by M cameras of different bands of the aircraft, and the orthorectified image of each of the M frames at the current time is determined based on the orthorectified image generation method according to any one of claims 1 to 3. Based on the orthophoto image corresponding to each of the M frames at the current time, the orthophoto index map at the current time is determined.

5. An orthophoto image generation apparatus, characterized in that, include: The acquisition module is configured to use M cameras based on different bands of the aircraft to acquire M frames of images corresponding to the M cameras at the current moment, where M is a positive integer greater than 1. The first determining module is configured to determine the imaging pose information corresponding to each of the M frames based on the M frames; The second determining module is configured to: divide the M frames of images to obtain N frames of reference images and MN frames of non-reference images, where N is a positive integer less than M; determine the dense point cloud data corresponding to each of the N frames of reference images; determine the elevation information corresponding to each of the N frames of reference images based on the dense point cloud data corresponding to each of the N frames of reference images; determine the image matrix template corresponding to each of the N frames of reference images based on the elevation information corresponding to each of the N frames of reference images; determine the image pixel data corresponding to each of the N frames of reference images based on the elevation information corresponding to each of the N frames of reference images; determine the orthorectified image corresponding to each of the N frames of reference images based on the image matrix template and image pixel data corresponding to each of the N frames of reference images; and determine the orthorectified image of each of the MN frames of non-reference images at the current time based on the imaging pose information, camera parameter information, and elevation information corresponding to each of the N frames of reference images.

6. An orthophoto graph generation device, characterized in that, include: The third determining module is configured to determine the orthophoto of each of the M frame images at the current time, wherein the M frame images are images of the current time collected by M cameras of different bands of the aircraft, and the orthophoto of each of the M frame images at the current time is determined based on the orthophoto generation method according to any one of claims 1 to 3. The fourth determining module is configured to determine the orthophoto index map at the current moment based on the orthophoto image corresponding to each of the M frames at the current moment.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1 to 4.

8. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to perform the method described in any one of claims 1 to 4.

Images