Image alignment method and device, storage medium and electronic device

Abstract

The present application provides an image alignment method and device, a storage medium and an electronic device, and relates to the field of image processing technology. The image alignment method includes: determining the camera depth map corresponding to each of the multiple cameras in the aircraft based on the depth data corresponding to the terrain data of the target scene; projecting the images captured by each of the multiple cameras onto the image plane of the virtual camera corresponding to the multiple cameras based on the camera depth map corresponding to the multiple cameras, and determining the aligned images corresponding to the images captured by each of the multiple cameras. The image alignment method provided in the embodiment of the present application integrates the depth data corresponding to the terrain data of the target scene. Compared with the multiple camera alignment method based on the high consistency assumption, the alignment accuracy is significantly improved and the scope of application is expanded. At the same time, it provides a prerequisite for the subsequent application of multiple cameras in the analysis of farmland operation information.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image alignment method and device, a storage medium, and an electronic device. Background Art

[0002] In agricultural scenarios, it's often necessary to analyze crop growth, pest and disease information, and other farmland operations using images captured by multiple cameras in an aircraft. This analysis requires aligning the images captured by these cameras. Existing multi-camera image alignment schemes typically assume minimal terrain undulations, approximating it to a flat surface. This means that the heights of the cameras relative to the ground are fixed and uniform. Consequently, existing schemes can lead to significant alignment errors, making them unsuitable for real-world applications. Summary of the Invention

[0003] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide an image alignment method and device, a storage medium, and an electronic device.

[0004] In a first aspect, an embodiment of the present application provides an image alignment method, comprising: determining a camera depth map corresponding to each of multiple cameras in an aircraft based on depth data corresponding to terrain data of a target scene; projecting images captured by each of the multiple cameras onto image planes of virtual cameras corresponding to the multiple cameras based on the camera depth maps corresponding to the multiple cameras, and determining aligned images corresponding to the images captured by each of the multiple cameras.

[0005] In combination with the first aspect, in certain implementations of the first aspect, based on the depth data corresponding to the terrain data of the target scene, determining the camera depth maps corresponding to each of the multiple cameras in the aircraft, including: determining the depth information of the image planes corresponding to each of the multiple cameras in the terrain data based on the depth data and the relative transformation relationship between the camera coordinate systems corresponding to each of the multiple cameras and the world coordinate system; determining the camera depth maps corresponding to each of the multiple cameras based on the depth information of the image planes corresponding to each of the multiple cameras in the terrain data.

[0006] In combination with the first aspect, in certain implementations of the first aspect, depth information of image planes corresponding to multiple cameras in terrain data is determined based on depth data and the relative transformation relationship between the camera coordinate systems corresponding to each of the multiple cameras and the world coordinate system, including: determining the camera center coordinates corresponding to each of the multiple cameras based on the depth data and the relative transformation relationship between the camera coordinate systems corresponding to each of the multiple cameras and the world coordinate system; determining the depth information of the camera center coordinates corresponding to each of the multiple cameras in terrain data based on the camera center coordinates corresponding to each of the multiple cameras and the depth data; determining the depth information of the image planes corresponding to each of the multiple cameras in terrain data based on the depth information of the camera center coordinates corresponding to each of the multiple cameras in terrain data.

[0007] In combination with the first aspect, in certain implementations of the first aspect, based on the depth information of the camera center coordinates corresponding to each of the multiple cameras in the terrain data, the depth information of the image planes corresponding to each of the multiple cameras in the terrain data is determined, including: for each of the multiple cameras, based on the depth information and coordinate offset information corresponding to the camera center coordinates corresponding to each camera in the terrain data, determining the depth information corresponding to each camera coordinate corresponding to each camera in the terrain data, wherein the coordinate offset information includes the offset of each other camera coordinate corresponding to each camera relative to the camera center coordinate; based on the depth information corresponding to each camera coordinate corresponding to each camera in the terrain data, determining the depth information corresponding to the image plane of each camera in the terrain data.

[0008] In combination with the first aspect, in certain implementations of the first aspect, before determining the camera depth maps corresponding to multiple cameras in the aircraft based on the depth data corresponding to the terrain data of the target scene, it also includes: determining three-dimensional reconstruction information corresponding to the target scene; and generating terrain data of the target scene based on the three-dimensional reconstruction information.

[0009] In combination with the first aspect, in certain implementations of the first aspect, determining camera depth maps corresponding to each of the multiple cameras in the aircraft based on the depth data corresponding to the terrain data of the target scene includes: determining the depth data corresponding to the terrain data of the target scene based on the depth data collected by the depth sensor in the aircraft; generating a depth map corresponding to the depth sensor based on the depth data corresponding to the terrain data of the target scene; and determining the camera depth map corresponding to each of the multiple cameras based on the depth map corresponding to the depth sensor, the relative transformation relationship between the depth sensor and the multiple cameras, and the intrinsic parameter information corresponding to each of the multiple cameras.

[0010] In combination with the first aspect, in certain implementations of the first aspect, based on the camera depth maps corresponding to each of the multiple cameras, the images captured by the multiple cameras are projected onto the image planes of the virtual cameras corresponding to the multiple cameras, and the aligned images corresponding to the images captured by the multiple cameras are determined, including: determining the virtual viewpoints of the virtual cameras corresponding to the multiple cameras based on the position information of each of the multiple cameras; determining the virtual relative transformation relationships corresponding to the multiple cameras based on the camera relative transformation relationships between the virtual viewpoints and the multiple cameras, wherein the virtual relative transformation relationships are the relative transformation relationships between each camera and the virtual viewpoint; based on the camera depth maps corresponding to the multiple cameras, the virtual relative transformation relationships, the camera internal parameter information corresponding to the virtual cameras, and the camera internal parameter information corresponding to the multiple cameras, the image coordinate system coordinate points corresponding to the images captured by the multiple cameras are projected onto the image planes of the virtual cameras to determine the aligned images.

[0011] In a second aspect, an embodiment of the present application provides an image alignment device, which includes: a first determination module for determining a camera depth map corresponding to each of multiple cameras in an aircraft based on depth data corresponding to terrain data of a target scene; a second determination module for projecting images captured by each of the multiple cameras onto image planes of virtual cameras corresponding to the multiple cameras based on the camera depth maps corresponding to the multiple cameras, and determining aligned images corresponding to the images captured by each of the multiple cameras.

[0012] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program for executing the method mentioned in the first aspect.

[0013] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; and the processor is configured to execute the method described in the first aspect.

[0014] The image alignment method and apparatus, storage medium, and electronic device provided in the embodiments of the present application determine the camera depth maps corresponding to each of the multiple cameras in the aircraft based on the depth data corresponding to the terrain data of the target scene. Subsequently, based on the camera depth maps corresponding to the multiple cameras, the images captured by each of the multiple cameras are projected onto the image planes of the virtual cameras corresponding to the multiple cameras, thereby achieving the purpose of determining the aligned images corresponding to the images captured by the multiple cameras. The image alignment method provided in the embodiments of the present application integrates the depth data corresponding to the terrain data of the target scene. Compared to multiple camera alignment methods based on the high consistency assumption, the alignment accuracy is significantly improved, the scope of application is expanded, and it also provides a prerequisite foundation for the subsequent application of multiple cameras to analyze farmland operation information. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 Shown is a schematic diagram of a scenario applicable to an embodiment of the present application.

[0017] Figure 2 Shown is another schematic diagram of a scenario applicable to the embodiment of the present application.

[0018] Figure 3 Shown is a flow chart of an image alignment method provided by an exemplary embodiment of the present application.

[0019] Figure 4 Shown is a flowchart of an image alignment method provided by another exemplary embodiment of the present application.

[0020] Figure 5 FIG2 is a flow chart of determining depth information of image planes corresponding to multiple cameras in terrain data provided by an exemplary embodiment of the present application.

[0021] Figure 6 FIG2 is a schematic diagram of a process for determining depth information of image planes corresponding to multiple cameras in terrain data provided by another exemplary embodiment of the present application.

[0022] Figure 7 Shown is a flowchart of an image alignment method provided by yet another exemplary embodiment of the present application.

[0023] Figure 8 FIG2 is a schematic diagram of a process for determining camera depth maps corresponding to multiple cameras in an aircraft according to an exemplary embodiment of the present application.

[0024] Figure 9 FIG2 is a schematic diagram showing the relative transformation of a depth sensor and a multispectral camera provided by an exemplary embodiment of the present application.

[0025] Figure 10 The figure shows a flow chart of determining aligned images corresponding to images captured by multiple cameras, provided by an exemplary embodiment of the present application.

[0026] Figure 11 Shown is a schematic structural diagram of an image alignment device provided by an exemplary embodiment of the present application.

[0027] Figure 12 Shown is a structural diagram of a first determination module provided by an exemplary embodiment of the present application.

[0028] Figure 13 FIG2 is a schematic structural diagram of a depth information determination unit provided by an exemplary embodiment of the present application.

[0029] Figure 14 FIG. 1 is a schematic structural diagram of a third determining subunit provided by an exemplary embodiment of the present application.

[0030] Figure 15 Shown is a structural schematic diagram of an image alignment device provided by another exemplary embodiment of the present application.

[0031] Figure 16 Shown is a structural diagram of a first determination module provided by another exemplary embodiment of the present application.

[0032] Figure 17 Shown is a structural diagram of a second determination module provided by an exemplary embodiment of the present application.

[0033] Figure 18 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0035] Figure 1 The following is a schematic diagram of a scenario applicable to the embodiment of the present application. Figure 1 As shown, the scenario to which the embodiment of the present application is applicable is an aircraft scenario. Specifically, the scenario includes an aircraft 2 equipped with an image acquisition device 20 and a server 1 connected to the image acquisition device 20.

[0036] Image acquisition device 20 includes multiple cameras, each of which is used to capture images corresponding to a target scene. Server 1 is used to determine camera depth maps corresponding to each of the multiple cameras in the aircraft based on depth data corresponding to terrain data of the target scene. Based on the depth maps, the images captured by each of the multiple cameras are projected onto the image planes of the virtual cameras corresponding to the multiple cameras, thereby determining aligned images corresponding to the images captured by the multiple cameras. This scenario implements an image alignment method. Multiple aircraft can share a single server, which can receive data uploaded by different aircraft. This server can be updated for multiple aircraft simultaneously, thus conserving resources.

[0037] It should be noted that this application is also applicable to another scenario. Figure 2 FIG2 is a schematic diagram of another scenario applicable to an embodiment of the present application. Specifically, the scenario includes an aircraft 2, wherein the aircraft 2 includes an image acquisition module 201 and a computing module 202, and a communication connection exists between the image acquisition module 201 and the computing module 202.

[0038] Specifically, the image acquisition module 201 in the aircraft 2 includes multiple cameras, each of which is used to acquire images corresponding to the target scene. The calculation module 202 in the aircraft 2 is used to determine the camera depth map corresponding to each of the multiple cameras in the aircraft based on the depth data corresponding to the terrain data of the target scene. Then, based on the camera depth map corresponding to each of the multiple cameras, the images acquired by each of the multiple cameras are projected onto the image planes of the virtual cameras corresponding to the multiple cameras, thereby determining the aligned images corresponding to the images acquired by each of the multiple cameras. That is, this scenario implements an image alignment method. Figure 1 Compared with the scenario shown in FIG, this scenario does not require data transmission operations with related devices such as a server. Therefore, this scenario can ensure the real-time performance of the image alignment method.

[0039] Exemplary Methods

[0040] Figure 3 FIG. 1 is a flow chart of an image alignment method provided by an exemplary embodiment of the present application. Figure 3 As shown, the image alignment method provided in the embodiment of the present application includes the following steps.

[0041] Step 100: Determine a camera depth map corresponding to each of a plurality of cameras in an aircraft based on depth data corresponding to terrain data of a target scene.

[0042] For example, the multiple cameras mentioned in step 100 can be binocular cameras or multi-cameras onboard an aircraft, or can be multispectral cameras onboard an aircraft. The aircraft can be a drone or other flying equipment, and this application does not make specific limitations.

[0043] Exemplarily, the camera depth map may be determined according to depth values ​​of pixels on an image plane corresponding to the camera.

[0044] Exemplarily, the terrain data is data used to characterize the undulating state of the terrain surface in the target scene, that is, data with elevation information.

[0045] Step 200 : Based on the camera depth maps corresponding to the multiple cameras, project the images captured by the multiple cameras onto image planes of virtual cameras corresponding to the multiple cameras, and determine aligned images corresponding to the images captured by the multiple cameras.

[0046] For example, the images captured by the multiple cameras mentioned in step 200 include, but are not limited to, two frames of images of the target scene simultaneously captured by an aircraft's onboard binocular camera, multiple frames of images of the target scene simultaneously captured by an aircraft's onboard multi-camera, or multiple frames of images of the target scene captured by an aircraft's onboard multispectral camera in different wavelength bands. This embodiment of the present application does not specifically limit this.

[0047] Specifically, a virtual camera is constructed using a virtual photography method. The relative positional relationship between multiple cameras is utilized to reproject the images captured by each camera into an ideal virtual image, thereby aligning the images of multiple cameras.

[0048] The image alignment method provided in the embodiment of the present application determines the camera depth maps corresponding to each of the multiple cameras in the aircraft based on the depth data corresponding to the terrain data of the target scene, and then projects the images captured by each of the multiple cameras onto the image planes of the virtual cameras corresponding to the multiple cameras based on the camera depth maps corresponding to the multiple cameras, thereby achieving the purpose of determining the aligned images corresponding to the images captured by each of the multiple cameras. The embodiment of the present application performs image alignment of multiple cameras when the depth data corresponding to the terrain data of the target scene is known, thereby avoiding the problem of large image alignment errors of multiple cameras due to the assumption of high consistency, thereby effectively improving the image alignment accuracy. In addition, the embodiment of the present application has the advantage of a wide range of applications.

[0049] In particular, the image alignment method provided in the embodiments of this application is applied to agricultural drone operation scenarios in farmland scenes. Especially for low-altitude scenes and non-flat operation scenes such as mountains, where drones typically fly at low altitudes, the image alignment method provided in the embodiments of this application does not require the assumption that the elevation of the onboard multispectral camera relative to the ground is a fixed value. Instead, it only requires depth data corresponding to the terrain data of the target scene to determine the aligned images corresponding to the images captured by each multispectral camera. Compared to multispectral camera alignment methods based on the assumption of high consistency, the alignment accuracy is significantly improved, providing a prerequisite foundation for the subsequent application of multispectral cameras in the analysis of farmland operation information.

[0050] As you can understand, multispectral cameras are used to provide multi-band spectral data for agricultural remote sensing. They consist of multiple independent imagers, each equipped with a specialized filter that allows each imager to capture spectra in a different wavelength range. A multispectral camera is used to capture farmland scenes, capturing images in different spectral bands, such as red, green, blue, red-edge, and near-infrared. These spectral images, collected by cameras in different spectral bands, are then aligned to create an aligned image.

[0051] Figure 4 The figure shows a flow chart of an image alignment method provided by another exemplary embodiment of the present application. Figure 3 The present application is extended based on the embodiment shown Figure 4 The embodiment shown is described below in detail. Figure 4 The embodiment shown is Figure 3 The differences and similarities between the illustrated embodiments are not described in detail.

[0052] like Figure 4As shown, in the image alignment method provided in the embodiment of the present application, based on the depth data corresponding to the terrain data of the target scene, the camera depth map corresponding to each of the multiple cameras in the aircraft is determined (step 100), which includes the following steps.

[0053] Step 101 : Determine depth information of image planes corresponding to the plurality of cameras in terrain data based on depth data and a relative transformation relationship between a camera coordinate system corresponding to the plurality of cameras and a world coordinate system.

[0054] Specifically, the drone's onboard real-time kinematic (RTK) module acquires the target scene's spatial position in the world coordinate system. This determines the world coordinates and elevation of the target scene's spatial position. Once the positions of multiple cameras and the RTK module are determined, the relative transformation relationship between each camera and the RTK module can be determined.

[0055] In one embodiment, multiple cameras and the real-time differential positioning module are fixed on the drone during installation. Based on the rigid connection, the connection relationship between the multiple cameras and the real-time differential positioning module is fixed. The relative transformation relationship between each of the multiple cameras and the real-time differential positioning module can be determined through the external parameter calibration method.

[0056] For example, the positional relationship between camera C1 and the real-time differential positioning module is known. The relative transformation relationship between camera C1 and the real-time differential positioning module is determined by the external parameter calibration method. That is, the relative transformation relationship between the coordinate system information corresponding to camera C1 among the multiple cameras and the world coordinate system information. Similarly, the relative transformation relationship between the other cameras among the multiple cameras and the real-time differential positioning module can be obtained.

[0057] Step 102 : determining a camera depth map corresponding to each of the multiple cameras based on depth information of the image planes corresponding to each of the multiple cameras in the terrain data.

[0058] Specifically, the coordinates of the points on the image planes corresponding to the multiple cameras are two-dimensional coordinates. Terrain data can be considered a map containing three-dimensional coordinates, where each pixel in the map records the elevation and longitude and latitude of the terrain. The coordinate points on the camera's image plane are mapped in the terrain data, with each coordinate point corresponding to an elevation. By mapping the image planes corresponding to the multiple cameras in the terrain data and performing a depth value query based on the longitude and latitude information of the terrain data, the depth information corresponding to the coordinates of the points on the image planes corresponding to the multiple cameras can be determined, thereby determining the camera depth maps corresponding to the multiple cameras.

[0059] The image alignment method provided in the embodiment of the present application determines the depth information of the image planes corresponding to the multiple cameras in the terrain data based on the depth data and the relative transformation relationship between the camera coordinate system corresponding to the multiple cameras and the world coordinate system. Based on the depth information of the image planes corresponding to the multiple cameras in the terrain data, the purpose of determining the camera depth map corresponding to the multiple cameras is achieved, providing a prerequisite for subsequent image alignment.

[0060] Figure 5 The figure shows a flow chart of determining the depth information of the image planes corresponding to multiple cameras in the terrain data provided by an exemplary embodiment of the present application. Figure 4 The present application is extended based on the embodiment shown Figure 5 The embodiment shown is described below in detail. Figure 5 The embodiment shown is Figure 4 The differences and similarities between the illustrated embodiments are not described in detail.

[0061] like Figure 5 As shown, in the image alignment method provided in the embodiment of the present application, based on the depth data and the relative transformation relationship between the camera coordinate system corresponding to each of the multiple cameras and the world coordinate system, the depth information of the image plane corresponding to each of the multiple cameras in the terrain data is determined (step 101), which includes the following steps.

[0062] Step 1011 : determining the camera center coordinates corresponding to each of the multiple cameras based on the depth data and the relative transformation relationship between the camera coordinate system corresponding to each of the multiple cameras and the world coordinate system.

[0063] Step 1012 : Determine depth information of the camera center coordinates corresponding to the plurality of cameras in the terrain data based on the camera center coordinates and depth data corresponding to the plurality of cameras.

[0064] Step 1013 : Determine the depth information of the image planes corresponding to the multiple cameras in the terrain data based on the depth information of the camera center coordinates corresponding to the multiple cameras in the terrain data.

[0065] For example, assume that the spatial position corresponding to the positioning information collected by the real-time differential positioning module is p G =(x G ,y G ,z G ), based on the relative transformation relationship between the calibrated camera C1 and the real-time differential positioning module Determine the camera coordinate center of camera C1 among multiple cameras for: The camera coordinate center corresponding to camera C1 Map the terrain data to the target scene to find the depth value and determine the camera coordinate center Depth information. At the camera coordinate center After the depth information of the camera is determined, the depth information of the image planes corresponding to the multiple cameras in the terrain data is further determined, thereby further obtaining the camera depth map Similarly, the camera depth maps of the remaining cameras can be further obtained.

[0066] The image alignment method provided in the embodiment of the present application determines the camera center coordinates corresponding to each of the multiple cameras based on depth data and the relative transformation relationship between the camera coordinate system corresponding to each of the multiple cameras and the world coordinate system. Then, based on the camera center coordinates corresponding to each of the multiple cameras and the depth data, the depth information of the camera center coordinates corresponding to each of the multiple cameras in the terrain data is determined. Finally, based on the depth information of the camera center coordinates corresponding to each of the multiple cameras in the terrain data, the purpose of determining the depth information of the image planes corresponding to each of the multiple cameras in the terrain data is achieved, which is facilitating the subsequent determination of the camera depth maps corresponding to each of the multiple cameras.

[0067] Figure 6 The figure shows a flow chart of determining the depth information of the image planes corresponding to multiple cameras in the terrain data provided by another exemplary embodiment of the present application. Figure 5 The present application is extended based on the embodiment shown Figure 6 The embodiment shown is described below in detail. Figure 6 The embodiment shown is Figure 5 The differences and similarities between the illustrated embodiments are not described in detail.

[0068] like Figure 6 As shown, in the image alignment method provided in the embodiment of the present application, based on the depth information of the camera center coordinates corresponding to each of the multiple cameras in the terrain data, the depth information of the image plane corresponding to each of the multiple cameras in the terrain data is determined (step 1013), including the following steps. It will be understood that the following steps need to be performed for each of the multiple cameras.

[0069] Step 10130: Determine the depth information corresponding to each camera coordinate in the terrain data for each camera based on the depth information and coordinate offset information corresponding to the camera center coordinate in the terrain data for each camera. The coordinate offset information includes the offset of each camera coordinate relative to the camera center coordinate.

[0070] Step 10131 : Determine the depth information corresponding to the image plane of each camera in the terrain data based on the depth information corresponding to each camera coordinate in the terrain data.

[0071] Specifically, the camera center is equivalent to the center position of the camera's image. By mapping the most central position in the terrain data, the depth information of the camera coordinate center can be determined. Once the depth information of the camera coordinate center is determined, the depth information of the other camera coordinates can be determined based on the offset between the coordinates of other cameras on the image plane corresponding to camera C1 and the camera coordinate center. Based on the depth information of all camera coordinates on the image plane corresponding to camera C1, the depth information corresponding to the image plane of camera C1 in the terrain data can be determined. Similarly, the depth information corresponding to the image planes of each of the multiple cameras in the terrain data can be determined.

[0072] For example, the camera's image plane is a 640*480 image, and the pixel coordinates of the image center are (320, 240). When the depth information of the image center is determined based on the longitude and latitude in the terrain data, the depth information of other pixels on the image in the terrain data can be determined based on the offset between other pixels on the image and the image center. In this way, the depth information of the image corresponding to the camera's image plane can be obtained.

[0073] The image alignment method provided in the embodiment of the present application determines the depth information corresponding to each camera coordinate of each camera in the terrain data based on the depth information and coordinate offset information corresponding to the camera center coordinates of each camera in the terrain data. Based on the depth information corresponding to each camera coordinate of each camera in the terrain data, the purpose of determining the depth information corresponding to the image plane of each camera in the terrain data is achieved, and the scene depth information corresponding to the image plane of the two-dimensional camera is restored, providing a prerequisite for the subsequent determination of the camera depth map.

[0074] Figure 7 The figure shows a flow chart of an image alignment method provided by another exemplary embodiment of the present application. Figure 3 The present application is extended based on the embodiment shown Figure 7 The embodiment shown is described below in detail. Figure 7 The embodiment shown is Figure 3 The differences and similarities between the illustrated embodiments are not described in detail.

[0075] like Figure 7 As shown, in the image alignment method provided in the embodiment of the present application, before determining the camera depth maps corresponding to each of the multiple cameras in the aircraft based on the depth data corresponding to the terrain data of the target scene (step 100), the following steps are included.

[0076] Step 80: Determine the 3D reconstruction information corresponding to the target scene.

[0077] For example, the three-dimensional reconstruction information mentioned in step 80 can be reconstructed by a monocular camera or binocular camera onboard the drone, combined with the positioning information collected by the real-time differential positioning module, to reconstruct the three-dimensional environment of the target scene, thereby determining the three-dimensional reconstruction information corresponding to the target scene.

[0078] Step 90: Generate terrain data of the target scene based on the 3D reconstruction information.

[0079] Specifically, the terrain data generated by the three-dimensional dense reconstruction result is used as the source of the depth value, and the depth value of each point in the terrain data can be queried using longitude and latitude.

[0080] In one embodiment, the flying altitude of the operating drone in the farmland scene is relatively low, and it cannot be assumed that the shooting scene is all in a plane. In particular, for scenes such as mountains and terraces, the high consistency assumption is even more unsatisfactory. A drone is equipped with a monocular camera and a real-time differential positioning module. When the drone shoots the farmland scene, the real-time differential positioning module records the positioning information of the shooting point. After the drone shoots the image, the farmland scene is reconstructed in three dimensions based on the shot image and the corresponding positioning information. The terrain data generated by the three-dimensional dense reconstruction result is used as the source of the depth value, so that the depth value of each point in the terrain data can be queried using longitude and latitude.

[0081] The image alignment method provided in the embodiments of the present application determines the 3D reconstruction information corresponding to the target scene and, based on this information, generates terrain data for the target scene. This allows the depth value of each point in the terrain data to be queried using longitude and latitude. When the depth data corresponding to the target scene is known, multiple camera images are aligned, significantly improving alignment accuracy compared to multiple camera alignment methods based on the assumption of high consistency.

[0082] Figure 8 The figure shows a flow chart of determining the camera depth map corresponding to each of the multiple cameras in the aircraft provided by an exemplary embodiment of the present application. Figure 3 The present application is extended based on the embodiment shown Figure 8 The embodiment shown is described below in detail. Figure 8 The embodiment shown is Figure 3 The differences and similarities between the illustrated embodiments are not described in detail.

[0083] like Figure 8 As shown, in the image alignment method provided in the embodiment of the present application, based on the depth data corresponding to the terrain data of the target scene, the camera depth map corresponding to each of the multiple cameras in the aircraft is determined (step 100), which includes the following steps.

[0084] Step 103 : Determine depth data corresponding to the terrain data of the target scene based on the depth data collected by the depth sensor in the aircraft.

[0085] For example, the depth sensor mentioned in step 103 may be a Kinect sensor or a Realsense sensor. This application does not specifically limit the type of depth sensor, as long as it can collect depth data.

[0086] Step 104 : Generate a depth map corresponding to the depth sensor based on the depth data corresponding to the terrain data of the target scene.

[0087] For example, the depth map mentioned in step 104 includes depth data of the target scene collected by the depth sensor. Each pixel of the depth map corresponds to a depth data in the terrain data.

[0088] Step 105 : determining a camera depth map corresponding to each of the multiple cameras based on the depth map corresponding to the depth sensor, the relative transformation relationship between the depth sensor and the multiple cameras, and the intrinsic parameter information corresponding to each of the multiple cameras.

[0089] In one embodiment, an agricultural drone equipped with a multispectral camera and a depth sensor performs farmland operations. Assume that the multispectral camera consists of four-band cameras: C1, C2, C3, and C4. Once the relative positions of the multispectral camera and depth sensor D are determined, the relative transformation relationships between depth sensor D and the multiple cameras can be determined based on calibration results.

[0090] Figure 9 The figure shows a schematic diagram of the relative transformation between the depth sensor and the multi-spectral camera provided by an exemplary embodiment of the present application. Figure 9 As shown, the relative transformation relationship between the depth sensor D and the camera C1 in the multispectral camera M is Similarly, the relative transformation relationships between the depth sensor D and the cameras C2, C3, and C4 can be determined.

[0091] Since the flying height of the drone in the farmland scene is relatively low, it is impossible to assume that the shooting scene is all on the same plane. Especially for scenes such as mountains and terraces, the height consistency assumption is even more unsatisfactory. The depth data corresponding to the terrain data of the target scene is obtained in real time through the depth sensor D, and the depth map I corresponding to the depth sensor D is generated. D For the depth map I D Any pixel p in i =(u i ,v i ,z i ), known depth map I D The corresponding camera intrinsic parameter is K D, the spatial point P in the depth sensor D coordinate system i D =(x i ,y i ,z i ) can be determined based on the following formula (1).

[0092]

[0093] Among them, the internal parameter of the camera is K D It can be the following matrix.

[0094]

[0095] It is known that the relative transformation relationship between the depth sensor D and the camera C1 in the multispectral camera M is Based on the following formula (2), the depth sensor D is transformed into the coordinate system of the camera C1, and the spatial point in the coordinate system of the camera C1 can be obtained:

[0096]

[0097] Assume that the intrinsic parameter matrix of camera C1 is Based on the following formula (3), the spatial point in the coordinate system of camera C1 is Projected onto the camera C1 image plane In the image plane of camera C1, we can get Coordinate points in

[0098]

[0099] The depth map I constructed by the depth sensor D D All pixels are projected onto the camera C1 image plane In the example, we can construct the camera depth map corresponding to camera C1.

[0100] Similarly, according to the transfer effect of relative pose transformation, the images of camera C2, camera C3 and camera C4 in the multispectral camera M are Build camera depth map

[0101] The image alignment method provided in the embodiment of the present application determines the depth data corresponding to the terrain data of the target scene based on the depth data collected by the depth sensor in the aircraft, and then generates a depth map corresponding to the depth sensor based on the depth data corresponding to the terrain data of the target scene. Finally, based on the depth map corresponding to the depth sensor, the relative transformation relationship between the depth sensor and multiple cameras, and the intrinsic parameter information corresponding to each of the multiple cameras, the camera depth map corresponding to each of the multiple cameras is determined, which is conducive to the subsequent implementation of image alignment of multiple cameras.

[0102] Figure 10 The figure shows a flow chart of determining the alignment images corresponding to the images captured by multiple cameras according to an exemplary embodiment of the present application. Figure 3 The present application is extended based on the embodiment shown Figure 10 The embodiment shown is described below in detail. Figure 10 The embodiment shown is Figure 3 The differences and similarities between the illustrated embodiments are not described in detail.

[0103] like Figure 10 As shown, in the image alignment method provided in the embodiment of the present application, based on the camera depth maps corresponding to the multiple cameras, the images captured by the multiple cameras are projected onto the image planes of the virtual cameras corresponding to the multiple cameras, and the aligned images corresponding to the images captured by the multiple cameras are determined (step 200), which includes the following steps.

[0104] Step 201 : determining virtual viewpoints of virtual cameras corresponding to the multiple cameras based on respective position information of the multiple cameras.

[0105] For example, multiple cameras correspond to one virtual camera, and the virtual camera has a virtual viewpoint. The virtual viewpoint can be located at the center of the multiple cameras, or at other reference points corresponding to the multiple cameras. For example, the virtual viewpoint of a virtual camera corresponding to a multispectral camera can be located at the center of the multispectral camera. This application does not impose specific limitations on this.

[0106] Step 202 : determining virtual relative transformation relationships corresponding to the multiple cameras based on the camera relative transformation relationships between the virtual viewpoint and the multiple cameras, wherein the virtual relative transformation relationship is the relative transformation relationship between the camera and the virtual viewpoint.

[0107] Specifically, the position information of each of the multiple cameras and the relative transformation relationship between the multiple cameras are known, and based on the virtual viewpoint of the assumed virtual camera, the relative transformation relationship between each of the multiple cameras and the virtual viewpoint is determined.

[0108] In step 203, based on the camera depth maps corresponding to the multiple cameras, the virtual relative transformation relationship, the camera internal parameter information corresponding to the virtual camera, and the camera internal parameter information corresponding to the multiple cameras, the image coordinate system coordinate points corresponding to the images captured by the multiple cameras are projected onto the image plane of the virtual camera to determine the aligned images.

[0109] For example, the positions of four cameras in a multispectral camera M are determined, wherein the relative transformations between the four cameras in the multispectral camera M are known: Indicates the transformation from camera C1 to camera C2; Indicates the transformation from camera C1 to camera C3; Represents the transformation from camera C1 to camera C4. Since the transformation matrix T is a homogeneous matrix, the relative transformation can be transferred by multiplication, so we can get: the relative transformation between camera C2 and camera C3 is The relative transformation between camera C2 and camera C4 is: The relative transformation between camera C3 and camera C4 is: The relative transformation relationship between the four cameras is known, so it is assumed that the virtual viewpoint V is located at the center of the multispectral camera, and the virtual camera internal parameter is set to K V . And the relative transformation between camera C1 and virtual viewpoint V is Similarly, determine the relative transformation between camera C2 and virtual viewpoint V Relative transformation between camera C3 and virtual viewpoint V The relative transformation between camera C4 and virtual viewpoint V is

[0110] According to the overlapping parts of the images taken by the four cameras, the overlapping parts corresponding to the images taken by camera C1 are determined. Assume that any coordinate point p in the overlapping parts corresponding to the images taken by camera C1 is i =(u i ,v i ,1) Projected to virtual camera C V Image plane, according to the following formula (4) we get p i In the virtual camera C V Coordinates corresponding to the image plane Then we have:

[0111]

[0112] In the above formula (4), z i is the coordinate point p i Similarly, all points in all cameras are projected into the virtual camera image plane, thereby aggregating the information of the four bands into the same virtual camera.

[0113] The image alignment method provided in the embodiment of the present application first determines the virtual viewpoints of the virtual cameras corresponding to the multiple cameras based on the position information of each of the multiple cameras; then determines the virtual relative transformation relationship corresponding to the multiple cameras based on the virtual viewpoint and the camera relative transformation relationship between the multiple cameras; finally, based on the camera depth maps corresponding to the multiple cameras, the virtual relative transformation relationship, the camera internal parameter information corresponding to the virtual camera, and the camera internal parameter information corresponding to the multiple cameras, the image coordinate system coordinate points corresponding to the images captured by the multiple cameras are projected onto the image plane of the virtual camera, thereby determining the aligned image. A virtual photography method is adopted, and the mutual positional relationship between the multiple cameras is utilized to project the images captured by the multiple cameras into the virtual camera, thereby achieving alignment of the images captured by the multiple cameras and ensuring that the alignment error is controlled at the sub-pixel level.

[0114] Exemplary devices

[0115] Combined with the above Figures 1 to 10 , describes the method embodiment of the present application in detail, and the following is combined with Figures 11 to 18 , the device embodiment of the present application is described in detail. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so for parts not described in detail, reference can be made to the previous method embodiment.

[0116] Figure 11 FIG. 1 is a schematic diagram of the structure of an image alignment device provided by an exemplary embodiment of the present application. Figure 11 As shown, the image alignment apparatus provided in an embodiment of the present application includes a first determination module 300 and a second determination module 400. The first determination module 300 is configured to determine a camera depth map corresponding to each of multiple cameras in an aircraft based on depth data corresponding to terrain data of a target scene. The second determination module 400 is configured to project the images captured by each of the multiple cameras onto the image planes of the virtual cameras corresponding to the multiple cameras based on the camera depth maps corresponding to the multiple cameras, and determine an aligned image corresponding to the images captured by the multiple cameras.

[0117] Figure 12 FIG. 1 is a schematic diagram showing the structure of a first determination module provided by an exemplary embodiment of the present application. Figure 12 As shown, in the image alignment device provided in the embodiment of the present application, the first determination module 300 includes a depth information determination unit 301 and a camera depth map determination unit 302. The depth information determination unit 301 is configured to determine the depth information of the image plane corresponding to each of the multiple cameras in the terrain data based on the depth data and the relative transformation relationship between the camera coordinate system corresponding to each of the multiple cameras and the world coordinate system. The camera depth map determination unit 302 is configured to determine the camera depth map corresponding to each of the multiple cameras based on the depth information of the image plane corresponding to each of the multiple cameras in the terrain data.

[0118] Figure 13 FIG. 1 is a schematic diagram of a depth information determination unit provided by an exemplary embodiment of the present application. Figure 13 As shown, in the image alignment device provided in the embodiment of the present application, the depth information determination unit 301 includes a first determination subunit 3011, a second determination subunit 3012, and a third determination subunit 3013. The first determination subunit 3011 is configured to determine the camera center coordinates corresponding to each of the multiple cameras based on the depth data and the relative transformation relationship between the camera coordinate system corresponding to each of the multiple cameras and the world coordinate system. The second determination subunit 3012 is configured to determine the depth information of the camera center coordinates corresponding to each of the multiple cameras in the terrain data based on the camera center coordinates corresponding to each of the multiple cameras and the depth data. The third determination subunit 3013 is configured to determine the depth information of the image plane corresponding to each of the multiple cameras in the terrain data based on the depth information of the camera center coordinates corresponding to each of the multiple cameras in the terrain data.

[0119] Figure 14 FIG. 1 is a schematic diagram showing the structure of a third determining subunit provided by an exemplary embodiment of the present application. Figure 14 As shown, in the image alignment device provided in an embodiment of the present application, for each of the multiple cameras, the third determining subunit 3013 includes a fourth determining subunit 30130 and a fifth determining subunit 30131. The fourth determining subunit 30130 is configured to determine the depth information corresponding to each camera coordinate corresponding to each camera in the terrain data based on the depth information and coordinate offset information corresponding to the camera center coordinates corresponding to each camera in the terrain data, wherein the coordinate offset information includes the offset of each other camera coordinate corresponding to each camera relative to the camera center coordinates. The fifth determining subunit 30131 is configured to determine the depth information corresponding to the image plane of each camera in the terrain data based on the depth information corresponding to the camera center coordinates corresponding to each camera in the terrain data.

[0120] Figure 15 FIG. 1 is a schematic diagram of the structure of an image alignment device provided by another exemplary embodiment of the present application. Figure 15 As shown, in the image alignment device provided in the embodiment of the present application, the image alignment device also includes a 3D reconstruction information determination module 500 and a terrain data determination module 600. The 3D reconstruction information determination module 500 is configured to determine 3D reconstruction information corresponding to the target scene. The terrain data determination module 600 is configured to generate terrain data of the target scene based on the 3D reconstruction information.

[0121] Figure 16 FIG. 1 is a schematic diagram showing the structure of a first determination module provided by another exemplary embodiment of the present application. Figure 16As shown, in the image alignment device provided in the embodiment of the present application, the first determination module 300 includes a depth data determination unit 303, a sixth determination unit 304, and a seventh determination unit 305. The depth data determination unit 303 is configured to determine the depth data corresponding to the terrain data of the target scene based on the depth data collected by the depth sensor in the aircraft. The sixth determination unit 304 is configured to generate a depth map corresponding to the depth sensor based on the depth data corresponding to the terrain data of the target scene. The seventh determination unit 305 is configured to determine the camera depth map corresponding to each of the multiple cameras based on the depth map corresponding to the depth sensor, the relative transformation relationship between the depth sensor and the multiple cameras, and the intrinsic parameter information corresponding to each of the multiple cameras.

[0122] Figure 17 FIG. 1 is a schematic diagram showing the structure of a second determination module provided by an exemplary embodiment of the present application. Figure 17 As shown, in the image alignment device provided by the embodiment of the present application, the second determination module 400 includes a virtual viewpoint determination unit 401, a virtual relative transformation relationship determination unit 402, and an aligned image determination unit 403. The virtual viewpoint determination unit 401 is configured to determine the virtual viewpoints of the virtual cameras corresponding to the multiple cameras based on the position information of each of the multiple cameras. The virtual relative transformation relationship determination unit 402 is configured to determine the virtual relative transformation relationships corresponding to each of the multiple cameras based on the camera relative transformation relationships between the virtual viewpoint and the multiple cameras, wherein the virtual relative transformation relationships are the relative transformation relationships between each camera and the virtual viewpoint. The aligned image determination unit 403 is configured to project the image coordinate system coordinate points corresponding to the images captured by the multiple cameras onto the image plane of the virtual camera based on the camera depth maps corresponding to the multiple cameras, the virtual relative transformation relationships, the camera intrinsic parameter information corresponding to the virtual cameras, and the camera intrinsic parameter information corresponding to the multiple cameras, to determine the aligned image.

[0123] Exemplary electronic devices

[0124] Below, reference Figure 18 To describe the electronic device according to the embodiment of the present application. Figure 18 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.

[0125] like Figure 18 As shown, the electronic device 70 includes one or more processors 701 and a memory 702 .

[0126] The processor 701 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 70 to perform desired functions.

[0127] The memory 702 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 701 may execute the program instructions to implement the image alignment method of each embodiment of the present application described above and / or other desired functions. Various contents such as images captured by multiple cameras may also be stored in the computer-readable storage medium.

[0128] In one example, the electronic device 70 may further include an input device 703 and an output device 704 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0129] The input device 703 may include, for example, a keyboard, a mouse, and the like.

[0130] The output device 704 can output various information to the outside, including the determined integrity information of the target structure, etc. The output device 704 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0131] Of course, to simplify, Figure 18 Only some of the components related to the present application in the electronic device 70 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 70 may further include any other appropriate components according to specific application scenarios.

[0132] Exemplary computer-readable storage media

[0133] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the image alignment method according to various embodiments of the present application described above in this specification.

[0134] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0135] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the image alignment method according to various embodiments of the present application described above in this specification.

[0136] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0137] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0138] The block diagrams of the devices, devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0139] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0140] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0141] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An image alignment method, characterized in that: include: Determining a camera depth map corresponding to each of a plurality of cameras in the aircraft based on depth data corresponding to terrain data of the target scene; Determining virtual viewpoints of virtual cameras corresponding to the multiple cameras based on position information of each of the multiple cameras; Determining, based on a camera relative transformation relationship between the virtual viewpoint and the multiple cameras, a virtual relative transformation relationship corresponding to each of the multiple cameras, wherein the virtual relative transformation relationship is a relative transformation relationship between each camera and the virtual viewpoint; Based on the camera depth maps corresponding to each of the multiple cameras, the virtual relative transformation relationship, the camera internal parameter information corresponding to the virtual camera, and the camera internal parameter information corresponding to each of the multiple cameras, the image coordinate system coordinate points corresponding to the images captured by each of the multiple cameras are projected onto the image plane of the virtual camera to determine the aligned image.

2. The image alignment method according to claim 1, wherein: The step of determining a camera depth map corresponding to each of the plurality of cameras in the aircraft based on the depth data corresponding to the terrain data of the target scene includes: Determining depth information of image planes corresponding to each of the multiple cameras in the terrain data based on the depth data and a relative transformation relationship between a camera coordinate system corresponding to each of the multiple cameras and a world coordinate system; A camera depth map corresponding to each of the plurality of cameras is determined based on depth information of image planes corresponding to each of the plurality of cameras in the terrain data.

3. The image alignment method according to claim 2, characterized in that: The determining, based on the depth data and a relative transformation relationship between a camera coordinate system corresponding to each of the multiple cameras and a world coordinate system, depth information of image planes corresponding to each of the multiple cameras in the terrain data includes: Determining the camera center coordinates corresponding to each of the multiple cameras based on the depth data and a relative transformation relationship between the camera coordinate system corresponding to each of the multiple cameras and the world coordinate system; determining, based on the camera center coordinates corresponding to each of the plurality of cameras and the depth data, depth information of the camera center coordinates corresponding to each of the plurality of cameras in the terrain data; Depth information of image planes corresponding to each of the plurality of cameras in the terrain data is determined based on depth information of camera center coordinates corresponding to each of the plurality of cameras in the terrain data.

4. The image alignment method according to claim 3, wherein: The determining, based on the depth information of the camera center coordinates corresponding to each of the multiple cameras in the terrain data, the depth information of the image planes corresponding to each of the multiple cameras in the terrain data includes: For each camera of the plurality of cameras, Determining, based on depth information and coordinate offset information corresponding to the camera center coordinates corresponding to each camera in the terrain data, depth information corresponding to each camera coordinate corresponding to each camera in the terrain data, wherein the coordinate offset information includes an offset of each other camera coordinate corresponding to each camera relative to the camera center coordinate; Determine the depth information corresponding to the image plane of each camera in the terrain data based on the depth information corresponding to each camera coordinate corresponding to each camera in the terrain data.

5. The image alignment method according to claim 1, wherein: Before determining the camera depth maps corresponding to the plurality of cameras in the aircraft based on the depth data corresponding to the terrain data of the target scene, the method further includes: Determining three-dimensional reconstruction information corresponding to the target scene; Based on the three-dimensional reconstruction information, terrain data of the target scene is generated.

6. The image alignment method according to claim 1, wherein: The step of determining a camera depth map corresponding to each of the plurality of cameras in the aircraft based on the depth data corresponding to the terrain data of the target scene includes: Determining depth data corresponding to the terrain data of the target scene based on depth data collected by the depth sensor in the aircraft; generating a depth map corresponding to the depth sensor based on depth data corresponding to the terrain data of the target scene; Determine a camera depth map corresponding to each of the multiple cameras based on the depth map corresponding to the depth sensor, a relative transformation relationship between the depth sensor and the multiple cameras, and intrinsic parameter information corresponding to each of the multiple cameras.

7. An image alignment device, characterized in that: include: A first determining module is configured to determine a camera depth map corresponding to each of a plurality of cameras in the aircraft based on depth data corresponding to terrain data of a target scene; A second determining module is configured to determine virtual viewpoints of virtual cameras corresponding to the plurality of cameras based on position information of each of the plurality of cameras; Based on the camera relative transformation relationship between the virtual viewpoint and the multiple cameras, determine the virtual relative transformation relationship corresponding to each of the multiple cameras, wherein the virtual relative transformation relationship is the relative transformation relationship between each camera and the virtual viewpoint; based on the camera depth maps corresponding to each of the multiple cameras, the virtual relative transformation relationship, the camera internal parameter information corresponding to the virtual camera, and the camera internal parameter information corresponding to each of the multiple cameras, project the image coordinate system coordinate points corresponding to the images captured by each of the multiple cameras onto the image plane of the virtual camera to determine the aligned images.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and the computer program is used to execute the image alignment method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to execute the image alignment method described in any one of claims 1 to 6.

Images