Image alignment method and device, electronic device, and computer-readable storage medium

Abstract

The present application relates to the field of image processing technology, and specifically to an image alignment method and image alignment device, as well as an electronic device and a computer-readable storage medium, which solve the problem of poor alignment accuracy of existing image alignment methods. The image alignment method provided in an embodiment of the present application determines a depth map corresponding to the image acquisition device based on the geographic location information of the image acquisition device in the aircraft and the target scene image acquired by the image acquisition device, thereby providing real depth data for multiple cameras of different bands in the aircraft, so that the images acquired by the multiple cameras can be aligned based on the real depth data. Compared with the method in the prior art that assumes that the depth data is a fixed and uniform value, the present application greatly improves the accuracy of image alignment.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image alignment method and an image alignment device, as well as an electronic device and a computer-readable storage medium. Background Art

[0002] Because a multispectral camera includes multiple cameras with different spectral bands, the spectral images captured by these cameras need to be aligned to obtain an aligned image. Current methods for aligning spectral images captured by cameras with different spectral bands ignore terrain undulations and approximate the ground as a flat surface. This assumes that the elevation of the multispectral cameras relative to the ground is fixed and uniform, thus achieving alignment of the spectral images captured by multiple cameras.

[0003] However, in agricultural scenarios, when aircraft conduct crop growth and pest analysis, they generally fly at a low altitude. At this time, if the terrain undulations are ignored and the ground is approximated as a plane, the assumed aircraft flight altitude will have a large error, resulting in a large error in the obtained aligned images. Summary of the Invention

[0004] In view of this, embodiments of the present application provide an image alignment method and apparatus, an electronic device, and a computer-readable storage medium to solve the problem of poor alignment accuracy of existing image alignment methods.

[0005] In a first aspect, an embodiment of the present application provides an image alignment method, comprising: determining a depth map corresponding to the image acquisition device based on geographic location information of the image acquisition device in an aircraft and an image of a target scene acquired by the image acquisition device; and determining aligned images corresponding to the images acquired by each of the multiple cameras based on the depth map corresponding to the image acquisition device, images acquired by each of the multiple cameras of different bands in the aircraft, and a relative transformation relationship between the image acquisition device and the multiple cameras.

[0006] In combination with the first aspect, in certain implementations of the first aspect, determining a depth map corresponding to the image acquisition device based on the geographic location information of the image acquisition device in the aircraft and the target scene image captured by the image acquisition device includes: determining at least two frames of the target scene image captured by the image acquisition device; determining, based on the at least two frames of the target scene image, first camera pose information corresponding to the image acquisition device and first three-dimensional point cloud data corresponding to the target scene in the camera coordinate system of the image acquisition device; and determining the depth map corresponding to the image acquisition device based on the geographic location information, the first camera pose information and the first three-dimensional point cloud data of the image acquisition device.

[0007] In combination with the first aspect, in certain implementations of the first aspect, based on at least two frames of target scene images, first camera pose information corresponding to the image acquisition device and first three-dimensional point cloud data corresponding to the target scene are determined in the camera coordinate system of the image acquisition device, including: extracting feature point sets corresponding to each of the at least two frames of target scene images; performing feature matching and motion estimation based on the feature point sets corresponding to each of the at least two frames of target scene images to determine motion estimation data; and determining the first camera pose information and first three-dimensional point cloud data based on the motion estimation data.

[0008] In combination with the first aspect, in certain implementations of the first aspect, a depth map corresponding to the image acquisition device is determined based on the geographic location information, the first camera posture information and the first three-dimensional point cloud data of the image acquisition device, including: determining the second three-dimensional point cloud data corresponding to the target scene in the world coordinate system based on the geographic location information, the first camera posture information and the first three-dimensional point cloud data of the image acquisition device; and determining the depth map corresponding to the image acquisition device based on the geographic location information and the second three-dimensional point cloud data of the image acquisition device.

[0009] In combination with the first aspect, in certain implementations of the first aspect, before determining the depth map corresponding to the image acquisition device based on the geographic location information of the image acquisition device in the aircraft and the target scene image captured by the image acquisition device, it also includes: obtaining positioning data obtained based on the positioning device in the aircraft; and determining the geographic location information of the image acquisition device based on the relative transformation relationship between the positioning data, the image acquisition device and the positioning device.

[0010] In combination with the first aspect, in certain implementations of the first aspect, based on the depth map corresponding to the image acquisition device, the images captured by multiple cameras of different bands in the aircraft, and the position coordinate transformation data between the image acquisition device and the multiple cameras, the alignment images corresponding to the images captured by the multiple cameras are determined, including: determining the depth map corresponding to the multiple cameras based on the depth map corresponding to the image acquisition device, the relative transformation relationship between the image acquisition device and the multiple cameras, and the internal parameters of the image acquisition device and the multiple cameras; determining the alignment images corresponding to the images captured by the multiple cameras based on the depth maps corresponding to the multiple cameras and the images captured by the multiple cameras.

[0011] In combination with the first aspect, in certain implementations of the first aspect, based on the depth map corresponding to the image acquisition device, the relative transformation relationship between the image acquisition device and the multiple cameras, and the internal parameters of the image acquisition device and the multiple cameras, the depth maps corresponding to each of the multiple cameras are determined, including: determining the first coordinate data of the image acquisition device in the camera coordinate system of the image acquisition device based on the depth map of the image acquisition device and the internal parameters of the image acquisition device; for each of the multiple cameras, determining the second coordinate data of the camera in the camera coordinate system of the camera based on the first coordinate data and the position coordinate transformation data between the image acquisition device and the camera; and determining the depth maps corresponding to each of the multiple cameras based on the second coordinate data of each of the multiple cameras and the internal parameters of each of the multiple cameras.

[0012] In combination with the first aspect, in certain implementations of the first aspect, based on the depth maps corresponding to the multiple cameras and the images captured by the multiple cameras, the aligned images corresponding to the images captured by the multiple cameras are determined, including: based on the geographic location information of the multiple cameras, the virtual viewpoints of the virtual cameras corresponding to the multiple cameras are determined; based on the virtual viewpoints and the geographic location information of the multiple cameras, the virtual relative transformation relationships corresponding to the multiple cameras are determined, wherein the virtual relative transformation relationships are the relative transformation relationships between the cameras and the virtual viewpoints; based on the intrinsic parameters of the multiple cameras, the intrinsic parameters of the virtual cameras, the depth maps corresponding to the multiple cameras and the virtual relative transformation relationships corresponding to the multiple cameras, the image coordinate system coordinate points corresponding to the images captured by the multiple cameras are projected onto the image plane of the virtual camera to determine the aligned images.

[0013] In a second aspect, an image alignment device provided by an embodiment of the present application includes: a determination module, configured to determine a depth map corresponding to the image acquisition device based on the geographic location information of the image acquisition device in the aircraft and the target scene image acquired by the image acquisition device; an alignment module, configured to determine aligned images corresponding to the images acquired by each of the multiple cameras based on the depth map corresponding to the image acquisition device, images acquired by each of the multiple cameras of different bands in the aircraft, and the relative transformation relationship between the image acquisition device and the multiple cameras.

[0014] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed by a processor of an electronic device, the electronic device can execute the image alignment method mentioned in the first aspect above.

[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes: a processor; a memory for storing computer-executable instructions; and a processor for executing computer-executable instructions to implement the image alignment method mentioned in the first aspect above.

[0016] An image alignment method provided in an embodiment of the present application determines a depth map corresponding to an image acquisition device in an aircraft based on the geographic location information of the image acquisition device and the target scene image acquired by the image acquisition device, thereby providing real depth data for multiple cameras of different bands in the aircraft, so that the images acquired by the multiple cameras can be aligned according to the real depth data. Compared with the method in the prior art that assumes that the depth data is a fixed and uniform numerical value, the present application greatly improves the accuracy of image alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same structure or step.

[0018] Figure 1 FIG2 is a flow chart of an image alignment method provided in an embodiment of the present application.

[0019] Figure 1a Shown is a schematic diagram of the relative transformation relationship provided by an embodiment of the present application.

[0020] Figure 2 Shown is a flow chart of an image alignment method provided in another embodiment of the present application.

[0021] Figure 3 Shown is a flow chart of an image alignment method provided in another embodiment of the present application.

[0022] Figure 4 Shown is a flow chart of an image alignment method provided in another embodiment of the present application.

[0023] Figure 5 Shown is a flow chart of an image alignment method provided in another embodiment of the present application.

[0024] Figure 6 Shown is a flow chart of an image alignment method provided in another embodiment of the present application.

[0025] Figure 7 Shown is a flow chart of an image alignment method provided in another embodiment of the present application.

[0026] Figure 8 Shown is a flow chart of an image alignment method provided in another embodiment of the present application.

[0027] Figure 9Shown is a structural schematic diagram of an image alignment device provided in one embodiment of the present application.

[0028] Figure 10 Shown is a structural diagram of a determination module provided in one embodiment of the present application.

[0029] Figure 11 Shown is a structural schematic diagram of a first point cloud determination unit provided in one embodiment of the present application.

[0030] Figure 12 FIG2 is a schematic structural diagram of a depth map determination unit provided in an embodiment of the present application.

[0031] Figure 13 Shown is a structural schematic diagram of an image alignment device provided in another embodiment of the present application.

[0032] Figure 14 Shown is a structural diagram of an alignment module provided in one embodiment of the present application.

[0033] Figure 15 FIG2 is a schematic structural diagram of a camera depth map determination unit provided in one embodiment of the present application.

[0034] Figure 16 FIG. 1 is a schematic structural diagram of an alignment unit provided in an embodiment of the present application.

[0035] Figure 17 Shown is a structural schematic diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0037] The scenarios to which the embodiments of the present application are applicable may include an aircraft, an image acquisition device, and multiple cameras of different wavelengths. The image acquisition device and the multiple cameras of different wavelengths are fixedly mounted on the aircraft. A computing module is provided in the aircraft. A communication connection relationship exists between the computing module, the image acquisition device, and the multiple cameras of different wavelengths. The image acquisition device and the multiple cameras of different wavelengths can acquire images of a target scene. The computing module in the aircraft can determine a depth map corresponding to the image acquisition device based on the geographic location information of the image acquisition device in the aircraft and the image of the target scene acquired by the image acquisition device, and then determine aligned images corresponding to the images acquired by the multiple cameras based on the depth map corresponding to the image acquisition device, the images acquired by the multiple cameras of different wavelengths in the aircraft, and the relative transformation relationship between the image acquisition device and the multiple cameras.

[0038] Another scenario applicable to the embodiments of the present application may include a server, an image acquisition device, and multiple cameras of different wavelengths. A communication connection relationship exists between the server, the image acquisition device, and the multiple cameras of different wavelengths. The image acquisition device and the multiple cameras of different wavelengths can capture images of a target scene. The server can determine a depth map corresponding to the image acquisition device based on the geographic location information of the image acquisition device in the aircraft and the target scene image captured by the image acquisition device, and then determine aligned images corresponding to the images captured by the multiple cameras based on the depth map corresponding to the image acquisition device, the images captured by the multiple cameras of different wavelengths in the aircraft, and the relative transformation relationship between the image acquisition device and the multiple cameras.

[0039] Figure 1 FIG. 1 is a flow chart of an image alignment method provided by an embodiment of the present application. Figure 1 As shown, the image alignment method provided in the embodiment of the present application includes the following steps.

[0040] Step 110 : determining a depth map corresponding to the image acquisition device based on the geographical location information of the image acquisition device in the aircraft and the target scene image acquired by the image acquisition device.

[0041] Specifically, the geographic location information of the image acquisition device may be the location information of the image acquisition device in a world coordinate system. The geographic location information of the image acquisition device may be obtained by a positioning device mounted on an aircraft. The target scene may be a farmland scene. For example, the target scene may be a scene with large variations in ground undulations, such as mountains or terraced fields. The target scene image may be an image captured by an image acquisition device mounted on the aircraft during flight. The depth map corresponding to the image acquisition device refers to an image in which the distance from the image acquisition device to each point in the target scene is used as a pixel value.

[0042] For example, the aircraft may be a drone or other flying equipment, and this application does not make any specific restrictions. The image acquisition device may be a monocular camera or a multi-camera, as long as it can capture the target scene image, and this application does not make any specific restrictions.

[0043] Step 120 , based on the depth map corresponding to the image acquisition device, the images acquired by the multiple cameras of different bands in the aircraft, and the relative transformation relationship between the image acquisition device and the multiple cameras, determine the aligned images corresponding to the images acquired by the multiple cameras.

[0044] For example, the multiple cameras of different wavelength bands may be multiple cameras of different wavelength bands in a multispectral camera. The images captured by each of the multiple cameras of different wavelength bands may be spectral images. The multispectral camera may be composed of four cameras of different wavelength bands, or six cameras of different wavelength bands. This application does not specifically limit the number of cameras of different wavelength bands that a multispectral camera may comprise.

[0045] For example, multispectral cameras can provide multi-band spectral data for agricultural remote sensing. A multispectral camera consists of multiple independent imagers, each equipped with a specialized filter that allows each imager to capture a spectrum of different wavelengths. Using a multispectral camera, farmland images can be acquired in different spectral bands, such as red, green, blue, red-edge, and near-infrared. The spectral images captured by these cameras in different spectral bands are then aligned to produce an aligned image.

[0046] For example, the image acquisition device and the multiple cameras are fixedly mounted on the aircraft. The relative transformation relationship between the image acquisition device and the multiple cameras can be obtained by calibrating the image acquisition device and the multiple cameras. The relative transformation relationship between the multiple cameras of different wavelength bands included in the multispectral camera is known.

[0047] For example, Figure 1a The figure shows a schematic diagram of the relative transformation relationship provided by an embodiment of the present application. Figure 1a As shown, taking four cameras as an example, four cameras with different wavelengths can be represented by C1, C2, C3, and C4. The image acquisition device can be represented by C. The relative transformation relationship between the image acquisition device C and the camera C1 can be represented by T CC1 .

[0048] Specifically, based on the depth map corresponding to the image acquisition device, images captured by multiple cameras of different wavelengths in the aircraft, and the relative transformation relationship between the image acquisition device and the multiple cameras, alignment images corresponding to the images captured by the multiple cameras are determined. This can be achieved by utilizing the relative transformation relationship between the image acquisition device and the multiple cameras and using the depth map corresponding to the image acquisition device to determine the depth maps corresponding to the multiple cameras. A virtual camera is then constructed using a virtual projection method. Using the relative transformation relationship between the multiple cameras, the images captured by the multiple cameras are fused with their corresponding depth maps and then projected onto the image plane of the virtual camera, thereby achieving alignment of the images captured by the multiple cameras.

[0049] An image alignment method provided in an embodiment of the present application determines a depth map corresponding to an image acquisition device in an aircraft based on the geographic location information of the image acquisition device and the target scene image acquired by the image acquisition device, thereby providing real depth data for multiple cameras of different bands in the aircraft, so that the images acquired by the multiple cameras can be aligned according to the real depth data. Compared with the method in the prior art that assumes that the depth data is a fixed and uniform numerical value, the present application greatly improves the accuracy of image alignment.

[0050] Figure 2 FIG. 1 is a flow chart of an image alignment method provided by another embodiment of the present application. Figure 1 Based on the embodiment shown Figure 2 The embodiment shown is described below in detail. Figure 2 The embodiment shown and Figure 1 The differences and similarities between the illustrated embodiments are not described in detail.

[0051] like Figure 2 As shown, in an embodiment of the present application, based on the geographic location information of the image acquisition device in the aircraft and the target scene image acquired by the image acquisition device, the step of determining the depth map corresponding to the image acquisition device includes the following steps.

[0052] Step 210: Determine at least two frames of target scene images captured by the image capture device.

[0053] For example, the image acquisition device can capture two consecutive frames of target scene images, or two discontinuous frames of target scene images. Those skilled in the art can select the appropriate number based on actual needs. The number of image frames captured by the image acquisition device can also be selected based on actual needs and is not specifically limited in this application.

[0054] Step 220 : Based on at least two frames of target scene images, determine first camera pose information corresponding to the image acquisition device and first three-dimensional point cloud data corresponding to the target scene in the camera coordinate system of the image acquisition device.

[0055] For example, a simultaneous localization and mapping (SLAM) algorithm can be used to determine, based on two frames of target scene images, first camera pose information corresponding to the image acquisition device and first three-dimensional point cloud data corresponding to the target scene in the camera coordinate system of the image acquisition device.

[0056] Step 230 : Determine a depth map corresponding to the image acquisition device based on the geographic location information of the image acquisition device, the first camera pose information, and the first three-dimensional point cloud data.

[0057] Exemplarily, the geographic location information of the image acquisition device can be fused with the first camera pose information and the first three-dimensional point cloud data to determine the camera pose and three-dimensional point cloud data of the image acquisition device in the world coordinate system, thereby determining the depth map corresponding to the image acquisition device.

[0058] By having the image acquisition device capture at least two frames of target scene images and fusing them with the geographic location information of the image acquisition device to obtain a depth map corresponding to the image acquisition device, the depth map corresponding to the image acquisition device can be obtained in real time, thereby providing real depth information for real-time alignment of images captured by multiple cameras of different bands, thereby improving the real-time and accuracy of image alignment.

[0059] Figure 3 FIG. 1 is a flow chart of an image alignment method provided by another embodiment of the present application. Figure 2 Based on the embodiment shown Figure 3 The embodiment shown is described below in detail. Figure 3 The embodiment shown and Figure 2 The differences and similarities between the illustrated embodiments are not described in detail.

[0060] like Figure 3 As shown, in an embodiment of the present application, based on at least two frames of target scene images, the steps of determining the first camera pose information corresponding to the image acquisition device and the first three-dimensional point cloud data corresponding to the target scene in the camera coordinate system of the image acquisition device include the following steps.

[0061] Step 310 : extracting feature point sets corresponding to at least two frames of target scene images respectively.

[0062] Specifically, multiple feature points of each frame of the target scene image are extracted to form a feature point set. That is, each frame of the target scene image corresponds to a feature point set.

[0063] Step 320 : performing feature matching and motion estimation based on feature point sets corresponding to at least two frames of target scene images to determine motion estimation data.

[0064] For example, an image acquisition device captures two consecutive frames of target scene images, namely target scene image 1 and target scene image 2. Target scene image 1 corresponds to feature point set 1, and target scene image 2 corresponds to feature point set 2. Feature matching and motion estimation are performed on feature point set 1 and feature point set 2 to obtain motion estimation data during the process of the image acquisition device capturing target scene image 1 and target scene image 2.

[0065] Step 330 : Determine first camera pose information and first three-dimensional point cloud data based on the motion estimation data.

[0066] For example, based on the motion estimation data of the image acquisition device in the process of acquiring target scene image 1 and target scene image 2, the first camera pose information and first three-dimensional point cloud data of the image acquisition device can be obtained in the camera coordinate system of the image acquisition device.

[0067] By performing feature point extraction, feature matching and motion estimation on at least two frames of target scene images acquired by an image acquisition device, first camera pose information and first three-dimensional point cloud data of the image acquisition device are obtained. The method is simple, reliable and efficient.

[0068] Figure 4 FIG. 1 is a flow chart of an image alignment method provided by another embodiment of the present application. Figure 2 Based on the embodiment shown Figure 4 The embodiment shown is described below in detail. Figure 4 The embodiment shown and Figure 2 The differences and similarities between the illustrated embodiments are not described in detail.

[0069] like Figure 4 As shown, in an embodiment of the present application, based on the geographic location information of the image acquisition device, the first camera posture information and the first three-dimensional point cloud data, the step of determining the depth map corresponding to the image acquisition device includes the following steps.

[0070] Step 410 : Determine second three-dimensional point cloud data corresponding to the target scene in a world coordinate system based on the geographic location information of the image acquisition device, the first camera pose information, and the first three-dimensional point cloud data.

[0071] Specifically, the image acquisition device's geographic location information is its position in the world coordinate system, while the first camera pose information and the first 3D point cloud data are the image acquisition device's pose information and 3D point cloud data in the image acquisition device's camera coordinate system. Therefore, by fusing the image acquisition device's geographic location information, the first camera pose information, and the first 3D point cloud data, second 3D point cloud data corresponding to the target scene in the world coordinate system can be obtained.

[0072] Exemplarily, an objective function of the first camera pose information, the first three-dimensional point cloud data and the geographic location information of the image acquisition device can be constructed to map the first camera pose information and the first three-dimensional point cloud data in the camera coordinate system of the image acquisition device to the world coordinate system to obtain the second three-dimensional point cloud data corresponding to the target scene.

[0073] Step 420: Determine a depth map corresponding to the image acquisition device based on the geographic location information of the image acquisition device and the second three-dimensional point cloud data.

[0074] In practical applications, a depth map corresponding to the image acquisition device can be constructed based on the geographic location information of the image acquisition device and the second three-dimensional point cloud data in the world coordinate system, providing real depth data for subsequent image alignment.

[0075] Figure 5 FIG. 1 is a flow chart of an image alignment method provided by another embodiment of the present application. Figure 1 Based on the embodiment shown Figure 5 The embodiment shown is described below in detail. Figure 5 The embodiment shown and Figure 1 The differences and similarities between the illustrated embodiments are not described in detail.

[0076] like Figure 5 As shown, in an embodiment of the present application, before the step of determining the depth map corresponding to the image acquisition device based on the geographic location information of the image acquisition device in the aircraft and the target scene image acquired by the image acquisition device, the following steps are also included.

[0077] Step 510: Acquire positioning data obtained based on a positioning device in the aircraft.

[0078] For example, the positioning device may be a real-time kinematic (RTK) device. The positioning data obtained by the positioning device is the position data of the positioning device in a world coordinate system.

[0079] Step 520: Determine the geographic location information of the image acquisition device based on the positioning data, the relative transformation relationship between the image acquisition device and the positioning device.

[0080] Exemplarily, the positioning data and the image acquisition device are both fixedly arranged in the aircraft, that is, the positional relationship between the positioning data and the image acquisition device is fixed and unchanged. Therefore, the relative transformation relationship between the image acquisition device and the positioning device can be determined by the external parameter calibration method.

[0081] For example, Figure 1a As shown, the positioning device is represented by G, and the image acquisition device is represented by C. After calibration, the relative transformation relationship between the image acquisition device C and the positioning device G is T CG . T CG is a homogeneous matrix.

[0082] In practical applications, the relative transformation relationship between the image acquisition device and the positioning device can be used to calculate the image acquisition device's geographic location based on the positioning data. Because the relative transformation relationship between the image acquisition device and the positioning device is fixed, the image acquisition device's geographic location can be calculated in real time based on real-time positioning data, which is convenient, fast, and highly efficient.

[0083] Figure 6 FIG. 1 is a flow chart of an image alignment method provided by another embodiment of the present application. Figure 1 Based on the embodiment shown Figure 6 The embodiment shown is described below in detail. Figure 6 The embodiment shown and Figure 1 The differences and similarities between the illustrated embodiments are not described in detail.

[0084] like Figure 6 As shown, in an embodiment of the present application, based on the depth map corresponding to the image acquisition device, the images acquired by multiple cameras of different bands in the aircraft, and the position coordinate transformation data between the image acquisition device and the multiple cameras, the step of aligning the images corresponding to the images acquired by the multiple cameras is determined, and includes the following steps.

[0085] Step 610 : Determine the depth maps corresponding to the multiple cameras based on the depth map corresponding to the image acquisition device, the relative transformation relationship between the image acquisition device and the multiple cameras, and the internal parameters of the image acquisition device and the multiple cameras.

[0086] Specifically, by utilizing the relative transformation relationship between the image acquisition device and the multiple cameras and the internal parameters of the image acquisition device and the multiple cameras, any pixel in the depth map corresponding to the image acquisition device can be transformed into the image plane corresponding to the multiple cameras, thereby constructing the depth maps corresponding to the multiple cameras.

[0087] Step 620 : Determine aligned images corresponding to the images captured by the multiple cameras based on the depth maps corresponding to the multiple cameras and the images captured by the multiple cameras.

[0088] Specifically, a virtual projection method can be used to construct a virtual camera, and the images captured by multiple cameras are fused with their corresponding depth maps and then projected onto the image plane of the virtual camera, thereby achieving alignment of the images captured by multiple cameras.

[0089] Based on the depth map corresponding to the image acquisition device, a series of coordinate transformations are performed to obtain the depth maps corresponding to the multiple cameras, thereby obtaining the real depth data of the multiple cameras in the world coordinate system. Then, the real depth data of the multiple cameras in the world coordinate system are used to align the images captured by the multiple cameras, obtaining aligned images and improving the accuracy of image alignment.

[0090] Figure 7 FIG. 1 is a flow chart of an image alignment method provided by another embodiment of the present application. Figure 6 Based on the embodiment shown Figure 7 The embodiment shown is described below in detail. Figure 7 The embodiment shown and Figure 6 The differences and similarities between the illustrated embodiments are not described in detail.

[0091] like Figure 7 As shown, in an embodiment of the present application, based on the depth map corresponding to the image acquisition device, the relative transformation relationship between the image acquisition device and the multiple cameras, and the internal parameters of the image acquisition device and the multiple cameras, the step of determining the depth map corresponding to each of the multiple cameras includes the following steps.

[0092] Step 710 : Determine first coordinate data of the image acquisition device in a camera coordinate system of the image acquisition device based on the depth map of the image acquisition device and the internal parameters of the image acquisition device.

[0093] For example, the first coordinate data may be a spatial point P in the camera coordinate system of the image acquisition device C. i C coordinate data.

[0094] For example, a multispectral camera is composed of four cameras with different wavelengths. The four cameras with different wavelengths are represented by C1, C2, C3, and C4 respectively, and the image acquisition device is represented by C.

[0095] The depth map corresponding to the image acquisition device is represented by I D The internal parameter of the image acquisition device is K C For the depth map I D Any pixel p in i =(ui ,v i ,z i ), known depth map I D The internal parameter of the corresponding image acquisition device is K C Based on the following formula (1), the spatial point P in the camera coordinate system of the image acquisition device C can be determined i C =(x i ,y i ,z i ).

[0096]

[0097] Among them, the internal parameter of the image acquisition device is K C It can be the following matrix.

[0098]

[0099] Step 720 : For each camera among the plurality of cameras, determine second coordinate data of the camera in the camera coordinate system based on the first coordinate data and the position coordinate transformation data between the image acquisition device and the camera.

[0100] For example, the second coordinate data may be a spatial point in the coordinate system of the camera C1. By calibrating the image acquisition device C and the camera C1, the relative transformation relationship between the image acquisition device C and the camera C1 can be obtained.

[0101] It is known that the relative transformation relationship between the image acquisition device C and the camera C1 in the multispectral camera M is: Based on the following formula (2), the spatial point in the camera coordinate system of the image acquisition device C can be transformed into the coordinate system of the camera C1, and the spatial point in the coordinate system of the camera C1 can be obtained.

[0102]

[0103] Step 730 : Determine depth maps corresponding to the multiple cameras based on the respective second coordinate data of the multiple cameras and the respective intrinsic parameters of the multiple cameras.

[0104] Assume that the intrinsic parameter matrix of camera C1 is Based on the following formula (3), the spatial point in the coordinate system of camera C1 is Projected onto the image plane of camera C1 In the , we can get the coordinate point

[0105]

[0106] By projecting all pixels in the depth map corresponding to the image acquisition device onto the image plane of the camera C1 In the example, the depth map corresponding to camera C1 can be constructed

[0107] Similarly, according to the transfer effect of relative pose transformation, the depth maps corresponding to cameras C2, C3 and C4 in the multispectral camera M can be constructed.

[0108] According to the transmission effect of relative pose transformation, the depth maps of multiple cameras are constructed using the same method, which improves the efficiency of constructing the depth maps of multiple cameras.

[0109] Figure 8 FIG. 1 is a flow chart of an image alignment method provided by another embodiment of the present application. Figure 6 Based on the embodiment shown Figure 8 The embodiment shown is described below in detail. Figure 8 The embodiment shown and Figure 6 The differences and similarities between the illustrated embodiments are not described in detail.

[0110] like Figure 8 As shown, in an embodiment of the present application, based on the depth maps corresponding to the multiple cameras and the images captured by the multiple cameras, the step of determining the alignment images corresponding to the images captured by the multiple cameras includes the following steps.

[0111] Step 810 : determining virtual viewpoints of virtual cameras corresponding to the multiple cameras based on the respective geographic location information of the multiple cameras.

[0112] For example, the virtual camera may be a camera placed at a virtual viewpoint. The virtual viewpoint of the virtual camera may be the center position of multiple cameras. Figure 1a As shown, the virtual viewpoint is represented by V. The virtual viewpoint corresponding to the multispectral camera can be the center of the multispectral camera. The virtual viewpoint can also be other reference point positions corresponding to multiple cameras, which is not specifically limited in this application.

[0113] Step 820 : determining virtual relative transformation relationships corresponding to the multiple cameras based on the camera relative transformation relationships between the virtual viewpoint and the multiple cameras.

[0114] Specifically, the virtual relative transformation relationship is the relative transformation relationship between the camera and the virtual viewpoint. The relative transformation relationship between multiple cameras is known. For example, four cameras are used. The relative transformation relationship between four cameras of different wavelengths, C1, C2, C3, and C4, is as follows.

[0115] The relative transformation relationship from camera C1 to camera C2 is:

[0116] The relative transformation relationship from camera C1 to camera C3 is:

[0117] The relative transformation relationship from camera C1 to camera C4 is:

[0118] Due to the relative transformation relationship They are all homogeneous matrices, so the relative transformation relationship can be transferred by multiplication, so the following relative transformation relationship can be obtained.

[0119] The relative transformation relationship from camera C2 to camera C3 is:

[0120] The relative transformation relationship from camera C2 to camera C4 is:

[0121] The relative transformation relationship from camera C3 to camera C4 is:

[0122] Assume that the virtual viewpoint V is located at the center of the multispectral camera, and set the virtual camera internal parameter to K V The relative transformation between camera C1 and virtual viewpoint V is Similarly, determine the relative transformation between camera C2 and virtual viewpoint V Relative transformation between camera C3 and virtual viewpoint V The relative transformation between camera C4 and virtual viewpoint V is

[0123] In step 830 , based on the intrinsic parameters of each of the multiple cameras, the intrinsic parameters of the virtual camera, the depth maps corresponding to the multiple cameras, and the virtual relative transformation relationships corresponding to the multiple cameras, the image coordinate system coordinate points corresponding to the images captured by the multiple cameras are projected onto the image plane of the virtual camera to determine the aligned images.

[0124] For example, the overlapping portion of the image captured by the four cameras is used to determine the overlapping portion of the image captured by the camera C1. Assume that any coordinate point p in the overlapping portion of the image captured by the camera C1 is i =(u i ,v i ,1) Projected to virtual camera C V The image plane is obtained according to the following formula (4): i In the virtual camera C V The coordinate point corresponding to the image plane

[0125]

[0126] In the above formula (4), z i is the coordinate point p i Similarly, all points in all cameras are projected onto the virtual camera C V The image plane is formed by the four bands, thereby aggregating the information of the four bands into the same virtual camera. The above calculation method can control the error of image alignment to the sub-pixel level, greatly improving the accuracy of image alignment.

[0127] Combined with the above Figures 1 to 8 , describes the method embodiment of the present application in detail, and the following is combined with Figures 9 to 16 , the device embodiment of the present application is described in detail. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so for parts not described in detail, reference can be made to the previous method embodiment.

[0128] Figure 9 The figure shows a schematic diagram of the structure of an image alignment device provided by an embodiment of the present application. Figure 9 As shown, the image alignment device 900 provided in the embodiment of the present application includes a determination module 910 and an alignment module 920 .

[0129] Specifically, the determination module 910 is configured to determine a depth map corresponding to the image acquisition device based on the geographic location information of the image acquisition device in the aircraft and the target scene image captured by the image acquisition device. The alignment module 920 is configured to determine an aligned image corresponding to the images captured by the multiple cameras based on the depth map corresponding to the image acquisition device, images captured by the multiple cameras in different bands in the aircraft, and the relative transformation relationship between the image acquisition device and the multiple cameras.

[0130] Figure 10 The figure shows a schematic diagram of the structure of the determination module provided in one embodiment of the present application. Figure 9 Based on the embodiment shown Figure 10 The embodiment shown is described below in detail. Figure 10 The embodiment shown and Figure 9 The differences and similarities between the illustrated embodiments are not described in detail.

[0131] like Figure 10 As shown, in the embodiment of the present application, the determination module 910 includes an image acquisition unit 911 , a first point cloud determination unit 912 and a depth map determination unit 913 .

[0132] Specifically, the image acquisition unit 911 is configured to determine at least two frames of target scene images acquired by the image acquisition device. The first point cloud determination unit 912 is configured to determine, based on the at least two frames of target scene images, first camera pose information corresponding to the image acquisition device and first three-dimensional point cloud data corresponding to the target scene in the camera coordinate system of the image acquisition device. The depth map determination unit 913 is configured to determine a depth map corresponding to the image acquisition device based on the geographic location information of the image acquisition device, the first camera pose information, and the first three-dimensional point cloud data.

[0133] Figure 11 The figure shows a schematic diagram of the structure of the first point cloud determination unit provided in one embodiment of the present application. Figure 10 Based on the embodiment shown Figure 11 The embodiment shown is described below in detail. Figure 11 The embodiment shown and Figure 10 The differences and similarities between the illustrated embodiments are not described in detail.

[0134] like Figure 11 As shown, in this embodiment of the present application, the first point cloud determination unit 912 includes a feature extraction subunit 9121, a motion estimation subunit 9122 and a first point cloud determination subunit 9123.

[0135] Specifically, the feature extraction subunit 9121 is configured to extract feature point sets corresponding to at least two frames of target scene images. The motion estimation subunit 9122 is configured to perform feature matching and motion estimation based on the feature point sets corresponding to at least two frames of target scene images, and determine motion estimation data. The first point cloud determination subunit 9123 is configured to determine first camera pose information and first three-dimensional point cloud data based on the motion estimation data.

[0136] Figure 12 The figure shows a schematic diagram of the structure of a depth map determination unit provided by an embodiment of the present application. Figure 10 Based on the embodiment shown Figure 12 The embodiment shown is described below in detail. Figure 12 The embodiment shown and Figure 10 The differences and similarities between the illustrated embodiments are not described in detail.

[0137] like Figure 12 As shown, in this embodiment of the present application, the depth map determination unit 913 includes a second point cloud determination subunit 9131 and a depth map determination subunit 9132 .

[0138] Specifically, the second point cloud determination subunit 9131 is configured to determine, based on the geographic location information of the image acquisition device, the first camera pose information, and the first three-dimensional point cloud data, second three-dimensional point cloud data corresponding to the target scene in the world coordinate system. The depth map determination subunit 9132 is configured to determine, based on the geographic location information of the image acquisition device and the second three-dimensional point cloud data, a depth map corresponding to the image acquisition device.

[0139] Figure 13 The figure shows a schematic diagram of the structure of an image alignment device provided by another embodiment of the present application. Figure 9 Based on the embodiment shown Figure 13 The embodiment shown is described below in detail. Figure 13 The embodiment shown and Figure 9 The differences and similarities between the illustrated embodiments are not described in detail.

[0140] like Figure 13 As shown, in the embodiment of the present application, the image alignment device 900 further includes a positioning module 930 and a geographic location determination module 940 .

[0141] Specifically, the positioning module 930 is configured to obtain positioning data obtained by a positioning device in the aircraft. The geographic location determination module 940 is configured to determine the geographic location information of the image acquisition device based on the positioning data and the relative transformation relationship between the image acquisition device and the positioning device.

[0142] Figure 14 The figure shows a schematic diagram of the structure of the alignment module provided in one embodiment of the present application. Figure 9 Based on the embodiment shown Figure 14 The embodiment shown is described below in detail. Figure 14 The embodiment shown and Figure 9 The differences and similarities between the illustrated embodiments are not described in detail.

[0143] like Figure 14 As shown, in the embodiment of the present application, the alignment module 920 includes a camera depth map determination unit 921 and an alignment unit 922 .

[0144] Specifically, the camera depth map determination unit 921 is configured to determine the depth maps corresponding to the multiple cameras based on the depth maps corresponding to the image acquisition device, the relative transformation relationship between the image acquisition device and the multiple cameras, and the internal parameters of the image acquisition device and the multiple cameras. The alignment unit 922 is configured to determine the aligned images corresponding to the images captured by the multiple cameras based on the depth maps corresponding to the multiple cameras and the images captured by the multiple cameras.

[0145] Figure 15The figure shows a schematic diagram of the structure of a camera depth map determination unit provided in one embodiment of the present application. Figure 14 Based on the embodiment shown Figure 15 The embodiment shown is described below in detail. Figure 15 The embodiment shown and Figure 14 The differences and similarities between the illustrated embodiments are not described in detail.

[0146] like Figure 15 As shown, in the embodiment of the present application, the camera depth map determining unit 921 includes a first coordinate data determining subunit 9211 , a second coordinate data determining subunit 9212 and a camera depth map determining subunit 9213 .

[0147] Specifically, the first coordinate data determination subunit 9211 is configured to determine the first coordinate data of the image acquisition device in the camera coordinate system of the image acquisition device based on the depth map of the image acquisition device and the intrinsic parameters of the image acquisition device. The second coordinate data determination subunit 9212 is configured to determine, for each of the multiple cameras, the second coordinate data of the camera in the camera coordinate system of the camera based on the first coordinate data and the position coordinate transformation data between the image acquisition device and the camera. The camera depth map determination subunit 9213 is configured to determine the depth map corresponding to each of the multiple cameras based on the second coordinate data of each of the multiple cameras and the intrinsic parameters of each of the multiple cameras.

[0148] Figure 16 The figure shows a schematic diagram of the structure of the alignment unit provided in one embodiment of the present application. Figure 14 Based on the embodiment shown Figure 16 The embodiment shown is described below in detail. Figure 16 The embodiment shown and Figure 14 The differences and similarities between the illustrated embodiments are not described in detail.

[0149] like Figure 16 As shown, in the embodiment of the present application, the alignment unit 922 includes a virtual viewpoint determination subunit 9221 , a virtual relative transformation relationship determination subunit 9222 and an alignment subunit 9223 .

[0150] Specifically, the virtual viewpoint determination subunit 9221 is configured to determine the virtual viewpoints of the virtual cameras corresponding to the multiple cameras based on the respective geographic location information of the multiple cameras. The virtual relative transformation relationship determination subunit 9222 is configured to determine the virtual relative transformation relationships corresponding to the multiple cameras based on the camera relative transformation relationships between the virtual viewpoint and the multiple cameras, wherein the virtual relative transformation relationship is the relative transformation relationship between the camera and the virtual viewpoint. The alignment subunit 9223 is configured to project the image coordinate system coordinate points corresponding to the images captured by the multiple cameras onto the image plane of the virtual camera based on the respective intrinsic parameters of the multiple cameras, the intrinsic parameters of the virtual camera, the depth maps corresponding to the multiple cameras, and the virtual relative transformation relationships corresponding to the multiple cameras, so as to determine the aligned images.

[0151] Below, reference Figure 17 To describe the electronic device according to the embodiment of the present application. Figure 17 Shown is a structural schematic diagram of an electronic device provided in one embodiment of the present application.

[0152] like Figure 17 As shown, the electronic device 170 includes: one or more processors 1701 and a memory 1702; and computer program instructions stored in the memory 1702, which, when executed by the processor 1701, enable the processor 1701 to execute the image alignment method as described in any of the above embodiments.

[0153] The processor 1701 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0154] Memory 1702 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and processor 1701 may execute the program instructions to implement the steps of the image alignment method of each embodiment of the present application described above and / or other desired functions.

[0155] In one example, the electronic device 170 may further include: an input device 1703 and an output device 1704, which are connected via a bus system and / or other forms of connection mechanisms ( Figure 17 not shown) interconnected.

[0156] In addition, the input device 1703 may also include, for example, a keyboard, a mouse, a microphone, etc.

[0157] The output device 1704 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output devices, etc.

[0158] Of course, to simplify, Figure 17 Only some of the components related to the present application in the electronic device 170 are shown, and components such as a bus, an input device / output interface, etc. are omitted. In addition, the electronic device 170 may further include any other appropriate components according to specific application conditions.

[0159] In addition to the above methods and devices, embodiments of the present application may also be computer program products, including computer program instructions, which, when executed by a processor, enable the processor to perform the steps in the image alignment method of any of the above embodiments.

[0160] The computer program product may be written in any combination of one or more programming languages to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0161] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the image alignment method according to various embodiments of the present application described in the above “Exemplary Method” section of this specification.

[0162] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0163] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0164] The block diagrams of the devices, devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0165] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0166] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0167] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

[0168] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image alignment method, characterized in that: include: Determining a depth map corresponding to an image acquisition device in an aircraft based on geographic location information of an image acquisition device and an image of a target scene acquired by the image acquisition device, wherein the target scene is a farmland scene having a ground relief change value greater than a preset relief threshold, and the flight altitude of the aircraft is lower than a preset altitude threshold; Determining, based on a depth map corresponding to the image acquisition device, images acquired by a plurality of cameras of different wavelength bands in the aircraft, and a relative transformation relationship between the image acquisition device and the plurality of cameras, aligned images corresponding to the images acquired by the plurality of cameras, wherein the plurality of cameras of different wavelength bands are plurality of cameras of different spectral bands in a multispectral camera; The determining of a depth map corresponding to an image acquisition device based on geographic location information of an image acquisition device in an aircraft and a target scene image acquired by the image acquisition device includes: Determining at least two frames of the target scene image captured by the image capture device; Based on the at least two frames of target scene images, determining first camera pose information corresponding to the image acquisition device and first three-dimensional point cloud data corresponding to the target scene in a camera coordinate system of the image acquisition device; Based on the geographic location information of the image acquisition device, the first camera pose information and the first three-dimensional point cloud data, a depth map corresponding to the image acquisition device is determined.

2. The image alignment method according to claim 1, wherein: The determining, based on the at least two frames of target scene images, first camera pose information corresponding to the image acquisition device and first three-dimensional point cloud data corresponding to the target scene in a camera coordinate system of the image acquisition device includes: respectively extracting feature point sets corresponding to the at least two frames of target scene images; Performing feature matching and motion estimation based on feature point sets corresponding to the at least two frames of target scene images to determine motion estimation data; Based on the motion estimation data, the first camera pose information and the first three-dimensional point cloud data are determined.

3. The image alignment method according to claim 1, wherein: The determining of a depth map corresponding to the image acquisition device based on the geographic location information of the image acquisition device, the first camera pose information, and the first three-dimensional point cloud data includes: Determining, based on the geographic location information of the image acquisition device, the first camera pose information, and the first three-dimensional point cloud data, second three-dimensional point cloud data corresponding to the target scene in a world coordinate system; Based on the geographic location information of the image acquisition device and the second three-dimensional point cloud data, a depth map corresponding to the image acquisition device is determined.

4. The image alignment method according to any one of claims 1 to 3, characterized in that: Before determining the depth map corresponding to the image acquisition device based on the geographical location information of the image acquisition device in the aircraft and the target scene image acquired by the image acquisition device, the method further includes: Acquiring positioning data based on a positioning device in the aircraft; Based on the positioning data, the relative transformation relationship between the image acquisition device and the positioning device, the geographical location information of the image acquisition device is determined.

5. The image alignment method according to any one of claims 1 to 3, characterized in that: The step of determining, based on a depth map corresponding to the image acquisition device, images acquired by multiple cameras of different wavelength bands in the aircraft, and position coordinate transformation data between the image acquisition device and the multiple cameras, an aligned image corresponding to the images acquired by the multiple cameras comprises: Determining a depth map corresponding to each of the multiple cameras based on a depth map corresponding to the image acquisition device, a relative transformation relationship between the image acquisition device and the multiple cameras, and internal parameters of the image acquisition device and the multiple cameras; Based on the depth maps corresponding to the multiple cameras and the images captured by the multiple cameras, aligned images corresponding to the images captured by the multiple cameras are determined.

6. The image alignment method according to claim 5, characterized in that: The determining, based on the depth map corresponding to the image acquisition device, the relative transformation relationship between the image acquisition device and the multiple cameras, and the internal parameters of the image acquisition device and the multiple cameras, the depth map corresponding to each of the multiple cameras includes: Determining first coordinate data of the image acquisition device in a camera coordinate system of the image acquisition device based on the depth map of the image acquisition device and an internal parameter of the image acquisition device; For each camera of the plurality of cameras, determining second coordinate data of the camera in a camera coordinate system of the camera based on the first coordinate data and position coordinate transformation data between the image acquisition device and the camera; Determine depth maps corresponding to the cameras based on the second coordinate data of the cameras and the intrinsic parameters of the cameras.

7. The image alignment method according to claim 5, characterized in that: The determining, based on the depth maps corresponding to the multiple cameras and the images captured by the multiple cameras, aligned images corresponding to the images captured by the multiple cameras, includes: Determining virtual viewpoints of virtual cameras corresponding to the multiple cameras based on the respective geographic location information of the multiple cameras; Determining, based on a camera relative transformation relationship between the virtual viewpoint and the multiple cameras, a virtual relative transformation relationship corresponding to each of the multiple cameras, wherein the virtual relative transformation relationship is a relative transformation relationship between the camera and the virtual viewpoint; Based on the intrinsic parameters of each of the multiple cameras, the intrinsic parameters of the virtual camera, the depth maps corresponding to each of the multiple cameras, and the virtual relative transformation relationships corresponding to each of the multiple cameras, the image coordinate system coordinate points corresponding to the images captured by each of the multiple cameras are projected onto the image plane of the virtual camera to determine the aligned image.

8. An image alignment device, characterized in that: include: a determination module configured to determine a depth map corresponding to an image acquisition device in an aircraft based on geographic location information of the image acquisition device and an image of a target scene acquired by the image acquisition device, wherein the target scene is a farmland scene having a ground relief change value greater than a preset relief threshold, and the flight altitude of the aircraft is lower than a preset altitude threshold; an alignment module configured to determine, based on a depth map corresponding to the image acquisition device, images acquired by a plurality of cameras of different wavelength bands in the aircraft, and a relative transformation relationship between the image acquisition device and the plurality of cameras, an alignment image corresponding to the images acquired by the plurality of cameras, wherein the plurality of cameras of different wavelength bands are multiple cameras of different spectral bands in a multispectral camera; The determining module is further configured to determine at least two frames of the target scene image captured by the image capture device; Based on the at least two frames of target scene images, determining first camera pose information corresponding to the image acquisition device and first three-dimensional point cloud data corresponding to the target scene in a camera coordinate system of the image acquisition device; Based on the geographic location information of the image acquisition device, the first camera pose information and the first three-dimensional point cloud data, a depth map corresponding to the image acquisition device is determined. 9 . A computer-readable storage medium storing instructions, wherein when the instructions are executed by a processor of an electronic device, the electronic device is enabled to execute the image alignment method according to claim 1 .

10. An electronic device, comprising: processor; memory for storing computer-executable instructions; The processor is configured to execute the computer-executable instructions to implement the image alignment method according to any one of claims 1 to 7.

Images