Image alignment method and device, electronic device, and computer-readable storage medium

Abstract

The present application relates to the field of image processing technology, and more specifically to an image alignment method and an image alignment device, as well as an electronic device and a computer-readable storage medium, to solve the problem of poor alignment accuracy of existing image alignment methods. The image alignment method provided in an embodiment of the present application determines the depth map corresponding to the target scene based on the depth measurement device in the aircraft, thereby providing real and accurate depth data for multiple cameras of different bands in the aircraft, so that the images collected by the multiple cameras can be aligned according to the real and accurate depth data. Compared with the method in the prior art that assumes that the depth data is a fixed and uniform value, the present application greatly improves the accuracy of image alignment.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image alignment method and an image alignment device, as well as an electronic device and a computer-readable storage medium. Background Art

[0002] Because a multispectral camera includes multiple cameras with different spectral bands, the spectral images captured by these cameras need to be aligned to obtain an aligned image. Current methods for aligning spectral images captured by cameras with different spectral bands ignore terrain undulations and approximate the ground as a flat surface. This assumes that the elevation of the multispectral cameras relative to the ground is fixed and uniform, thus achieving alignment of the spectral images captured by multiple cameras.

[0003] However, in agricultural scenarios, when aircraft conduct crop growth and pest analysis, they generally fly at a low altitude. At this time, if the terrain undulations are ignored and the ground is approximated as a plane, the assumed aircraft flight altitude will have a large error, resulting in a large error in the obtained aligned images. Summary of the Invention

[0004] In view of this, embodiments of the present application provide an image alignment method and apparatus, an electronic device, and a computer-readable storage medium to solve the problem of poor alignment accuracy of existing image alignment methods.

[0005] In a first aspect, an image alignment method provided by an embodiment of the present application includes: determining a depth map corresponding to a target scene based on a depth measurement device in an aircraft, wherein the depth measurement device includes at least two depth measurement devices with different measurement principles; based on the depth map corresponding to the target scene, images captured by multiple cameras of different bands in the aircraft, and the relative transformation relationship between the depth measurement device and the multiple cameras, determining an aligned image of the images captured by the multiple cameras.

[0006] In combination with the first aspect, in certain implementations of the first aspect, the depth measuring device includes a first depth measuring device and a second depth measuring device, and determining a depth map corresponding to the target scene based on the depth measuring device in the aircraft includes: determining a first depth value based on the first depth measuring device; and determining the depth map corresponding to the target scene based on the first depth value and a preset depth threshold.

[0007] In combination with the first aspect, in certain implementations of the first aspect, a depth map corresponding to the target scene is determined based on a first depth value and a preset depth threshold, including: if the first depth value is greater than or equal to the preset depth threshold, determining the depth map corresponding to the target scene based on a first depth measurement device; if the first depth value is less than the preset depth threshold, determining the depth map corresponding to the target scene based on a second depth measurement device.

[0008] In combination with the first aspect, in certain implementations of the first aspect, based on the depth map corresponding to the target scene, images captured by multiple cameras of different bands in the aircraft, and the relative transformation relationship between the depth measurement device and the multiple cameras, an aligned image of the images captured by the multiple cameras is determined, including: determining the depth maps corresponding to the multiple cameras based on the depth map corresponding to the target scene, the relative transformation relationship between the depth measurement device and the multiple cameras, and the internal parameters of the multiple cameras; and determining the aligned image of the images captured by the multiple cameras based on the depth maps corresponding to the multiple cameras and the images captured by the multiple cameras.

[0009] In combination with the first aspect, in certain implementations of the first aspect, determining the depth maps corresponding to each of the multiple cameras based on the depth map corresponding to the target scene, the relative transformation relationship between the depth measurement device and the multiple cameras, and the internal parameters of each of the multiple cameras includes: determining the coordinate data of each of the multiple cameras in the camera coordinate system based on the depth map corresponding to the target scene, the relative transformation relationship between the depth measurement device and the multiple cameras; and determining the depth maps corresponding to each of the multiple cameras based on the coordinate data of each of the multiple cameras in the camera coordinate system and the internal parameters of each of the multiple cameras.

[0010] In combination with the first aspect, in certain implementations of the first aspect, based on the depth maps corresponding to each of the multiple cameras and the images captured by each of the multiple cameras, an aligned image of the images captured by each of the multiple cameras is determined, including: based on the geographic location information of each of the multiple cameras, a virtual viewpoint of the virtual camera corresponding to the multiple cameras is determined; based on the camera relative transformation relationship between the virtual viewpoint and the multiple cameras, a virtual relative transformation relationship corresponding to each of the multiple cameras is determined, wherein the virtual relative transformation relationship is a relative transformation relationship between the camera and the virtual viewpoint; based on the intrinsic parameters of each of the multiple cameras, the intrinsic parameters of the virtual camera, the depth maps corresponding to each of the multiple cameras, and the virtual relative transformation relationship corresponding to the multiple cameras, the image coordinate system coordinate points corresponding to the images captured by each of the multiple cameras are projected onto the image plane of the virtual camera to determine the aligned image.

[0011] In combination with the first aspect, in some implementations of the first aspect, the first depth measuring device includes a laser radar, and the second depth measuring device includes a binocular camera.

[0012] In a second aspect, an image alignment device provided by an embodiment of the present application includes: a determination module, configured to determine a depth map corresponding to a target scene based on a depth measurement device in an aircraft, wherein the depth measurement device includes at least two depth measurement devices with different measurement principles; an alignment module, configured to determine an aligned image of images captured by multiple cameras based on the depth map corresponding to the target scene, images captured by multiple cameras of different bands in the aircraft, and a relative transformation relationship between the depth measurement device and the multiple cameras.

[0013] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed by a processor of an electronic device, the electronic device can execute the image alignment method mentioned in the first aspect above.

[0014] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes: a processor; a memory for storing computer-executable instructions; and a processor for executing computer-executable instructions to implement the image alignment method mentioned in the first aspect above.

[0015] An image alignment method provided in an embodiment of the present application determines the depth map corresponding to the target scene based on a depth measurement device in an aircraft, thereby providing real depth data for multiple cameras in different bands in the aircraft. This allows the images captured by each of the multiple cameras to be aligned based on the real depth data. Compared with the prior art method that assumes that the depth data is a fixed and uniform value, the present application greatly improves the accuracy of image alignment. In addition, because the depth measurement device includes at least two depth measurement devices with different measurement principles, it is possible to select the appropriate depth measurement device based on the advantages of the different measurement principles, thereby obtaining more accurate depth data, providing a more accurate data foundation for subsequent image alignment, and further improving the accuracy of image alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 FIG2 is a flow chart of an image alignment method provided in an embodiment of the present application.

[0017] Figure 2 Shown is a flow chart of an image alignment method provided in another embodiment of the present application.

[0018] Figure 3 Shown is a flow chart of an image alignment method provided in another embodiment of the present application.

[0019] Figure 4 Shown is a flow chart of an image alignment method provided in another embodiment of the present application.

[0020] Figure 5Shown is a flow chart of an image alignment method provided in another embodiment of the present application.

[0021] Figure 5a Shown is a schematic diagram of a relative transformation relationship provided by an embodiment of the present application.

[0022] Figure 6 Shown is a flow chart of an image alignment method provided in another embodiment of the present application.

[0023] Figure 7 Shown is a structural schematic diagram of an image alignment device provided in one embodiment of the present application.

[0024] Figure 8 Shown is a structural diagram of a determination module provided in one embodiment of the present application.

[0025] Figure 9 FIG2 is a schematic structural diagram of a depth map determination unit of a measurement device provided in an embodiment of the present application.

[0026] Figure 10 Shown is a structural diagram of an alignment module provided in one embodiment of the present application.

[0027] Figure 11 FIG2 is a schematic structural diagram of a camera depth map determination unit provided in one embodiment of the present application.

[0028] Figure 12 FIG. 1 is a schematic structural diagram of an alignment unit provided in an embodiment of the present application.

[0029] Figure 13 Shown is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] The scenarios to which the embodiments of the present application are applicable may include an aircraft, a depth measuring device, and multiple cameras of different wavelengths. The depth measuring device and the multiple cameras of different wavelengths are fixedly mounted on the aircraft. A computing module is provided in the aircraft. A communication connection relationship exists between the computing module, the depth measuring device, and the multiple cameras of different wavelengths. The multiple cameras of different wavelengths can capture images of the target scene. The computing module in the aircraft can determine a depth map corresponding to the target scene based on the depth measuring device in the aircraft, wherein the depth measuring device includes at least two depth measuring devices with different measurement principles, and then determine an aligned image of the images captured by the multiple cameras based on the depth map corresponding to the target scene, the images captured by the multiple cameras of different wavelengths in the aircraft, and the relative transformation relationship between the depth measuring device and the multiple cameras.

[0032] Another scenario to which the embodiments of the present application are applicable may include a server, a depth measurement device, and multiple cameras of different bands. A communication connection relationship exists between the server, the depth measurement device, and the multiple cameras of different bands. The multiple cameras of different bands can capture images of the target scene. The server can determine a depth map corresponding to the target scene based on the depth measurement device in the aircraft, wherein the depth measurement device includes at least two depth measurement devices with different measurement principles, and then determine an aligned image of the images captured by the multiple cameras based on the depth map corresponding to the target scene, the images captured by the multiple cameras of different bands in the aircraft, and the relative transformation relationship between the depth measurement device and the multiple cameras.

[0033] Figure 1 FIG. 1 is a flow chart of an image alignment method provided by an embodiment of the present application. Figure 1 As shown, the image alignment method provided in the embodiment of the present application includes the following steps.

[0034] Step 110 : Determine a depth map corresponding to the target scene based on a depth measurement device in the aircraft.

[0035] Specifically, the depth measuring device includes at least two depth measuring devices with different measurement principles. For example, the depth measuring device may include two depth measuring devices with different measurement principles. One depth measuring device may be a device that uses the time difference method (TOF) to measure depth values, such as a lidar. Another depth measuring device may be a measuring device that implements depth recovery based on the triangulation principle, such as a binocular camera. As long as the depth measuring device can obtain a depth map for the target scene, this application does not make any specific restrictions. The depth map corresponding to the target scene refers to an image that uses the distance from the depth measuring device to each point in the target scene as a pixel value. The target scene may be a farmland scene. For example, the target scene may be a scene with large ground undulations such as mountains and terraces.

[0036] For example, the aircraft may be a drone or other flying device, and this application does not impose any specific restrictions. The depth measurement device may include two depth measurement devices or three depth measurement devices, and this application does not impose any specific restrictions on the number of depth measurement devices included in the depth measurement device. Among the multiple depth measurement devices included in the depth measurement device, as long as one depth measurement device has a different measurement principle from the other depth measurement devices, it is sufficient.

[0037] Step 120 : determining an aligned image of the images captured by the multiple cameras based on the depth map corresponding to the target scene, images captured by the multiple cameras of different bands in the aircraft, and a relative transformation relationship between the depth measurement device and the multiple cameras.

[0038] For example, the multiple cameras of different wavelength bands may be multiple cameras of different wavelength bands in a multispectral camera. The images captured by each of the multiple cameras of different wavelength bands may be spectral images. The multispectral camera may be composed of four cameras of different wavelength bands, or six cameras of different wavelength bands. This application does not specifically limit the number of cameras of different wavelength bands that a multispectral camera may comprise.

[0039] For example, multispectral cameras can provide multi-band spectral data for agricultural remote sensing. A multispectral camera consists of multiple independent imagers, each equipped with a specialized filter that allows each imager to capture a spectrum of light in a different wavelength range. Using a multispectral camera, images of farmland can be acquired in different spectral bands, such as red, green, blue, infrared, and near-infrared. The spectral images captured by these cameras in different spectral bands are then aligned to produce an aligned image.

[0040] For example, the depth measurement device and the multiple cameras are fixedly mounted on the aircraft. The relative transformation relationship between the depth measurement device and the multiple cameras can be obtained by calibrating the depth measurement device and the multiple cameras. The relative transformation relationship between the multiple cameras of different wavelength bands included in the multispectral camera is known.

[0041] Specifically, based on a depth map corresponding to the target scene, images captured by multiple cameras in different wavelength bands in the aircraft, and the relative transformation relationship between the depth measurement device and the multiple cameras, an alignment image of the images captured by the multiple cameras is determined. This can be achieved by utilizing the relative transformation relationship between the depth measurement device and the multiple cameras to determine the depth maps corresponding to the multiple cameras using the depth map corresponding to the target scene. A virtual projection method is then used to construct a virtual camera. Using the relative transformation relationship between the multiple cameras, the images captured by the multiple cameras are fused with their corresponding depth maps and then projected onto the image plane of the virtual camera, thereby achieving alignment of the images captured by the multiple cameras.

[0042] An image alignment method provided in an embodiment of the present application determines the depth map corresponding to the target scene based on a depth measurement device in an aircraft, thereby providing real depth data for multiple cameras in different bands in the aircraft. This allows the images captured by each of the multiple cameras to be aligned based on the real depth data. Compared with the prior art method that assumes that the depth data is a fixed and uniform value, the present application greatly improves the accuracy of image alignment. In addition, because the depth measurement device includes at least two depth measurement devices with different measurement principles, it is possible to select the appropriate depth measurement device based on the advantages of the different measurement principles, thereby obtaining more accurate depth data, providing a more accurate data foundation for subsequent image alignment, and further improving the accuracy of image alignment.

[0043] Figure 2 FIG. 1 is a flow chart of an image alignment method provided by another embodiment of the present application. Figure 1 Based on the embodiment shown Figure 2 The embodiment shown is described below in detail. Figure 2 The embodiment shown is Figure 1 The differences and similarities between the illustrated embodiments are not described in detail.

[0044] like Figure 2 As shown, in an embodiment of the present application, the step of determining a depth map corresponding to a target scene based on a depth measurement device in an aircraft includes the following steps.

[0045] Step 210: Determine a first depth value based on a first depth measurement device.

[0046] Specifically, the depth measurement device includes a first depth measurement device and a second depth measurement device. The depth map corresponding to the target scene can be determined only by the first depth measurement device, only by the second depth measurement device, or jointly by the first depth measurement device and the second depth measurement device. The depth map corresponding to the target scene is jointly determined by the first depth measurement device and the second depth measurement device, and a portion of the depth map corresponding to the target scene can be determined by the first depth measurement device, while the other portion can be determined by the second depth measurement device. The first depth measurement device can be a device that actively measures the depth of the target scene, thereby directly outputting the measured depth value.

[0047] For example, the first depth value may be a depth value directly output by the first depth measuring device, or it may be an average value of multiple depth values output by the first depth measuring device within a time period. Those skilled in the art may determine the calculation method of the first depth value according to actual needs, and this application does not make any specific limitations.

[0048] Step 220: Determine a depth map corresponding to the target scene based on the first depth value and a preset depth threshold.

[0049] For example, the first and second depth measurement devices may each have different advantages. For example, the second depth measurement device may be more accurate than the first depth measurement device for close-range measurements, but more accurate than the second depth measurement device for long-range measurements. For another example, the second depth measurement device may capture more data than the first depth measurement device.

[0050] For example, the preset depth threshold can be determined based on the ranging advantages of the first depth measuring device and the second depth measuring device. For example, if the first depth measuring device has better accuracy than the second depth measuring device when measuring a distance greater than or equal to 50 meters, and the second depth measuring device has better accuracy than the first depth measuring device when measuring a distance less than 50 meters, then the preset depth threshold can be set to 50 meters. Those skilled in the art can set the preset depth threshold based on actual needs, and this application does not impose any specific limitations.

[0051] By comparing the first depth value and the preset depth threshold to determine the depth map corresponding to the target scene, the ranging advantages of the first depth measurement device and the second depth measurement device can be fully considered, so as to select a more accurate depth measurement device to obtain a more accurate depth map corresponding to the target scene.

[0052] Figure 3 FIG. 1 is a flow chart of an image alignment method provided by another embodiment of the present application. Figure 2 Based on the embodiment shown Figure 3 The embodiment shown is described below in detail. Figure 3 The embodiment shown is Figure 2 The differences and similarities between the illustrated embodiments are not described in detail.

[0053] like Figure 3 As shown, in an embodiment of the present application, the step of determining a depth map corresponding to a target scene based on a first depth value and a preset depth threshold includes the following steps.

[0054] Step 310: Determine whether the first depth value is greater than or equal to a preset depth threshold.

[0055] For example, in actual application, if step 310 is judged to be yes, that is, the first depth value is greater than or equal to the preset depth threshold, step 320 is executed. If step 310 is judged to be no, that is, the first depth value is less than the preset depth threshold, step 330 is executed.

[0056] Step 320: Determine a depth map corresponding to the target scene based on the first depth measurement device.

[0057] Step 330: Determine a depth map corresponding to the target scene based on the second depth measurement device.

[0058] Illustratively, the first depth measurement device can be a device that uses TOF to measure depth, such as a lidar. A lidar is a radar system that uses a laser beam to detect characteristic quantities such as the position and velocity of a target. The lidar operates by transmitting a detection signal (laser beam) toward the target and then comparing the received signal reflected from the target (target echo) with the transmitted detection signal to obtain relevant target information, such as target distance, direction, altitude, speed, attitude, shape, and other parameters.

[0059] For example, the second depth measurement device can be a device that uses triangulation to achieve depth recovery, such as a binocular camera. Alternatively, the second depth measurement device can be a multi-camera. When measuring at close range, depth recovery based on triangulation has low uncertainty, high depth value reliability, and a large amount of data information. Therefore, a device such as a binocular camera or a multi-camera that uses triangulation to achieve depth recovery can obtain accurate and reliable depth values and rich data information when measuring at close range.

[0060] Specifically, the depth map corresponding to the target scene can be determined only based on the first depth measurement device, can be determined only based on the second depth measurement device, or part of the area in the depth map corresponding to the depth measurement device can be determined based on the first depth measurement device, and another part of the area can be determined based on the first depth measurement device. This application does not make specific limitations.

[0061] By comparing the first depth value with the preset depth threshold, the depth map corresponding to the target scene is determined, thereby automatically selecting a more accurate depth measurement device for the depth map corresponding to the target scene, so as to obtain a more accurate depth map corresponding to the target scene, providing more accurate depth data for subsequent image alignment.

[0062] Figure 4 FIG. 1 is a flow chart of an image alignment method provided by another embodiment of the present application. Figure 1 Based on the embodiment shown Figure 4 The embodiment shown is described below in detail. Figure 4 The embodiment shown is Figure 1 The differences and similarities between the illustrated embodiments are not described in detail.

[0063] like Figure 4 As shown, in an embodiment of the present application, based on the depth map corresponding to the target scene, the images captured by multiple cameras of different bands in the aircraft, and the relative transformation relationship between the depth measurement device and the multiple cameras, the image alignment step of the images captured by the multiple cameras is determined, including the following steps.

[0064] Step 410 : Determine depth maps corresponding to the multiple cameras based on the depth map corresponding to the target scene, the relative transformation relationship between the depth measurement device and the multiple cameras, and the internal parameters of the multiple cameras.

[0065] Specifically, by utilizing the relative transformation relationship between the depth measurement device and multiple cameras and the internal parameters of each of the multiple cameras, any spatial point in the depth map corresponding to the target scene can be transformed into the image plane corresponding to each of the multiple cameras, thereby constructing the depth maps corresponding to each of the multiple cameras.

[0066] Step 420 : determining an aligned image of the images captured by the multiple cameras based on the depth maps corresponding to the multiple cameras and the images captured by the multiple cameras.

[0067] Specifically, a virtual projection method can be used to construct a virtual camera, and the images captured by multiple cameras are fused with their corresponding depth maps and then projected onto the image plane of the virtual camera, thereby achieving alignment of the images captured by multiple cameras.

[0068] Based on the depth map corresponding to the target scene, a series of coordinate transformations are performed to obtain the depth maps corresponding to multiple cameras, thereby obtaining the real depth data of multiple cameras in the world coordinate system. Then, the real depth data of multiple cameras in the world coordinate system is used to align the images collected by multiple cameras, obtaining aligned images and improving the accuracy of image alignment.

[0069] Figure 5 FIG. 1 is a flow chart of an image alignment method provided by another embodiment of the present application. Figure 4 Based on the embodiment shown Figure 5 The embodiment shown is described below in detail. Figure 5 The embodiment shown is Figure 4 The differences and similarities between the illustrated embodiments are not described in detail.

[0070] like Figure 5 As shown, in an embodiment of the present application, the step of determining the depth maps corresponding to the multiple cameras based on the depth map corresponding to the target scene, the relative transformation relationship between the depth measurement device and the multiple cameras, and the internal parameters of the multiple cameras includes the following steps.

[0071] Step 510 : determining coordinate data of each of the multiple cameras in a camera coordinate system based on a depth map corresponding to the target scene, a relative transformation relationship between the depth measurement device and the multiple cameras.

[0072] For example, Figure 5a FIG. 1 is a schematic diagram of a relative transformation relationship provided by an embodiment of the present application. Figure 5aAs shown in the figure, the multispectral camera consists of four cameras with different wavelengths. The four cameras with different wavelengths are represented by C1, C2, C3, and C4 respectively, the first depth measuring device is represented by L, and the second depth measuring device is represented by S. The first depth measuring device L, the second depth measuring device S, and the four cameras with different wavelengths are all fixedly installed on the aircraft. Therefore, the positional relationship between the first depth measuring device L, the second depth measuring device S, and the four cameras with different wavelengths is fixed. Therefore, by calibrating the first depth measuring device L and camera C1, the relative transformation relationship between the first depth measuring device L and camera C1 can be obtained. By calibrating the second depth measuring device S and the camera C1, the relative transformation relationship between the second depth measuring device S and the camera C1 can be obtained. The relative transformation relationship between the multiple cameras of different wavelength bands included in the multispectral camera is known. Therefore, the relative transformation relationship between the first depth measurement device L, the second depth measurement device S and the cameras C2, C3, and C4 can be obtained. In addition, there is also a relative transformation relationship T between the second depth measurement device S and the first depth measurement device L. SL .

[0073] The depth map corresponding to the target scene is represented by I D For the depth map I D Any point P in i =(x i ,y i ,z i ), based on the following formula (1) or (2), the depth map I D Any point P in i Transform to the coordinate system of camera C1 to obtain the spatial point in the coordinate system of camera C1

[0074]

[0075]

[0076] Specifically, if the depth map I D Point P in i is obtained by the first depth measuring device L, then the spatial point in the coordinate system of the camera C1 is obtained based on formula (1): If the depth map I D Point P in i is obtained by the second depth measuring device S, then the spatial point in the coordinate system of the camera C1 is obtained based on formula (2):

[0077] Step 520 : Determine depth maps corresponding to the multiple cameras based on the coordinate data of the multiple cameras in their respective camera coordinate systems and the internal parameters of the multiple cameras.

[0078] Assume that the intrinsic parameter matrix of camera C1 is Based on the following formula (3), the spatial point in the coordinate system of camera C1 is Projected onto the image plane of camera C1 In the , we can get the coordinates of the points

[0079]

[0080] Among them, the intrinsic parameter matrix of camera C1 is It can be the following matrix.

[0081]

[0082] By projecting all pixels in the depth map corresponding to the image acquisition device onto the image plane of the camera C1 In the example, the depth map corresponding to camera C1 can be constructed

[0083] Similarly, according to the transfer effect of relative pose transformation, the depth maps corresponding to cameras C2, C3 and C4 in the multispectral camera M can be constructed.

[0084] According to the transmission effect of relative pose transformation, the depth maps of multiple cameras are constructed using the same method, which improves the efficiency of constructing the depth maps of multiple cameras.

[0085] Figure 6 FIG. 1 is a flow chart of an image alignment method provided by another embodiment of the present application. Figure 4 Based on the embodiment shown Figure 6 The embodiment shown is described below in detail. Figure 6 The embodiment shown is Figure 4 The differences and similarities between the illustrated embodiments are not described in detail.

[0086] like Figure 6 As shown, in an embodiment of the present application, based on the depth maps corresponding to the multiple cameras and the images captured by the multiple cameras, the step of aligning the images captured by the multiple cameras includes the following steps.

[0087] Step 610 : Determine virtual viewpoints of virtual cameras corresponding to the multiple cameras based on the respective geographic location information of the multiple cameras.

[0088] For example, multiple cameras correspond to one virtual camera, and the virtual camera has one virtual viewpoint. The virtual viewpoint can be located at the center of the multiple cameras. Figure 5a As shown, the virtual viewpoint is represented by V. The virtual viewpoint of the virtual camera corresponding to the multispectral camera can be located at the center of the multispectral camera, or at other reference points corresponding to multiple cameras, which is not specifically limited in this application.

[0089] Step 620 : determining virtual relative transformation relationships corresponding to the multiple cameras based on the camera relative transformation relationships between the virtual viewpoint and the multiple cameras.

[0090] Specifically, the virtual relative transformation relationship is the relative transformation relationship between the camera and the virtual viewpoint. The relative transformation relationship between multiple cameras is known. For example, four cameras are used. The relative transformation relationship between four cameras of different wavelengths, C1, C2, C3, and C4, is as follows.

[0091] The relative transformation relationship from camera C1 to camera C2 is:

[0092] The relative transformation relationship from camera C1 to camera C3 is:

[0093] The relative transformation relationship from camera C1 to camera C4 is:

[0094] Due to the relative transformation relationship They are all homogeneous matrices, so the relative transformation relationship can be transferred by multiplication, so the following relative transformation relationship can be obtained.

[0095] The relative transformation relationship from camera C2 to camera C3 is:

[0096] The relative transformation relationship from camera C2 to camera C4 is:

[0097] The relative transformation relationship from camera C3 to camera C4 is:

[0098] Assume that the virtual viewpoint V is located at the center of the multispectral camera, and set the virtual camera internal parameter to K V The relative transformation between camera C1 and virtual viewpoint V is Similarly, determine the relative transformation between camera C2 and virtual viewpoint V Relative transformation between camera C3 and virtual viewpoint V The relative transformation between camera C4 and virtual viewpoint V is

[0099] In step 630 , based on the intrinsic parameters of each of the multiple cameras, the intrinsic parameters of the virtual camera, the depth maps corresponding to each of the multiple cameras, and the virtual relative transformation relationships corresponding to each of the multiple cameras, the image coordinate system coordinate points corresponding to the images captured by each of the multiple cameras are projected onto the image plane of the virtual camera to determine the aligned images.

[0100] For example, the overlapping portion of the image captured by the four cameras is used to determine the overlapping portion of the image captured by the camera C1. Assume that any coordinate point p in the overlapping portion of the image captured by the camera C1 is i =(u i ,v i ,1) Projected to virtual camera C V The image plane is obtained according to the following formula (4): i In the virtual camera C V The coordinate point corresponding to the image plane

[0101]

[0102] In the above formula (4), z i is the coordinate point p i The depth value can be obtained based on the depth map corresponding to each of the multiple cameras. Similarly, all points in all cameras are projected onto the virtual camera C V The image plane is formed by the four bands, thereby aggregating the information of the four bands into the same virtual camera. The above calculation method can control the error of image alignment to the sub-pixel level, greatly improving the accuracy of image alignment.

[0103] Combined with the above Figures 1 to 6 , describes the method embodiment of the present application in detail, and the following is combined with Figures 7 to 12 , the device embodiment of the present application is described in detail. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so for parts not described in detail, reference can be made to the previous method embodiment.

[0104] Figure 7 The figure shows a schematic diagram of the structure of an image alignment device provided by an embodiment of the present application. Figure 7 As shown, the image alignment device 700 provided in the embodiment of the present application includes a determination module 710 and an alignment module 720 .

[0105] Determination module 710 is configured to determine a depth map corresponding to a target scene based on a depth measurement device in the aircraft, wherein the depth measurement device includes at least two depth measurement devices with different measurement principles. Alignment module 720 is configured to determine an aligned image of the images captured by the multiple cameras based on the depth map corresponding to the target scene, images captured by the multiple cameras in different wavelength bands in the aircraft, and the relative transformation relationship between the depth measurement device and the multiple cameras.

[0106] Figure 8 The figure shows a schematic diagram of the structure of the determination module provided in one embodiment of the present application. Figure 7 Based on the embodiment shown Figure 8 The embodiment shown is described below in detail. Figure 8 The embodiment shown is Figure 7 The differences and similarities between the illustrated embodiments are not described in detail.

[0107] like Figure 8 As shown, in the embodiment of the present application, the determination module 710 includes a depth value determination unit 711 and a depth map determination unit 712 .

[0108] Specifically, the depth value determining unit 711 is configured to determine a first depth value based on the first depth measuring device. The depth map determining unit 712 is configured to determine a depth map corresponding to the target scene based on the first depth value and a preset depth threshold.

[0109] Figure 9 The figure shows a schematic diagram of the structure of a depth map determination unit of a measurement device provided by an embodiment of the present application. Figure 8 Based on the embodiment shown Figure 9 The embodiment shown is described below in detail. Figure 9 The embodiment shown is Figure 8 The differences and similarities between the illustrated embodiments are not described in detail.

[0110] like Figure 9 As shown, in the embodiment of the present application, the measuring device depth map determining unit 712 includes a first measuring device depth map determining subunit 7121 and a second measuring device depth map determining subunit 7122 .

[0111] Specifically, the first measurement device depth map determination subunit 7121 is configured to, if the first depth value is greater than or equal to a preset depth threshold, determine a depth map corresponding to the target scene based on the first depth measurement device. The second measurement device depth map determination subunit 7122 is configured to, if the first depth value is less than the preset depth threshold, determine a depth map corresponding to the target scene based on the second depth measurement device.

[0112] Figure 10The figure shows a schematic diagram of the structure of the alignment module provided in one embodiment of the present application. Figure 7 Based on the embodiment shown Figure 10 The embodiment shown is described below in detail. Figure 10 The embodiment shown is Figure 7 The differences and similarities between the illustrated embodiments are not described in detail.

[0113] like Figure 10 As shown, in the embodiment of the present application, the alignment module 720 includes: a camera depth map determination unit 721 and an alignment unit 722.

[0114] Specifically, the camera depth map determination unit 721 is configured to determine the depth maps corresponding to the multiple cameras based on the depth map corresponding to the target scene, the relative transformation relationship between the depth measurement device and the multiple cameras, and the intrinsic parameters of the multiple cameras. The alignment unit 722 is configured to determine an aligned image of the images captured by the multiple cameras based on the depth maps corresponding to the multiple cameras and the images captured by the multiple cameras.

[0115] Figure 11 The figure shows a schematic diagram of the structure of a camera depth map determination unit provided in one embodiment of the present application. Figure 10 Based on the embodiment shown Figure 11 The embodiment shown is described below in detail. Figure 11 The embodiment shown is Figure 10 The differences and similarities between the illustrated embodiments are not described in detail.

[0116] like Figure 11 As shown, in the embodiment of the present application, the camera depth map determining unit 721 includes a coordinate determining subunit 7211 and a camera depth map determining subunit 7212 .

[0117] Specifically, the coordinate determination subunit 7211 is configured to determine the coordinate data of each of the multiple cameras in the camera coordinate system based on the depth map corresponding to the target scene, the relative transformation relationship between the depth measurement device and the multiple cameras. The camera depth map determination subunit 7212 is configured to determine the depth map corresponding to each of the multiple cameras based on the coordinate data of each of the multiple cameras in the camera coordinate system and the intrinsic parameters of each of the multiple cameras.

[0118] Figure 12 The figure shows a schematic diagram of the structure of the alignment unit provided in one embodiment of the present application. Figure 10 Based on the embodiment shown Figure 12 The embodiment shown is described below in detail. Figure 12 The embodiment shown is Figure 10 The differences and similarities between the illustrated embodiments are not described in detail.

[0119] like Figure 12 As shown, in the embodiment of the present application, the alignment unit 722 includes a virtual viewpoint determination subunit 7221 , a virtual relative transformation relationship determination subunit 7222 and an alignment subunit 7223 .

[0120] Specifically, the virtual viewpoint determination subunit 7221 is configured to determine the virtual viewpoints of the virtual cameras corresponding to the multiple cameras based on the respective geographic location information of the multiple cameras. The virtual relative transformation relationship determination subunit 7222 is configured to determine the virtual relative transformation relationship corresponding to each of the multiple cameras based on the camera relative transformation relationship between the virtual viewpoint and the multiple cameras, wherein the virtual relative transformation relationship is the relative transformation relationship between the camera and the virtual viewpoint. The alignment subunit 7223 is configured to project the image coordinate system coordinate points corresponding to the images captured by each of the multiple cameras onto the image plane of the virtual camera based on the respective intrinsic parameters of the multiple cameras, the intrinsic parameters of the virtual camera, the depth maps corresponding to the multiple cameras, and the virtual relative transformation relationship corresponding to the multiple cameras, so as to determine the aligned images.

[0121] Below, reference Figure 13 To describe the electronic device according to the embodiment of the present application. Figure 13 Shown is a schematic structural diagram of an electronic device provided in one embodiment of the present application.

[0122] like Figure 13 As shown, the electronic device 130 includes: one or more processors 1301 and a memory 1302; and computer program instructions stored in the memory 1302, which, when executed by the processor 1301, enable the processor 1301 to perform the image alignment method as described in any of the above embodiments.

[0123] The processor 1301 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0124] The memory 1302 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1301 may execute the program instructions to implement the steps of the image alignment method of each embodiment of the present application described above and / or other desired functions.

[0125] In one example, the electronic device 130 may further include: an input device 1303 and an output device 1304, which are connected via a bus system and / or other forms of connection mechanisms ( Figure 13 not shown) interconnected.

[0126] In addition, the input device 1303 may also include, for example, a keyboard, a mouse, a microphone, etc.

[0127] The output device 1304 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output devices, etc.

[0128] Of course, to simplify, Figure 13 Only some of the components related to the present application in the electronic device 130 are shown, and components such as a bus, an input device / output interface, etc. are omitted. In addition, the electronic device 130 may further include any other appropriate components according to specific application conditions.

[0129] In addition to the above methods and devices, embodiments of the present application may also be computer program products, including computer program instructions, which, when executed by a processor, enable the processor to perform the steps in the image alignment method of any of the above embodiments.

[0130] The computer program product may be written in any combination of one or more programming languages to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0131] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the image alignment method according to various embodiments of the present application described in the above “Exemplary Method” section of this specification.

[0132] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0133] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0134] The block diagrams of the devices, devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0135] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0136] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0137] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

[0138] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image alignment method, characterized in that: include: Determining a depth map corresponding to the target scene based on a depth measurement device in the aircraft, wherein the depth measurement device includes at least two depth measurement devices with different measurement principles; Determining depth maps corresponding to the multiple cameras based on the depth map corresponding to the target scene, the relative transformation relationship between the depth measurement device and multiple cameras of different wavelength bands in the aircraft, and the intrinsic parameters of each of the multiple cameras; determining virtual viewpoints of virtual cameras corresponding to the multiple cameras based on the geographic location information of each of the multiple cameras; determining virtual relative transformation relationships corresponding to the multiple cameras based on the camera relative transformation relationship between the virtual viewpoint and the multiple cameras; and projecting image coordinate system coordinate points corresponding to images captured by each of the multiple cameras onto an image plane of the virtual camera based on the intrinsic parameters of each of the multiple cameras, the intrinsic parameters of the virtual camera, the depth maps corresponding to the multiple cameras, and the virtual relative transformation relationships corresponding to the multiple cameras to determine aligned images. The virtual relative transformation relationship is a relative transformation relationship between the cameras and the virtual viewpoint, the multiple cameras of different wavelength bands are multiple cameras of different spectral bands in a multispectral camera, the relative transformation relationships between the multiple cameras of different spectral bands included in the multispectral camera are known, and the images captured by the multiple cameras of different wavelength bands are spectral images.

2. The image alignment method according to claim 1, wherein: The depth measurement device includes a first depth measurement device and a second depth measurement device, and determining a depth map corresponding to a target scene based on the depth measurement device in the aircraft includes: determining a first depth value based on the first depth measurement device; A depth map corresponding to the target scene is determined based on the first depth value and a preset depth threshold.

3. The image alignment method according to claim 2, characterized in that: The determining, based on the first depth value and a preset depth threshold, a depth map corresponding to the target scene includes: If the first depth value is greater than or equal to the preset depth threshold, determining a depth map corresponding to the target scene based on the first depth measurement device; If the first depth value is less than the preset depth threshold, A depth map corresponding to the target scene is determined based on the second depth measurement device.

4. The image alignment method according to claim 1, wherein: The determining of the depth maps corresponding to the multiple cameras based on the depth map corresponding to the target scene, the relative transformation relationship between the depth measurement device and the multiple cameras, and the internal parameters of the multiple cameras includes: Determining coordinate data of each of the multiple cameras in a camera coordinate system based on a depth map corresponding to the target scene and a relative transformation relationship between the depth measurement device and the multiple cameras; Determine depth maps corresponding to the cameras based on the coordinate data of the cameras in their respective camera coordinate systems and the intrinsic parameters of the cameras.

5. The image alignment method according to claim 2, wherein: The first depth measuring device includes a laser radar, and the second depth measuring device includes a binocular camera.

6. An image alignment device, characterized in that: include: a determination module configured to determine a depth map corresponding to a target scene based on a depth measurement device in the aircraft, wherein the depth measurement device includes at least two depth measurement devices with different measurement principles; An alignment module is configured to determine depth maps corresponding to each of the multiple cameras based on a depth map corresponding to the target scene, a relative transformation relationship between the depth measurement device and multiple cameras of different wavelength bands in the aircraft, and intrinsic parameters of each of the multiple cameras; determine virtual viewpoints of virtual cameras corresponding to the multiple cameras based on geographic location information of each of the multiple cameras; determine virtual relative transformation relationships corresponding to each of the multiple cameras based on a camera relative transformation relationship between the virtual viewpoint and the multiple cameras; and project image coordinate system coordinate points corresponding to images captured by each of the multiple cameras onto an image plane of the virtual camera based on the intrinsic parameters of each of the multiple cameras, the intrinsic parameters of the virtual camera, the depth maps corresponding to each of the multiple cameras, and the virtual relative transformation relationships corresponding to the multiple cameras to determine an aligned image. The virtual relative transformation relationship is a relative transformation relationship between the cameras and the virtual viewpoint, the multiple cameras of different wavelength bands are multiple cameras of different spectral bands in a multispectral camera, the relative transformation relationships between the multiple cameras of different spectral bands included in the multispectral camera are known, and the images captured by the multiple cameras of different wavelength bands are spectral images. 7 . A computer-readable storage medium storing instructions, wherein when the instructions are executed by a processor of an electronic device, the electronic device is enabled to execute the image alignment method according to claim 1 .

8. An electronic device, comprising: processor; memory for storing computer-executable instructions; The processor is configured to execute the computer-executable instructions to implement the image alignment method according to any one of claims 1 to 5.

Images