Multispectral ground observation device omni-directional orthoimage generation method, program product

Abstract

The present application belongs to a kind of panoramic orthographic image generation method, to solve the current, ground multi-spectral camera for single multi-spectral remote sensing image in the means of space-ground integration remote sensing monitoring, it is difficult to accurately position the observed target, so that space-ground integration remote sensing monitoring capability is restricted technical problem, provide a kind of multi-spectral ground observation device panoramic orthographic image generation method, computer program product, the multi-spectral ground observation device is collected to multiple different angle original observation image, after projection transformation, generate panoramic orthographic image under the bird's-eye view, solve the problem that the observation range of single multi-spectral original image is small and the observation angle is not unified, the panoramic orthographic image generated can directly show observation area and observation range, provide basis for spatial accurate positioning of the observed target of interest, provide support for space-ground integration remote sensing observation.

Description

Technical Field

[0001] This invention relates to a panoramic orthophoto generation method, specifically to a panoramic orthophoto generation method and computer program product for a multispectral ground observation device. Background Technology

[0002] Multispectral remote sensing technology refers to the technique of dividing the electromagnetic waves emitted by ground objects into several narrow spectral bands, and acquiring information about the same target in different bands simultaneously through photography or scanning. Multispectral remote sensing can identify ground objects not only based on differences in the shape and structure of images, but also based on differences in spectral characteristics, thus expanding the amount of information available in remote sensing. Therefore, it is widely used in fields such as agricultural monitoring, geological exploration, environmental monitoring, and marine research.

[0003] Integrating space-air-ground remote sensing monitoring methods, such as aerospace satellite remote sensing, aerial drone remote sensing, and ground-based observation remote sensing, can provide comprehensive, multi-scale, and multi-element data information support for multispectral remote sensing applications. These three types of remote sensing observation modes each have their own characteristics and can complement each other in various fields. Aerospace satellite remote sensing achieves spectral remote sensing observation through multispectral imaging payloads carried by satellites. Its advantages lie in its wide observation range (swath width can reach hundreds of kilometers) and unrestricted observation area (capable of monitoring any region domestically or internationally). However, its disadvantages include low spatial resolution (only tens of meters), slow observation cycle (satellite revisit cycle of one to two weeks), difficulty in radiometric correction (greatly affected by atmospheric factors), susceptibility to weather factors (such as inability to observe under thick cloud cover), and high overall cost. Aerial drone remote sensing typically uses a payload-capable drone platform (such as the DJI M600) to carry a satellite multispectral camera. Currently, most of these cameras operate in the visible-near-infrared range. Their advantages include a larger observation range, higher observation flexibility, and less susceptibility to weather factors compared to ground-based observations. However, their disadvantages include limitations imposed by the platform's payload capacity, limited performance of the miniature multispectral camera, observation duration restricted by the drone's flight time, and the need for professional pilots, resulting in high labor costs.

[0004] Ground-based remote sensing monitoring typically uses ground-based spectrometers to acquire spectral information of crops, while contact sensors or chemical methods are used to acquire physicochemical information. Its advantages are: (1) due to the short radiation transmission link and low interference, it can achieve quantitative analysis with the highest accuracy; (2) it can collect data in real time; (3) it can acquire physicochemical parameters of crops in the same phase; (4) compared with UAV remote sensing, it has a higher degree of automation; and (5) it has a relatively low cost. However, it still has some disadvantages, such as lower flexibility compared with UAV remote sensing and a limited measurement range of a single device.

[0005] To compensate for the limited field of view of ground-based multispectral cameras in remote sensing, a panoramic turntable can be configured to give the multispectral camera a 360-degree panoramic observation capability. However, for a single multispectral remote sensing image, it is difficult to intuitively see its spatial range and thus accurately locate the observed target, which limits the integrated air-space-ground remote sensing monitoring capability. Summary of the Invention

[0006] This invention addresses the technical problem that current integrated space-air-ground remote sensing monitoring methods struggle to accurately locate observation targets from single multispectral remote sensing images, thus limiting the capabilities of integrated space-air-ground remote sensing monitoring. The invention provides a method for generating panoramic orthophotos from a multispectral ground observation device and a computer program product.

[0007] To achieve the above objectives, the present invention employs the following technical solution:

[0008] A method for generating panoramic orthophotos from a multispectral ground observation device, characterized by the following steps:

[0009] S1, geometric registration and merging of the image data of each channel of each multispectral image from the multispectral camera are performed respectively to obtain the registered and merged multispectral image;

[0010] S2, crop the boundary region of the registered and merged multispectral image to obtain the cropped multispectral image;

[0011] S3, delineate the farthest observation boundary line on the cropped multispectral image to obtain the observation area of ​​each multispectral image;

[0012] S4, combined with the farthest observation boundary line, determines the focus of the multispectral camera and the coordinates of the four corner points of the multispectral camera sensor within the observation area in the world coordinate system;

[0013] S5, respectively obtain the world coordinate system vectors from the focus of the multispectral camera to the four corner points of the multispectral camera sensor in the observation area;

[0014] S6, respectively acquire the ground coordinates of the four corner points of the multispectral camera sensor in the observation area of ​​each multispectral image;

[0015] S7. Based on the ground coordinates obtained in step S6, generate the ground projection transformation matrix for each multispectral image.

[0016] S8 generates a ground base map for multispectral image projection mosaicking from different perspectives of the multispectral ground observation device;

[0017] S9. Based on the ground projection transformation matrix of each multispectral image obtained in step S7, the multispectral images from different perspectives of the multispectral ground observation device are projected and mosaicked onto the ground base map to generate a panoramic orthophoto.

[0018] Furthermore, step S1 specifically includes:

[0019] S1.1, calculate the feature points of each channel image data of each multispectral image;

[0020] S1.2, Select any channel image data as the reference image, and use the other channel image data as the images to be matched;

[0021] S1.3, Based on the feature points of each channel image data obtained in step S1.1, perform feature point matching between each image to be matched and the reference image;

[0022] S1.4, the least squares method is used to calculate the projection transformation matrix from each image to the reference image;

[0023] S1.5, according to the projection transformation matrix of each image to be matched to the reference image, perform projection transformation on each image to be matched, and merge all channel image data.

[0024] Further, step S2 specifically involves cropping the registered and merged multispectral image by a fixed width towards the center of the reference image.

[0025] Further, step S4 specifically involves obtaining the coordinates of the multispectral camera focus and the four corner points of the multispectral camera sensor in the world coordinate system using the following formula:

[0026] Foc1 = Rz·Rx·Foc + T;

[0027] C11 = Rz·Rx·C1 + T;

[0028] C21 = Rz·Rx·C2 + T;

[0029] C31 = Rz·Rx·C3 + T;

[0030] C41 = Rz·Rx·C4 + T;

[0031] Where Foc1 represents the coordinates of the multispectral camera focus in the world coordinate system, C11, C21, C31, and C41 represent the coordinates of the four corner points of the multispectral camera sensor in the world coordinate system, Rz represents the azimuth rotation matrix, Rx represents the pitch rotation matrix, and T represents the altitude vector.

[0032] Foc represents the initial coordinates of the multispectral camera focus in the world coordinate system:

[0033] Foc = [0, -f, 0]'

[0034] Where f represents the focal length of the multispectral camera sensor;

[0035] C1, C2, C3, and C4 represent the initial coordinates of the four corner points of the multispectral camera sensor in the world coordinate system:

[0036]

[0037]

[0038]

[0039]

[0040] Where coms_w represents the width of the multispectral camera sensor, coms_h represents the height of the multispectral camera sensor, img_h represents the height of the cropped multispectral image, and top_h represents the number of rows in the horizontal direction of the multispectral image from the farthest observation boundary line.

[0041] Further, step S5 specifically involves obtaining the vector from the multispectral camera focus to the four corner points of the multispectral camera sensor using the following formula:

[0042] V1 = C11 - Foc1

[0043] V2=C21-Foc1

[0044] V3 = C31 - Foc1

[0045] V4 = C41 - Foc1

[0046] Where V1 represents the world coordinate system vector from the multispectral camera focus to C11, V2 represents the world coordinate system vector from the multispectral camera focus to C21, V31 represents the world coordinate system vector from the multispectral camera focus to C31, and V4 represents the world coordinate system vector from the multispectral camera focus to C41.

[0047] Further, step S6 specifically involves obtaining the ground coordinates of the four corner points of the observation area for each multispectral image using the following formula:

[0048] G1(1)=Foc1(1)-Foc1(3)×(C11(1)-Foc1(1)) / (C11(3)-Foc1(3))

[0049] G1(2)=Foc1(2)-Foc1(3)×(C11(2)-Foc1(2)) / (C11(3)-Foc1(3))

[0050] G2(1)=Foc1(1)-Foc1(3)×(C21(1)-Foc1(1)) / (C21(3)-Foc1(3))

[0051] G2(2)=Foc1(2)-Foc1(3)×(C21(2)-Foc1(2)) / (C21(3)-Foc1(3))

[0052] G3(1)=Foc1(1)-Foc1(3)×(C31(1)-Foc1(1)) / (C31(3)-Foc1(3))

[0053] G3(2)=Foc1(2)-Foc1(3)×(C31(2)-Foc1(2)) / (C31(3)-Foc1(3))

[0054] G4(1)=Foc1(1)-Foc1(3)×(C41(1)-Foc1(1)) / (C41(3)-Foc1(3))

[0055] G4(2)=Foc1(2)-Foc1(3)×(C41(2)-Foc1(2)) / (C41(3)-Foc1(3))

[0056] Wherein, G1(1) represents the X-axis coordinate of the ground coordinate G1 in the world coordinate system, G1(2) represents the Y-axis coordinate of the ground coordinate G1 in the world coordinate system, G2(1) represents the X-axis coordinate of the ground coordinate G2 in the world coordinate system, G2(2) represents the Y-axis coordinate of the ground coordinate G2 in the world coordinate system, G3(1) represents the X-axis coordinate of the ground coordinate G3 in the world coordinate system, G3(2) represents the Y-axis coordinate of the ground coordinate G3 in the world coordinate system, G4(1) represents the X-axis coordinate of the ground coordinate G4 in the world coordinate system, and G4(2) represents the Y-axis coordinate of the ground coordinate G4 in the world coordinate system. The coordinate system is defined by the Y-axis coordinate; Foc1(1) represents the final world coordinate of the multispectral camera focus, Foc1's X-axis coordinate; Foc1(2) represents the final world coordinate of the multispectral camera focus, Foc1's Y-axis coordinate; Foc1(3) represents the final world coordinate of the multispectral camera focus, Foc1's Z-axis coordinate; C11(1) represents the final world coordinate of the corner point on the sensor, C11's X-axis coordinate; C11(2) represents the final world coordinate of the corner point on the sensor, C11(3) represents the final world coordinate of the corner point on the sensor, C11(4) represents the final world coordinate of the corner point on the sensor, C11(5) represents the final world coordinate of the corner point on the sensor, C11(6) represents the final world coordinate of the corner point on the sensor, C11(7) represents the final world coordinate of the corner point on the sensor, C11(8) represents the final world coordinate of the corner point on the sensor, C11(9) represents the final world coordinate of the corner point on the sensor, C11(10 ... The Z-axis coordinate of the corner point with coordinate C11; C21(1) represents the X-axis coordinate of the corner point with final world coordinate C21 on the sensor, C21(2) represents the Y-axis coordinate of the corner point with final world coordinate C21 on the sensor, C21(3) represents the Z-axis coordinate of the corner point with final world coordinate C21 on the sensor; C31(1) represents the X-axis coordinate of the corner point with final world coordinate C31 on the sensor, C31(2) represents the Y-axis coordinate of the corner point with final world coordinate C31 on the sensor, C31(3) represents the Z-axis coordinate of the corner point with final world coordinate C21 on the sensor. The Z-axis coordinate of the corner point marked C31; ground coordinate G1 is the ground coordinate corresponding to C11, ground coordinate G2 is the ground coordinate corresponding to C21, ground coordinate G3 is the ground coordinate corresponding to C31, ground coordinate G4 is the ground coordinate corresponding to C41; C41(1) represents the X-axis coordinate of the corner point with final world coordinates C41 on the sensor, C41(2) represents the Y-axis coordinate of the corner point with final world coordinates C41 on the sensor, and C41(3) represents the Z-axis coordinate of the corner point with final world coordinates C41 on the sensor.

[0057] Further, step S7 specifically involves obtaining the projection transformation matrix TForm from the image coordinates of the four corner points of the multispectral image to the ground coordinates of the four corner points of the multispectral camera sensor within the corresponding observation area using the least squares method, according to the following formula:

[0058] G1 = TForm·I1

[0059] G2 = TForm·I2

[0060] G3 = TForm·I3

[0061] G4 = TForm·I4;

[0062] Wherein, I1 represents the corner image coordinates of the multispectral image corresponding to G1, I2 represents the corner image coordinates of the multispectral image corresponding to G2, I3 represents the corner image coordinates of the multispectral image corresponding to G3, and I4 represents the corner image coordinates of the multispectral image corresponding to G4.

[0063] Further, in step S7, the coordinates of the four corner points of the multispectral image are as follows:

[0064] I1=[1,1]′

[0065] I2 = [img_w, 1]'

[0066] I3 = [ing_w, img_h - top_h]'

[0067] I4 = [1, img_h - top_h]'

[0068] Where img_w represents the width of the cropped multispectral image.

[0069] Furthermore, in step S9, before generating the panoramic orthophoto, the oversampled or undersampled pixel regions are filled using a linear fitting method.

[0070] In addition, the present invention also provides a computer program product, including a computer program, which is characterized in that when the program is executed by a processor, it implements the steps of the above-described method for generating panoramic orthophotos of a multispectral ground observation device.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] 1. The panoramic orthophoto generation method for multispectral ground observation devices proposed in this invention overcomes the problems of small field of view and unclear spatial location information of the observed target in the original multispectral images. The spatial location information and observation range of the observation area can be intuitively seen from the panoramic orthophoto, providing support for integrated air-space-ground remote sensing observation. Furthermore, the output multispectral panoramic orthophoto includes an azimuth compass and scale bar, which helps in interpreting the azimuth and distance of the observation area.

[0073] 2. In the panoramic orthophoto generation method of the present invention, the farthest observation boundary line is delineated on the multispectral remote sensing image, which helps to eliminate observation areas with too few image pixels and too far distances. At the same time, it also avoids the problem that areas near the horizon and in the air cannot be projected and transformed onto the ground coordinate system.

[0074] 3. The panoramic orthophoto generation method of the present invention requires few parameters, has a fast calculation speed and a robust algorithm, and can realize automated generation of multispectral ground observation panoramic orthophotos.

[0075] 4. The present invention also provides a computer program product capable of executing the above method steps, enabling the application of the method of the present invention and achieving integration on corresponding hardware devices. Attached Figure Description

[0076] Figure 1 This is a flowchart illustrating an embodiment of the panoramic orthophoto generation method for a multispectral ground observation device according to the present invention.

[0077] Figure 2 The data for each channel of a single multispectral image in this embodiment of the invention are shown below; where (a) is the red band channel, (b) is the green band channel, (c) is the blue band channel, (d) is the red edge band channel, and (e) is the near-infrared band channel.

[0078] Figure 3 This invention relates to the matching of feature points in two channels of image data in an embodiment of the invention.

[0079] Figure 4 This is a schematic diagram of boundary cropping after registration and merging of each channel of a single multispectral image in an embodiment of the present invention;

[0080] Figure 5 This is a schematic diagram showing the dimensions of a single multispectral image after registration and merging of each channel and subsequent boundary cropping, as described in an embodiment of the present invention.

[0081] Figure 6 The image shown is a multispectral image with full-view registration and merging obtained in an embodiment of the present invention; wherein, 1.tif-12.tif are multispectral images acquired at different horizontal angles.

[0082] Figure 7 This is a schematic diagram illustrating the delineation of the farthest observation boundary line of multispectral images in an embodiment of the present invention;

[0083] Figure 8 This is a schematic diagram showing the positional relationship of the four corner points of the multispectral camera sensor and thus the focal point of the multispectral camera in an embodiment of the present invention;

[0084] Figure 9 This is a schematic diagram of the ground coordinates of the four corner points of the multispectral image observation area in an embodiment of the present invention;

[0085] Figure 10 This is a single multispectral ground orthophoto obtained in an embodiment of the present invention;

[0086] Figure 11 This is a panoramic orthophoto image of a multispectral ground observation device obtained in an embodiment of the present invention. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0088] This invention proposes a method for generating panoramic orthophotos from a multispectral ground observation device. This method transforms multiple original observation images from different angles acquired by the multispectral ground observation device into a panoramic orthophoto from a top-down perspective through projection transformation. This solves the problems of small observation range and inconsistent observation angles of single multispectral original images. The generated panoramic orthophoto can intuitively show the observation area and observation range, providing a basis for precise spatial positioning of targets of interest and supporting integrated air, space, and ground remote sensing observation.

[0089] For the panoramic orthorectified image generation method of the present invention, the inputs are multiple original image data of each channel acquired by a multispectral ground observation device, multispectral camera observation geometric data, and multispectral camera parameter data, and the output is multispectral panoramic orthorectified image data. Specifically, the multispectral ground observation device in this invention can achieve 360° panoramic horizontal and vertical observation. The acquired M original multispectral images need to cover all observation angles in the horizontal direction, and the field of view of each image needs to overlap. The multispectral ground observation device is equipped with a multispectral camera; the multispectral camera contains N spectral channels, and there are field-of-view geometric deviations between channels, requiring channel registration and fusion; the multispectral camera observation geometric data includes the camera pitch angle Ax, camera azimuth angle Az, and camera ground height H; the multispectral camera parameter data includes the sensor size CMOS_w × CMOS_h and focal length f.

[0090] Specific steps are as follows Figure 1 As shown:

[0091] S1, Registration of each channel in a single multispectral image

[0092] Geometric registration and merging of N channels of a single multispectral image is performed as follows: ① Calculate the feature points of each of the N channels, such as using the SURF algorithm; ② Select a channel image as the reference image, such as the N / 2th image; ③ Match the reference image with the feature points of other channels; ④ Calculate the projection transformation matrix from the reference image to the other channels using the least squares method; ⑤ Perform projection transformation on the other channels and merge all channels.

[0093] S2, cropping the boundary region of the registered multispectral image.

[0094] The boundary regions of the registered and merged multispectral image are cropped to remove the influence of unregistered boundary areas. The cropping method is to use the center of the single-channel reference image as a reference, and crop a fixed width L from the periphery of the registered and merged multispectral image towards the center. The size of the cropped multispectral image is denoted as img_w×img_h.

[0095] S3, Delineate the farthest observation boundary line for the cropped multispectral image.

[0096] The farthest observation boundary line is delineated on the cropped multispectral image. Observations that are too far from the observation area and occupy too few pixels can be excluded. The farthest observation boundary line is parallel to the horizontal direction of the multispectral image, and its row number is denoted as top_h. The farthest extent of the multispectral image is then determined by this farthest observation boundary line during subsequent projection transformations onto the ground map.

[0097] S4 transforms the multispectral camera coordinate system to the world coordinate system.

[0098] The world coordinate system described in this invention, namely the East-North-Sky coordinate system, specifically has the positive X-axis pointing due east, the positive Y-axis pointing due north, and the positive Z-axis pointing to the zenith.

[0099] Using geometric data and camera parameter data observed by a multispectral camera, the multispectral camera coordinate system is transformed to the world coordinate system. The sensor center of the multispectral camera is set as the origin of the world coordinate system, the initial world coordinates of the camera focus are denoted as Foc, and the initial world coordinates of the four corner points on the sensor are C1, C2, C3, and C4. Since a boundary line for the farthest observation of the image is defined, C1 and C2 are the sensor endpoints corresponding to this boundary line, as shown in the following formula:

[0100] Foc = [0, -f, 0]'

[0101]

[0102]

[0103]

[0104]

[0105] Considering the observation geometry of the multispectral camera, namely the camera's pitch angle Ax, azimuth angle Az, and ground height H, the initial world coordinates of the focal point and corner points need to be multiplied by the pitch rotation matrix Rx and the azimuth rotation matrix Rz, and then added to the height vector T. Therefore, the final world coordinates of the multispectral camera's focal point are Foc1, and the final world coordinates of the four corner points on the sensor are C11, C21, C31, and C41.

[0106] Foc1 = Rz·Rx·Foc + T

[0107] C11=Rz·Rx·C1+T

[0108] C21=Rz·Rx·C2+T

[0109] C31=Rz·Rx·C3+T

[0110] C41=Rz·Rx·C4+T

[0111]

[0112]

[0113] T = [0 0 H]′

[0114] S5, vector generation from the focus of the multispectral camera to the corner point of the sensor;

[0115] Generate the world coordinate system vectors V1, V2, V3, and V4 from the multispectral camera focus to the sensor corner point, as follows:

[0116] V1 = C11 - Foc1

[0117] V2=C21-Foc1

[0118] V3 = C31 - Foc1

[0119] V4 = C41 - Foc1.

[0120] S6, Generation of Ground Coordinates from a Single Multispectral Image

[0121] The ground coordinates G1, G2, G3, and G4 of the four corner points of the sensor within the observation area of ​​a single multispectral image are generated using the following formula for calculating the intersection of a spatial vector and a plane:

[0122] G1(1)=Foc1(1)-Foc1(3)×(C11(1)-Foc1(1)) / (C11(3)-Foc1(3))

[0123] G1(2)=Foc1(2)-Foc1(3)×(C11(2)-Foc1(2)) / (C11(3)-Foc1(3))

[0124] G2(1)=Foc1(1)-Foc1(3)×(C21(1)-Foc1(1)) / (C21(3)-Foc1(3))

[0125] G2(2)=Foc1(2)-Foc1(3)×(C21(2)-Foc1(2)) / (C21(3)-Foc1(3))

[0126] G3(1)=Foc1(1)-Foc1(3)×(C31(1)-Foc1(1)) / (C31(3)-Foc1(3))

[0127] G3(2)=Foc1(2)-Foc1(3)×(C31(2)-Foc1(2)) / (C31(3)-Foc1(3))

[0128] G4(1)=Foc1(1)-Foc1(3)×(C41(1)-Foc1(1)) / (C41(3)-Foc1(3))

[0129] G4(2)=Foc1(2)-Foc1(3)×(C41(2)-Foc1(2)) / (C41(3)-Foc1(3))

[0130] Wherein, G1(1) represents the X-axis coordinate of the world coordinate system of ground coordinate G1, G1(2) represents the Y-axis coordinate of the world coordinate system of ground coordinate G1, G2(1) represents the X-axis coordinate of the world coordinate system of ground coordinate G2, G2(2) represents the Y-axis coordinate of the world coordinate system of ground coordinate G2, G3(1) represents the X-axis coordinate of the world coordinate system of ground coordinate G3, and G3(2) represents the Y-axis coordinate of the world coordinate system of ground coordinate G3; Foc1(1) represents the X-axis coordinate of the final world coordinate of the multispectral camera focus Foc1, Foc1(2) represents the Y-axis coordinate of the final world coordinate of the multispectral camera focus Foc1, and Foc1(3) represents the Z-axis coordinate of the final world coordinate of the multispectral camera focus Foc1; C11(1) represents the X-axis coordinate of the corner point on the sensor whose final world coordinate is C11, and C11(2) represents the Y-axis coordinate of the corner point on the sensor whose final world coordinate is C11. C11(3) represents the Z-axis coordinate of the corner point on the sensor whose final world coordinates are C11; C21(1) represents the X-axis coordinate of the corner point on the sensor whose final world coordinates are C21; C21(2) represents the Y-axis coordinate of the corner point on the sensor whose final world coordinates are C21; C21(3) represents the Z-axis coordinate of the corner point on the sensor whose final world coordinates are C21; C31(1) represents the X-axis coordinate of the corner point on the sensor whose final world coordinates are C31; C31(2) represents the Y-axis coordinate of the corner point on the sensor whose final world coordinates are C31; C31(3) represents the Z-axis coordinate of the corner point on the sensor whose final world coordinates are C31; C41(1) represents the X-axis coordinate of the corner point on the sensor whose final world coordinates are C41; C41(2) represents the Y-axis coordinate of the corner point on the sensor whose final world coordinates are C41; C41(3) represents the Z-axis coordinate of the corner point on the sensor whose final world coordinates are C41.

[0131] S7, Generation of Ground Projection Transformation Matrix for Single Multispectral Image

[0132] The least squares method is used to calculate the projection transformation matrix TForm from the image coordinates I1, I2, I3, I4 at the four corner points of the multispectral image to the corresponding ground coordinates G1, G2, G3, G4 of the observation area. The image coordinates are in pixels, and the ground coordinates are in centimeters. Therefore, the ground coordinates need to be multiplied by a transformation coefficient rate (unit: pixels / cm). The rate value can be empirically given based on the size of the transformed image.

[0133] G1 = TForm·I1

[0134] G2 = TForm·I2

[0135] G3 = TForm·I3

[0136] G4 = TForm·I4

[0137] I1=[1,1]′

[0138] I2 = [img_w, 1]'

[0139] I3 = [ing_w, img_h - top_h]'

[0140] I4 = [1, img_h - top_h]'.

[0141] S8, Ground Map Generation

[0142] Generate a ground base map for multispectral images from each viewpoint to be projected and mosaicked onto it. The size of the ground base map must exceed the ground projection range of the multispectral images from each viewpoint. The specific value can be adjusted based on experience.

[0143] S9, multispectral imagery is projected and mosaicked onto a ground base map.

[0144] The multispectral images from various perspectives of the multispectral ground observation device are projected and mosaicked onto the ground base map. This involves multiplying each pixel of the multispectral image from each perspective by its corresponding ground projection transformation matrix, transforming and assigning the transformation values ​​to the ground image, and then filling oversampled or undersampled pixel areas using a linear fitting method.

[0145] The panoramic orthophoto generation method of this invention outputs multispectral panoramic orthophoto data with azimuth and scale information. The Y-axis of the ground base map is oriented towards true north, thus an azimuth compass is added. Then, based on the transformation coefficient 'rate' from the ground world physical coordinate system to the pixel coordinate system of the ground base map image, the scale corresponding to the number of pixels in the base map image is determined.

[0146] The following is a specific embodiment of the panoramic orthophoto generation method using the multispectral ground observation device of the present invention:

[0147] An experiment was conducted using a multispectral ground observation device for agricultural monitoring in a certain region. The device is equipped with a 5-channel multispectral camera, which is located inside a panoramic observation turntable and can achieve 360° horizontal azimuth and 0-90° elevation angle observations.

[0148] like Figure 2 This involves acquiring 5-channel data from a single multispectral image, including red, green, blue, red-edge, and near-infrared channels. For each single multispectral image, SURF feature points are calculated for each of the 5 channels. Using the 3rd channel as the reference, feature points are matched with the other 4 channels. The projection transformation matrix is ​​calculated using the least squares method. Then, the other 4 channels are registered to the 3rd channel, and multi-channel merging is performed. Figure 3 This is a schematic diagram illustrating feature point matching between two channels of image data. (Example) Figure 4For a single multispectral image after channel registration and merging, its boundary region is cropped. Using the center of the third channel as a reference, the original resolution of the single-channel image is 1280×960, and the cropping size from all sides towards the center is L = 40 pixels, as shown below. Figure 5 The cropped multispectral image size is img_w×img_h = 1200×880. For example... Figure 6 To obtain panoramic data of the observation area, this embodiment acquired a total of 12 multispectral images at different horizontal angles, namely... Figure 6 Images numbered 1 to 12. (Example) Figure 7 The farthest observation boundary line is delineated on the registered and merged multispectral image. Figure 7 (Solid line at the top), the farthest observation boundary line is parallel to the horizontal direction of the multispectral image, and the vertical coordinate of the farthest observation boundary line is the 200th pixel. For example... Figure 8 Using geometric data and camera parameter data observed by a multispectral camera, the multispectral camera coordinate system is transformed to the world coordinate system. The initial world coordinates Foc of the camera focus are [0, -0.55, 0]. The initial world coordinates C1, C2, C3, and C4 of the four corner points on the sensor are [-0.24, 0, 0.0986], [0.24, 0, 0.0986], [-0.24, 0, -0.18], and [0.24, 0, 0.18], respectively. The altitude vector T is [0, 0, 300], the pitch angle Ax is -15°, and the azimuth angle Az is 15°. Then, the pitch rotation matrix Rx and the azimuth rotation matrix Rz are as follows:

[0149]

[0150]

[0151] After calculating the pitch, azimuth, and altitude factors, Foc1 is [0.1375, -0.5132, 300.1424], and C11, C21, C31, and C41 are [-0.2384, -0.0375, 300.0952], [0.2252, 0.868, 300.0952], [-0.2198, -0.1071, 299.8261], and [0.2439, 0.0171, 299.8261], respectively.

[0152] The world coordinate system vectors V1, V2, V3, and V4 from the focus of the multispectral camera to the corner of the sensor are [-0.3759, 0.4757, -0.0471], [0.0877, 0.5999, -0.0471], [-0.3573, 0.4060, -0.3162], and [0.1064, 0.5303, -0.3162], respectively.

[0153] like Figure 9Using the formula for calculating the intersection of spatial vectors and planes, the ground coordinates G1, G2, G3, and G4 of the four corner points corresponding to the observation area of ​​a single multispectral image are generated as [-2394.5, 3029.6], [558.9, 3820.9], [-338.9, 384.9], and [101.1, 502.8], respectively. Using the least squares method, the projection transformation matrix TForm from the image coordinates I1, I2, I3, and I4 of the four corner points of the multispectral image to the corresponding ground coordinates G1, G2, G3, and G4 of the observation area is calculated, where I1, I2, I3, and I4 are [1,1], [1200,1], [1200,681], and [1,681], respectively. The resulting TForm is as follows:

[0154]

[0155] For each pixel of a single multispectral image, multiply it by its corresponding projection transformation matrix TForm, and assign the transformed values ​​to the base image. Oversampled or undersampled pixel regions are then filled using linear fitting to obtain the result. Figure 10 The above is a ground orthophoto of a single multispectral image.

[0156] like Figure 11 The same method was applied to 12 multispectral images to generate panoramic orthophoto data of the multispectral ground observation device, along with azimuth and scale information.

[0157] In addition, the panoramic orthophoto generation method of the multispectral ground observation device of the present invention can also form a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the panoramic orthophoto generation method of the multispectral ground observation device.

[0158] The panoramic orthophoto generation method of the present invention generates a panoramic orthophoto from a top-down perspective by projecting and transforming multiple original observation images from different angles collected by a multispectral ground observation device. This solves the problems of small observation range and inconsistent observation angles of a single multispectral original image. The generated panoramic orthophoto can intuitively show the observation area and observation range, providing a basis for accurate spatial positioning of the target of interest and supporting integrated air, space, and ground remote sensing observation.

[0159] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating a panoramic orthographic image of a multispectral ground observation device, characterized in that, The method comprises the following steps: S1, respectively, each channel image data of each multi-spectral image of the multi-spectral camera is geometrically registered and merged to obtain a registered and merged multi-spectral image; S2, the boundary region of the registered and merged multi-spectral image is cropped to obtain a cropped multi-spectral image; S3, the farthest observation boundary line is drawn on the cropped multi-spectral image to obtain an observation region of each multi-spectral image; S4, in combination with the farthest observation boundary line, the coordinates of the multi-spectral camera focus point and the four corner points of the multi-spectral camera sensor in the world coordinate system are determined; S5, the world coordinate system vectors from the multi-spectral camera focus point to the four corner points of the multi-spectral camera sensor in the observation region are respectively obtained; S6, the ground coordinates of the four corner points of the multi-spectral camera sensor in the observation region of each multi-spectral image are respectively obtained; S7, according to the ground coordinates obtained in step S6, the ground projection transformation matrix of each multi-spectral image is respectively generated; S8, a ground base map for multi-spectral image projection and inlaying of the multi-spectral ground observation device under different viewing angles is generated; S9, according to the ground projection transformation matrix of each multi-spectral image obtained in step S7, the multi-spectral images of the multi-spectral ground observation device under different viewing angles are projected and inlaid on the ground base map to generate a panoramic orthographic image.

2. The method of claim 1, wherein the panoramic orthographic image is generated by: Step S1 is specifically: S1.1, the feature points of each channel image data of each multi-spectral image are respectively calculated; S1.2, any channel image data is selected as a reference image, and the other channel image data is selected as a matching image; S1.3, according to the feature points of each channel image data obtained in step S1.1, each matching image is matched with the reference image respectively; S1.4, the least square method is used to calculate the projection transformation matrix of each matching image to the reference image respectively; S1.5, according to the projection transformation matrix of each matching image to the reference image, the projection transformation of each matching image is carried out respectively, and all channel image data are combined.

3. The method for generating panoramic orthophotos of a multispectral ground observation device according to claim 2, characterized in that: Step S2 is specifically cutting a fixed width to the center of the reference image of the registered and merged multi-spectral image.

4. The method according to any one of claims 1 to 3, wherein the multi-spectral ground observation device orthoimage generation method is characterized by, Step S4 is specifically obtaining the coordinates of the multi-spectral camera focus point and the four corner points of the multi-spectral camera sensor in the world coordinate system by the following formula: Foc1=Rz·Rx·Foc+T; C11=Rz·Rx·C1+T; C21=Rz·Rx·C2+T; C31=Rz·Rx·C3+T; C41=Rz·Rx·C4+T; Wherein, Foc1 represents the coordinates of the multi-spectral camera focus point in the world coordinate system, C11, C21, C31, C41 respectively represent the coordinates of the four corner points of the multi-spectral camera sensor in the world coordinate system, Rz represents the azimuth rotation matrix, Rx represents the pitch rotation matrix, T represents the height vector; Foc represents the initial coordinates of the multi-spectral camera focus point in the world coordinate system: Foc=[0,-f,0]′ Wherein, f represents the focal length of the multi-spectral camera sensor; C1, C2, C3, C4 respectively represent the initial coordinates of the four corner points of the multi-spectral camera sensor in the world coordinate system: Wherein, coms_w represents the width of the multispectral camera sensor, coms_h represents the height of the multispectral camera sensor, img_h represents the height of the cropped multispectral image, and top_h represents the number of rows in the horizontal direction of the multispectral image from the farthest observation boundary line.

5. The method of claim 4, wherein the panoramic orthographic image is generated by: Step S5 is specifically obtaining the vector from the focus of the multispectral camera to the four corner points of the multispectral camera sensor by the following formula: V1=C11-Foc1 V2=C21-Foc1 V3=C31-Foc1 V4=C41-Foc1 Wherein, V1 represents the world coordinate system vector from the focus of the multispectral camera to C11, V2 represents the world coordinate system vector from the focus of the multispectral camera to C21, V31 represents the world coordinate system vector from the focus of the multispectral camera to C31, and V4 represents the world coordinate system vector from the focus of the multispectral camera to C41.

6. The method of claim 5, wherein the panoramic orthographic image is generated by: Step S6 is specifically obtaining the ground coordinates of the four corner points of the observation area of each multispectral image by the following formula: G1(1)=Foc1(1)-Foc1(3)×(C11(1)-Foc1(1)) / (C11(3)-Foc1(3)) G1(2)=Foc1(2)-Foc1(3)×(C11(2)-Foc1(2)) / (C11(3)-Foc1(3)) G2(1)=Foc1(1)-Foc1(3)×(C21(1)-Foc1(1)) / (C21(3)-Foc1(3)) G2(2)=Foc1(2)-Foc1(3)×(C21(2)-Foc1(2)) / (C21(3)-Foc1(3)) G3(1)=Foc1(1)-Foc1(3)×(C31(1)-Foc1(1)) / (C31(3)-Foc1(3)) G3(2)=Foc1(2)-Foc1(3)×(C31(2)-Foc1(2)) / (C31(3)-Foc1(3)) G4(1)=Foc1(1)-Foc1(3)×(C41(1)-Foc1(1)) / (C41(3)-Foc1(3)) G4(2)=Foc1(2)-Foc1(3)×(C41(2)-Foc1(2)) / (C41(3)-Foc1(3)) Wherein, G1(1) represents the world coordinate system X axis coordinate of ground coordinate G1, G1(2) represents the world coordinate system Y axis coordinate of ground coordinate G1, G2(1) represents the world coordinate system X axis coordinate of ground coordinate G2, G2(2) represents the world coordinate system Y axis coordinate of ground coordinate G2, G3(1) represents the world coordinate system X axis coordinate of ground coordinate G3, G3(2) represents the world coordinate system Y axis coordinate of ground coordinate G3, G4(1) represents the world coordinate system X axis coordinate of ground coordinate G4, G4(2) represents the world coordinate system Y axis coordinate of ground coordinate G4;Foc1(1) represents the X axis coordinate of the final world coordinate Foc1 of the focus point of the multispectral camera, Foc1(2) represents the Y axis coordinate of the final world coordinate Foc1 of the focus point of the multispectral camera, Foc1(3) represents the Z axis coordinate of the final world coordinate Foc1 of the focus point of the multispectral camera;C11(1) represents the X axis coordinate of the corner point with the final world coordinate C11 on the sensor, C11(2) represents the Y axis coordinate of the corner point with the final world coordinate C11 on the sensor, C11(3) represents the Z axis coordinate of the corner point with the final world coordinate C11 on the sensor;C21(1) represents the X axis coordinate of the corner point with the final world coordinate C21 on the sensor, C21(2) represents the Y axis coordinate of the corner point with the final world coordinate C21 on the sensor, C21(3) represents the Z axis coordinate of the corner point with the final world coordinate C21 on the sensor;C31(1) represents the X axis coordinate of the corner point with the final world coordinate C31 on the sensor, C31(2) represents the Y axis coordinate of the corner point with the final world coordinate C31 on the sensor, C31(3) represents the Z axis coordinate of the corner point with the final world coordinate C31 on the sensor;The ground coordinate G1 is the ground coordinate corresponding to C11, the ground coordinate G2 is the ground coordinate corresponding to C21, the ground coordinate G3 is the ground coordinate corresponding to C31, and the ground coordinate G4 is the ground coordinate corresponding to C41;C41(1) represents the X axis coordinate of the corner point with the final world coordinate C41 on the sensor, C41(2) represents the Y axis coordinate of the corner point with the final world coordinate C41 on the sensor, and C41(3) represents the Z axis coordinate of the corner point with the final world coordinate C41 on the sensor.

7. The method of claim 6, wherein the panoramic orthographic image is generated by: Step S7 is specifically, by the following formula, using the least square method, the projection transformation matrix TForm of the four corner point image coordinates of the multispectral image to the ground coordinates of the four corner points of the multispectral camera sensor in the corresponding observation area is obtained: G1=TForm·I1 G2=TForm·I2 G3=TForm·I3 G4=TForm·I4; Wherein, I1 represents the corner point image coordinate of the multispectral image corresponding to G1, I2 represents the corner point image coordinate of the multispectral image corresponding to G2, I3 represents the corner point image coordinate of the multispectral image corresponding to G3, and I4 represents the corner point image coordinate of the multispectral image corresponding to G4.

8. The method of claim 7, wherein the panoramic orthographic image is generated by: In step S7, the four corner point image coordinates of the multispectral image are respectively: I1＝[1，1]′ I2 = [img_w, 1]' I3 = [img_w, img_h - top_h]' I4 = [1, img_h - top_h]' wherein img_w denotes the width of the cropped multi-spectral image.

9. The method of claim 8, wherein the method further comprises: generating a panoramic orthographic image from the plurality of images. 9 Before the step of generating the panoramic ortho-image, the step of filling the over-sampled or under-sampled pixel region by linear fitting is further included in step S9.

10. A computer program product comprising a computer program, characterized in that: The program is executed by the processor to implement the steps of the panoramic ortho-image generation method of the multi-spectral ground observation device according to any one of claims 1 to 9.

Images