Footprint positioning and visualization method for non-imaging hyperspectral observation based on pinhole camera

Abstract

The present invention provides a non-imaging hyperspectral observation footprint positioning and visualization method based on a pinhole camera, which solves the problem that traditional non-imaging spectrometers are difficult to locate observation footprints; the method comprises: synchronously collecting spectral data and image data of up-going radiation through a non-imaging hyperspectral observation module and a pinhole camera module, and the non-imaging hyperspectral observation module sends the spectral data and image data to a control module; the control module performs spatial registration on the spectral data and image data of up-going radiation through an image processing module and a spectral data processing module provided therein, accurately locates the observation footprint, calculates phenological parameters such as greenness index and vegetation coverage, generates footprint positioning data, and sends the footprint positioning data to a visualization display module; the present invention is suitable for ecological monitoring and carbon sink assessment in complex environments such as forest canopies, has the characteristics of high efficiency, reliability and low cost, and provides accurate data support for agriculture, forestry and ecological research.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, and in particular to a pinhole camera-based non-imaging hyperspectral observation footprint positioning and visualization method. Background Art

[0002] In the fields of ecological and environmental monitoring and agricultural assessment, sunlight-induced chlorophyll fluorescence (SIF), as a direct indicator of plant photosynthesis, has become an important means of studying plant physiological status. Traditional SIF observations rely primarily on non-imaging spectrometers, such as ocean optical spectrometers. These devices can provide data with ultra-high spectral resolution, suitable for SIF inversion and calculation of vegetation reflectance indices. However, due to its non-imaging nature, existing technologies have shortcomings in the spatial positioning and visualization of SIF observations. The lack of spatial information during the observation process makes it difficult to accurately locate the observation footprint, especially when performing multi-angle rotation observations, making it even more difficult to determine the specific observed ground objects.

[0003] In order to accurately locate and visualize the observation footprints and obtain more vegetation parameter information, researchers use high-resolution cameras and spectrometers or non-imaging hyperspectral synchronous observations to obtain spatial information, and use image processing technology to achieve the positioning and visualization of the observation footprints. Summary of the Invention

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A pinhole camera-based non-imaging hyperspectral observation footprint positioning and visualization method includes: a non-imaging hyperspectral observation module, a pinhole camera module, a multi-angle pan-tilt module, and a control module; wherein the multi-angle pan-tilt module adjusts the rotation angle to achieve multi-angle observation;

[0006] The pinhole camera module captures images of the observation area in real time, records spatial information, and sends the image data to the non-imaging hyperspectral observation module. The image data can be used to calculate the phenological parameters greenness index (GCC) and / or fractional vegetation cover (FVC);

[0007] The non-imaging hyperspectral observation module collects spectral data of the target area, which can be used to invert sunlight-induced chlorophyll fluorescence (SIF) and calculate vegetation reflectance index;

[0008] The non-imaging hyperspectral observation module sends the spectral data and image data to the control module;

[0009] The control module controls the non-imaging hyperspectral observation module and the pinhole camera module to synchronously collect spectral data and image data of the uplink radiation, and spatially aligns the spectral data and image data of the uplink radiation through the image processing module and spectral data processing module provided therein, marks the calculated sunlight-induced chlorophyll fluorescence (SIF), vegetation reflectance index, greenness index (GCC) and / or fractional vegetation cover (FVC) at the corresponding spatial position, generates footprint positioning data, and sends the footprint positioning data to the visualization display module;

[0010] The visual display module receives the footprint positioning data, and the control module controls the visual display module to display the footprint positioning data, and updates and stores the footprint positioning data in real time.

[0011] Furthermore, the non-imaging hyperspectral observation module includes at least: a spectrometer, a multi-path switcher, a bare optical fiber, a cosine receiver, and optical fibers, which are used to collect spectral data from the target area, which can be used to invert sunlight-induced chlorophyll fluorescence (SIF) and calculate the vegetation reflectance index. The pinhole camera module includes at least a pinhole camera probe, an optical fiber, a light-proof container, and a photosensitive element (such as a CCD or CMOS sensor), which is used to capture images of the observation area in real time, record spatial information, and calculate phenological parameters such as the greenness index (GCC) and fractional vegetation cover (FVC) from the image data. The control module includes at least an image processing module and a spectral data processing module, which are used to control the spectrometer and camera to collect data and perform storage and analysis. The multi-angle pan-tilt module includes at least a rotating pan-tilt head, a pan-tilt head controller, and corresponding transmission interfaces, which are used to support and adjust the angle of the device to achieve multi-angle observation.

[0012] Specifically, a bare optical fiber is fixed on a multi-angle pan / tilt head to measure the uplink radiation.

[0013] The cosine receiver is used to measure the downward radiation from the sky to the ground. To ensure that vertically incident radiation is received, it is fixed vertically upward next to the multi-angle gimbal.

[0014] The pinhole camera probe is fixed on a multi-angle pan-tilt head, maintaining the same observation angle as the bare optical fiber, and synchronously acquiring uplink radiation and photos of ground objects with the bare optical fiber.

[0015] The multi-path switcher is connected to the cosine receiver, spectrometer, camera, and pinhole camera probe via optical fibers. It switches between different observation modes. When observing downlink radiation, only one optical path is open, leading to the spectrometer. The downlink radiation measured by the cosine receiver is transmitted to the spectrometer via optical fiber to observe the incident solar spectrum. When observing uplink radiation, the spectrometer and camera optical paths are connected simultaneously, and the uplink radiation and pinhole camera image are recorded simultaneously.

[0016] The camera and spectrometer are connected to the control module via data cables to achieve data communication.

[0017] The multi-angle pan / tilt head and the pan / tilt head controller are connected via a data cable and connected to the control module.

[0018] The control module circuit board is equipped with corresponding circuit modules, including an image processing module, a spectral data processing module and a control module, which are used to process, analyze and store the collected images and spectral data, record the observation position, control the observation angle and manage the data acquisition process.

[0019] The multi-optical path switcher includes a solar incident light observation optical path, an upward ground object incident light observation optical path, and a pinhole camera image acquisition optical path.

[0020] On the basis of the above solution, the present invention can also make the following improvements:

[0021] Furthermore, the spectrometer is manufactured by Ocean Insight, a US company, and is a QEPRO spectrometer with a spectral range of 650-800 nm and a spectral resolution of 0.3 nm.

[0022] Furthermore, the spectrometer and camera are placed inside a constant temperature box to reduce the impact of temperature changes on the observation data.

[0023] Furthermore, the bare optical fiber and the pinhole camera probe are fixed on the pan-tilt head, maintaining the same observation angle and rotating synchronously with the pan-tilt head.

[0024] Furthermore, the multi-optical path switcher uses a switcher independently developed and produced by Nanjing Aigesafe Company that supports 8 channel selections.

[0025] Furthermore, the optical fiber adopts a 600-micron single-core optical fiber produced by Ocean Insight, a US company.

[0026] Furthermore, the cosine receiver is manufactured by Ocean Insight, USA, model CC-3.

[0027] Furthermore, the multi-angle pan-tilt module is composed of a rotating pan-tilt head and a pan-tilt head controller of model PTU-D46 produced by FLIR Corporation of the United States.

[0028] Furthermore, the pan-tilt head and the pan-tilt head controller are connected via a data cable, and the signal transmission interface thereof is RS232, which is converted into a USB interface through the transmission interface to achieve connection with the control module.

[0029] Furthermore, the pinhole camera-based non-imaging hyperspectral observation footprint positioning and visualization method further includes the following observation steps:

[0030] Step S1: When the set observation time arrives, the multi-angle pan / tilt module is started and rotated to a preset angle.

[0031] Specifically, the observation time parameters and preset angles can be set according to specific factors such as the environment, season, and coverage area. The observation time parameters may include parameters such as observation time, frequency, and duration. The preset angle can be obtained through the correspondence between the observation time and the preset angle recorded in the historical record information table, or adjusted through manual input by the user.

[0032] When the detection reaches the preset observation time T, such as 7 am, the control module is started, and the control module sends instructions to the multi-angle gimbal. According to the instructions of the control module, the multi-angle gimbal is driven to rotate according to the preset angle until it reaches the first preset observation angle. When the preset angle is reached, the control module controls the switch to the corresponding observation mode.

[0033] Furthermore, the control module may adopt the Android system or other software for controlling the observation module.

[0034] Step S2: Acquire downlink radiation data, optimize integration time and record dark current.

[0035] Specifically, obtaining downlink radiation data, optimizing the integration time, and recording dark current include the following steps:

[0036] In step S21, when the multi-angle gimbal rotates to a preset angle, the control module issues an instruction to use the optical path switching switch to switch the multi-optical path switch to the solar incident light observation optical path, and measures the downlink radiation from the sky to the ground through the cosine receiver to obtain the downlink radiation data.

[0037] Step S22: observe a solar incident spectrum according to the optimized integration time and record the data, then turn off the internal optical path switch of the spectrometer and record a dark current according to the optimized integration time.

[0038] The dark current is the noise data generated by the spectrometer itself when no light enters the spectrometer.

[0039] Step S3: Synchronously acquire uplink radiation and image data through a multi-optical path switch.

[0040] Specifically, synchronously acquiring uplink radiation and image data through a multi-optical path switch includes the following steps:

[0041] Step S31 : The multi-optical path switcher switches to the uplink ground object incident light observation optical path and the pinhole camera image acquisition optical path.

[0042] In step S32, the bare optical fiber measures the uplink radiation, transmits the optical signal to the spectrometer through the uplink ground object incident light observation optical path of the multi-optical path switcher, and transmits the pinhole camera collected image data to the camera through the pinhole camera image collection optical path of the multi-optical path switcher.

[0043] In step S33 , the spectrometer and the camera synchronously record the spectral data and the photo data.

[0044] Step S4: Process the collected data to achieve observation footprint positioning.

[0045] The collected data is processed to realize the observation footprint positioning, including the following steps:

[0046] Step S41 : sending the collected spectral data to the spectral data processing module in the control module, and sending the collected image data to the image processing module in the control module.

[0047] Step S42 , combining the hyperspectral data with the image data of the pinhole camera, and accurately locating the spatial footprint of the hyperspectral observation through image processing and spatial registration technology.

[0048] Furthermore, the image data can be processed by performing pre-processing such as denoising, contrast enhancement, and distortion correction on the acquired images through an image processing module to improve the image quality in preparation for subsequent spatial registration.

[0049] Spatial registration works by finding common feature points between spectral data and image data, spatially aligning them to ensure that observations from the same geographic location are consistent. In addition to the methods mentioned above, spatial registration of spectral data and visible light images can also be achieved through a variety of image vision-based algorithms, such as spatial registration based on image feature points, spatial registration based on similarity metrics between images, and image registration methods based on fusion depth.

[0050] Furthermore, the obtained spectral data are used to invert sunlight-induced chlorophyll fluorescence (SIF) and calculate vegetation reflectance index.

[0051] Furthermore, the spectral data after spatial registration is obtained, and the spectral data is combined with the image data after spatial positioning registration, and the calculated sunlight-induced chlorophyll fluorescence (SIF) inversion and vegetation reflectance index are marked at the corresponding spatial position.

[0052] Furthermore, the spectral data and image data after spatial registration can be combined, and according to the different landform conditions of the image data, the influence of factors such as terrain environment, vegetation type, soil type, and weather conditions on the calculation results of sunlight-induced chlorophyll fluorescence (SIF) inversion and vegetation reflectance index can be considered.

[0053] Step S5: Process the collected data to calculate phenological parameters.

[0054] Furthermore, the image data from the pinhole camera were used to calculate phenological parameters such as the greenness index GCC and vegetation coverage FVC of the observation area.

[0055] Step S6: Visualization result generation: The observation footprints and phenological parameters are visualized to generate images showing the spatial distribution and vegetation conditions of the observation area.

[0056] Furthermore, the extracted phenological parameters and observation footprints can be visualized through professional visualization software to generate images showing the spatial distribution and vegetation conditions of the observation area.

[0057] Furthermore, the algorithm set in the module can be used to update the visual image of the observation footprint in real time, accurately displaying the observation position and observation signal at each angle.

[0058] Furthermore, the time of this observation and the angle of the pan-tilt rotation are stored in the historical record information table, and the spectral data, image data, visualization display data, phenological parameters and other data obtained this time are stored.

[0059] Step S7: Determine whether the preset stop time has been reached. If not, repeat step S1, and the pan / tilt head continues to rotate to the next set angle, and continues a new round of observation and calculation until the preset stop observation time is reached.

[0060] Compared with the prior art, the advantages of the present invention are:

[0061] 1. By combining the non-imaging hyperspectral observation module with the pinhole camera, the spectral data obtained by the ultra-high spectral observation module is combined with the image data obtained by the pinhole camera, and the spatial footprint of the ultra-high spectral observation is accurately located through spatial registration technology.

[0062] 2. By controlling the horizontal and vertical rotation of the multi-angle gimbal, spectral data of different angles, different species, and different canopy heights in the observation area can be obtained, and image data can be obtained through the pinhole camera module. Based on the spectral data and image data, the corresponding phenological parameters such as sunlight-induced chlorophyll fluorescence (SIF) inversion, vegetation reflectance index, phenological parameters, greenness index GCC, and vegetation cover fraction FVC are calculated, providing more comprehensive information for the assessment of vegetation growth status and multi-faceted assessment of the carbon sequestration and health status of vegetation such as forests.

[0063] 3. By visualizing the observation footprints and storing and updating the visual images of the footprints in real time, the observation position and observation signal at each angle can be accurately displayed. The phenological parameters of the corresponding position can be presented through the visualization module, effectively visualizing the observation footprints and phenological parameters, providing intuitive and accurate information for fields such as ecological environment monitoring and agricultural assessment, and realizing dynamic monitoring and analysis of regional vegetation, and displaying the characteristics of vegetation coverage and its changes in real time.

[0064] 4. By setting the preset observation time and preset angle, and storing the observation time and preset angle after each observation, combined with the control module's control of the multi-angle pan / tilt head, and the synergy of cameras, spectrometers and other equipment, the observation time and observation angle can be accurately controlled to ensure the quality and accuracy of the observation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Non-imaging hyperspectral observation module based on pinhole camera

[0066] Figure 2 Operation flow chart of the non-imaging hyperspectral multi-angle observation module based on pinhole camera

[0067] Figure 3 Schematic diagram of module field observation

[0068] Figure 4 Actual field installation diagram of the module DETAILED DESCRIPTION

[0069] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings:

[0070] As attached Figure 1 As shown, the present invention is mainly composed of the following host parts and external devices;

[0071] The main unit part (in the dotted box) includes: constant temperature box: a spectrometer and a camera are set inside it. Due to the uncertainty of outdoor weather, the constant temperature box can ensure that the equipment works in a stable temperature environment. Circuit board and module control module: responsible for the overall control and data processing of the module. Multi-light path switcher module: connects the camera and spectrometer through optical fiber to realize the switching of multi-light path signals. External equipment includes: optical fiber: used to transmit optical signals. Pinhole camera probe: captures images of the observation area. Cosine receiver: fixed vertically upward, used to measure downlink radiation. Among them, the multi-light path switcher is connected to the camera and spectrometer through optical fiber; the camera and spectrometer communicate with the control module through a data cable. The bare optical fiber and the pinhole camera probe for measuring uplink radiation are fixed at the same angle position on the pan-tilt head, and the cosine receiver is fixed vertically upward (as shown in the attached Figure 4 shown).

[0072] Furthermore, it also includes: a non-imaging hyperspectral observation module, a pinhole camera module, a multi-angle gimbal module and a control module;

[0073] Among them, the multi-angle pan-tilt module adjusts the rotation angle to achieve multi-angle observation;

[0074] The pinhole camera module captures images of the observation area in real time, records spatial information, and sends the image data to the non-imaging hyperspectral observation module. The image data can be used to calculate the phenological parameters greenness index (GCC) and / or fractional vegetation cover (FVC);

[0075] The non-imaging hyperspectral observation module collects spectral data of the target area, which can be used to invert sunlight-induced chlorophyll fluorescence (SIF) and calculate vegetation reflectance index;

[0076] The non-imaging hyperspectral observation module sends the spectral data and image data to the control module;

[0077] The control module controls the non-imaging hyperspectral observation module and the pinhole camera module to synchronously collect spectral data and image data of the uplink radiation, and spatially aligns the spectral data and image data of the uplink radiation through the image processing module and spectral data processing module provided therein, marks the calculated sunlight-induced chlorophyll fluorescence (SIF), vegetation reflectance index, greenness index (GCC) and / or fractional vegetation cover (FVC) at the corresponding spatial position, generates footprint positioning data, and sends the footprint positioning data to the visualization display module;

[0078] The visual display module receives the footprint positioning data, and the control module controls the visual display module to display the footprint positioning data, and updates and stores the footprint positioning data in real time.

[0079] The non-imaging hyperspectral observation module includes at least a spectrometer, a multi-path switcher, a bare optical fiber, a cosine receiver, and optical fibers. These modules are used to collect spectral data from the target area, which can be used to invert sunlight-induced chlorophyll fluorescence (SIF) and calculate vegetation reflectance indices. The pinhole camera module includes at least a pinhole camera probe, optical fibers, a light-proof container, and a photosensitive element (such as a CCD or CMOS sensor). These modules are used to capture real-time images of the observation area, record spatial information, and calculate phenological parameters such as the greenness index (GCC) and fractional vegetation cover (FVC) from the image data. The control module includes at least an image processing module and a spectral data processing module, which are used to control the spectrometer and camera to collect, store, and analyze data. The multi-angle pan / tilt module includes at least a rotating pan / tilt, a pan / tilt controller, and corresponding transmission interfaces, which are used to support and adjust the device's angle for multi-angle observation.

[0080] Specifically, a bare optical fiber is fixed on a multi-angle pan / tilt head to measure the uplink radiation.

[0081] The cosine receiver is used to measure the downward radiation from the sky to the ground. To ensure that vertically incident radiation is received, it is fixed vertically upward next to the multi-angle gimbal.

[0082] The pinhole camera probe is fixed on a multi-angle pan-tilt head, maintaining the same observation angle as the bare optical fiber, and synchronously acquiring uplink radiation and photos of ground objects with the bare optical fiber.

[0083] The multi-path switcher is connected to the cosine receiver, spectrometer, camera, and pinhole camera probe via optical fibers. It switches between different observation modes. When observing downlink radiation, only one optical path is open, leading to the spectrometer. The downlink radiation measured by the cosine receiver is transmitted to the spectrometer via optical fiber to observe the incident solar spectrum. When observing uplink radiation, the spectrometer and camera optical paths are connected simultaneously, and the uplink radiation and pinhole camera image are recorded simultaneously.

[0084] The camera and spectrometer are connected to the control module via a data line to achieve data communication.

[0085] The multi-angle pan / tilt head and the pan / tilt head controller are connected via a data cable and connected to the control module.

[0086] The control module circuit board is equipped with corresponding circuit modules, including an image processing module, a spectral data processing module and a control module, which are used to process, analyze and store the collected images and spectral data, record the observation position, control the observation angle and manage the data acquisition process.

[0087] The multi-optical path switcher includes a solar incident light observation optical path, an upward ground object incident light observation optical path, and a pinhole camera image acquisition optical path.

[0088] On the basis of the above solution, the present invention can also make the following improvements:

[0089] Furthermore, the spectrometer is manufactured by Ocean Insight, a US company, and is a QEPRO spectrometer with a spectral range of 650-800 nm and a spectral resolution of 0.3 nm.

[0090] Furthermore, the spectrometer and camera are placed inside a constant temperature box to reduce the impact of temperature changes on the observation data.

[0091] Furthermore, the bare optical fiber and the pinhole camera probe are fixed on the pan-tilt head, maintaining the same observation angle and rotating synchronously with the pan-tilt head.

[0092] Furthermore, the multi-optical path switcher uses a switcher independently developed and produced by Nanjing Aigesafe Company that supports 8 channel selections.

[0093] Furthermore, the optical fiber adopts a 600-micron single-core optical fiber produced by Ocean Insight, a US company.

[0094] Furthermore, the cosine receiver is manufactured by Ocean Insight, USA, model CC-3.

[0095] Furthermore, the multi-angle pan-tilt module is composed of a rotating pan-tilt head and a pan-tilt head controller of model PTU-D46 produced by FLIR Corporation of the United States.

[0096] Furthermore, the pan-tilt head and pan-tilt head controller are connected via a data cable, and the signal transmission interface thereof is RS232, which is converted into a USB interface through the transmission interface to achieve connection with the Android control module.

[0097] Attachment Figure 2 is another embodiment of the present application:

[0098] Combined with attachment Figure 2 It can be seen that the operation process of the non-imaging hyperspectral multi-angle observation module based on the pinhole camera is mainly as follows:

[0099] Observation start: At the set observation time (for example, 7:00 a.m.), the multi-angle gimbal starts rotating according to the preset angle.

[0100] Data acquisition: For downlink radiation measurement: The control module issues instructions to obtain the sun's downlink radiation and optimizes the integration time to record data. For uplink radiation and image acquisition: Uplink radiation and photos of ground objects are synchronously obtained through bare optical fiber and pinhole camera.

[0101] Data processing and storage: The collected data is transmitted to the control module for storage, processing and visualization.

[0102] Loop observation: Repeat the above steps to obtain data from the next observation angle until the set stop observation time (for example, 6 pm) ends the observation.

[0103] Furthermore, the pinhole camera-based non-imaging hyperspectral observation footprint positioning and visualization method further includes the following observation steps:

[0104] Step S1: When the set observation time arrives, the multi-angle gimbal is activated and rotated to a preset angle.

[0105] Specifically, the observation time parameters and preset angles can be set according to specific factors such as the environment, season, and coverage area. The observation time parameters may include parameters such as observation time, frequency, and duration. The preset angle can be obtained through the correspondence between the observation time and the preset angle recorded in the historical record information table, or adjusted through manual input by the user.

[0106] When the detection reaches the preset observation time T, such as 7 am, the control module is started, and the control module sends instructions to the multi-angle gimbal. According to the instructions of the control module, the multi-angle gimbal is driven to rotate according to the preset angle until it reaches the first preset observation angle. When the preset angle is reached, the control module controls the switch to the corresponding observation mode.

[0107] Furthermore, the control module may adopt the Android system or other software for controlling the observation module.

[0108] Step S2: Acquire downlink radiation data, optimize integration time and record dark current.

[0109] Specifically, obtaining downlink radiation data, optimizing the integration time, and recording dark current include the following steps:

[0110] In step S21, when the multi-angle gimbal rotates to a preset angle, the control module issues an instruction to use the optical path switching switch to switch the multi-optical path switch to the solar incident light observation optical path, and measures the downlink radiation from the sky to the ground through the cosine receiver to obtain the downlink radiation data.

[0111] Step S22: observe a solar incident spectrum according to the optimized integration time and record the data, then turn off the internal optical path switch of the spectrometer and record a dark current according to the optimized integration time.

[0112] The dark current is the noise data generated by the spectrometer itself when no light enters the spectrometer.

[0113] Furthermore, the optimized integration time can be achieved using the following formula:

[0114] is the customized initial integration time, Record values ​​for a user-defined ideal spectrometer, For user-defined The maximum spectrometer record value of the spectrum collected within the time. At the same time, the maximum integration time is set to prevent the integration time from being infinite.

[0115] Step S3: Synchronously acquire uplink radiation and image data through a multi-optical path switch.

[0116] Specifically, synchronously acquiring uplink radiation and image data through a multi-optical path switch includes the following steps:

[0117] Step S31 : The multi-optical path switcher switches to the uplink ground object incident light observation optical path and the pinhole camera image acquisition optical path.

[0118] In step S32, the bare optical fiber measures the uplink radiation, transmits the optical signal to the spectrometer through the uplink ground object incident light observation optical path of the multi-optical path switcher, and transmits the pinhole camera collected image data to the camera through the pinhole camera image collection optical path of the multi-optical path switcher.

[0119] In step S33 , the spectrometer and the camera synchronously record the spectral data and the photo data.

[0120] Step S4: Process the collected data to achieve observation footprint positioning.

[0121] The collected data is processed to realize the observation footprint positioning, including the following steps:

[0122] Step S41 : sending the collected spectral data to the spectral data processing module in the control module, and sending the collected image data to the image processing module in the control module.

[0123] Step S42 , combining the hyperspectral data with the image data of the pinhole camera, and accurately locating the spatial footprint of the hyperspectral observation through image processing and spatial registration technology.

[0124] Furthermore, the image data can be processed by performing pre-processing such as denoising, contrast enhancement, and distortion correction on the acquired images through an image processing module to improve the image quality in preparation for subsequent spatial registration.

[0125] Furthermore, the obtained spectral data can also be used for the inversion of sunlight-induced chlorophyll fluorescence (SIF) and the calculation of vegetation reflectance index.

[0126] Furthermore, the specific spatial registration process is as follows:

[0127] S1. Extract depth information from the image to be registered and the reference image, generate a depth matrix corresponding to each image, and diffract to generate a grayscale image corresponding to each depth matrix.

[0128] S2. Extract feature points based on different scales for each grayscale image, and extract feature points based on a fixed slider for each depth matrix to obtain feature points in each grayscale image;

[0129] S3. Position-encode each grayscale image, and append a numerical matrix consisting of the position codes to the original grayscale image to obtain a grayscale image with the position codes appended thereto, and correspond the feature points to the grayscale image with the position codes appended thereto to obtain feature points in each grayscale image with the position codes appended thereto;

[0130] S4, converting the feature points in each grayscale image with the additional position code into a feature vector, and obtaining, through an attention mechanism, a first feature point pair that matches the grayscale image with the additional position code to be registered and the reference grayscale image with the additional position code;

[0131] S5. Obtain feature descriptors of the remaining unmatched feature points, and calculate the shortest Euclidean distance between the feature descriptors based on the depth information to obtain a second pair of feature points that matches each other between the original grayscale image to be registered and the original reference grayscale image;

[0132] S6. Calculate a corresponding affine transformation matrix based on the first feature point pair and the second feature point pair, and perform affine transformation according to the affine transformation matrix to complete the registration of the image to be registered and the reference image.

[0133] Spatial registration ensures that observations from the same geographic location can be matched by finding common feature points between spectral data and image data and spatially aligning them. In addition to the methods mentioned above, spatial registration of spectral data images and visible light images can also be achieved through a variety of image vision-based algorithms, such as spatial registration based on image feature points, spatial registration based on similarity metrics between images, and image registration methods based on fusion depth. Furthermore, the obtained spectral data is used for the inversion of sunlight-induced chlorophyll fluorescence (SIF) and the calculation of vegetation reflectance index.

[0134] Furthermore, the spectral data after spatial registration is obtained, and the spectral data is combined with the image data after spatial positioning registration, and the calculated sunlight-induced chlorophyll fluorescence (SIF) inversion and vegetation reflectance index are marked at the corresponding spatial position.

[0135] Furthermore, the spectral data and image data after spatial registration can be combined, and according to the different landform conditions of the image data, the influence of factors such as terrain environment, vegetation type, soil type, and weather conditions on the calculation results of sunlight-induced chlorophyll fluorescence (SIF) inversion and vegetation reflectance index can be considered.

[0136] Step S5: Process the collected data to calculate phenological parameters.

[0137] Furthermore, the image data from the pinhole camera were used to calculate phenological parameters such as the greenness index GCC and vegetation coverage FVC of the observation area.

[0138] Furthermore, GCC is a widely used indicator for measuring canopy greenness. The specific calculation process can be as follows:

[0139] S1: Preprocess the image data acquired by the pinhole camera.

[0140] S2: The model is trained through a neural network model to extract the region of interest (ROI) from the input image.

[0141] S3: Occlude the image region of interest (ROI) and calculate the RGB color channel information of each pixel.

[0142] S4: According to the formula Calculate the value of the average greenness index, where 、 、 are the averages of the RGB spectral bands of the pixels in the ROI, respectively.

[0143] Furthermore, FVC is an important indicator for measuring the degree of vegetation coverage on the ground surface. It is usually used to describe the percentage of the vertical projection area of ​​vegetation on the ground to the total area. It can be estimated by analyzing the image data obtained by the pinhole camera. The specific process is as follows:

[0144] S1: Preprocess the image data acquired by the pinhole camera.

[0145] S2: Calculate the Normalized Difference Vegetation Index (NDVI).

[0146] S3: The minimum and maximum values ​​of NDVI are selected to represent the extreme cases of soil and pure vegetation cover, respectively.

[0147] S4: According to the formula To calculate vegetation cover, 、 are the maximum and minimum NDVI values ​​in the region, respectively.

[0148] Step S6: Visualization result generation: The observation footprints and phenological parameters are visualized to generate images showing the spatial distribution and vegetation conditions of the observation area.

[0149] Furthermore, the extracted phenological parameters and observation footprints can be visualized through professional visualization software to generate images showing the spatial distribution and vegetation conditions of the observation area.

[0150] Furthermore, the algorithm set in the module can be used to update the visual image of the observation footprint in real time, accurately displaying the observation position and observation signal at each angle.

[0151] Furthermore, the time of this observation and the angle of the pan-tilt rotation are stored in the historical record information table, and the spectral data, image data, visualization display data, phenological parameters and other data obtained this time are stored.

[0152] Step S7: Determine whether the preset stop time has been reached. If not, repeat step S1, and the pan / tilt head continues to rotate to the next set angle, and continues a new round of observation and calculation until the preset stop observation time is reached.

[0153] Further, attached Figure 3 A schematic diagram of a module field observation is given, Figure 4 A schematic diagram of the module's field installation is provided. This module can be installed above the forest canopy, such as on a high tower or crane, based on the actual vegetation environment. By controlling the horizontal and vertical rotation of the pan / tilt, spectral data can be acquired from different angles, species, and canopy heights of the forest, thereby assessing the forest's carbon sequestration and health. The module's main unit can be placed below the tower, and external equipment can be installed on the top of the tower (e.g., attached). Figure 4 The bare optical fiber and pinhole camera probe are fixed on the gimbal, maintaining the same observation angle and rotating synchronously with the gimbal; the cosine receiver is fixed next to the gimbal, mounted vertically upward.

[0154] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications, replacements or improvements can be made without departing from the spirit and principles of the present invention, and all should be considered to fall within the scope of protection of the present invention.

Claims

1. A non-imaging hyperspectral observation footprint positioning and visualization method based on a pinhole camera, characterized in that: include: Non-imaging hyperspectral observation module, pinhole camera module, control module and multi-angle gimbal module; Among them, the multi-angle pan-tilt module adjusts the rotation angle to achieve multi-angle observation; The pinhole camera module captures images of the observation area in real time, records spatial information, and sends the image data to the non-imaging hyperspectral observation module. The image data can be used to calculate the phenological parameters greenness index (GCC) and / or fractional vegetation cover (FVC); The non-imaging hyperspectral observation module collects spectral data of the target area, which can be used to invert sunlight-induced chlorophyll fluorescence (SIF) and calculate vegetation reflectance index; The non-imaging hyperspectral observation module sends the spectral data and image data to the control module; The control module controls the non-imaging hyperspectral observation module and the pinhole camera module to synchronously collect spectral data and image data of the uplink radiation, and spatially aligns the spectral data and image data of the uplink radiation through the image processing module and spectral data processing module provided therein, marks the calculated sunlight-induced chlorophyll fluorescence (SIF), vegetation reflectance index, greenness index (GCC) and / or fractional vegetation cover (FVC) at the corresponding spatial position, generates footprint positioning data, and sends the footprint positioning data to the visualization display module; The visual display module receives the footprint positioning data, and the control module controls the visual display module to display the footprint positioning data, and updates and stores the footprint positioning data in real time; The spatial registration process is as follows: S1. Extract depth information from the image to be registered and the reference image, generate a depth matrix corresponding to each image, and diffract to generate a grayscale image corresponding to each depth matrix; S2. Extract feature points based on different scales for each grayscale image, and extract feature points based on a fixed slider for each depth matrix to obtain feature points in each grayscale image; S3. Position-encode each grayscale image, and append a numerical matrix consisting of the position codes to the original grayscale image to obtain a grayscale image with the position codes appended thereto, and correspond the feature points to the grayscale image with the position codes appended thereto to obtain feature points in each grayscale image with the position codes appended thereto; S4, converting the feature points in each grayscale image with the additional position code into a feature vector, and obtaining, through an attention mechanism, a first feature point pair that matches the grayscale image with the additional position code to be registered and the reference grayscale image with the additional position code; S5. Obtain feature descriptors of the remaining unmatched feature points, and calculate the shortest Euclidean distance between the feature descriptors based on the depth information to obtain a second pair of feature points that matches each other between the original grayscale image to be registered and the original reference grayscale image; S6. Calculate a corresponding affine transformation matrix based on the first feature point pair and the second feature point pair, and perform affine transformation according to the affine transformation matrix to complete the registration of the image to be registered and the reference image.

2. The method according to claim 1, characterized in that The non-imaging hyperspectral observation module includes at least a spectrometer, a multi-light path switcher, a bare optical fiber, a cosine receiver and an optical fiber, and is used to collect spectral data of the target area; The pinhole camera module comprises at least a pinhole camera probe, a photosensitive element and an optical fiber, and is used to capture images of the observation area in real time; The control module includes at least an image processing module and a spectral data processing module, which are used to control the collection, storage and analysis of spectral data and image data; The multi-angle pan-tilt module at least includes a rotating pan-tilt head, a pan-tilt head controller and a corresponding transmission interface, which is used to support and adjust the angle of the device to achieve multi-angle observation.

3. The method according to claim 2, characterized in that The multi-path switch is connected to the cosine receiver, the spectrometer, the camera and the pinhole camera probe respectively through optical fibers, and the control module controls the multi-path switch to switch between downlink radiation and uplink radiation observation modes.

4. The method according to claim 3, characterized in that The multi-optical path switcher is used to switch between different observation modes. When observing downlink radiation, only one optical path channel is opened to the spectrometer. The downlink radiation measured by the cosine receiver is transmitted to the spectrometer through optical fiber to observe the incident solar spectrum. When observing uplink radiation, the spectrometer optical path and the camera optical path are connected at the same time, and the spectral data of the uplink radiation and the image data collected by the pinhole camera module are recorded at the same time.

5. The method according to claim 2, characterized in that The control module uses the Android system to control the collection of spectral data and image data, and is connected to the pan / tilt controller via the RS232 interface to achieve multi-angle rotation control.

6. The method according to claim 2, characterized in that The bare optical fiber and the pinhole camera probe are fixed on the multi-angle platform, maintain the same observation angle, and rotate synchronously with the multi-angle platform.

7. The method according to claim 2, characterized in that The cosine receiver is used to measure downward radiation from the sky to the ground, and is fixed vertically upward next to the multi-angle pan-tilt platform.

8. The method according to any one of claims 1 to 7, characterized in that The pinhole camera-based non-imaging hyperspectral observation footprint positioning and visualization method further includes the following observation steps: Step S1: At the set observation time, start the multi-angle pan / tilt module and rotate it to a preset angle; Step S2: Acquire downlink radiation data, optimize integration time and record dark current; Step S3: synchronously acquiring uplink radiation and image data through a multi-optical path switch; Step S4: Processing the collected data to achieve observation footprint positioning; Step S5: Processing the collected data to calculate phenological parameters; Step S6: Visualization result generation: Visualize the observed footprints and phenological parameters to generate an image showing the spatial distribution and vegetation status of the observed area; Step S7: Determine whether the preset stop time has been reached. If not, repeat step S1, and the multi-angle gimbal continues to rotate to the next set angle, and continues a new round of observation and calculation until the preset stop observation time is reached.

Images