Non-imaging hyperspectral observation footprint positioning and visualization method based on pinhole camera

Abstract

The invention provides a non-imaging hyperspectral observation footprint positioning and visualization method based on a pinhole camera, and solves the problem that a traditional non-imaging spectrometer is difficult to position an observation footprint. The method comprises the steps that spectrum data and image data of uplink radiation are synchronously acquired through a non-imaging hyperspectral observation module and a pinhole camera module, and the spectrum data and the image data are sent to a control module by the non-imaging hyperspectral observation module; the control module carries out spatial registration on spectrum data and image data of uplink radiation through an image processing module and a spectrum data processing module which are arranged in the control module, accurately locates and observes footprints, calculates phenological parameters such as a greenness index and vegetation coverage, generates footprint positioning data, and sends the footprint positioning data to the server. The footprint positioning module is used for receiving footprint positioning data and sending the footprint positioning data to the visual display module; the method is suitable for ecological monitoring and carbon sink evaluation of complex environments such as forest canopies, has the characteristics of high efficiency, reliability and low cost, and provides accurate data support for agriculture and forestry and ecological research.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, and in particular to a non-imaging hyperspectral observation footprint positioning and visualization method based on a pinhole camera. Background Art

[0002] In the field of ecological environment monitoring and agricultural assessment, sunlight-induced chlorophyll fluorescence (SIF), as a direct indicator of plant photosynthesis, has become an important means of studying the physiological state of plants. Traditional SIF observations mainly rely on non-imaging spectrometers, such as ocean optical spectrometers, which can provide data with ultra-high spectral resolution and are suitable for SIF inversion and calculation of vegetation reflectance index. However, due to its non-imaging characteristics, existing technologies are insufficient in the spatial positioning and visualization of SIF observations. The lack of spatial information during the observation process makes it difficult to accurately locate the observation footprint, especially when performing multi-angle rotation observations, making it even more difficult to determine the specific observation objects.

[0003] In order to accurately locate and visualize the observation footprints and obtain more vegetation parameter information, the researchers used high-resolution cameras and spectrometers or non-imaging hyperspectral synchronous observations to obtain spatial information, and used image processing technology to locate and visualize the observation footprints. Summary of the invention

[0004] To achieve the above object, the present invention provides the following technical solutions: A pinhole camera-based non-imaging hyperspectral observation footprint positioning and visualization method, comprising: a non-imaging hyperspectral observation module, a pinhole camera module, a multi-angle pan-tilt module and a control module; wherein the multi-angle pan-tilt module adjusts the rotation angle to achieve multi-angle observation; The pinhole camera module takes images of the observation area in real time, records spatial information, and sends the image data to the non-imaging hyperspectral observation module. The image data can be used to calculate the phenological parameter greenness index (GCC) and / or fractional vegetation coverage (FVC); The non-imaging hyperspectral observation module collects spectral data of the target area, which can be used for the inversion of sunlight-induced chlorophyll fluorescence (SIF) and the calculation of vegetation reflectance index; The non-imaging hyperspectral observation module sends the spectral data and the image data to the control module; The control module controls the non-imaging hyperspectral observation module and the pinhole camera module to synchronously collect the spectral data and image data of the uplink radiation, and performs spatial registration on the spectral data and image data of the uplink radiation through the image processing module and the spectral data processing module provided therein, marks the calculated daylight-induced chlorophyll fluorescence (SIF), vegetation reflectance index, greenness index (GCC) and / or vegetation coverage (FVC) at the corresponding spatial position, generates footprint positioning data, and sends the footprint positioning data to the visualization display module; The visual display module receives the footprint positioning data, and the control module controls the visual display module to display the footprint positioning data, and updates and stores the footprint positioning data in real time.

[0005] Furthermore, the non-imaging hyperspectral observation module includes at least: a spectrometer, a multi-path switcher, a bare optical fiber, a cosine receiver, and an optical fiber, which are used to collect spectral data of the target area, which can be used for the inversion of sunlight-induced chlorophyll fluorescence (SIF) and the calculation of vegetation reflectance index. The pinhole camera module includes at least a pinhole camera probe, an optical fiber, an opaque container, and a photosensitive element (such as a CCD or CMOS sensor), which is used to capture images of the observation area in real time, record spatial information, and calculate phenological parameters such as greenness index (GCC) and vegetation coverage (FVC) through image data. The control module includes at least an image processing module and a spectral data processing module, which are used to control the spectrometer and camera to collect data, and store and analyze it. The multi-angle pan-tilt module includes at least a rotating pan-tilt, a pan-tilt controller, and corresponding transmission interfaces, etc., which are used to support and adjust the angle of the equipment to achieve multi-angle observation.

[0006] Specifically, the bare optical fiber is fixed on a multi-angle pan / tilt head to measure the uplink radiation.

[0007] The cosine receiver is used to measure the downward radiation from the sky to the ground. To ensure that vertically incident radiation is received, it is fixed vertically upward next to the multi-angle gimbal.

[0008] The pinhole camera probe is fixed on a multi-angle pan-tilt head, maintaining the same observation angle as the bare optical fiber, and synchronously acquiring the uplink radiation and photos of the ground objects with the bare optical fiber.

[0009] The multi-optical path switcher is connected to the cosine receiver, spectrometer, camera and pinhole camera probe through optical fibers. The multi-optical path switcher is used to switch between different observation modes. When observing downlink radiation, only one optical path channel is opened to the spectrometer. The downlink radiation measured by the cosine receiver is transmitted to the spectrometer through optical fiber to observe the incident solar spectrum. When observing uplink radiation, the spectrometer optical path and the camera optical path are connected at the same time, and the uplink radiation and pinhole camera photos are recorded at the same time.

[0010] The camera and the spectrometer are connected to the control module via a data line to achieve data communication.

[0011] The multi-angle pan / tilt and pan / tilt controller are connected via a data cable and connected to the control module.

[0012] The control module circuit board is provided with corresponding circuit modules, including an image processing module, a spectral data processing module and a control module, which are used to process, analyze and store the collected images and spectral data, record the observation position, control the observation angle and manage the data collection process.

[0013] The multi-optical path switch includes a solar incident light observation optical path, an upward ground object incident light observation optical path, and a pinhole camera image acquisition optical path.

[0014] On the basis of the above solution, the present invention can also make the following improvements: Furthermore, the spectrometer is manufactured by Ocean Insight, USA, model QEPRO spectrometer, with a spectral range of 650-800 nm and a spectral resolution of 0.3 nm.

[0015] Furthermore, the spectrometer and camera are placed inside a constant temperature box to reduce the impact of temperature changes on the observation data.

[0016] Furthermore, the bare optical fiber and the pinhole camera probe are fixed on the gimbal, maintaining the same observation angle and rotating synchronously with the gimbal.

[0017] Furthermore, the multi-optical path switcher adopts a switcher that supports 8 channel selections and is independently developed and produced by Nanjing Aigesef Company.

[0018] Furthermore, the optical fiber adopts a 600-micron single-core optical fiber produced by Ocean Insight, a US company.

[0019] Furthermore, the cosine receiver is manufactured by Ocean Insight, Inc. of the United States, model CC-3.

[0020] Furthermore, the multi-angle pan / tilt module is composed of a rotating pan / tilt head and a pan / tilt controller of model PTU-D46 produced by FLIR Corporation of the United States.

[0021] Furthermore, the pan / tilt and pan / tilt controller are connected via a data cable, and the signal transmission interface thereof is RS232, which is converted into a USB interface through the transmission interface to achieve connection with the control module.

[0022] Furthermore, the pinhole camera-based non-imaging hyperspectral observation footprint positioning and visualization method also includes the following observation steps: Step S1: When the set observation time arrives, the multi-angle gimbal module is started and rotated to a preset angle.

[0023] Specifically, the observation time parameters and preset angles can be set according to specific factors such as the environment, season, coverage area, etc. The observation time parameters may include parameters such as observation time, frequency, and duration. The preset angle can be obtained through the correspondence between the observation time and the preset angle recorded in the historical record information table, or adjusted through manual input by the user.

[0024] When the detection reaches the preset observation time T, such as 7 am, the control module is started, and the control module sends instructions to the multi-angle gimbal. According to the instructions of the control module, the multi-angle gimbal is driven to rotate according to the preset angle until it reaches the first preset observation angle. When the preset angle is reached, the control module controls the switch to the corresponding observation mode.

[0025] Furthermore, the control module may adopt the Android system or other software for controlling the observation module.

[0026] Step S2: Obtain downlink radiation data, optimize integration time and record dark current.

[0027] Specifically, obtaining downlink radiation data, optimizing integration time and recording dark current include the following steps: Step S21, when the multi-angle gimbal rotates to a preset angle, the control module issues a command to use the optical path switching switch to switch the multi-optical path switch to the solar incident light observation optical path, and measures the downlink radiation from the sky to the ground through the cosine receiver to obtain the downlink radiation data.

[0028] Step S22, observe a solar incident spectrum according to the optimized integration time and record the data, then turn off the internal optical path switch of the spectrometer, and record a dark current according to the optimized integration time.

[0029] The dark current is the noise data generated by the spectrometer itself when no light enters the spectrometer.

[0030] Step S3: synchronously acquiring uplink radiation and image data through a multi-optical path switch.

[0031] Specifically, synchronously acquiring uplink radiation and image data through a multi-optical path switch includes the following steps: Step S31, the multi-optical path switcher switches to the uplink ground object incident light observation optical path and the pinhole camera image acquisition optical path.

[0032] Step S32, the bare optical fiber measures the uplink radiation, transmits the optical signal to the spectrometer through the uplink ground object incident light observation optical path of the multi-optical path switcher, and transmits the pinhole camera collected image data to the camera through the pinhole camera image collection optical path of the multi-optical path switcher.

[0033] Step S33: the spectrometer and the camera synchronously record the spectrum data and the photo data.

[0034] Step S4: Process the collected data to achieve observation footprint positioning.

[0035] The collected data is processed to realize the observation footprint positioning, including the following steps: Step S41, sending the collected spectral data to the spectral data processing module in the control module, and sending the collected image data to the image processing module in the control module.

[0036] Step S42, combining the ultra-high spectral data with the image data of the pinhole camera, and accurately locating the spatial footprint of the ultra-high spectral observation through image processing and spatial registration technology.

[0037] Furthermore, the image data can be processed by performing pre-processing such as denoising, contrast enhancement, and distortion correction on the acquired images through an image processing module to improve the image quality in preparation for subsequent spatial registration.

[0038] Spatial registration ensures that the observation data of the two at the same geographical location can correspond by finding common feature points between spectral data and image data and aligning them in space. Among them, for the spatial registration of spectral data images and visible light images, in addition to the methods mentioned above, it can also be achieved through a variety of algorithms based on image vision, such as spatial registration based on image feature points, spatial registration based on similarity metrics between images, and image registration methods based on fusion depth.

[0039] Furthermore, the obtained spectral data are used for the inversion of sunlight-induced chlorophyll fluorescence (SIF) and the calculation of vegetation reflectance index.

[0040] Furthermore, the spectral data after spatial registration are obtained, and the spectral data are combined with the image data after spatial positioning registration, and the calculated daylight-induced chlorophyll fluorescence (SIF) inversion and vegetation reflectance index are marked at the corresponding spatial position.

[0041] Furthermore, the spectral data and image data after spatial registration can be combined to consider the influence of terrain environment, vegetation type, soil type, weather conditions and other factors on the calculation results of sunlight-induced chlorophyll fluorescence (SIF) inversion and vegetation reflectance index according to the different geomorphological conditions of the image data.

[0042] Step S5: Process the collected data to calculate phenological parameters.

[0043] Furthermore, the image data from the pinhole camera were used to calculate phenological parameters such as the greenness index GCC and vegetation coverage FVC of the observation area.

[0044] Step S6: Visualization result generation: Visualize the observation footprints and phenological parameters to generate images showing the spatial distribution and vegetation conditions of the observation area.

[0045] Furthermore, the extracted phenological parameters and observation footprints can be visualized through professional visualization software to generate images showing the spatial distribution and vegetation conditions of the observation area.

[0046] Furthermore, the algorithm set in the module can be used to update the visual image of the observation footprint in real time, accurately displaying the observation position and observation signal at each angle.

[0047] Furthermore, the time of this observation and the angle of the pan / tilt rotation are stored in the historical record information table, and the spectral data, image data, visualization display data, phenological parameters and other data obtained this time are stored.

[0048] Step S7: Determine whether the preset stop time has been reached. If not, repeat step S1, and the gimbal continues to rotate to the next set angle, and continues a new round of observation and calculation until the preset stop observation time is reached.

[0049] Compared with the prior art, the advantages of the present invention are: 1. By combining the non-imaging hyperspectral observation module with the pinhole camera, the spectral data obtained by the ultra-high spectral observation module is combined with the image data obtained by the pinhole camera, and the spatial footprint of the ultra-high spectral observation is accurately located through spatial registration technology.

[0050] 2. By controlling the horizontal and vertical rotation of the multi-angle gimbal, spectral data of different angles, different species, and different canopy heights in the observation area can be obtained, and image data can be obtained through the pinhole camera module. Through the spectral data and image data, the corresponding phenological parameters such as daylight-induced chlorophyll fluorescence (SIF) inversion, vegetation reflectance index, phenological parameters, greenness index GCC and vegetation coverage FVC are calculated, which provides more comprehensive information for the evaluation of vegetation growth conditions and multi-faceted evaluation of the carbon sink and health status of vegetation such as forests.

[0051] 3. The observation footprints are presented visually, and the visual images of the footprints are stored and updated in real time to accurately display the observation position and observation signal at each angle. The phenological parameters of the corresponding position are presented through the visualization module, and the observation footprints and phenological parameters are effectively visualized to provide intuitive and accurate information for fields such as ecological environment monitoring and agricultural assessment, and to realize dynamic monitoring and analysis of regional vegetation, and to display the characteristics of vegetation coverage and its changes in real time.

[0052] 4. By setting the preset observation time and preset angle, and storing the observation time and preset angle after each observation, combined with the control module's control of the multi-angle gimbal, and the synergy of cameras, spectrometers and other equipment, the observation time and observation angle can be accurately controlled to ensure the quality and accuracy of the observation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Non-imaging hyperspectral observation module based on pinhole camera Figure 2 Operation flow chart of non-imaging hyperspectral multi-angle observation module based on pinhole camera Figure 3 Module field observation diagram Figure 4 Module actual field installation diagram DETAILED DESCRIPTION

[0054] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings: As attached Figure 1 As shown, the present invention is mainly composed of the following host parts and external devices; The main unit part (within the dotted box) includes: Constant temperature box: a spectrometer and a camera are set inside it. Due to the uncertainty of outdoor weather, the constant temperature box can ensure that the equipment works in a stable temperature environment. Circuit board and module control module: responsible for the overall control and data processing of the module. Multi-light path switcher module: connects the camera and the spectrometer through optical fiber to realize the switching of multi-light path signals. External devices include: Optical fiber: used to transmit optical signals. Pinhole camera probe: captures images of the observation area. Cosine receiver: fixed vertically upward, used to measure downlink radiation. Among them, the multi-light path switcher is connected to the camera and spectrometer through optical fiber; the camera and spectrometer communicate with the control module through a data cable. The bare optical fiber and the pinhole camera probe for measuring the uplink radiation are fixed at the same angle position on the gimbal, and the cosine receiver is fixed vertically upward (as shown in the attached Figure 4 as shown).

[0055] Furthermore, it also includes: a non-imaging hyperspectral observation module, a pinhole camera module, a multi-angle gimbal module and a control module; Among them, the multi-angle gimbal module adjusts the rotation angle to achieve multi-angle observation; The pinhole camera module takes images of the observation area in real time, records spatial information, and sends the image data to the non-imaging hyperspectral observation module. The image data can be used to calculate the phenological parameter greenness index (GCC) and / or fractional vegetation coverage (FVC); The non-imaging hyperspectral observation module collects spectral data of the target area, which can be used for the inversion of sunlight-induced chlorophyll fluorescence (SIF) and the calculation of vegetation reflectance index; The non-imaging hyperspectral observation module sends the spectral data and the image data to the control module; The control module controls the non-imaging hyperspectral observation module and the pinhole camera module to synchronously collect the spectral data and image data of the uplink radiation, and performs spatial registration on the spectral data and image data of the uplink radiation through the image processing module and the spectral data processing module provided therein, marks the calculated daylight-induced chlorophyll fluorescence (SIF), vegetation reflectance index, greenness index (GCC) and / or vegetation coverage (FVC) at the corresponding spatial position, generates footprint positioning data, and sends the footprint positioning data to the visualization display module; The visual display module receives the footprint positioning data, and the control module controls the visual display module to display the footprint positioning data, and updates and stores the footprint positioning data in real time.

[0056] Among them, the non-imaging hyperspectral observation module at least includes: a spectrometer, a multi-path switcher, a bare optical fiber, a cosine receiver, and an optical fiber, which are used to collect spectral data of the target area, which can be used for the inversion of sunlight-induced chlorophyll fluorescence (SIF) and the calculation of vegetation reflectance index. The pinhole camera module at least includes a pinhole camera probe, an optical fiber, an opaque container, and a photosensitive element (such as a CCD or CMOS sensor), which is used to capture images of the observation area in real time, record spatial information, and calculate phenological parameters such as greenness index (GCC) and vegetation coverage (FVC) through image data. The control module at least includes an image processing module and a spectral data processing module, which are used to control the spectrometer and camera to collect data, and store and analyze it. The multi-angle pan-tilt module at least includes a rotating pan-tilt, a pan-tilt controller, and corresponding transmission interfaces, etc., which are used to support and adjust the angle of the equipment to achieve multi-angle observation.

[0057] Specifically, the bare optical fiber is fixed on a multi-angle pan / tilt head to measure the uplink radiation.

[0058] The cosine receiver is used to measure the downward radiation from the sky to the ground. To ensure that vertically incident radiation is received, it is fixed vertically upward next to the multi-angle gimbal.

[0059] The pinhole camera probe is fixed on a multi-angle pan-tilt head, maintaining the same observation angle as the bare optical fiber, and synchronously acquiring the uplink radiation and photos of the ground objects with the bare optical fiber.

[0060] The multi-optical path switcher is connected to the cosine receiver, spectrometer, camera and pinhole camera probe through optical fibers. The multi-optical path switcher is used to switch between different observation modes. When observing downlink radiation, only one optical path channel is opened to the spectrometer. The downlink radiation measured by the cosine receiver is transmitted to the spectrometer through optical fiber to observe the incident solar spectrum. When observing uplink radiation, the spectrometer optical path and the camera optical path are connected at the same time, and the uplink radiation and pinhole camera photos are recorded at the same time.

[0061] The camera and the spectrometer are connected to the control module via a data line to achieve data communication.

[0062] The multi-angle pan / tilt and pan / tilt controller are connected via a data cable and connected to the control module.

[0063] The control module circuit board is provided with corresponding circuit modules, including an image processing module, a spectral data processing module and a control module, which are used to process, analyze and store the collected images and spectral data, record the observation position, control the observation angle and manage the data collection process.

[0064] The multi-optical path switch includes a solar incident light observation optical path, an upward ground object incident light observation optical path, and a pinhole camera image acquisition optical path.

[0065] On the basis of the above solution, the present invention can also make the following improvements: Furthermore, the spectrometer is manufactured by Ocean Insight, USA, model QEPRO spectrometer, with a spectral range of 650-800 nm and a spectral resolution of 0.3 nm.

[0066] Furthermore, the spectrometer and camera are placed inside a constant temperature box to reduce the impact of temperature changes on the observation data.

[0067] Furthermore, the bare optical fiber and the pinhole camera probe are fixed on the gimbal, maintaining the same observation angle and rotating synchronously with the gimbal.

[0068] Furthermore, the multi-optical path switcher adopts a switcher that supports 8 channel selections and is independently developed and produced by Nanjing Aigesef Company.

[0069] Furthermore, the optical fiber adopts a 600-micron single-core optical fiber produced by Ocean Insight, a US company.

[0070] Furthermore, the cosine receiver is manufactured by Ocean Insight, Inc. of the United States, model CC-3.

[0071] Furthermore, the multi-angle pan / tilt module is composed of a rotating pan / tilt head and a pan / tilt controller of model PTU-D46 produced by FLIR Corporation of the United States.

[0072] Furthermore, the pan / tilt and pan / tilt controller are connected via a data cable, and the signal transmission interface thereof is RS232, which is converted into a USB interface through the transmission interface to achieve connection with the Android control module.

[0073] Attached Figure 2 is another embodiment of the present application: Combined with Figure 2 It can be seen that the operation process of the non-imaging hyperspectral multi-angle observation module based on the pinhole camera is mainly as follows: Observation start: At the set observation time (for example, 7 a.m.), the multi-angle gimbal starts rotating according to the preset angle.

[0074] Data acquisition: Downward radiation measurement: The control module issues instructions to obtain the sun's downward radiation and optimizes the integration time to record data. Upward radiation and image acquisition: The upward radiation and photos of the ground objects are synchronously obtained through bare optical fiber and pinhole camera.

[0075] Data processing and storage: The collected data is transmitted to the control module for storage, processing and visualization.

[0076] Loop observation: Repeat the above steps to obtain data from the next observation angle until the set stop observation time (for example, 6 pm) ends the observation.

[0077] Furthermore, the pinhole camera-based non-imaging hyperspectral observation footprint positioning and visualization method also includes the following observation steps: Step S1: When the set observation time arrives, the multi-angle gimbal is started and rotated to a preset angle.

[0078] Specifically, the observation time parameters and preset angles can be set according to specific factors such as the environment, season, coverage area, etc. The observation time parameters may include parameters such as observation time, frequency, and duration. The preset angle can be obtained through the correspondence between the observation time and the preset angle recorded in the historical record information table, or adjusted through manual input by the user.

[0079] When the detection reaches the preset observation time T, such as 7 am, the control module is started, and the control module sends instructions to the multi-angle gimbal. According to the instructions of the control module, the multi-angle gimbal is driven to rotate according to the preset angle until it reaches the first preset observation angle. When the preset angle is reached, the control module controls the switch to the corresponding observation mode.

[0080] Furthermore, the control module may adopt the Android system or other software for controlling the observation module.

[0081] Step S2: Obtain downlink radiation data, optimize integration time and record dark current.

[0082] Specifically, obtaining downlink radiation data, optimizing integration time and recording dark current include the following steps: Step S21, when the multi-angle gimbal rotates to a preset angle, the control module issues a command to use the optical path switching switch to switch the multi-optical path switch to the solar incident light observation optical path, and measures the downlink radiation from the sky to the ground through the cosine receiver to obtain the downlink radiation data.

[0083] Step S22, observe a solar incident spectrum according to the optimized integration time and record the data, then turn off the internal optical path switch of the spectrometer, and record a dark current according to the optimized integration time.

[0084] The dark current is the noise data generated by the spectrometer itself when no light enters the spectrometer.

[0085] Furthermore, the optimized integration time can be achieved using the following formula:

[0086] is the customized initial integration time, Records values ​​for a user-defined ideal spectrometer, For user-defined The maximum spectrometer record value of the spectrum collected within the time. At the same time, the maximum integration time is set to prevent the integration time from being infinite.

[0087] Step S3: synchronously acquiring uplink radiation and image data through a multi-optical path switch.

[0088] Specifically, synchronously acquiring uplink radiation and image data through a multi-optical path switch includes the following steps: Step S31, the multi-optical path switcher switches to the uplink ground object incident light observation optical path and the pinhole camera image acquisition optical path.

[0089] Step S32, the bare optical fiber measures the uplink radiation, transmits the optical signal to the spectrometer through the uplink ground object incident light observation optical path of the multi-optical path switcher, and transmits the pinhole camera collected image data to the camera through the pinhole camera image collection optical path of the multi-optical path switcher.

[0090] Step S33: the spectrometer and the camera synchronously record the spectrum data and the photo data.

[0091] Step S4: Process the collected data to achieve observation footprint positioning.

[0092] The collected data is processed to realize the observation footprint positioning, including the following steps: Step S41, sending the collected spectral data to the spectral data processing module in the control module, and sending the collected image data to the image processing module in the control module.

[0093] Step S42, combining the ultra-high spectral data with the image data of the pinhole camera, and accurately locating the spatial footprint of the ultra-high spectral observation through image processing and spatial registration technology.

[0094] Furthermore, the image data can be processed by performing pre-processing such as denoising, contrast enhancement, and distortion correction on the acquired images through an image processing module to improve the image quality in preparation for subsequent spatial registration.

[0095] Furthermore, the obtained spectral data can also be used for the inversion of sunlight-induced chlorophyll fluorescence (SIF) and the calculation of vegetation reflectance index.

[0096] Furthermore, the specific spatial registration process is as follows: S1. Extract depth information from the image to be registered and the reference image, generate a depth matrix corresponding to each image, and diffract to generate a grayscale image corresponding to each depth matrix.

[0097] S2, extracting feature points based on different scales for each grayscale image, and extracting feature points based on a fixed slider for each depth matrix, to obtain feature points in each grayscale image; S3, position coding each grayscale image, and attaching a numerical matrix composed of the position coding to the original grayscale image to obtain a grayscale image with additional position coding, and corresponding the feature points in the grayscale image with additional position coding to obtain feature points in each grayscale image with additional position coding; S4, converting feature points in each grayscale image with additional position coding into feature vectors, and obtaining first feature point pairs that match each other between the grayscale image with additional position coding to be registered and the grayscale image with additional position coding as a reference through an attention mechanism; S5, obtaining feature descriptors of the remaining unmatched feature points, and calculating the shortest Euclidean distance between the feature descriptors according to the depth information, to obtain a second feature point pair that matches the original grayscale image to be registered and the original reference grayscale image; S6. Based on the first feature point pair and the second feature point pair, calculate the corresponding affine transformation matrix, and perform affine transformation according to the affine transformation matrix to complete the registration of the image to be registered and the reference image.

[0098] Spatial registration ensures that the observation data of the two at the same geographical location can correspond by finding common feature points between spectral data and image data and aligning spectral data with image data in space. Among them, for the spatial registration of spectral data images and visible light images, in addition to the methods mentioned above, it can also be achieved through a variety of algorithms based on image vision, such as spatial registration based on image feature points, spatial registration based on similarity metrics between images, and image registration methods based on fusion depth. Furthermore, the obtained spectral data is used for the inversion of sunlight-induced chlorophyll fluorescence (SIF) and the calculation of vegetation reflectance index.

[0099] Furthermore, the spectral data after spatial registration are obtained, and the spectral data are combined with the image data after spatial positioning registration, and the calculated daylight-induced chlorophyll fluorescence (SIF) inversion and vegetation reflectance index are marked at the corresponding spatial position.

[0100] Furthermore, the spectral data and image data after spatial registration can be combined to consider the influence of terrain environment, vegetation type, soil type, weather conditions and other factors on the calculation results of sunlight-induced chlorophyll fluorescence (SIF) inversion and vegetation reflectance index according to the different geomorphological conditions of the image data.

[0101] Step S5: Process the collected data to calculate phenological parameters.

[0102] Furthermore, the image data from the pinhole camera were used to calculate phenological parameters such as the greenness index GCC and vegetation coverage FVC of the observation area.

[0103] Furthermore, GCC is a widely used indicator for measuring canopy greenness, and the specific calculation process can be as follows: S1: Preprocess the image data acquired by the pinhole camera.

[0104] S2: Train the model through the neural network model and extract the region of interest ROI from the input image.

[0105] S3: Mask the region of interest (ROI) of the image and calculate the RGB color channel information of each pixel.

[0106] S4: According to the formula Calculate the value of the average greenness index, where , , are the averages of the RGB spectral bands of the pixels in the ROI, respectively.

[0107] Furthermore, FVC is an important indicator to measure the coverage of vegetation on the ground surface. It is usually used to describe the percentage of the vertical projection area of ​​vegetation on the ground to the total area. It can be estimated by analyzing the image data obtained by the pinhole camera. The specific process is as follows: S1: Preprocess the image data acquired by the pinhole camera.

[0108] S2: Calculate the Normalized Difference Vegetation Index (NDVI).

[0109] S3: The minimum and maximum values ​​of NDVI are selected to represent the extreme cases of soil and pure vegetation cover, respectively. S4: According to the formula To calculate vegetation coverage, , are the maximum and minimum NDVI values ​​in the region, respectively.

[0110] Step S6: Visualization result generation: Visualize the observation footprints and phenological parameters to generate images showing the spatial distribution and vegetation conditions of the observation area.

[0111] Furthermore, the extracted phenological parameters and observation footprints can be visualized through professional visualization software to generate images showing the spatial distribution and vegetation conditions of the observation area.

[0112] Furthermore, the algorithm set in the module can be used to update the visual image of the observation footprint in real time, accurately displaying the observation position and observation signal at each angle.

[0113] Furthermore, the time of this observation and the angle of the pan / tilt rotation are stored in the historical record information table, and the spectral data, image data, visualization display data, phenological parameters and other data obtained this time are stored.

[0114] Step S7: Determine whether the preset stop time has been reached. If not, repeat step S1, and the gimbal continues to rotate to the next set angle, and continues a new round of observation and calculation until the preset stop observation time is reached.

[0115] Further, attached Figure 3 A schematic diagram of a module field observation is given, and Figure 4 A schematic diagram of field installation of the module is given. The module can be installed above the forest canopy, such as a high tower or a crane, in combination with the actual vegetation environment. By controlling the horizontal and vertical rotation of the pan / tilt, spectral data of different angles, species, and canopy heights of the forest can be acquired, thereby evaluating the carbon sink and health of the forest. The host of the module can be placed below the tower, and the external equipment can be installed on the top of the tower (such as the attached Figure 4The bare optical fiber and pinhole camera probe are fixed on the gimbal, maintaining the same observation angle and rotating synchronously with the gimbal; the cosine receiver is fixed next to the gimbal and installed vertically upward.

[0116] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications, replacements or improvements can be made without departing from the spirit and principle of the present invention, and all should be deemed to fall within the scope of protection of the present invention.

Claims

1. A non-imaging hyperspectral observation footprint positioning and visualization method based on a pinhole camera, characterized in that: include: Non-imaging hyperspectral observation module, pinhole camera module, control module and multi-angle gimbal module; Among them, the multi-angle gimbal module adjusts the rotation angle to achieve multi-angle observation; The pinhole camera module takes images of the observation area in real time, records spatial information, and sends the image data to the non-imaging hyperspectral observation module. The image data can be used to calculate the phenological parameter greenness index (GCC) and / or fractional vegetation coverage (FVC); The non-imaging hyperspectral observation module collects spectral data of the target area, which can be used for the inversion of sunlight-induced chlorophyll fluorescence (SIF) and the calculation of vegetation reflectance index; The non-imaging hyperspectral observation module sends the spectral data and the image data to the control module; The control module controls the non-imaging hyperspectral observation module and the pinhole camera module to synchronously collect the spectral data and image data of the uplink radiation, and performs spatial registration on the spectral data and image data of the uplink radiation through the image processing module and the spectral data processing module provided therein, marks the calculated daylight-induced chlorophyll fluorescence (SIF), vegetation reflectance index, greenness index (GCC) and / or vegetation coverage (FVC) at the corresponding spatial position, generates footprint positioning data, and sends the footprint positioning data to the visualization display module; The visual display module receives the footprint positioning data, and the control module controls the visual display module to display the footprint positioning data, and updates and stores the footprint positioning data in real time.

2. The method according to claim 1, characterized in that The non-imaging hyperspectral observation module includes at least a spectrometer, a multi-light path switcher, a bare optical fiber, a cosine receiver and an optical fiber, and is used to collect spectral data of the target area; The pinhole camera module at least includes a pinhole camera probe, a photosensitive element and an optical fiber, and is used to capture images of the observation area in real time; The control module includes at least an image processing module and a spectral data processing module, which are used to control the collection, storage and analysis of spectral data and image data; The multi-angle pan-tilt module at least includes a rotating pan-tilt, a pan-tilt controller and a corresponding transmission interface, which are used to support and adjust the angle of the device to achieve multi-angle observation.

3. The method according to claim 2, characterized in that The multi-light path switch is respectively connected to the cosine receiver, the spectrometer, the camera and the pinhole camera probe through optical fibers, and the control module controls the multi-light path switch to switch between the downlink radiation and uplink radiation observation modes.

4. The method according to claim 3, characterized in that The multi-optical path switcher is used to switch between different observation modes. When observing downlink radiation, only one optical path channel is opened to the spectrometer. The downlink radiation measured by the cosine receiver is transmitted to the spectrometer through optical fiber to observe the incident solar spectrum. When observing uplink radiation, the spectrometer optical path and the camera optical path are connected at the same time, and the spectral data of the uplink radiation and the image data collected by the pinhole camera module are recorded at the same time.

5. The method according to claim 2, characterized in that: The control module adopts the Android system to control the collection of spectral data and image data, and is connected to the pan / tilt controller through an RS232 interface to achieve multi-angle rotation control.

6. The method according to claim 2, characterized in that The bare optical fiber and the pinhole camera probe are fixed on the multi-angle pan-tilt platform, maintain the same observation angle, and rotate synchronously with the multi-angle pan-tilt platform.

7. The method according to claim 2, characterized in that: The cosine receiver is used to measure the downward radiation from the sky to the ground, and is fixed vertically upward beside the multi-angle pan-tilt platform.

8. The method according to any one of claims 1 to 7, characterized in that: The pinhole camera-based non-imaging hyperspectral observation footprint positioning and visualization method also includes the following observation steps: Step S1: At the set observation time, start the multi-angle gimbal module and rotate it to a preset angle; Step S2: Obtain downlink radiation data, optimize integration time and record dark current; Step S3: synchronously acquiring uplink radiation and image data through a multi-optical path switch; Step S4: Process the collected data to achieve observation footprint positioning; Step S5: Process the collected data to calculate phenological parameters; Step S6: Visualization result generation: Visualize the observed footprints and phenological parameters to generate images showing the spatial distribution and vegetation conditions of the observed area; Step S7: Determine whether the preset stop time has been reached. If not, repeat step S1, and the multi-angle gimbal continues to rotate to the next set angle, and continues a new round of observation and calculation until the preset stop observation time is reached.

Images