Device and method for extracting spatial position and reflectivity information based on hyperspectral camera

Abstract

The present application discloses a device and method for extracting spatial position and reflectivity information based on a hyperspectral camera, relating to the field of hyperspectral imaging, including: a host computer, a three-dimensional imaging module, a sliding module, a pan-tilt head and a connecting frame; the three-dimensional imaging module, the connecting frame, the pan-tilt head and the sliding module are connected in sequence. The three-dimensional imaging module includes a first line-scanning hyperspectral camera and a second line-scanning hyperspectral camera; the host computer is respectively connected to the first line-scanning hyperspectral camera and the second line-scanning hyperspectral camera; the host computer determines the three-dimensional spatial position information of the target object based on the characteristic bands according to the first hyperspectral image and the second hyperspectral image; determines the spectral irradiance of the 100% white board by using the white board height correction formula according to the height value of the target object; and determines the reflectivity of the target object by using the reflectivity calculation formula. The present application can adapt to the measurement of the reflectivity of objects at different heights in different light intensity environments without the need for additional equipment for data fusion.

Description

Technical Field

[0001] The present application relates to the field of hyperspectral imaging, and particularly to an apparatus and method for extracting spatial position and reflectance information based on a hyperspectral camera. Background Art

[0002] Currently, three-dimensional space information acquisition technology and hyperspectral imaging technology have been widely applied in multiple fields. However, there are still certain limitations in realizing synchronous acquisition of spatial information and spectral data. Usually, additional three-dimensional measurement devices (such as lidar or structured light sensors) are required to collect data with a hyperspectral camera, and then complex post-processing of the data is carried out to achieve the fusion of three-dimensional point clouds and spectral reflectance data. For example, a method for generating and calibrating a three-dimensional hyperspectral point cloud model of plants, which generates a three-dimensional hyperspectral point cloud model of plants based on the multimodal fusion of a multispectral camera and an RGB-D camera, but it requires an additional RGB-D camera and the registration calculation is complex. There are also related methods that create a 3D whiteboard database by integrating a hyperspectral camera and an RGB-D depth camera to correct the spatial position and tilt angle of the leaves, but its operation is complex and the calibration efficiency is low, which cannot meet the requirements of real-time applications.

[0003] Therefore, there are certain technical shortcomings in the existing hyperspectral imaging technology for obtaining three-dimensional information and reflectance information of the object to be measured. Summary of the Invention

[0004] The purpose of the present application is to provide an apparatus and method for extracting spatial position and reflectance information based on a hyperspectral camera to improve the measurement accuracy of the reflectance of objects at different heights in different light intensity environments.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides an apparatus for extracting spatial position and reflectance information based on a hyperspectral camera, including: a host computer, a three-dimensional imaging module, a sliding module, a pan-tilt head, and a connecting frame; the connecting frame is connected to the sliding module through the pan-tilt head; the connecting frame includes an upper plate and two side plates; the two side plates are respectively connected to both ends of the upper plate; the side plates are perpendicular to the upper plate;

[0007] The three-dimensional imaging module includes a first line-scanning hyperspectral camera, a second line-scanning hyperspectral camera, two fiber optic line light sources, and two free-stop hinges; the first line-scanning hyperspectral camera and the second line-scanning hyperspectral camera are fixed side by side on the lower surface of the upper plate, and the connection line between the lens centers of the first line-scanning hyperspectral camera and the second line-scanning hyperspectral camera is perpendicular to the movement direction; the fiber optic line light source is arranged below the side plate through the free-stop hinge;

[0008] The sliding module is used to adjust the position of the three-dimensional imaging module;

[0009] The pan-tilt is used to adjust the angle of the three-dimensional imaging module;

[0010] The fiber optic line light source is used to provide a set light intensity;

[0011] The first line-scanning hyperspectral camera is used to collect the first hyperspectral image of the target object under the set height and the set light intensity conditions; the second line-scanning hyperspectral camera is used to collect the second hyperspectral image of the target object under the set height and the set light intensity conditions;

[0012] The host computer is respectively connected to the first line-scanning hyperspectral camera and the second line-scanning hyperspectral camera; the host computer is used for:

[0013] Based on the first hyperspectral image and the second hyperspectral image, and based on the characteristic band, determine the height value of the target object, so as to obtain the three-dimensional spatial position information of the target object; the characteristic band is 956.4 nm;

[0014] According to the height value of the target object, use the whiteboard height correction formula to determine the 100% whiteboard spectral irradiance under the set height and the set light intensity conditions;

[0015] According to the target object spectral irradiance and the 100% whiteboard spectral irradiance under the set height and the set light intensity conditions, use the reflectivity calculation formula to determine the reflectivity of the target object; the target object spectral irradiance is determined according to the first hyperspectral image and the second hyperspectral image.

[0016] Optionally, the sliding module includes: a first servo motor, a motor controller, a guide rail slider, a guide rail fixing plate, a lead screw guide rail and a connecting plate;

[0017] The lead screw guide rail is fixed on the guide rail fixing plate; the first servo motor is connected to one end of the lead screw guide rail; the guide rail slider is threadedly connected to the lead screw guide rail; the connecting plate includes an upper connecting plate and a lower connecting plate; the upper connecting plate is fixed below the guide rail slider; the lower connecting plate and the upper connecting plate are connected by bolts; the lower connecting plate is connected to the pan-tilt below.

[0018] Optionally, the pan-tilt includes: a second servo motor, a pan-tilt bracket and a pan-tilt controller;

[0019] The second servo motor is installed on the pan-tilt bracket; the lower part of the pan-tilt bracket is fixedly connected to the upper plate; the pan-tilt controller is connected to the second servo motor; the pan-tilt controller is used to control the rotation of the second servo motor, and the rotation of the second servo motor drives the pan-tilt bracket to rotate so as to adjust the angle of the three-dimensional imaging module.

[0020] In a second aspect, the present application provides a method for extracting spatial position and reflectivity information based on a hyperspectral camera. The method for extracting spatial position and reflectivity information based on a hyperspectral camera is implemented based on the above-mentioned device for extracting spatial position and reflectivity information based on a hyperspectral camera; the method for extracting spatial position and reflectivity information based on a hyperspectral camera includes:

[0021] Obtain a first hyperspectral image and a second hyperspectral image;

[0022] Based on the first hyperspectral image and the second hyperspectral image, and based on the characteristic band, determine the height value of the target object, so as to obtain the three-dimensional spatial position information of the target object; the characteristic band is 956.4 nm;

[0023] According to the height value of the target object, use the whiteboard height correction formula to determine the 100% whiteboard spectral irradiance under the conditions of the set height and the set illumination intensity;

[0024] According to the spectral irradiance of the target object and the 100% whiteboard spectral irradiance under the conditions of the set height and the set illumination intensity, use the reflectivity calculation formula to determine the reflectivity of the target object; the spectral irradiance of the target object is determined according to the first hyperspectral image and the second hyperspectral image.

[0025] Optionally, the whiteboard height correction formula is:

[0026] ;

[0027] Wherein, is the whiteboard spectral irradiance of the i-th band; and represent the coefficient of the relationship between the spectral irradiance of the i-th band and the height; and represent the coefficient of the relationship between the spectral irradiance of the i-th band and the illumination intensity; is the average illumination intensity directly below the lens when the vertical distance from the lens plane of the hyperspectral camera is ; is the vertical distance from the lens plane of the hyperspectral camera at any light field environment with a height of The average light intensity directly below the camera at that time; H is the height value of the target object.

[0028] Optionally, based on the first hyperspectral image and the second hyperspectral image, and based on the characteristic band, determine the height value of the target object, so as to obtain the three-dimensional spatial position information of the target object, specifically including:

[0029] Based on the characteristic band, generate a first grayscale image and a second grayscale image from the first hyperspectral image and the second hyperspectral image respectively;

[0030] According to the first grayscale image and the second grayscale image, use a feature matching algorithm to determine the parallax;

[0031] According to the parallax, determine the height value of the target object, so as to obtain the three-dimensional spatial position information of the target object.

[0032] In a third aspect, the present application provides a system for extracting spatial position and reflectance information based on a hyperspectral camera, including:

[0033] A data acquisition module for acquiring a first hyperspectral image and a second hyperspectral image;

[0034] A height determination module for determining the height value of a target object based on the first hyperspectral image and the second hyperspectral image and based on a characteristic band, so as to obtain the three-dimensional spatial position information of the target object; the characteristic band is 956.4 nm;

[0035] A whiteboard height correction module for determining the 100% whiteboard spectral irradiance under the conditions of a set height and a set light intensity according to the height value of the target object by using a whiteboard height correction formula;

[0036] A reflectance determination module for determining the reflectance of the target object according to the spectral irradiance of the target object and the 100% whiteboard spectral irradiance under the conditions of the set height and the set light intensity by using a reflectance calculation formula; the spectral irradiance of the target object is determined according to the first hyperspectral image and the second hyperspectral image.

[0037] In a fourth aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the method for extracting spatial position and reflectance information based on a hyperspectral camera described in any one of the above.

[0038] Fifth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for extracting spatial position and reflectivity information based on a hyperspectral camera described in any one of the above is implemented.

[0039] Sixth aspect, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for extracting spatial position and reflectivity information based on a hyperspectral camera described in any one of the above is implemented.

[0040] According to the specific embodiments provided by the present application, the present application has the following technical effects:

[0041] The present application provides a device and method for extracting spatial position and reflectivity information based on a hyperspectral camera, including: a host computer, a three-dimensional imaging module, a sliding module, a pan-tilt head, and a connecting frame; the connecting frame is connected to the sliding module through the pan-tilt head; the three-dimensional imaging module is connected to the connecting frame. The three-dimensional imaging module includes a first line-scanning hyperspectral camera, a second line-scanning hyperspectral camera, two fiber optic line light sources, and two free-stop hinges; the first line-scanning hyperspectral camera and the second line-scanning hyperspectral camera are fixedly arranged side by side on the lower surface of the connecting frame, and the line connecting the lens centers of the first line-scanning hyperspectral camera and the second line-scanning hyperspectral camera is perpendicular to the moving direction; the fiber optic line light sources are arranged under the side plates of the connecting frame through the free-stop hinges; the first line-scanning hyperspectral camera acquires a first hyperspectral image, and the second line-scanning hyperspectral camera acquires a second hyperspectral image; the host computer is respectively connected to the first line-scanning hyperspectral camera and the second line-scanning hyperspectral camera; the host computer is configured to: determine the height value of the target object according to the first hyperspectral image and the second hyperspectral image, so as to obtain the three-dimensional spatial position information of the target object; determine the 100% white board spectral irradiance according to the height value of the target object by using the white board height correction formula; the target object spectral irradiance and the 100% white board spectral irradiance, and determine the reflectivity of the target object by using the reflectivity calculation formula. The present application adopts a dual-hyperspectral camera stereoscopic imaging technology, which can calculate the spatial information of the target while obtaining the spectral information of the measured object in a wider wavelength range, avoiding the system complexity and inaccurate registration problems brought by multi-source information fusion such as additional stereoscopic imaging devices in the past. The mathematical relationship between the change of the system spectral irradiance and the height and light intensity is explored, and a white board height correction method under different heights and light intensities is constructed, effectively improving the measurement accuracy of the reflectivity of objects at different heights in different light intensity environments, and realizing the output of four-dimensional data including three-dimensional spatial position information and reflectivity information with one acquisition. Description of the Drawings

[0042] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0043] Figure 1 Schematic diagram of a device for extracting spatial position and reflectivity information based on a hyperspectral camera;

[0044] Figure 2 Flowchart of the process for establishing the whiteboard height correction formula;

[0045] Figure 3 For collecting the light intensity at a height perpendicular to the camera lens plane with a height of Schematic diagram;

[0046] Figure 4 Flowchart of the height adaptive whiteboard correction method;

[0047] Figure 5 Fitting schematic diagram of the height and DN value at 549.91nm, 681.05nm, and 750.72nm in the first line-scanning hyperspectral camera;

[0048] Figure 6 Fitting schematic diagram of the height and DN value at 1199.54nm, 1449.44nm, and 1640.03nm in the second line-scanning hyperspectral camera;

[0049] Figure 7 Fitting schematic diagram of the light intensity and DN value at 549.91nm, 681.05nm, and 750.72nm in the first line-scanning hyperspectral camera;

[0050] Figure 8 Fitting schematic diagram of the light intensity and DN value at 1199.54nm, 1449.44nm, and 1640.03nm in the second line-scanning hyperspectral camera;

[0051] Figure 9 Schematic diagram of 3D point cloud generation image and correction;

[0052] Figure 10 Schematic diagram of the structure of a computer device provided by an embodiment of the present application.

[0053] Reference numerals: 1 - lead screw guide rail, 2 - first servo motor, 3 - first line-scanning hyperspectral camera, 4 - second line-scanning hyperspectral camera, 5 - free stop hinge, 6 - guide rail slider, 7 - connecting plate, 8 - pan-tilt head, 9 - connecting frame, 10 - fiber optic line light source. Detailed implementation manners

[0054] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0055] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below with reference to the drawings and specific implementation manners.

[0056] Existing hyperspectral imaging technologies have certain technical shortcomings in obtaining three-dimensional information and reflectance information of the object to be measured. To solve the above problems, the present application proposes a device and method for obtaining spatial position and reflectance information based on a hyperspectral camera. By combining two hyperspectral cameras with different wavelength ranges and stereoscopic imaging technology, it is possible to calculate the spatial information of the target while obtaining spectral information of the object to be measured in a wider wavelength range, avoiding the system complexity and inaccurate registration problems brought by additional multi-source information fusion such as stereoscopic imaging in the past. At the same time, the present application has explored the irradiance and reflectance characteristics of the system and developed a whiteboard spatial calibration method, effectively improving the measurement accuracy of the reflectance of objects at different heights in different light intensity environments and avoiding the process of separately collecting whiteboard data. The system can finally directly output four-dimensional data including three-dimensional spatial position information and reflectance information and has the characteristics of one-time acquisition and synchronous processing.

[0057] In an exemplary embodiment, as Figure 1 shown, a device for extracting spatial position and reflectance information based on a hyperspectral camera is provided, including: a host computer (not shown in Figure 1 ), a three-dimensional imaging module, a sliding module, a pan-tilt head 8, and a connecting frame 9; the connecting frame 9 is connected to the sliding module through the pan-tilt head 8; the three-dimensional imaging module is installed on the connecting frame 9 and is used to obtain hyperspectral image data of a sample to be measured (target object) and calculate spatial information; the sliding module performs linear sliding and position adjustment of the device. The connecting frame 9 includes an upper plate and two side plates; the two side plates are respectively connected to both ends of the upper plate; the side plates are perpendicular to the upper plate.

[0058] In practical applications, the device for extracting spatial position and reflectivity information based on a hyperspectral camera according to the present application includes a first line-scanning hyperspectral camera 3 (band range 400 - 1000 nm), a second line-scanning hyperspectral camera 4 (band range 900 - 1700 nm), a fiber optic line light source 10, a pan-tilt head 8, a free-stop hinge 5, a first servo motor 2, a lead screw guide rail 1, a guide rail slider 6, a connecting plate 7, and a connecting frame 9. In this device, the field of view angle (38°) and focal length (15 mm) of the lenses of the two line-scanning hyperspectral cameras are the same, and they are at the same height and the line connecting the centers of the lenses is perpendicular to the moving direction of the guide rail. During the scanning process, the first servo motor 2 operates to drive the guide rail slider 6 to move forward, and the guide rail slider 6 drives the two line-scanning hyperspectral cameras to move forward through the connecting plate 7. At the same time, the two line-scanning hyperspectral cameras start to scan line by line simultaneously at the same frequency to obtain the spectral information of the object, and the corresponding lines of each scan are controlled.

[0059] The three-dimensional imaging module includes a first line-scanning hyperspectral camera 3, a second line-scanning hyperspectral camera 4, two fiber optic line light sources 10, and two free-stop hinges 5; the first line-scanning hyperspectral camera 3 and the second line-scanning hyperspectral camera 4 are fixedly arranged side by side on the lower surface of the upper plate, maintaining the same height and being aligned on the same horizontal line, and the line connecting the center of the lens of the first line-scanning hyperspectral camera 3 and the center of the lens of the second line-scanning hyperspectral camera 4 is perpendicular to the moving direction; the fiber optic line light source 10 is arranged below the side plate through the free-stop hinge 5, and the free-stop hinge 5 can rotate to control the angle between the fiber optic line light source 10 and the horizontal direction.

[0060] The sliding module is used to adjust the position of the three-dimensional imaging module.

[0061] As an optional implementation manner, the sliding module includes: a first servo motor 2, a motor controller, a guide rail slider 6, a guide rail fixing plate, a lead screw guide rail 1, and a connecting plate 7.

[0062] The lead screw guide rail 1 is fixed on the guide rail fixing plate; the power output end of the first servo motor 2 is connected to one end of the lead screw guide rail 1; the motor controller is used to control the rotation of the first servo motor 2. The guide rail slider 6 is threadedly connected to the lead screw guide rail 1; the connecting plate 7 includes an upper connecting plate and a lower connecting plate; the upper connecting plate is fixed below the guide rail slider 6 and slides together with the guide rail slider 6; the lower connecting plate and the upper connecting plate are connected by bolts; the lower part of the lower connecting plate is connected to the pan-tilt head 8.

[0063] The pan-tilt head 8 is used to adjust the angle of the three-dimensional imaging module.

[0064] As an optional implementation manner, the pan-tilt head 8 includes: a second servo motor, a pan-tilt head bracket, and a pan-tilt head controller.

[0065] The second servo motor is installed on the pan-tilt bracket; the lower part of the pan-tilt bracket is fixedly connected to the upper plate; the pan-tilt controller is connected to the second servo motor; the pan-tilt controller is used to control the rotation of the second servo motor, and the rotation of the second servo motor drives the pan-tilt bracket to rotate, so as to adjust the angle of the connecting frame 9 and adjust the angle of the three-dimensional imaging module.

[0066] The optical fiber line light source 10 is used to provide a set light intensity.

[0067] The first line-scanning hyperspectral camera 3 is used to collect the first hyperspectral image of the target object under the conditions of a set height and the set light intensity; the second line-scanning hyperspectral camera 4 is used to collect the second hyperspectral image of the target object under the conditions of a set height and the set light intensity.

[0068] The host computer is respectively connected to the first line-scanning hyperspectral camera 3 and the second line-scanning hyperspectral camera 4; the host computer is used for:

[0069] Based on the first hyperspectral image and the second hyperspectral image, and based on the characteristic band, determine the height value of the target object, so as to obtain the three-dimensional spatial position information of the target object; the characteristic band is 956.4 nm.

[0070] According to the height value of the target object, use the whiteboard height correction formula to determine the 100% whiteboard spectral irradiance under the conditions of the set height and the set light intensity.

[0071] According to the target object spectral irradiance and the 100% whiteboard spectral irradiance under the conditions of the set height and the set light intensity, use the reflectivity calculation formula to determine the reflectivity of the target object; the target object spectral irradiance is determined according to the first hyperspectral image and the second hyperspectral image.

[0072] In an exemplary embodiment, the present application provides a method for extracting spatial position and reflectivity information based on a hyperspectral camera. The method for extracting spatial position and reflectivity information based on a hyperspectral camera is implemented based on the above-mentioned device for extracting spatial position and reflectivity information based on a hyperspectral camera; the method for extracting spatial position and reflectivity information based on a hyperspectral camera includes:

[0073] Obtain a first hyperspectral image and a second hyperspectral image.

[0074] Based on the first hyperspectral image and the second hyperspectral image, determine the height value of the target object based on the characteristic band, so as to obtain the three-dimensional spatial position information of the target object; the characteristic band is 956.4 nm.

[0075] Based on the first hyperspectral image and the second hyperspectral image, determine the height value of the target object based on the characteristic band, so as to obtain the three-dimensional spatial position information of the target object, specifically including:

[0076] Based on the characteristic band, generate a first grayscale image and a second grayscale image from the first hyperspectral image and the second hyperspectral image respectively.

[0077] Based on the first grayscale image and the second grayscale image, use a feature matching algorithm to determine the parallax.

[0078] Based on the parallax, determine the height value of the target object, so as to obtain the three-dimensional spatial position information of the target object.

[0079] Based on the height value of the target object, use the whiteboard height correction formula to determine the 100% whiteboard spectral irradiance under the conditions of the set height and the set light intensity.

[0080] The whiteboard height correction formula is:

[0081] 。

[0082] Where is the whiteboard spectral irradiance of the i-th band; and represent the coefficient of the relationship between the spectral irradiance of the i-th band and the height; and represent the coefficient of the relationship between the spectral irradiance of the i-th band and the light intensity; is the average light intensity directly below the lens when the vertical distance height between the high-spectral camera lens plane and the ground is ; is the average light intensity directly below the lens when the vertical distance height between the high-spectral camera lens plane and the ground is in any light field environment; H is the height value of the target object.

[0083] Based on the spectral irradiance of the target object and the 100% whiteboard spectral irradiance under the conditions of the set height and the set light intensity, use the reflectivity calculation formula to determine the reflectivity of the target object; the spectral irradiance of the target object is determined according to the first hyperspectral image and the second hyperspectral image.

[0084] The reflectivity calculation formula is:

[0085] 。

[0086] Among them, is the reflectivity of the target object; is the spectral irradiance of a 100% white board, is the spectral irradiance of the target object, which can be directly read from the first hyperspectral image and the second hyperspectral image, is the spectral irradiance of the dark field.

[0087] In practical applications, as Figure 2 shown, the establishment process of the white board height correction formula includes:

[0088] S1: Calibrate two hyperspectral cameras with different wavelength ranges based on a calibration board to obtain internal and external parameters and relative external parameters.

[0089] To ensure the accuracy of height measurement of the line-scanning hyperspectral camera, it is first necessary to calibrate the internal and external parameters of the two cameras, including focal length, optical center, rotation matrix, translation vector, and baseline distance. Since the line-scanning hyperspectral camera scans and images line by line, each line of the image has independent external parameters. Therefore, it is necessary to obtain the rotation matrix and the translation vector at the position of each pixel point of each line when the two cameras are shooting, where is the scanning line index number. In this camera system, the camera moves along the track in a uniform straight line. Therefore, the rotation matrix is a fixed value, which can be calculated from the tangent vector and normal vector of the track. The translation vector at the position of each pixel point of each line is derived from the geometric model of the line-scanning camera movement. The specific formula is that the translation vector of the th scanning line is . Among them, is the track direction, is the initial position, is the track movement speed, is the camera frame rate. The relative external parameters of the two cameras are , , where , , , are the rotation matrix and translation vector of the first line-scanning hyperspectral camera and the second line-scanning hyperspectral camera respectively, and the baseline distance between the two cameras is .

[0090] The internal parameters of the camera remain unchanged during the scanning process. During the calibration process, two line-scanning hyperspectral cameras start shooting simultaneously at the same frequency, and acquire hyperspectral images of the calibration board at multiple angles and distances. To eliminate the differences in optical and physical characteristics of the cameras in different bands, specific bands (such as 956.4 nm) within the overlapping band range (930 - 1000 nm) of the hyperspectral images of the calibration board obtained by the two cameras are synthesized into grayscale images and subjected to denoising preprocessing. Since the camera scans line by line and only some scan lines contain checkerboard corner points, the Harris corner detection combined with the sub-pixel accuracy optimization algorithm in the OpenCV library is used to extract the corner coordinates , and measure the actual physical coordinates of the corner points. For each scan line containing corner points, the correspondence between pixel coordinates and physical coordinates is established. The internal parameters of the camera are calculated through the projection formula , and the projection formula is , where is the scan line number, is the pixel column coordinate of the corner point in the th row, is the actual physical coordinate of the corner point, is the external parameter of the th row of the camera, is the internal parameter of the camera, , where and are the focal lengths in the camera movement direction and the direction perpendicular to it respectively, and are the optical center coordinates. The internal parameter results calculated for each scan line are averaged to obtain the global internal parameters of the two cameras.

[0091] S2: In a darkroom, measure the average light intensity directly below the lens when the vertical height from the lens plane of the hyperspectral camera is .

[0092] S3: Acquire the spectral data of a standard whiteboard at different heights, and fit the relationship between irradiance and height in this light field environment through a regression model.

[0093] S4: Acquire the spectral data of a standard whiteboard at different light intensities, and fit the relationship between irradiance and light intensity when the whiteboard is away from the camera through a regression model.

[0094] S5: Based on the relationships between spectral irradiance and height and light intensity for each band, a whiteboard height correction formula for any light field environment is constructed.

[0095] As Figure 3 and Figure 4 shown, in a darkroom, measure the average light intensity directly below the lens when the vertical distance height from the lens plane of the hyperspectral camera is ​ The spectral data of a 100% standard whiteboard at different heights from the camera were collected using a hyperspectral camera, as Figure 5 and Figure 6 shown. The spectral irradiance of each band of the spectral data collected by the two cameras in this light field environment was fitted using a regression model versus the height to obtain a relational expression . Among them, represents the band, and and represent the coefficients of the relational expressions between each band and the height . Further, the spectral data of a 100% standard whiteboard at a height of from the camera were collected using a hyperspectral camera under different light intensities, as shown. The spectral irradiance of each band of the spectral data collected by the two cameras at this height was fitted using a regression model Figure 7 and Figure 8 to obtain a relational expression versus the light intensity to obtain a relational expression . Among them, and represent the coefficients of the relational expressions between each band and the light intensity .

[0096] The coefficients of the relational expressions corresponding to each band are shown in Table 1.

[0097] Table 1 Coefficient table of relational expressions corresponding to each band

[0098]

[0099] Through experiments, it was found that the spectral irradiance of each band is linearly related to the height and the light intensity respectively. Based on this, a whiteboard height correction formula for any light field environment was constructed in this application: . Among them, is the average light intensity directly below the lens when the height perpendicular to the lens plane of the hyperspectral camera is in any light field environment. According to the whiteboard height correction formula, when the correction parameters , , and are measured, if the hyperspectral camera first measures before starting the measurement, and then calculates the height information of each pixel point on the hyperspectral image, the 100% standard whiteboard DN value of each pixel on the hyperspectral image at the same height information can be obtained, and then the whiteboard height adaptive correction can be realized.

[0100] S6: Calculate the three-dimensional information of the object based on the matching corresponding points and the triangulation method, combine the camera spectral data and substitute it into the correction formula to obtain the reflectivity, and finally obtain the spatial position and reflectivity information of the object to be measured.

[0101] In actual measurement, the samples to be measured (tea seedling plants) with different acquisition heights are collected. First, in the overlapping area within the wavelength range of 930 - 1000 nm by using two hyperspectral cameras, a certain wavelength band (such as 956.4 nm) is selected to generate the corresponding grayscale image. Subsequently, the feature matching algorithm in the OpenCV library is used to perform feature matching on the image, and the corresponding points between the same rows in the image are matched row by row to calculate the parallax. The parallax is , where and are the pixel positions of the corresponding matching points of the first line-scanning hyperspectral camera and the second line-scanning hyperspectral camera in the th row respectively. According to the depth formula , the height value of each matching point is calculated point by point . Finally, substitute it into the whiteboard height correction formula to obtain the 100% whiteboard spectral irradiance corresponding to the light intensity and height of the object. The whiteboard height correction formula: , and then according to the whiteboard correction formula (reflectivity calculation formula) , the object reflectivity is obtained. Among them, is the parameter automatically obtained by the camera. Through the whiteboard height correction, the spectral information can be dynamically adjusted according to the three-dimensional shape of the target object, so as to maintain the accuracy and stability of the spectral data under complex environments and diverse target height conditions. As Figure 9 shown, the spatial position and reflectivity information of the object to be measured of the four-dimensional data are finally obtained.

[0102] This application has the following advantages:

[0103] Adopting the stereo imaging technology generated by dual hyperspectral cameras can calculate the spatial information of the target while obtaining the spectral information of a wider wavelength band range of the object to be measured, avoiding the system complexity and inaccurate registration problems brought by the multi-source information fusion such as adding additional stereo imaging equipment in the past.

[0104] The mathematical relationship between the change of the system irradiation intensity and the height and light intensity is explored, and the automatic whiteboard correction method under different heights and light intensities is constructed, effectively improving the measurement accuracy of the reflectivity of objects at different heights under different light intensity environments, and realizing the output of four-dimensional data containing three-dimensional spatial position information and reflectivity information with one acquisition.

[0105] Based on the same inventive concept, an embodiment of the present application further provides a system for extracting spatial position and reflectivity information based on a hyperspectral camera for implementing the method for extracting spatial position and reflectivity information based on a hyperspectral camera involved above. The implementation solution provided by this system for solving problems is similar to the implementation solution described in the above method. Therefore, the specific limitations in the embodiment of the system for extracting spatial position and reflectivity information based on a hyperspectral camera provided below can refer to the limitations on the method for extracting spatial position and reflectivity information based on a hyperspectral camera in the above text, and will not be elaborated here.

[0106] In an exemplary embodiment, a system for extracting spatial position and reflectivity information based on a hyperspectral camera is provided, including:

[0107] A data acquisition module, configured to acquire a first hyperspectral image and a second hyperspectral image.

[0108] An altitude determination module, configured to determine the altitude value of the target object based on the characteristic band according to the first hyperspectral image and the second hyperspectral image, so as to obtain the three-dimensional spatial position information of the target object; the characteristic band is 956.4 nm.

[0109] A whiteboard altitude correction module, configured to determine the spectral irradiance of a 100% whiteboard under the conditions of the set altitude and the set light intensity by using a whiteboard altitude correction formula according to the altitude value of the target object.

[0110] A reflectivity determination module, configured to determine the reflectivity of the target object by using a reflectivity calculation formula according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the conditions of the set altitude and the set light intensity; the spectral irradiance of the target object is determined according to the first hyperspectral image and the second hyperspectral image.

[0111] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the method for extracting spatial position and reflectivity information based on a hyperspectral camera described above is implemented.

[0112] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the method for extracting spatial position and reflectivity information based on a hyperspectral camera described above is implemented.

[0113] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for extracting spatial position and reflectivity information based on a hyperspectral camera described above is implemented.

[0114] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal, and its internal structure diagram can be as shown in Figure 10 . The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for extracting spatial position and reflectivity information based on a hyperspectral camera.

[0115] Those skilled in the art can understand that Figure 10 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0116] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0117] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0118] The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0119] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0120] Specific examples are used in this article to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A device for extracting spatial position and reflectivity information based on a hyperspectral camera, characterized in that: include: A host computer, a three-dimensional imaging module, a sliding module, a pan-tilt and a connecting frame; the connecting frame is connected to the sliding module through the pan-tilt; the connecting frame includes an upper plate and two side plates; the two side plates are respectively connected to the two ends of the upper plate; the side plates are perpendicular to the upper plate; The three-dimensional imaging module includes a first line scanning hyperspectral camera, a second line scanning hyperspectral camera, two optical fiber line light sources and two free stop hinges; the first line scanning hyperspectral camera and the second line scanning hyperspectral camera are fixed side by side on the lower surface of the upper plate, and the line connecting the lens center of the first line scanning hyperspectral camera and the lens center of the second line scanning hyperspectral camera is perpendicular to the movement direction; the optical fiber line light source is arranged below the side plate through the free stop hinge; The sliding module is used to adjust the position of the three-dimensional imaging module; The pan / tilt platform is used to adjust the angle of the three-dimensional imaging module; The optical fiber line light source is used to provide a set light intensity; The first line scanning hyperspectral camera is used to collect a first hyperspectral image of the target object under the conditions of a set height and the set light intensity; the second line scanning hyperspectral camera is used to collect a second hyperspectral image of the target object under the conditions of a set height and the set light intensity; The host computer is connected to the first line scanning hyperspectral camera and the second line scanning hyperspectral camera respectively; the host computer is used for: According to the first hyperspectral image and the second hyperspectral image, based on characteristic bands, a height value of the target object is determined, thereby obtaining three-dimensional spatial position information of the target object; The characteristic band is 956.4nm; According to the height value of the target object, using a whiteboard height correction formula, determine the 100% whiteboard spectral irradiance under the set height and the set light intensity conditions; Determine the reflectivity of the target object using a reflectivity calculation formula according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the set height and the set light intensity conditions; The target object spectral irradiance is determined based on the first hyperspectral image and the second hyperspectral image.

2. The device for extracting spatial position and reflectivity information based on a hyperspectral camera according to claim 1, characterized in that: The sliding module comprises: a first servo motor, a motor controller, a guide rail slider, a guide rail fixing plate, a lead screw guide rail and a connecting plate; The lead screw guide is fixed on the guide fixing plate; the first servo motor is connected to one end of the lead screw guide; the guide rail slider is connected to the lead screw guide by threads; the connecting plate includes an upper connecting plate and a lower connecting plate; the upper connecting plate is fixed below the guide rail slider; the lower connecting plate and the upper connecting plate are connected by bolts; the lower connecting plate is connected to the pan-tilt head.

3. The device for extracting spatial position and reflectivity information based on a hyperspectral camera according to claim 2, characterized in that: The pan / tilt head comprises: a second servo motor, a pan / tilt head bracket and a pan / tilt head controller; The second servo motor is installed on the gimbal bracket; the bottom of the gimbal bracket is fixedly connected to the upper plate; the gimbal controller is connected to the second servo motor; the gimbal controller is used to control the rotation of the second servo motor, and the rotation of the second servo motor drives the gimbal bracket to rotate to adjust the angle of the three-dimensional imaging module.

4. A method for extracting spatial position and reflectivity information based on a hyperspectral camera, characterized in that: The method for extracting spatial position and reflectivity information based on a hyperspectral camera is implemented based on the device for extracting spatial position and reflectivity information based on a hyperspectral camera according to any one of claims 1 to 3; The method for extracting spatial position and reflectivity information based on a hyperspectral camera includes: Acquire a first hyperspectral image and a second hyperspectral image; According to the first hyperspectral image and the second hyperspectral image, based on a characteristic band, a height value of the target object is determined, thereby obtaining three-dimensional spatial position information of the target object; the characteristic band is 956.4 nm; According to the height value of the target object, using a whiteboard height correction formula, determine the 100% whiteboard spectral irradiance under the set height and the set light intensity conditions; The reflectivity of the target object is determined according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the set height and the set light intensity conditions using a reflectivity calculation formula; the spectral irradiance of the target object is determined according to the first hyperspectral image and the second hyperspectral image.

5. The method for extracting spatial position and reflectivity information based on a hyperspectral camera according to claim 4, characterized in that: The whiteboard height correction formula is: ； in, is the spectral irradiance of the white board in the i-th band; and The coefficient representing the relationship between the spectral irradiance and height of the i-th band; and Represents the coefficient of the relationship between the spectral irradiance and light intensity of the i-th band; The vertical distance height of the hyperspectral camera lens plane is The average light intensity directly below the lens at 100 Hz; is the vertical distance from the hyperspectral camera lens plane in any light field environment. is the average light intensity directly below the lens; H is the height of the target object.

6. The method for extracting spatial position and reflectivity information based on a hyperspectral camera according to claim 4, characterized in that: Determining the height value of the target object based on the first hyperspectral image and the second hyperspectral image and the characteristic band, thereby obtaining the three-dimensional spatial position information of the target object, specifically includes: Generate a first grayscale image and a second grayscale image from the first hyperspectral image and the second hyperspectral image based on characteristic bands; Determine the disparity by using a feature matching algorithm according to the first grayscale image and the second grayscale image; The height value of the target object is determined according to the parallax, thereby obtaining the three-dimensional spatial position information of the target object.

7. A spatial position and reflectivity information extraction system based on a hyperspectral camera, characterized in that: include: A data acquisition module, used for acquiring a first hyperspectral image and a second hyperspectral image; A height determination module, used to determine the height value of the target object based on the first hyperspectral image and the second hyperspectral image and a characteristic band, so as to obtain the three-dimensional spatial position information of the target object; the characteristic band is 956.4nm; A whiteboard height correction module is used to determine the 100% whiteboard spectral irradiance under the conditions of a set height and a set light intensity according to the height value of the target object using a whiteboard height correction formula; A reflectivity determination module, used to determine the reflectivity of the target object according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the conditions of the set height and the set light intensity, using a reflectivity calculation formula; The target object spectral irradiance is determined based on the first hyperspectral image and the second hyperspectral image.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for extracting spatial position and reflectivity information based on a hyperspectral camera according to any one of claims 4 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting spatial position and reflectivity information based on a hyperspectral camera according to any one of claims 4 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for extracting spatial position and reflectivity information based on a hyperspectral camera according to any one of claims 4 to 6 is implemented.

Images