Spatial position and reflectivity information extraction device and method based on hyperspectral camera

Abstract

The invention discloses a spatial position and reflectivity information extraction device and method based on a hyperspectral camera, and relates to the field of hyperspectral imaging, and the device comprises an upper computer, a three-dimensional imaging module, a sliding module, a holder and a connecting frame. The three-dimensional imaging module, the connecting frame, the holder and the sliding module are connected in sequence. The three-dimensional imaging module comprises a first line scanning hyperspectral camera and a second line scanning hyperspectral camera; the upper computer is respectively connected with the first line scanning hyperspectral camera and the second line scanning hyperspectral camera; the upper computer determines three-dimensional space position information of the target object based on the characteristic wave band according to the first hyperspectral image and the second hyperspectral image; according to the height value of the target object, determining 100% white board spectral irradiance by using a white board height correction formula; and determining the reflectivity of the target object by using a reflectivity calculation formula. The method can adapt to object reflectivity measurement in different light intensity environments and at different heights, and extra equipment is not needed for data fusion.

Description

Technical Field

[0001] The present application relates to the field of hyperspectral imaging, and in particular to a device and method for extracting spatial position and reflectivity information based on a hyperspectral camera. Background Art

[0002] At present, three-dimensional spatial information acquisition technology and hyperspectral imaging technology have been widely used in many fields. However, there are still certain limitations in realizing the synchronous collection of spatial information and spectral data. It is often necessary to use additional three-dimensional measurement equipment (such as lidar or structured light sensor) and hyperspectral camera for data collection, and then realize the fusion of three-dimensional point cloud and spectral reflectance data through complex post-data processing. For example, a method for generating and correcting a three-dimensional hyperspectral point cloud model of a plant is based on a three-dimensional hyperspectral point cloud model generated by multimodal fusion of a multispectral camera and an RGB-D camera, but it requires the addition of an RGB-D camera and the registration calculation is complex. There are also related methods that create a 3D whiteboard database by integrating a hyperspectral camera and an RGB-D depth camera to correct the spatial position and tilt angle of the leaves, but its operation is complicated and the correction efficiency is low, which cannot meet the needs of real-time applications.

[0003] Therefore, the existing hyperspectral imaging technology has certain technical shortcomings in obtaining three-dimensional information and reflectivity information of the object being measured. Summary of the invention

[0004] The purpose of this application is to provide a device and method for extracting spatial position and reflectivity information based on a hyperspectral camera to improve the reflectivity measurement accuracy of objects at different heights under different light intensity environments.

[0005] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides a device for extracting spatial position and reflectivity information based on a hyperspectral camera, comprising: a host computer, a three-dimensional imaging module, a sliding module, a pan-tilt and a connecting frame; the connecting frame is connected to the sliding module through the pan-tilt; the connecting frame comprises an upper plate and two side plates; the two side plates are respectively connected to the two ends of the upper plate; the side plates are perpendicular to the upper plate; The three-dimensional imaging module includes a first line scanning hyperspectral camera, a second line scanning hyperspectral camera, two optical fiber line light sources and two free stop hinges; the first line scanning hyperspectral camera and the second line scanning hyperspectral camera are fixed side by side on the lower surface of the upper plate, and the line connecting the lens center of the first line scanning hyperspectral camera and the lens center of the second line scanning hyperspectral camera is perpendicular to the movement direction; the optical fiber line light source is arranged below the side plate through the free stop hinge; The sliding module is used to adjust the position of the three-dimensional imaging module; The pan / tilt platform is used to adjust the angle of the three-dimensional imaging module; The optical fiber line light source is used to provide a set light intensity; The first line scanning hyperspectral camera is used to collect a first hyperspectral image of the target object under the conditions of a set height and the set light intensity; the second line scanning hyperspectral camera is used to collect a second hyperspectral image of the target object under the conditions of a set height and the set light intensity; The host computer is connected to the first line scanning hyperspectral camera and the second line scanning hyperspectral camera respectively; the host computer is used for: According to the first hyperspectral image and the second hyperspectral image, based on a characteristic band, a height value of the target object is determined, thereby obtaining three-dimensional spatial position information of the target object; the characteristic band is 956.4 nm; According to the height value of the target object, using a whiteboard height correction formula, determine the 100% whiteboard spectral irradiance under the set height and the set light intensity conditions; The reflectivity of the target object is determined according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the set height and the set light intensity conditions using a reflectivity calculation formula; the spectral irradiance of the target object is determined according to the first hyperspectral image and the second hyperspectral image.

[0006] Optionally, the sliding module includes: a first servo motor, a motor controller, a guide rail slider, a guide rail fixing plate, a lead screw guide rail and a connecting plate; The lead screw guide is fixed on the guide fixing plate; the first servo motor is connected to one end of the lead screw guide; the guide rail slider is connected to the lead screw guide by threads; the connecting plate includes an upper connecting plate and a lower connecting plate; the upper connecting plate is fixed below the guide rail slider; the lower connecting plate and the upper connecting plate are connected by bolts; the lower connecting plate is connected to the pan-tilt head.

[0007] Optionally, the pan-tilt head comprises: a second servo motor, a pan-tilt head bracket and a pan-tilt head controller; The second servo motor is installed on the gimbal bracket; the bottom of the gimbal bracket is fixedly connected to the upper plate; the gimbal controller is connected to the second servo motor; the gimbal controller is used to control the rotation of the second servo motor, and the rotation of the second servo motor drives the gimbal bracket to rotate to adjust the angle of the three-dimensional imaging module.

[0008] In a second aspect, the present application provides a method for extracting spatial position and reflectance information based on a hyperspectral camera, wherein the method for extracting spatial position and reflectance information based on a hyperspectral camera is implemented based on the above-mentioned device for extracting spatial position and reflectance information based on a hyperspectral camera; the method for extracting spatial position and reflectance information based on a hyperspectral camera comprises: Acquire a first hyperspectral image and a second hyperspectral image; According to the first hyperspectral image and the second hyperspectral image, based on a characteristic band, a height value of the target object is determined, thereby obtaining three-dimensional spatial position information of the target object; the characteristic band is 956.4 nm; According to the height value of the target object, using a whiteboard height correction formula, determine the 100% whiteboard spectral irradiance under the set height and the set light intensity conditions; The reflectivity of the target object is determined according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the set height and the set light intensity conditions using a reflectivity calculation formula; the spectral irradiance of the target object is determined according to the first hyperspectral image and the second hyperspectral image.

[0009] Optionally, the whiteboard height correction formula is: ; in, is the spectral irradiance of the white board in the i-th band; and The coefficient representing the relationship between the spectral irradiance and height of the i-th band; and Represents the coefficient of the relationship between the spectral irradiance and light intensity of the i-th band; The vertical distance height of the hyperspectral camera lens plane is The average light intensity directly below the lens at 100 Hz; is the vertical distance from the hyperspectral camera lens plane in any light field environment. is the average light intensity directly below the lens; H is the height of the target object.

[0010] Optionally, determining the height value of the target object based on the first hyperspectral image and the second hyperspectral image and based on a characteristic band, thereby obtaining the three-dimensional spatial position information of the target object, specifically includes: Generate a first grayscale image and a second grayscale image from the first hyperspectral image and the second hyperspectral image based on characteristic bands; Determine the disparity by using a feature matching algorithm according to the first grayscale image and the second grayscale image; The height value of the target object is determined according to the parallax, thereby obtaining the three-dimensional spatial position information of the target object.

[0011] In a third aspect, the present application provides a spatial position and reflectivity information extraction system based on a hyperspectral camera, comprising: A data acquisition module, used for acquiring a first hyperspectral image and a second hyperspectral image; A height determination module, used to determine the height value of the target object based on the first hyperspectral image and the second hyperspectral image and a characteristic band, so as to obtain the three-dimensional spatial position information of the target object; the characteristic band is 956.4nm; A whiteboard height correction module is used to determine the 100% whiteboard spectral irradiance under the conditions of a set height and a set light intensity according to the height value of the target object using a whiteboard height correction formula; A reflectivity determination module is used to determine the reflectivity of the target object according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the set height and the set light intensity conditions, using a reflectivity calculation formula; the spectral irradiance of the target object is determined based on the first hyperspectral image and the second hyperspectral image.

[0012] In a fourth aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-mentioned methods for extracting spatial position and reflectivity information based on a hyperspectral camera.

[0013] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned methods for extracting spatial position and reflectivity information based on a hyperspectral camera.

[0014] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for extracting spatial position and reflectivity information based on a hyperspectral camera.

[0015] According to the specific embodiments provided in this application, this application has the following technical effects: The present application provides a device and method for extracting spatial position and reflectivity information based on a hyperspectral camera, comprising: a host computer, a three-dimensional imaging module, a sliding module, a pan-tilt head and a connecting frame; the connecting frame is connected to the sliding module through the pan-tilt head; and the three-dimensional imaging module is connected to the connecting frame. The three-dimensional imaging module includes a first line scanning hyperspectral camera, a second line scanning hyperspectral camera, two optical fiber line light sources and two free stop hinges; the first line scanning hyperspectral camera and the second line scanning hyperspectral camera are fixed side by side on the lower surface of the connecting frame, and the line connecting the lens center of the first line scanning hyperspectral camera and the lens center of the second line scanning hyperspectral camera is perpendicular to the movement direction; the optical fiber line light source is arranged under the side plate of the connecting frame through the free stop hinge; the first line scanning hyperspectral camera collects the first hyperspectral image, and the second line scanning hyperspectral camera collects the second hyperspectral image; the host computer is connected to the first line scanning hyperspectral camera and the second line scanning hyperspectral camera respectively; the host computer is used to: determine the height value of the target object according to the first hyperspectral image and the second hyperspectral image, so as to obtain the three-dimensional spatial position information of the target object; determine the 100% white board spectral irradiance according to the height value of the target object by using the white board height correction formula; determine the reflectivity of the target object by using the spectral irradiance of the target object and the spectral irradiance of the 100% white board by using the reflectivity calculation formula. This application uses dual hyperspectral cameras to generate stereo imaging technology, which can obtain spectral information of the object under test in a wider band range while calculating the spatial information of the target, avoiding the system complexity and inaccurate registration problems caused by the fusion of multi-source information such as the addition of stereo imaging equipment in the past. The mathematical relationship between the change in the system's spectral irradiance and the height and light intensity was explored, and a whiteboard height correction method under different heights and light intensities was constructed, which effectively improved the reflectivity measurement accuracy of objects at different heights under different light intensity environments, and achieved the output of four-dimensional data containing three-dimensional spatial position information and reflectivity information in one acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 Schematic diagram of a device for extracting spatial position and reflectivity information based on a hyperspectral camera; Figure 2 Create a process flow chart for the whiteboard height correction formula; Figure 3 To collect the vertical distance from the camera lens plane Light intensity Schematic diagram; Figure 4 A flow chart of a highly adaptive whiteboard correction method; Figure 5 This is a schematic diagram of the fitting of the height and DN value at 549.91nm, 681.05nm and 750.72nm in the first line scan hyperspectral camera; Figure 6 It is a schematic diagram of the fitting of the height and DN value at 1199.54nm, 1449.44nm and 1640.03nm in the second line scanning hyperspectral camera; Figure 7 This is a schematic diagram of the fitting of light intensity and DN value at 549.91nm, 681.05nm and 750.72nm in the first line scanning hyperspectral camera; Figure 8 This is a schematic diagram of the fitting of the light intensity and DN value at 1199.54nm, 1449.44nm and 1640.03nm in the second line scanning hyperspectral camera; Fig. 9 Generate images and correction diagrams for 3D point clouds; Fig.10 A schematic diagram of the structure of a computer device provided in one embodiment of the present application.

[0018] Figure numerals: 1-screw guide rail, 2-first servo motor, 3-first line scanning hyperspectral camera, 4-second line scanning hyperspectral camera, 5-free stop hinge, 6-guide rail slider, 7-connecting plate, 8-pan head, 9-connecting frame, 10-optical fiber line light source. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0021] The existing hyperspectral imaging technology has certain technical shortcomings in acquiring three-dimensional information and reflectivity information of the object under test. In order to solve the above problems, the present application proposes a device and method for acquiring spatial position and reflectivity information based on a hyperspectral camera. By using two hyperspectral cameras with different band ranges combined with stereo imaging technology, it is possible to obtain spectral information of the object under test in a wider band range while calculating the spatial information of the target, avoiding the system complexity and inaccurate registration problems caused by the addition of multi-source information fusion such as stereo imaging. At the same time, the present application has explored the irradiation and reflection characteristics of the system and developed a whiteboard space correction method, which effectively improves the reflectivity measurement accuracy of objects at different heights in different light intensity environments, avoiding the process of collecting whiteboard data separately. The system can eventually directly output data containing three-dimensional spatial position information and reflectivity information The four-dimensional data has the characteristics of one-time collection and synchronous processing.

[0022] In an exemplary embodiment, Figure 1 As shown, a device for extracting spatial position and reflectivity information based on a hyperspectral camera is provided, comprising: a host computer (not shown in Figure 1 ), a three-dimensional imaging module, a sliding module, a pan-tilt 8 and a connecting frame 9; the connecting frame 9 is connected to the sliding module through the pan-tilt 8; the three-dimensional imaging module is installed on the connecting frame 9, and is used to obtain the hyperspectral image data of the sample to be tested (target object) and calculate the spatial information; the sliding module performs the linear sliding and position adjustment of the device. The connecting frame 9 includes an upper plate and two side plates; the two side plates are respectively connected to the two ends of the upper plate; the side plates are perpendicular to the upper plate.

[0023] In practical applications, the spatial position and reflectivity information extraction device based on the hyperspectral camera of the present application includes a first line scanning hyperspectral camera 3 (band range 400-1000nm), a second line scanning hyperspectral camera 4 (band range 900-1700nm), an optical fiber line light source 10, a pan / tilt 8, a free stop hinge 5, a first servo motor 2, a lead screw guide rail 1, a guide rail slider 6, a connecting plate 7 and a connecting frame 9. The field of view angle (38°) and focal length (15mm) of the two line scanning hyperspectral cameras in the device are the same, the same height is maintained and the center line of the lens is perpendicular to the direction of movement of the guide rail. The scanning process is that the first servo motor 2 works to drive the guide rail slider 6 to move forward, and the guide rail slider 6 drives the two line scanning hyperspectral cameras to move forward through the connecting plate 7. At the same time, the two line scanning hyperspectral cameras start scanning line by line at the same frequency to obtain the spectral information of the object, and control the correspondence of each scanning line.

[0024] The three-dimensional imaging module includes a first line scanning hyperspectral camera 3, a second line scanning hyperspectral camera 4, two optical fiber line light sources 10 and two free stop hinges 5; the first line scanning hyperspectral camera 3 and the second line scanning hyperspectral camera 4 are fixed side by side on the lower surface of the upper plate, the two are kept at the same height and aligned on the same horizontal line, and the line connecting the lens center of the first line scanning hyperspectral camera 3 and the lens center of the second line scanning hyperspectral camera 4 is perpendicular to the movement direction; the optical fiber line light source 10 is arranged below the side plate through the free stop hinge 5, and the free stop hinge 5 can be rotated to control the angle between the optical fiber line light source 10 and the horizontal direction.

[0025] The sliding module is used to adjust the position of the three-dimensional imaging module.

[0026] As an optional implementation, the sliding module includes: a first servo motor 2 , a motor controller, a guide rail slider 6 , a guide rail fixing plate, a lead screw guide rail 1 and a connecting plate 7 .

[0027] The lead screw guide rail 1 is fixed to the guide rail fixing plate; the power output end of the first servo motor 2 is connected to one end of the lead screw guide rail 1; the motor controller is used to control the rotation of the first servo motor 2. The guide rail slider 6 is connected to the lead screw guide rail 1 by threads; the connecting plate 7 includes an upper connecting plate and a lower connecting plate; the upper connecting plate is fixed to the bottom of the guide rail slider 6 and slides with the guide rail slider 6; the lower connecting plate and the upper connecting plate are connected by bolts; the bottom of the lower connecting plate is connected to the pan / tilt head 8.

[0028] The pan / tilt platform 8 is used to adjust the angle of the three-dimensional imaging module.

[0029] As an optional implementation, the pan / tilt head 8 includes: a second servo motor, a pan / tilt head bracket and a pan / tilt head controller.

[0030] The second servo motor is installed on the gimbal bracket; the bottom of the gimbal bracket is fixedly connected to the upper plate; the gimbal controller is connected to the second servo motor; the gimbal controller is used to control the rotation of the second servo motor, and the rotation of the second servo motor drives the gimbal bracket to rotate, thereby realizing the angle adjustment of the connecting frame 9 to adjust the angle of the three-dimensional imaging module.

[0031] The optical fiber line light source 10 is used to provide a set light intensity.

[0032] The first line scanning hyperspectral camera 3 is used to collect a first hyperspectral image of the target object under the set height and the set light intensity conditions; the second line scanning hyperspectral camera 4 is used to collect a second hyperspectral image of the target object under the set height and the set light intensity conditions.

[0033] The host computer is connected to the first line scanning hyperspectral camera 3 and the second line scanning hyperspectral camera 4 respectively; the host computer is used for: According to the first hyperspectral image and the second hyperspectral image, based on a characteristic band, the height value of the target object is determined, thereby obtaining the three-dimensional spatial position information of the target object; the characteristic band is 956.4 nm.

[0034] According to the height value of the target object, the 100% whiteboard spectral irradiance under the set height and the set light intensity conditions is determined using a whiteboard height correction formula.

[0035] The reflectivity of the target object is determined according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the set height and the set light intensity conditions using a reflectivity calculation formula; the spectral irradiance of the target object is determined according to the first hyperspectral image and the second hyperspectral image.

[0036] In an exemplary embodiment, the present application provides a method for extracting spatial position and reflectance information based on a hyperspectral camera, wherein the method for extracting spatial position and reflectance information based on a hyperspectral camera is implemented based on the above-mentioned device for extracting spatial position and reflectance information based on a hyperspectral camera; the method for extracting spatial position and reflectance information based on a hyperspectral camera includes: A first hyperspectral image and a second hyperspectral image are acquired.

[0037] According to the first hyperspectral image and the second hyperspectral image, based on a characteristic band, the height value of the target object is determined, thereby obtaining the three-dimensional spatial position information of the target object; the characteristic band is 956.4 nm.

[0038] Determining the height value of the target object based on the first hyperspectral image and the second hyperspectral image and the characteristic band, thereby obtaining the three-dimensional spatial position information of the target object, specifically includes: The first hyperspectral image and the second hyperspectral image are respectively used to generate a first grayscale image and a second grayscale image based on characteristic bands.

[0039] The disparity is determined according to the first grayscale image and the second grayscale image using a feature matching algorithm.

[0040] The height value of the target object is determined according to the parallax, thereby obtaining the three-dimensional spatial position information of the target object.

[0041] According to the height value of the target object, the 100% whiteboard spectral irradiance under the set height and the set light intensity conditions is determined using a whiteboard height correction formula.

[0042] The whiteboard height correction formula is: .

[0043] in, is the spectral irradiance of the white board in the i-th band; and The coefficient representing the relationship between the spectral irradiance and height of the i-th band; and Represents the coefficient of the relationship between the spectral irradiance and light intensity of the i-th band; The vertical distance height of the hyperspectral camera lens plane is The average light intensity directly below the lens at 100 Hz; is the vertical distance from the hyperspectral camera lens plane in any light field environment. is the average light intensity directly below the lens; H is the height of the target object.

[0044] The reflectivity of the target object is determined according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the set height and the set light intensity conditions using a reflectivity calculation formula; the spectral irradiance of the target object is determined according to the first hyperspectral image and the second hyperspectral image.

[0045] The reflectivity calculation formula is: .

[0046] in, is the reflectivity of the target object; is the 100% white board spectral irradiance, is the spectral irradiance of the target object, which can be directly read out through the first hyperspectral image and the second hyperspectral image. is the dark field spectral irradiance.

[0047] In practical applications, such as Figure 2 As shown in the figure, the process of establishing the whiteboard height correction formula includes: S1: Calibrate two hyperspectral cameras with different wavelength ranges based on the calibration plate to obtain internal and external parameters and relative external parameters.

[0048] To ensure the height measurement accuracy of the line scanning hyperspectral camera, the internal and external parameters of the two cameras need to be calibrated first, including focal length, optical center, rotation matrix, translation vector and baseline distance. Since the line scanning hyperspectral camera scans the image line by line, each line of the image has independent external parameters, so it is necessary to obtain the rotation matrix of the two cameras when shooting each line of pixel positions. and translation vector ,in, is the scan line index number. In this camera system, the camera moves along the track at a uniform speed in a straight line, so the rotation matrix is a fixed value, which can be calculated by the tangent vector and normal vector of the trajectory. The translation vector at the position of each row of pixels Derived from the geometric model of line scan camera motion, the specific formula is: The translation vector of the scan line is .in, is the trajectory direction, is the initial position, is the orbital velocity, is the camera frame rate. The relative external parameters of the two cameras are , ,in, , , , are the rotation matrix and translation vector of the first line scanning hyperspectral camera and the second line scanning hyperspectral camera respectively, and the baseline distance between the two cameras is .

[0049] The camera internal parameters remain unchanged during the scanning process. During the calibration process, two line-scanning hyperspectral cameras start shooting at the same frequency at the same time to collect hyperspectral images of the calibration plate at multiple angles and distances. In order to eliminate the differences in the optical and physical properties of the cameras in different bands, the two cameras obtain a specific band (such as 956.4nm) in the overlapping band range (930-1000nm) of the hyperspectral image of the calibration plate. The grayscale image is synthesized and denoised. Since only some of the scan lines contain checkerboard corners when the camera scans line by line, Harris corner detection is used in combination with the sub-pixel precision optimization algorithm in the OpenCV library to extract the corner coordinates. , and measure the actual physical coordinates of the corner points. For each scan line containing a corner point, establish the correspondence between the pixel coordinates and the physical coordinates. Calculate the camera's intrinsic parameters using the projection formula , the projection formula is ,in, is the scan line number, The corner point is The pixel column coordinates in the row, is the actual physical coordinate of the corner point, The camera is The external reference of the line, is the camera’s internal parameter, ,in, and are the focal lengths in the direction of camera movement and in the direction perpendicular to it, and is the optical center coordinate. The internal parameter results calculated for each scanning line are averaged to obtain the global internal parameters of the two cameras.

[0050] S2: In a dark room, measure the vertical height from the hyperspectral camera lens plane. Average light intensity directly below the lens .

[0051] S3: Collect spectral data of a standard white board at different heights, and use a regression model to fit the relationship between irradiance and height in the light field environment.

[0052] S4: Collect spectral data of a standard whiteboard under different light intensities, and use a regression model to fit the distance between the whiteboard and the camera. The relationship between irradiance and light intensity.

[0053] S5: Based on the relationship between the spectral irradiance of each band and the height and light intensity, a whiteboard height correction formula was constructed in any light field environment.

[0054] like Figure 3 and Figure 4 As shown, in the dark room, the vertical distance from the hyperspectral camera lens plane is measured to be Average light intensity directly below the lens , use a hyperspectral camera to collect spectral data of 100% standard white board at different heights from the camera, such as Figure 5 and Figure 6 As shown in the figure, the regression model is used to fit the spectral irradiance of each band of the spectral data collected by the two cameras in this light field environment. With height The relationship between ,in, Indicates the band, and Indicates each band With height Furthermore, a hyperspectral camera was used to collect 100% standard whiteboard images at a height of 1. The spectral data at Figure 7 and Figure 8 As shown in the figure, the regression model is used to fit the spectral irradiance of each band of the spectral data collected by the two cameras at this height. With light intensity The relationship between ,in, and Indicates each band With light intensity The relationship coefficient between .

[0055] The coefficients of the relationship corresponding to each band are shown in Table 1.

[0056] Table 1 Correlation coefficient table corresponding to each band

[0057] Through experiments, it is found that the spectral irradiance of each band With height and light intensity They are linearly related, based on this, this application constructs a whiteboard height correction formula in any light field environment: ,in, is the vertical distance from the hyperspectral camera lens plane in any light field environment. The average light intensity directly below the lens at this time. According to the whiteboard height correction formula, the correction parameters are measured. , , and In the case of , and then calculate the height information of each pixel on the hyperspectral image , the 100% standard white board DN value of each pixel point on the hyperspectral image under the same height information can be obtained, thereby realizing the adaptive correction of the white board height.

[0058] S6: Calculate the three-dimensional information of the object based on matching corresponding points and triangulation method, combine the camera spectrum data into the correction formula to obtain the reflectivity, and finally obtain the spatial position and reflectivity information of the object under test.

[0059] In the actual measurement, samples (tea seedlings) of different heights were collected. First, the overlapping area of ​​the two hyperspectral cameras in the band range of 930-1000nm was used to select a certain band (such as 956.4nm) to generate the corresponding grayscale image. Then, the feature matching algorithm in the OpenCV library was used to match the images, match the corresponding points between the same lines in the image line by line, and calculate the disparity. The disparity is ,in, and The first line scan hyperspectral camera and the second line scan hyperspectral camera are respectively The row corresponds to the pixel position of the matching point, according to the depth formula Calculate the height value of each matching point point by point Finally, it is substituted into the whiteboard height correction formula to obtain the 100% whiteboard spectral irradiance of the object corresponding to the light intensity and height. The whiteboard height correction formula is: , and then according to the whiteboard correction formula (reflectivity calculation formula) , get the object reflectivity .in, The parameters are automatically acquired by the camera. Through whiteboard height correction, the spectral information can be dynamically adjusted according to the three-dimensional shape of the target object, thereby maintaining the accuracy and stability of the spectral data under complex environments and diverse target height conditions. Fig. 9 As shown, the spatial position and reflectivity information of the object under test are finally obtained. Four-dimensional data.

[0060] This application has the following advantages: The use of dual hyperspectral cameras to generate stereo imaging technology can obtain spectral information of the object under test in a wider band range while calculating the spatial information of the target, avoiding the system complexity and inaccurate registration problems caused by multi-source information fusion such as the addition of additional stereo imaging equipment in the past.

[0061] The mathematical relationship between the change in the system's irradiation intensity and the height and light intensity was explored, and an automatic whiteboard correction method under different heights and light intensities was constructed, which effectively improved the reflectivity measurement accuracy of objects at different heights under different light intensity environments, and achieved the output of four-dimensional data containing three-dimensional spatial position information and reflectivity information in a single acquisition.

[0062] Based on the same inventive concept, the embodiment of the present application also provides a system for extracting spatial position and reflectivity information based on a hyperspectral camera for realizing the above-mentioned method for extracting spatial position and reflectivity information based on a hyperspectral camera. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiment of the system for extracting spatial position and reflectivity information based on a hyperspectral camera provided below can be referred to the limitations of the method for extracting spatial position and reflectivity information based on a hyperspectral camera above, and will not be repeated here.

[0063] In an exemplary embodiment, a system for extracting spatial position and reflectivity information based on a hyperspectral camera is provided, comprising: The data acquisition module is used to acquire a first hyperspectral image and a second hyperspectral image.

[0064] The height determination module is used to determine the height value of the target object based on the first hyperspectral image and the second hyperspectral image and a characteristic band, so as to obtain the three-dimensional spatial position information of the target object; the characteristic band is 956.4nm.

[0065] The whiteboard height correction module is used to determine the 100% whiteboard spectral irradiance under the set height and the set light intensity conditions according to the height value of the target object and using the whiteboard height correction formula.

[0066] A reflectivity determination module is used to determine the reflectivity of the target object according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the set height and the set light intensity conditions, using a reflectivity calculation formula; the spectral irradiance of the target object is determined based on the first hyperspectral image and the second hyperspectral image.

[0067] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the above-mentioned method for extracting spatial position and reflectivity information based on a hyperspectral camera is implemented.

[0068] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which, when executed by a processor, implements the above-mentioned method for extracting spatial position and reflectivity information based on a hyperspectral camera.

[0069] In an exemplary embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the above-mentioned method for extracting spatial position and reflectivity information based on a hyperspectral camera.

[0070] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for extracting spatial position and reflectivity information based on a hyperspectral camera is implemented.

[0071] Those skilled in the art will understand that Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0073] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0074] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0075] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A device for extracting spatial position and reflectivity information based on a hyperspectral camera, characterized in that: include: A host computer, a three-dimensional imaging module, a sliding module, a pan-tilt and a connecting frame; the connecting frame is connected to the sliding module through the pan-tilt; the connecting frame includes an upper plate and two side plates; the two side plates are respectively connected to the two ends of the upper plate; the side plates are perpendicular to the upper plate; The three-dimensional imaging module includes a first line scanning hyperspectral camera, a second line scanning hyperspectral camera, two optical fiber line light sources and two free stop hinges; the first line scanning hyperspectral camera and the second line scanning hyperspectral camera are fixed side by side on the lower surface of the upper plate, and the line connecting the lens center of the first line scanning hyperspectral camera and the lens center of the second line scanning hyperspectral camera is perpendicular to the movement direction; the optical fiber line light source is arranged below the side plate through the free stop hinge; The sliding module is used to adjust the position of the three-dimensional imaging module; The pan / tilt platform is used to adjust the angle of the three-dimensional imaging module; The optical fiber line light source is used to provide a set light intensity; The first line scanning hyperspectral camera is used to collect a first hyperspectral image of the target object under the conditions of a set height and the set light intensity; the second line scanning hyperspectral camera is used to collect a second hyperspectral image of the target object under the conditions of a set height and the set light intensity; The host computer is connected to the first line scanning hyperspectral camera and the second line scanning hyperspectral camera respectively; the host computer is used for: According to the first hyperspectral image and the second hyperspectral image, based on characteristic bands, a height value of the target object is determined, thereby obtaining three-dimensional spatial position information of the target object; The characteristic waveband is 956.4nm; According to the height value of the target object, using a whiteboard height correction formula, determine the 100% whiteboard spectral irradiance under the set height and the set light intensity conditions; Determine the reflectivity of the target object using a reflectivity calculation formula according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the set height and the set light intensity conditions; The target object spectral irradiance is determined based on the first hyperspectral image and the second hyperspectral image.

2. The device for extracting spatial position and reflectivity information based on a hyperspectral camera according to claim 1, characterized in that: The sliding module comprises: a first servo motor, a motor controller, a guide rail slider, a guide rail fixing plate, a lead screw guide rail and a connecting plate; The lead screw guide is fixed on the guide fixing plate; the first servo motor is connected to one end of the lead screw guide; the guide rail slider is connected to the lead screw guide by threads; the connecting plate includes an upper connecting plate and a lower connecting plate; the upper connecting plate is fixed below the guide rail slider; the lower connecting plate and the upper connecting plate are connected by bolts; the lower connecting plate is connected to the pan-tilt head.

3. The device for extracting spatial position and reflectivity information based on a hyperspectral camera according to claim 2, characterized in that: The pan / tilt head comprises: a second servo motor, a pan / tilt head bracket and a pan / tilt head controller; The second servo motor is installed on the gimbal bracket; the bottom of the gimbal bracket is fixedly connected to the upper plate; the gimbal controller is connected to the second servo motor; the gimbal controller is used to control the rotation of the second servo motor, and the rotation of the second servo motor drives the gimbal bracket to rotate to adjust the angle of the three-dimensional imaging module.

4. A method for extracting spatial position and reflectivity information based on a hyperspectral camera, characterized in that: The method for extracting spatial position and reflectivity information based on a hyperspectral camera is implemented based on the device for extracting spatial position and reflectivity information based on a hyperspectral camera according to any one of claims 1 to 3; The method for extracting spatial position and reflectivity information based on a hyperspectral camera includes: Acquire a first hyperspectral image and a second hyperspectral image; According to the first hyperspectral image and the second hyperspectral image, based on a characteristic band, a height value of the target object is determined, thereby obtaining three-dimensional spatial position information of the target object; the characteristic band is 956.4 nm; According to the height value of the target object, using a whiteboard height correction formula, determine the 100% whiteboard spectral irradiance under the set height and the set light intensity conditions; The reflectivity of the target object is determined according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the set height and the set light intensity conditions using a reflectivity calculation formula; the spectral irradiance of the target object is determined according to the first hyperspectral image and the second hyperspectral image.

5. The method for extracting spatial position and reflectivity information based on a hyperspectral camera according to claim 4, characterized in that: The whiteboard height correction formula is: ； in, is the spectral irradiance of the white board in the i-th band; and The coefficient representing the relationship between the spectral irradiance and height of the i-th band; and Represents the coefficient of the relationship between the spectral irradiance and light intensity of the i-th band; The vertical distance height of the hyperspectral camera lens plane is The average light intensity directly below the lens at 100 Hz; is the vertical distance from the hyperspectral camera lens plane in any light field environment. is the average light intensity directly below the lens; H is the height of the target object.

6. The method for extracting spatial position and reflectivity information based on a hyperspectral camera according to claim 4, characterized in that: Determining the height value of the target object based on the first hyperspectral image and the second hyperspectral image and the characteristic band, thereby obtaining the three-dimensional spatial position information of the target object, specifically includes: Generate a first grayscale image and a second grayscale image from the first hyperspectral image and the second hyperspectral image based on characteristic bands; Determine the disparity by using a feature matching algorithm according to the first grayscale image and the second grayscale image; The height value of the target object is determined according to the parallax, thereby obtaining the three-dimensional spatial position information of the target object.

7. A spatial position and reflectivity information extraction system based on a hyperspectral camera, characterized in that: include: A data acquisition module, used for acquiring a first hyperspectral image and a second hyperspectral image; A height determination module, used to determine the height value of the target object based on the first hyperspectral image and the second hyperspectral image and a characteristic band, so as to obtain the three-dimensional spatial position information of the target object; the characteristic band is 956.4nm; A whiteboard height correction module is used to determine the 100% whiteboard spectral irradiance under the conditions of a set height and a set light intensity according to the height value of the target object and using a whiteboard height correction formula; A reflectivity determination module, used to determine the reflectivity of the target object according to the spectral irradiance of the target object and the spectral irradiance of the 100% whiteboard under the conditions of the set height and the set light intensity, using a reflectivity calculation formula; The target object spectral irradiance is determined based on the first hyperspectral image and the second hyperspectral image.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for extracting spatial position and reflectivity information based on a hyperspectral camera according to any one of claims 4 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting spatial position and reflectivity information based on a hyperspectral camera according to any one of claims 4 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for extracting spatial position and reflectivity information based on a hyperspectral camera according to any one of claims 4 to 6 is implemented.

Images