Multi-view hyperspectral three-dimensional reconstruction and spectral mapping method

Abstract

The present application belongs to the technical field of three-dimensional reconstruction, and particularly relates to a multi-view hyperspectral three-dimensional reconstruction and spectral mapping method. The present application establishes a unified geometric reference through explicit three-dimensional reconstruction, establishes a strict projection relationship between a space point and a hyperspectral pixel based on camera internal and external parameters, selects a target view angle in combination with a normal vector geometric constraint, and finally realizes accurate, stable and physically credible mapping of hyperspectral data on a three-dimensional point cloud through nearest surface point discrimination and voxel grid constraint screening of target pixel points. The present application can strengthen the constraint relationship between a space point and a real surface position, inhibit mixed pixel interference and same-pixel multi-depth ambiguity, and further ensure the physical authenticity of the spectral mapping result.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional reconstruction technology, and in particular relates to a multi-view hyperspectral three-dimensional reconstruction and spectral mapping method. Background Technology

[0002] Hyperspectral imaging technology can acquire the reflectance characteristics of a target in a continuous spectral band, accurately characterizing the target's material composition and physicochemical properties; three-dimensional measurement technology can reconstruct the target's precise spatial geometry and local morphological features. The hyperspectral three-dimensional point cloud formed by effectively fusing these two types of information can simultaneously carry the target's geometric structure and material property information, serving as a core foundational technology for fields such as three-dimensional digital modeling, material identification, quantitative inversion, and three-dimensional scene understanding.

[0003] Currently, mainstream hyperspectral 3D reconstruction and spectral mapping schemes can be divided into three categories. The first category is band-by-band reconstruction and registration schemes. This method performs 3D reconstruction on each hyperspectral band image separately, and then fuses the multi-band reconstruction results through point cloud registration to form 3D point cloud data containing full-band spectral information. The second category is multimodal data registration and mapping schemes. This method achieves intermodal registration through extrinsic parameter calibration between the RGB 3D model and the hyperspectral image, and then projects the spectral information onto the 3D point cloud to complete the fusion of the geometric model and spectral data. The third category is based on multi-view... Figure 3 The hyperspectral mapping scheme for 3D reconstruction first recovers the 3D geometric model from multi-view images, and then maps the hyperspectral information to the 3D point cloud according to the camera projection relationship, finally generating a hyperspectral 3D point cloud.

[0004] However, band-by-band reconstruction and registration schemes require separate reconstruction and registration fusion of each band, which easily leads to the accumulation of multi-band point cloud registration errors. This makes it difficult to guarantee the consistency between continuous spectrum and 3D geometry, and it cannot achieve a stable and accurate one-to-one correspondence between hyperspectral data and dense point clouds. Multimodal data registration and mapping schemes rely on extrinsic parameter calibration between the RGB 3D model and the hyperspectral image to complete modal alignment. The model coupling accuracy is significantly constrained by the performance of the registration algorithm, and registration deviations easily occur in scenes with weak textures and complex structures, leading to spectral and spatial misalignment. Based on multi-view... Figure 3 Hyperspectral mapping schemes for 3D reconstruction often rely on simple reprojection to complete spectral assignment, which does not make sufficient use of the local geometry of 3D and camera projection constraints, easily causing mixed pixel interference and ambiguity of the same pixel at multiple depths. At the same time, hyperspectral images are affected by complex illumination and anisotropic surface reflection, which makes the photometric consistency assumption on which dense reconstruction depends invalid, further reducing the accuracy and reliability of 3D reconstruction and spectral mapping. Summary of the Invention

[0005] In view of this, the present invention aims to provide a multi-view hyperspectral 3D reconstruction and spectral mapping method to address the common problems of existing traditional schemes for fusion and registration of hyperspectral data and 3D point clouds, such as accumulated registration errors, limited modal alignment accuracy, easy generation of mixed pixels and ambiguity of depth within the same pixel, and significant influence from scene texture, lighting, and surface reflection characteristics, making it difficult to achieve high-precision, stable one-to-one correspondence and collaborative reconstruction between spectral information and 3D geometric position. This invention improves the consistency between spectrum and geometric position by establishing a unified and stable one-to-one correspondence between continuous hyperspectral data and 3D point clouds, enhancing the alignment stability of geometric models and spectral data in complex scenes; it introduces a viewpoint geometric constraint and optimal orthographic viewpoint selection mechanism to fundamentally eliminate spectral deviations and misalignments caused by viewpoint tilt and projection distortion, significantly improving the accuracy and reliability of spectral mapping; and it strengthens the constraint relationship between spatial points and real surface positions, suppressing mixed pixel interference and ambiguity of depth within the same pixel, further ensuring the physical authenticity of the spectral mapping results.

[0006] To achieve the above objectives, the technical solution created by this invention is implemented as follows: A multi-view hyperspectral 3D reconstruction and spectral mapping method, specifically including the following steps: S1: Full coverage acquisition of the target from different perspectives to obtain a multi-view hyperspectral image dataset of the target, and radiometric correction of the multi-view hyperspectral image dataset to obtain a pseudo-color image that is pixel-level aligned with each hyperspectral image contained in the multi-view hyperspectral image dataset, thus obtaining a pseudo-color image set. S2: Perform 3D reconstruction of the target based on the pseudo-color image set to obtain the global 3D point cloud of the target surface; S3: Based on the camera intrinsic and extrinsic parameters corresponding to the hyperspectral images from each viewpoint, construct a mapping model for the global 3D point cloud projected onto the image plane of the corresponding hyperspectral image. S4: Based on the mapping model, the spatial points of the global 3D point cloud are projected onto the image plane of the hyperspectral image from the corresponding viewpoint to filter the effective spatial points in the global 3D point cloud; S5: Establish a standardized point cloud of the target surface based on effective spatial points, and calculate the unit normal vector of each effective spatial point surface based on the standardized point cloud of the target surface. S6: Project each effective spatial point onto the image plane of the hyperspectral image at each viewpoint, and calculate the target pixel points corresponding to each target viewpoint by combining the unit normal vector of the surface of each effective spatial point. S7: Extract the full-band reflectance information of the target pixels corresponding to each target viewpoint, and generate a hyperspectral 3D point cloud with accurate spectral and geometric registration.

[0007] Furthermore, in step S2, after the target is reconstructed in three dimensions based on the pseudo-color image set, redundant points and outliers are removed from the original three-dimensional point cloud obtained by the three-dimensional reconstruction to obtain the global three-dimensional point cloud of the target surface.

[0008] Furthermore, in step S3, the calculation formula used to project the spatial points contained in the global 3D point cloud in the world coordinate system onto the image plane of the hyperspectral image from the corresponding viewpoint is as follows: ; ; ; in, and These are the equivalent focal lengths of the camera in the horizontal and vertical directions of the image plane of the hyperspectral image at the corresponding viewpoint. , () represents the principal point coordinates of the image plane of the hyperspectral image from the corresponding viewpoint. K For the camera intrinsic parameter matrix, R Let be the orthogonal rotation matrix of the camera coordinate system relative to the world coordinate system, t be the spatial offset vector of the origin of the camera coordinate system relative to the world coordinate system, and (x, y, z) be the three-dimensional coordinates of the point in the space in the world coordinate system. , () represents the homogeneous coordinate components of a point in the camera coordinate system. This represents the depth value of a point in the spatial coordinate system. , , , , , , , and All are matrix elements. Let x be the translation of the camera in the x-direction of the world coordinate system. Let be the translation of the camera in the y-direction of the world coordinate system. This represents the camera's translation in the z-direction of the world coordinate system. By normalizing the homogeneous coordinate components of the spatial points in the camera coordinate system, we obtain the two-dimensional pixel coordinates of the spatial points projected onto the image plane of the hyperspectral image from the corresponding viewpoint. This completes the construction of the mapping model of the global 3D point cloud projected onto the image plane of the hyperspectral image from the corresponding viewpoint. ; in,( , The coordinates are two-dimensional pixel coordinates obtained by projecting a spatial point in the world coordinate system onto the image plane of a hyperspectral image at the corresponding viewpoint.

[0009] Furthermore, step S4 specifically includes the following steps: S41: Read the mask image of the current viewpoint, use the mask image to binarize the hyperspectral image corresponding to the current viewpoint, divide the target area and background area, generate a binary mask of the target area, and complete the mask preprocessing; S42: Based on the camera's intrinsic and extrinsic parameters corresponding to the current viewpoint, project the global 3D point cloud onto the image plane of the hyperspectral image of the current viewpoint, and remove invalid spatial points located behind the camera and outside the frame of the hyperspectral image corresponding to the current viewpoint. S43: Use a binary mask to process the pixels corresponding to the image plane of the hyperspectral image projected onto the current viewpoint from the global 3D point cloud, and remove the spatial points corresponding to the pixels falling in the background area. S44: From the current perspective, remove isolated noise points with abnormal neighborhood distance distribution within the target area; S45: Replace the mask image of the current view with the mask image of the next view. Repeat steps S41-S44 to remove all invalid spatial points in the global 3D point cloud and complete the selection of valid spatial points in the global 3D point cloud.

[0010] Furthermore, step S5 specifically includes: S51: Select effective spatial points in the normalized point cloud of the target surface, and search for neighborhood spatial points of the effective spatial points with a fixed radius to obtain a local neighborhood spatial point set, so as to construct the covariance matrix of the current effective spatial points: ; in, Let i be the covariance matrix of the current valid spatial points, and i be the index number of the neighboring spatial points. P i Let be the three-dimensional coordinates of the i-th neighborhood point. The three-dimensional coordinates of the current valid spatial point are given, N is the total number of neighborhood spatial points contained in the local neighborhood spatial point set, and T is the transpose. S52: Perform eigenvalue decomposition on the covariance matrix of the current valid space point, extract the smallest eigenvalue among the three eigenvalues ​​of the covariance matrix, and use the eigenvector corresponding to the smallest eigenvalue as the surface normal vector n of the current valid space point; S53: Normalize the surface normal vector of the current valid space point to obtain the unit normal vector of the surface of the current valid space point: ; in, The surface normal vector of the current valid spatial point. Let be the magnitude of the surface normal vector of the current valid spatial point. This is the unit normal vector of the surface of the currently valid spatial point; S54: Replace the current valid spatial point with the next valid spatial point, repeat steps S51-S53, obtain the unit normal vector of the surface of each valid spatial point, and establish the corresponding KDTree spatial index according to the valid spatial points corresponding to the hyperspectral images from different perspectives.

[0011] Furthermore, in step S6, the specific steps for calculating the target viewpoint include: S6A1: Select the spatial point to be processed in the normalized point cloud of the target surface, and calculate the unit incident ray vector from the camera optical center to the current spatial point to be processed from the current viewpoint: ; in, Let be the unit incident ray vector pointing from the camera's optical center to the current point in space to be processed. The three-dimensional coordinates of the point to be processed. The three-dimensional coordinates of the camera's optical center from the current perspective; S6A2: Calculate the angle between the unit incident ray vector from the camera's optical center to the current point in space and the unit normal vector to the surface of the current point in space. : ; ; in, Let r be the two-dimensional projection vector of the unit normal vector of the surface of the current point to be processed onto the XOZ plane of the world coordinate system. xoz Let be the two-dimensional projection vector of the unit incident ray vector from the camera's optical center to the current point in space onto the XOZ plane of the world coordinate system. and These are the X-axis and Z-axis components of the unit incident ray vector from the camera's optical center to the current point in space, respectively, in the world coordinate system. and These are the X-axis and Z-axis components of the unit normal vector of the surface of the current point to be processed in the world coordinate system, respectively. S6A3: Replace the current view with the next view. Repeat steps S6A1-S6A2 to calculate the angle between the unit incident ray vector of the camera optical center pointing to the current spatial point and the unit normal vector of the surface of the current spatial point under different view. Take the view with the smallest angle as the target view corresponding to the current spatial point. S6A4: Replace the current spatial point to be processed with the next spatial point to be processed, and repeat steps S6A1-S6A3 until the target viewpoints corresponding to all spatial points to be processed are obtained.

[0012] Furthermore, in step S6, the specific steps for calculating the target pixel points corresponding to the target viewpoint include: S6B1: Based on the KDTree spatial index corresponding to the current target viewpoint, retrieve and extract the candidate point cloud corresponding to each pixel of the image plane of each hyperspectral image under the current target viewpoint. S6B2: Calculate the Euclidean distance between the candidate spatial points contained in the candidate point cloud corresponding to the current pixel to be processed under the current target view and the camera optical center corresponding to the current target view, and take the candidate spatial point with the smallest Euclidean distance to the camera optical center as the target spatial point. S6B3: Replace the current pixel with the next pixel to be processed and repeat step S6B2 to obtain all target spatial points under the current target view. S6B4: Replace the current target view with the next target view, and repeat steps S6B1-S6B3 until all target spatial points under all target views are obtained. S6B5: Based on the standardized point cloud of the target surface in the world coordinate system, calculate the voxel size of each dimension according to the preset voxel grid number: ; in, , These represent the voxel dimensions of the target space point along the x-axis, y-axis, and z-axis, respectively; that is, the side length of each voxel along the corresponding coordinate axis. , and These represent the maximum x-axis, maximum y-axis, and maximum z-axis coordinates of the target point in the world coordinate system, respectively. , and Let x be the minimum x-axis, y-axis, and z-axis coordinates of the target point in the world coordinate system. , and The preset number of voxel grids for the target space point in the x-axis, y-axis, and z-axis directions of the world coordinate system; S6B6: Based on the calculation results of step S6B5, voxelize the standardized point cloud and calculate the voxel index corresponding to each target spatial point. , , ): ; in,( , , () represents the three-dimensional coordinates of the target point in space; S6B7: Calculate the ray vectors of the camera optical center pointing to the corresponding target spatial points under different target viewpoints, count the number of voxels occupied by each ray vector, and take the pixels corresponding to the target spatial points whose voxel count does not exceed the threshold as target pixels.

[0013] Furthermore, in step S7, the full-band reflectivity information of the target pixels is extracted from each target viewpoint: ; in, This is the full-band reflectance spectral vector. From the perspective of the target, These are the pixel coordinates of the target pixel from the target's perspective.

[0014] Compared with the prior art, the present invention can achieve the following beneficial effects: (1) The multi-view hyperspectral 3D reconstruction and spectral mapping method described in this invention addresses the core pain point in existing multi-view hyperspectral 3D reconstruction and point cloud spectral mapping technologies, where it is difficult to establish a one-to-one correspondence between continuous hyperspectral data and dense 3D point clouds. It solves the problems of accumulated multi-band point cloud registration errors and poor spectral-geometric consistency in traditional band-by-band reconstruction and registration schemes. It overcomes the defects of multi-view 3D reconstruction framework in hyperspectral scenarios, where the local structure of 3D point clouds and camera projection relationship are not fully utilized and spectral assignment lacks geometric constraints. It solves the problems of mixed pixel interference, multi-depth ambiguity of the same pixel, and inaccurate spatial neighborhood matching caused by relying solely on simple reprojection in traditional methods. It makes up for the shortcomings of existing technologies that have not established a unified mathematical mapping relationship between point cloud local geometry, camera view, and hyperspectral pixels, resulting in insufficient reliability of spectral matching. It provides a precise spectral mapping method for 3D point clouds for multi-view hyperspectral imaging.

[0015] (2) The multi-view hyperspectral three-dimensional reconstruction and spectral mapping method described in this invention eliminates multimodal registration errors from the root, and makes the spectral and geometric alignment accuracy reach the pixel level. This invention constructs an RGB band pseudo-color generation scheme for hyperspectral images, directly selects the red, green and blue bands from the hyperspectral data cube to generate pseudo-color images. The generated pseudo-color images naturally have a pixel-level one-to-one correspondence with the original hyperspectral images, with no spatial offset, no scale scaling, no geometric distortion, and no need for additional multimodal extrinsic parameter calibration operations, thus eliminating the spectral misalignment problem caused by registration deviation from the root.

[0016] (3) The multi-view hyperspectral 3D reconstruction and spectral mapping method described in this invention avoids the accumulation of errors in multi-band reconstruction and ensures the global consistency of spectrum and geometry. Existing band-by-band reconstruction and registration schemes require performing 3D reconstruction on each band of the hyperspectral image separately, and then completing the fusion of multi-band data through point cloud registration. The registration error will accumulate continuously with the increase of the number of bands, which cannot guarantee the consistency between the continuous spectrum and the 3D geometry. Moreover, the computational load is extremely large and the processing efficiency is extremely low. This invention completes an explicit 3D reconstruction based on a unified pseudo-color image. The hyperspectral data of all bands are mapped based on the same set of 3D geometric references and camera parameters. There is no need for separate reconstruction and registration of multiple bands, which fundamentally eliminates the problem of error accumulation in multi-band registration and greatly improves the processing efficiency.

[0017] (4) The multi-view hyperspectral three-dimensional reconstruction and spectral mapping method described in this invention reduces the spectral deviation caused by viewpoint tilt and ensures the physical authenticity of spectral data. This invention designs an optimal (target) frontal view selection mechanism based on XOZ plane projection. By using the geometric constraints of the unit normal vector of the point cloud surface and the incident ray of the camera, the projection components of the two on the XOZ plane are extracted to calculate the angle. The vertical incident view with the smallest angle is automatically selected as the optimal (target) frontal view. This eliminates the spectral reflectance deviation caused by viewpoint tilt and projection distortion from the root and solves the problem of spectral misalignment caused by viewpoint factors.

[0018] (5) The multi-view hyperspectral 3D reconstruction and spectral mapping method described in this invention solves the problems of mixed pixels, multi-depth ambiguity, and occlusion interference, significantly improving the reliability of spectral mapping. Existing simple reprojection schemes do not further constrain and filter the projected pixels, which easily leads to multi-depth ambiguity of the same pixel and interference from mixed pixels. At the same time, without an occlusion judgment mechanism, invalid points that are occluded will also be incorrectly assigned spectral data, resulting in a high error rate and poor reliability of spectral mapping. This invention designs a three-tiered progressive filtering mechanism of view-specific nearest neighbor search, nearest surface point discrimination, and voxel mesh occlusion constraint: it completes accurate search for the same pixel through a dedicated KDTree for a single view, resolves multi-depth ambiguity by filtering real surface points through the nearest optical center distance, and completes occlusion judgment through ray voxel occupancy statistics. This solves the three core problems of mixed pixels, multi-depth ambiguity, and occlusion interference, ensuring that each spatial point corresponds to a unique and physically reliable optimal surface pixel. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1A schematic flowchart of the multi-view hyperspectral three-dimensional reconstruction and spectral mapping method described in the embodiments of the present invention; Figure 2 A schematic diagram of the structure for acquiring the multi-view hyperspectral image dataset as described in an embodiment of the present invention; Figure 3 A schematic diagram of the target perspective screening structure described in the embodiments of the present invention; Figure 4 The spectral curves corresponding to the hyperspectral point cloud described in the embodiments of the present invention.

[0020] Explanation of reference numerals in the attached figures: 1. Hyperspectral image acquisition equipment; 2. Target; 3. Hyperspectral image. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0023] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0024] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0025] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] like Figure 1 As shown, this invention provides a multi-view hyperspectral three-dimensional reconstruction and spectral mapping method, which specifically includes the following steps: S1: Full coverage acquisition of target 2 from different perspectives to obtain a multi-view hyperspectral image dataset of the target, and radiometric correction of the multi-view hyperspectral image dataset to obtain a pseudo-color image that is pixel-level aligned with each hyperspectral image 3 contained in the multi-view hyperspectral image dataset, thus obtaining a pseudo-color image set. S2: Perform 3D reconstruction of target 2 based on the pseudo-color image set to obtain the global 3D point cloud of the target surface; S3: Based on the camera intrinsic and extrinsic parameters corresponding to the hyperspectral images 3 from various viewpoints, construct a mapping model for global 3D point cloud projection onto the image plane of the corresponding hyperspectral image 3. S4: Based on the mapping model, the spatial points of the global 3D point cloud are projected onto the image plane of the hyperspectral image 3 from the corresponding viewpoint, and the effective spatial points in the global 3D point cloud are filtered. S5: Establish a standardized point cloud of the target surface based on effective spatial points, and calculate the unit normal vector of each effective spatial point surface based on the standardized point cloud of the target surface. S6: Project each effective spatial point onto the image plane of the hyperspectral image 3 at each viewpoint, and calculate the target pixel points corresponding to each target viewpoint by combining the unit normal vector of the surface of each effective spatial point. S7: Extract the full-band reflectance information of the target pixels corresponding to each target viewpoint, and generate a hyperspectral 3D point cloud with accurate spectral and geometric registration.

[0027] It should be noted that this invention provides a multi-view-based... Figure 3The hyperspectral point cloud-image-spectral mapping method for 3D reconstruction can be applied to equipment and systems such as hyperspectral 3D modeling, crop phenotypic detection, and quantitative material inversion. The core is to generate pixel-level aligned pseudo-color images from hyperspectral images to complete explicit 3D reconstruction, rely on camera projection models to complete the reverse mapping between spatial points and hyperspectral pixels, and complete the optimal surface (target) pixel matching through target view selection and triple constraint mechanism. Finally, it realizes end-to-end fully automated closed-loop processing from multi-view hyperspectral data acquisition to hyperspectral 3D point cloud output, without any manual intervention.

[0028] This invention constructs a unified geometric benchmark through explicit 3D reconstruction, establishes a strict projection relationship between spatial points and hyperspectral pixels based on camera intrinsic and extrinsic parameters, selects the optimal (target) viewpoint by combining normal vector geometric constraints, and filters the optimal surface (target) pixel points by nearest surface point discrimination and voxel mesh constraints, ultimately achieving accurate, stable, and physically reliable mapping of hyperspectral data on 3D point clouds.

[0029] In terms of hardware, the present invention mainly includes a hyperspectral image acquisition device 1 and a data processing unit; in terms of software and data processing, it mainly includes functions such as hyperspectral image acquisition and preprocessing, pseudo-color image generation, explicit 3D point cloud construction, camera parameter analysis, single-view point cloud filtering, merged point cloud preprocessing, spatial point back projection, optimal viewing angle selection, optimal surface pixel selection, hyperspectral reflectance assignment and result output.

[0030] In step S1, a multi-view hyperspectral image dataset of the target is acquired and preprocessed. A hyperspectral image acquisition device 1 (such as a camera) is used to acquire the target from multiple different observation positions and perspectives, obtaining a multi-view hyperspectral image dataset. To ensure the integrity and robustness of subsequent explicit 3D reconstruction, the region of interest (ROI) overlap rate between adjacent view hyperspectral images 3 is no less than 30%, ensuring sufficient common observation areas among the multi-view hyperspectral images 3, providing sufficient matching features for camera pose calculation and 3D reconstruction. The acquired hyperspectral image 3 is a 3D data cube containing continuous spectral bands, in the format [height (H) × width (W) × number of bands (C)]. Each pixel position in the image corresponds to a complete set of continuous spectral reflectance information, providing the original data source for subsequent spectral mapping. A standard reflectance plate needs to be acquired simultaneously when acquiring each view hyperspectral image 3, and radiometric correction preprocessing is performed on the hyperspectral image 3 to obtain the hyperspectral image reflectance. A schematic diagram of multi-view hyperspectral image 3 acquisition is shown below. Figure 2 .

[0031] From the preprocessed multi-view hyperspectral image dataset, three typical bands—red (R), green (G), and blue (B)—are selected. A pseudo-color visualization image perfectly aligned with hyperspectral image 3 is generated through linear combination of these bands. The generated pseudo-color image is completely identical to the original hyperspectral image 3 in resolution, pixel coordinates, imaging viewpoint, and distortion parameters. That is, any pixel coordinate (u,v) on the pseudo-color image has a one-to-one mapping relationship with the pixel at the same coordinate (u,v) on the hyperspectral image 3, without spatial offset, scale scaling, or geometric distortion. No additional registration or calibration operations are required, providing a unified pixel reference for subsequent 3D reconstruction and spectral mapping.

[0032] A single-view hyperspectral image 3 is extracted from the preprocessed multi-view hyperspectral dataset. Three bands belonging to the corresponding spectral ranges of red, green, and blue are fixedly selected. The original reflectance values ​​of each pixel in the three bands are linearly normalized and stretched to the gray range of 0 to 255. Then, they are sequentially mapped to the RGB channels for band linear fusion to obtain a pseudo-color image that is strictly aligned pixel by pixel with the single-view hyperspectral image 3.

[0033] Replace the current single-view hyperspectral image 3 with the next single-view hyperspectral image 3, and repeat the above steps to obtain pseudo-color images that are pixel-level aligned with each hyperspectral image 3 contained in the multi-view hyperspectral image dataset, thus obtaining a pseudo-color image set.

[0034] This invention designs a natural pixel-level alignment scheme between a hyperspectral image 3 and a pseudo-color image. It selects RGB bands from the hyperspectral data cube to generate a pseudo-color image. The two images are completely identical in resolution, pixel coordinates, and imaging parameters, eliminating the need for additional registration and calibration operations. This eliminates alignment errors caused by multimodal data registration at the source and provides a unified benchmark for 3D reconstruction and spectral mapping.

[0035] The multi-view hyperspectral dataset acquisition in this invention adopts a multi-view surround acquisition scheme using an area array hyperspectral camera. Depending on the target scene type and acquisition efficiency requirements, it can be adjusted to an area array hyperspectral imaging device with different spectral ranges and spatial resolutions. Alternatively, it can be replaced by various hardware solutions that can achieve multi-view hyperspectral data cube acquisition, such as a linear array pushbroom hyperspectral imaging system, an UAV-borne hyperspectral imaging system, a multi-camera array synchronous hyperspectral acquisition system, or a snapshot hyperspectral imager. All of these solutions can acquire standardized raw hyperspectral data that is strictly bound to the imaging viewpoint, providing a data source for subsequent pixel-level alignment and precise mapping.

[0036] The pseudo-color image generation method in this invention uses a hyperspectral RGB three-band linear combination method, which can be adjusted to a combination of different feature bands based on the signal-to-noise ratio of hyperspectral data and the texture features of the target. Alternatively, it can be replaced by various visualization image generation algorithms that can achieve 3-pixel alignment with hyperspectral images, such as PCA dimensionality reduction principal component mapping and single-band grayscale image generation. All of these methods can provide a unified pixel benchmark for 3D reconstruction without registration error.

[0037] In some embodiments, in step S2, after the target is reconstructed in three dimensions based on the pseudo-color image set, redundant points and outliers are removed from the original three-dimensional point cloud obtained by the three-dimensional reconstruction to obtain the global three-dimensional point cloud of the target surface.

[0038] Using the pseudo-color image set of multiple views generated in the previous step as input, a dense 3D point cloud of the target surface is obtained through explicit 3D reconstruction. This point cloud is an explicit geometric representation that can directly characterize the spatial structure and surface position of the target. The spatial position of a point can be represented as: ; in, x, y, z These represent the 3D coordinates of the spatial points in the world coordinate system. The reconstructed original 3D point cloud undergoes denoising, deduplication, and filtering preprocessing to remove redundant and outlier points, improving geometric accuracy. Simultaneously, the voxel downsampling size can be set as needed to uniformly downsample the point cloud, reducing computational load while preserving geometric details. If a unified coordinate system is required, PCA (Principal Component Analysis) combined with RANSAC (Random Sample Consensus) algorithm can be used for plane fitting and coordinate system correction, aligning the target principal plane to the world coordinate system XOY plane to ensure coordinate system consistency between all views and the point cloud. The mathematical definitions, judgment formulas, and processing rules for redundant and outlier points are as follows: (1) Definition and Judgment Rules of Redundant Points Redundant points refer to invalid points in three-dimensional space that overlap geometrically, causing duplication of spatial geometric information. Their determination and removal rules are entirely based on the uniqueness of their spatial coordinates. Let any two distinct spatial points in the point cloud be: ; in, For the i-th spatial point The three-dimensional coordinates For the j-th spatial point The three-dimensional coordinates.

[0039] Calculate the Euclidean distance between two spatial points : ; The preset redundancy point distance threshold is When satisfied At that time, the judgment P i and P j The points are redundant. During the deduplication process, only one point is retained and the rest are removed to ensure the uniqueness of the point cloud spatial coordinates and avoid computational redundancy and geometric deviation caused by duplicate points.

[0040] (2) Definition and Judgment Rules of Outliers Outliers are noise points whose local neighborhood distance characteristics significantly deviate from the overall statistical distribution of the point cloud. They are identified using statistical filtering. For any spatial point... P i Select its K Nearest neighbor set N K (P i ) ( K (For the preset number of neighboring points), calculate the spatial point. P i Average distance to all nearest neighbor points : ; Based on spatial points P i Average distance to all nearest neighbor points Calculate global statistical characteristics: ; ; in, N This represents the total number of points in space. μ This is the global mean of the neighborhood average distances of all spatial points in the point cloud. σ This is the global standard deviation of the average neighborhood distance of all spatial points in the point cloud. The preset standard deviation threshold is [value missing]. λ When the following conditions are met: ; Judgment point P i Outliers are removed to eliminate the interference of noise points on subsequent normal vector calculations and projection mapping.

[0041] The explicit 3D point cloud construction method in this invention uses a dense reconstruction method of multi-view pseudo-color images. It can be adjusted to different multi-view stereo matching algorithms according to the complexity of the target shape and the conditions of the acquisition scene, or equivalently replaced by various technical solutions such as RGB-D depth camera point cloud registration and LiDAR point cloud acquisition that can realize the explicit geometric structure restoration of the target. All of these can provide a unified and accurate 3D geometric benchmark for spectral mapping.

[0042] In some embodiments, in step S3, the calculation formula used to project the spatial points contained in the global 3D point cloud in the world coordinate system onto the image plane of the hyperspectral image 3 from the corresponding viewpoint is: ; ; ; in, and These are the equivalent focal lengths of the camera in the horizontal and vertical directions of the image plane of the hyperspectral image at the corresponding viewpoint. , () represents the principal point coordinates of the image plane of the hyperspectral image from the corresponding viewpoint. K For the camera intrinsic parameter matrix, R Let be the orthogonal rotation matrix of the camera coordinate system relative to the world coordinate system, t be the spatial offset vector of the origin of the camera coordinate system relative to the world coordinate system, and (x, y, z) be the three-dimensional coordinates of the point in the space in the world coordinate system. , () represents the homogeneous coordinate components of a point in the camera coordinate system. This represents the depth value of a point in the spatial coordinate system. , , , , , , , and All are matrix elements. Let x be the translation of the camera in the x-direction of the world coordinate system. Let be the translation of the camera in the y-direction of the world coordinate system. This represents the camera's translation in the z-direction of the world coordinate system. By normalizing the homogeneous coordinate components of the spatial points in the camera coordinate system, the two-dimensional pixel coordinates of the spatial points projected onto the image plane of the hyperspectral image 3 at the corresponding viewpoint are obtained, thus completing the construction of the mapping model of the global 3D point cloud projected onto the image plane of the hyperspectral image 3 at the corresponding viewpoint. ; in,( , The two-dimensional pixel coordinates are obtained by projecting spatial points in the world coordinate system onto the image plane of the hyperspectral image 3 at the corresponding viewpoint.

[0043] It should be noted that during explicit 3D reconstruction, the camera intrinsic and extrinsic parameters corresponding to each view are simultaneously recorded and output to construct a complete projection relationship from the world coordinate system to the image pixel coordinate system. All parameters are strictly bound to the corresponding pseudo-color image and hyperspectral image. (Camera intrinsic parameter matrix) K Represented as: ; in, and This refers to the equivalent focal length of the camera in the horizontal and vertical directions of the image. c x and c y These are the coordinates of the principal point of the image.

[0044] (1) Camera extrinsic optical center calculation Camera extrinsic parameters are determined by the rotation matrix. R Translation vector t This is a composition used to describe the camera's pose in the world coordinate system. The coordinates of the camera's optical center in the world coordinate system are: ; Where R is a 3×3 orthogonal rotation matrix, describing the rotation attitude of the camera coordinate system relative to the world coordinate system; t is a 3×1 translation vector, describing the spatial offset of the origin of the camera coordinate system relative to the world coordinate system; o is the camera optical center, which is the starting point of the projection ray; R T This is the transpose of the rotation matrix R. The transformation matrix (c2w matrix) from the camera coordinate system to the world coordinate system is a 4×4 homogeneous matrix, in the form: ; This matrix is ​​used to transform spatial points in the camera coordinate system to the world coordinate system, while also providing a reference for the generation of projection rays.

[0045] (2) Projection formula from point cloud to pixel The pixel coordinates of a 3D point P projected onto the image plane in the world coordinate system. (u,v) The complete formula is: ; ; Where K is the camera intrinsic parameter matrix, R is the orthogonal rotation matrix of the camera coordinate system relative to the world coordinate system, t is the spatial offset vector of the origin of the camera coordinate system relative to the world coordinate system, and (x, y, z) are the three-dimensional coordinates of the spatial point in the world coordinate system, respectively. , () represents the homogeneous coordinate components of a point in the camera coordinate system. R is the depth value of a point in the camera coordinate system. iiThis represents the projection component of the i-th basis vector in the world coordinate system onto the i-th axis in the camera coordinate system, for example: R 11 Specifically, R represents the component of the X-axis of the world coordinate system on the X-axis of the camera coordinate system. 12 Specifically, R represents the component of the X-axis in the world coordinate system on the Y-axis in the camera coordinate system. 13 Specifically, it represents the component of the X-axis of the world coordinate system on the z-axis of the camera coordinate system, ...; Finally, the two-dimensional pixel coordinates of the spatial points projected onto the image plane of the hyperspectral image 3 at the corresponding viewpoint are obtained by homogeneous coordinate normalization: ; in, w′ The depth value in the camera coordinate system, only when w′ When the value is greater than 0, the spatial point is located in front of the camera, and the projection is valid.

[0046] In some embodiments, step S4 specifically includes the following steps: S41: Read the mask image of the current viewpoint, use the mask image to perform binarization processing on the hyperspectral image 3 corresponding to the current viewpoint, divide the target area and background area, generate a binary mask of the target area, and complete the mask preprocessing; S42: Based on the camera's intrinsic and extrinsic parameters corresponding to the current viewpoint, project the global 3D point cloud onto the image plane of the hyperspectral image 3 of the current viewpoint, and remove invalid spatial points located behind the camera and outside the frame of the hyperspectral image 3 corresponding to the current viewpoint. S43: Use a binary mask to process the pixels corresponding to the image plane of the hyperspectral image 3 projected onto the current viewpoint of the global 3D point cloud, and remove the spatial points corresponding to the pixels falling in the background area. S44: From the current perspective, remove isolated noise points with abnormal neighborhood distance distribution within the target area; S45: Replace the mask image of the current view with the mask image of the next view. Repeat steps S41-S44 to remove all invalid spatial points in the global 3D point cloud and complete the selection of valid spatial points in the global 3D point cloud.

[0047] It should be noted that, to avoid invalid spatial points interfering with subsequent mapping, for each view, the effective spatial points under a single view are filtered based on the camera view frustum, mask region, and statistical filtering. The steps are as follows: (1) Read the mask image of the current view, and obtain the target effective area mask through binarization processing, retaining only the spatial points corresponding to the pixels in the mask; (2) Based on the camera's internal and external parameters, the global point cloud is projected onto the image plane corresponding to the current view, and the point cloud located within the camera's view frustum and the effective area of ​​hyperspectral image 3 is selected. Invalid spatial points behind the camera and outside the frame of hyperspectral image 3 are removed. (3) Statistical outlier removal is performed on the filtered point cloud to remove noise points with excessive distance deviation in the neighborhood, and finally the effective spatial points of the current view are obtained.

[0048] (4) Construct a dedicated KDTree spatial index for each valid spatial point of the view, which is only used for filtering the same pixel points in the current view, avoiding interference from invalid points in the global point cloud, and greatly improving the efficiency of nearest neighbor search.

[0049] In some embodiments, step S5 specifically includes: S51: Select effective spatial points in the normalized point cloud of the target surface, and search for neighborhood spatial points of the effective spatial points with a fixed radius to obtain a local neighborhood spatial point set, so as to construct the covariance matrix of the current effective spatial points: ; in, Let i be the covariance matrix of the current valid spatial points, and i be the index number of the neighboring spatial points. P i Let be the three-dimensional coordinates of the i-th neighborhood point. The three-dimensional coordinates of the current valid spatial point are given, N is the total number of neighborhood spatial points contained in the local neighborhood spatial point set, and T is the transpose. S52: Perform eigenvalue decomposition on the covariance matrix of the current valid space point, extract the smallest eigenvalue among the three eigenvalues ​​of the covariance matrix, and use the eigenvector corresponding to the smallest eigenvalue as the surface normal vector n of the current valid space point; S53: Normalize the surface normal vector of the current valid space point to obtain the unit normal vector of the surface of the current valid space point: ; in, The surface normal vector of the current valid spatial point. Let be the magnitude of the surface normal vector of the current valid spatial point. This is the unit normal vector of the surface of the currently valid spatial point; S54: Replace the current valid spatial point with the next valid spatial point, repeat steps S51-S53, obtain the unit normal vector of the surface of each valid spatial point, and establish the corresponding KDTree spatial index according to the valid spatial points corresponding to the hyperspectral images 3 from different perspectives.

[0050] It should be noted that by merging all valid spatial points from all views and performing deduplication based on spatial coordinates, a globally unique dense point cloud is obtained. For the merged global point cloud, i.e., the standardized point cloud of the target surface, the surface normal vector of each valid spatial point is calculated through local neighborhood covariance analysis, as follows: (1) For each effective spatial point P, select a set of local neighborhood spatial points within a fixed radius and construct the covariance matrix: ; in, P i For the neighboring region i One effective spatial point, is the center point of the neighborhood point set, i.e., the effective spatial point to be processed, and N is the local neighborhood point set.

[0051] (2) Perform eigenvalue decomposition on the covariance matrix. The eigenvector corresponding to the smallest eigenvalue is the surface normal vector n of that point. (3) Normalize the surface normal vector of the point cloud to obtain the unit normal vector: ; in, This is the normalized unit normal vector of the surface of the current point in space to be processed. This represents the magnitude of the surface normal vector of the current point in space to be processed. The final result is a standardized explicit 3D point cloud containing spatial coordinates, color information, and normalized surface normal vectors. This serves as a unified geometric reference for subsequent spectral mapping. Simultaneously, a KDTree spatial index is constructed for the global point cloud (standardized point cloud) for rapid lookup of subsequent normal vectors.

[0052] In some embodiments, the specific steps for calculating the target viewpoint in step S6 include: S6A1: Select the spatial point to be processed in the normalized point cloud of the target surface, and calculate the unit incident ray vector from the camera optical center to the current spatial point to be processed from the current viewpoint: ; in, Let be the unit incident ray vector pointing from the camera's optical center to the current point in space to be processed. The three-dimensional coordinates of the point to be processed. The three-dimensional coordinates of the camera's optical center from the current perspective; S6A2: Calculate the angle between the unit incident ray vector from the camera's optical center to the current point in space and the unit normal vector to the surface of the current point in space. : ; ; in, This is the two-dimensional projection vector of the unit normal vector of the surface of the current point to be processed onto the XOZ plane of the world coordinate system. Let be the two-dimensional projection vector of the unit incident ray vector from the camera's optical center to the current point in space onto the XOZ plane of the world coordinate system. and These are the X-axis and Z-axis components of the unit incident ray vector from the camera's optical center to the current point in space, respectively, in the world coordinate system. and These are the X-axis and Z-axis components of the unit normal vector of the surface of the current point to be processed in the world coordinate system, respectively. S6A3: Replace the current view with the next view. Repeat steps S6A1-S6A2 to calculate the angle between the unit incident ray vector of the camera optical center pointing to the current spatial point and the unit normal vector of the surface of the current spatial point under different view. Take the view with the smallest angle as the target view corresponding to the current spatial point. S6A4: Replace the current spatial point to be processed with the next spatial point to be processed, and repeat steps S6A1-S6A3 until the target viewpoints corresponding to all spatial points to be processed are obtained.

[0053] In some embodiments, the specific steps for calculating the target pixel in step S6 include: S6B1: Based on the KDTree spatial index corresponding to the current target viewpoint, retrieve and extract the candidate point cloud corresponding to each pixel of the image plane of each hyperspectral image 3 under the current target viewpoint. S6B2: Calculate the Euclidean distance between the candidate spatial points contained in the candidate point cloud corresponding to the current pixel to be processed under the current target view and the camera optical center corresponding to the current target view, and take the candidate spatial point with the smallest Euclidean distance to the camera optical center as the target spatial point. S6B3: Replace the current pixel with the next pixel to be processed and repeat step S6B2 to obtain all target spatial points under the current target view. S6B4: Replace the current target view with the next target view, and repeat steps S6B1-S6B3 until all target spatial points under all target views are obtained. S6B5: Based on the standardized point cloud of the target surface in the world coordinate system, calculate the voxel size of each dimension according to the preset voxel grid number: ; in, , These represent the voxel dimensions of the target space point along the x-axis, y-axis, and z-axis, respectively; that is, the side length of each voxel along the corresponding coordinate axis. , and These represent the maximum x-axis, maximum y-axis, and maximum z-axis coordinates of the target point in the world coordinate system, respectively. , and Let x be the minimum x-axis, y-axis, and z-axis coordinates of the target point in the world coordinate system. , and The preset number of voxel grids for the target space point in the x-axis, y-axis, and z-axis directions of the world coordinate system; S6B6: Based on the calculation results of step S6B5, voxelize the standardized point cloud and calculate the voxel index corresponding to each target spatial point. , , ): ; in,( , , () represents the three-dimensional coordinates of the target point in space; S6B7: Calculate the ray vectors of the camera optical center pointing to the corresponding target spatial points under different target viewpoints, count the number of voxels occupied by each ray vector, and take the pixels corresponding to the target spatial points whose voxel count does not exceed the threshold as target pixels.

[0054] It should be noted that the following section will explain the back projection of the spatial point cloud onto the hyperspectral image 3 and the visibility determination: Based on the reconstructed camera intrinsic and extrinsic parameters, each spatial point in the global point cloud (normalized point cloud) is back-projected onto the image plane of all view hyperspectral images 3, establishing a visibility correspondence between spatial points and pixels in hyperspectral images 3. All valid views that pass the verification are summarized, and the corresponding view number, hyperspectral pixel coordinates, and depth value information are recorded to form a precise association relationship of [single spatial point → valid pixels in multiple views], providing a complete visibility basis for subsequent target view selection and spectral assignment.

[0055] (1) Optimal (target) perspective selection like Figure 3 As shown, to ensure the physical accuracy of spectral observations, this invention selects the target viewing angle with the optimal observation angle through the geometric constraints of the unit normal vector of the spatial point surface and the incident ray. The steps are as follows: 1) Calculate the unit incident ray vector from the camera's optical center to the point in space to be processed in the current view: ; 2) Extract the projection components of the unit incident ray vector pointing from the camera's optical center to the point to be processed and the unit normal vector of the surface of the point to be processed onto the XOZ plane of the world coordinate system, and calculate the angle between them. θ Adapting to the geometric features of the target observation scene: ; ; in, r xoz The two-dimensional projection vector of the unit incident ray vector of the point to be processed in the space from which the camera's optical center points onto the XOZ plane in the world coordinate system. The unit normal vector of the surface of the point to be processed is projected onto the XOZ plane of the world coordinate system in a two-dimensional manner. and These are the X-axis (horizontal) and Z-axis (depth) components of the unit incident ray vector pointing from the camera's optical center to the point to be processed in the world coordinate system. and These are the X-axis and Z-axis components of the unit normal vector of the surface of the point to be processed in the world coordinate system, respectively. θ Let be the angle between the unit incident ray vector pointing from the camera's optical center to the point in space to be processed and the unit normal vector of the surface of the point in space to be processed in the XOZ plane of the world coordinate system. θ The smaller the value, the closer the observation angle is to perpendicular incidence, the less the spectral observation value is affected by the tilt of the angle, and the more reliable the data.

[0056] 3) Select the included angle for all valid views corresponding to the spatial point to be processed (valid view refers to the view that can see the point, among which the most direct view is the optimal view). θ The smallest view is taken as the optimal (target) frontal viewing angle to obtain the most accurate spectral observation values, thus avoiding spectral deviation caused by viewing angle tilt at the source.

[0057] (2) Optimal surface (target) pixel selection To address the issues of multi-depth ambiguity within the same pixel, interference from mixed pixels, and occlusion, this invention employs a triple screening mechanism: view-specific nearest neighbor search, nearest surface point discrimination, and voxel occlusion constraint, to determine the unique optimal surface pixel.

[0058] To address the issues of multi-depth ambiguity within the same pixel, interference from mixed pixels, and occlusion, this invention employs a triple-screening mechanism: view-specific nearest neighbor search, nearest surface point discrimination, and voxel occlusion constraint, to determine the unique optimal surface pixel. The steps are as follows: 1) Nearest surface point screening Based on the KDTree spatial index corresponding to the current viewpoint, retrieve and extract the candidate point cloud corresponding to each pixel of the image plane of each hyperspectral image 3 under the current viewpoint. For all candidate spatial points in the candidate point set, calculate the Euclidean distance from each candidate spatial point to the optical center of the current view camera: ; Selecting the candidate spatial point with the smallest distance as the true effective surface point, which is the foremost surface point directly observed by the camera, can fundamentally solve the problem of multiple depths within the same pixel.

[0059] 2) Voxel mesh occlusion constraints First, automatic voxel size calculation is performed: Based on the spatial range of the global point cloud (normalized point cloud), the voxel size of each dimension is automatically calculated according to the preset number of voxel grids. The formula is as follows: ; The 3D space is divided into regular voxel grids according to the calculated voxel size. The global point cloud is voxelized to obtain the voxel indices of all points occupied by the point cloud: ; Using the camera's optical center as the starting point of the ray and the real effective surface point cloud as the ending point of the ray, the number of voxel grids occupied by the ray is counted. Only when the number of occupied voxels does not exceed a preset threshold is the candidate spatial point determined to be unoccluded and a valid optimal surface pixel, i.e., the target pixel. If the number exceeds the threshold, the candidate spatial point is determined to be occluded and the candidate point cloud is removed.

[0060] Through the above three-stage screening, a unique, unobstructed, and physically reliable optimal surface pixel is finally determined for each spatial point cloud, i.e., the target pixel, ensuring the uniqueness and stability of spectral mapping.

[0061] The point cloud denoising and deduplication method adopted in this invention is K-nearest neighbor statistical filtering + Euclidean distance threshold deduplication. It can be adjusted to different neighborhood parameters and distance thresholds according to the point cloud density and noise distribution characteristics, or equivalently replaced by various algorithm schemes that can achieve point cloud noise and redundant point removal, such as radius filtering outlier removal and voxel downsampling centroid deduplication. All of these can improve the geometric accuracy of the point cloud and ensure the stability of subsequent spectral mapping.

[0062] In some embodiments, in step S7, the full-band reflectance information of the target pixel is extracted from each target viewpoint: ; in, This is the full-band reflectance spectral vector. From the perspective of the target; The pixel coordinates of the target pixel from the target's perspective.

[0063] It should be noted that the assignment of hyperspectral reflectance values ​​and the output of results will be explained below: (1) Assignment of hyperspectral reflectance Under the optimal (target) viewpoint, based on the two-pixel coordinates of the image plane of the hyperspectral image 3 corresponding to the optimal surface (target) pixel point. (u,v) Extract the full-band reflectance information for this location from the hyperspectral data cube: ; (2) Result output The three-dimensional coordinates of each point cloud in space, the unit normal vector of each point cloud surface, the RGB information of the corresponding hyperspectral image 3, the optimal (target) viewpoint information, and the full-band hyperspectral reflectance information are fused to finally output a hyperspectral point cloud, which contains structured spectral and geometric data, specifically the fields {reflectance, wavelengths, x / y / z, r / g / b}, where: reflectance: an N×C dimension reflectance matrix, where N is the total number of points in the point cloud and C is the number of spectral bands; wavelengths: a wavelength vector of 1×C dimensions, corresponding to the center wavelength of each spectral band; x / y / z: N×1 dimensional vectors, representing the three-dimensional spatial coordinates of each point cloud; r / g / b: An N×1 dimensional vector, representing the RGB color values ​​for each point cloud.

[0064] Each point in a hyperspectral point cloud contains its spectral information, such as... Figure 4 As shown.

[0065] This invention designs a single-view effective spatial point filtering and dedicated KDTree spatial indexing scheme. It performs effective spatial point filtering for each view separately based on view frustum and mask constraints, and constructs a dedicated KDTree index for the effective spatial points of each view. This index is only used for filtering pixels of the same pixel in the current view, which avoids interference from globally invalid points and greatly improves the efficiency of nearest neighbor search.

[0066] This invention designs an optimal (target) viewing angle selection mechanism based on the XOZ plane projection of the world coordinate system. It extracts the projection components of the incident ray and the unit normal vector of the point cloud surface in the XOZ plane of the world coordinate system to calculate the angle, selects the view with the smallest angle as the optimal frontal viewing angle, eliminates the spectral deviation caused by viewing angle tilt and projection distortion from the root, and ensures the physical accuracy of spectral observation.

[0067] This invention designs a triple progressive constraint screening mechanism for optimal surface (target) pixels. Addressing three core issues—multi-depth ambiguity of the same pixel, interference from mixed pixels, and occlusion interference—it sequentially employs three layers of screening: view-specific nearest neighbor search, nearest surface point discrimination, and voxel grid occlusion constraint. This ensures that each spatial point has a unique, unobstructed, and physically reliable optimal surface (target) pixel, guaranteeing the uniqueness and stability of spectral mapping.

[0068] This invention designs a quantitative judgment scheme for point cloud preprocessing. It automatically judges and removes redundant points based on the Euclidean distance between spatial points, and automatically judges and removes outliers based on K-nearest neighbor statistical features. At the same time, it supports coordinate system adaptive correction based on PCA+RANSAC, ensuring the geometric accuracy of point clouds and the consistency of the coordinate system, without the need for manual parameter adjustment.

[0069] This invention supports multi-threaded / multi-process parallel full point cloud spectral assignment processing, which can be adapted to the batch automated processing of large-scale dense point clouds. The final output is standardized hyperspectral 3D point cloud data that is compatible with both visualization and scientific research applications. Only basic parameters need to be adjusted to adapt to the hyperspectral 3D modeling needs of multiple scenarios such as crops, industrial parts, cultural relics, and remote sensing, and it has extremely strong generalization ability.

[0070] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0071] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-view hyperspectral three-dimensional reconstruction and spectral mapping method, characterized in that: Specifically, the steps include the following: S1: Full coverage acquisition of the target from different perspectives to obtain a multi-view hyperspectral image dataset of the target, and radiometric correction of the multi-view hyperspectral image dataset to obtain a pseudo-color image that is pixel-level aligned with each hyperspectral image contained in the multi-view hyperspectral image dataset, thus obtaining a pseudo-color image set. S2: Perform 3D reconstruction of the target based on the pseudo-color image set to obtain the global 3D point cloud of the target surface; S3: Based on the camera intrinsic and extrinsic parameters corresponding to the hyperspectral images from each viewpoint, construct a mapping model for the global 3D point cloud projected onto the image plane of the corresponding hyperspectral image. S4: Based on the mapping model, the spatial points of the global 3D point cloud are projected onto the image plane of the hyperspectral image from the corresponding viewpoint to filter the effective spatial points in the global 3D point cloud; S5: Establish a standardized point cloud of the target surface based on effective spatial points, and calculate the unit normal vector of each effective spatial point surface based on the standardized point cloud of the target surface. S6: Project each effective spatial point onto the image plane of the hyperspectral image at each viewpoint, and calculate the target pixel points corresponding to each target viewpoint by combining the unit normal vector of the surface of each effective spatial point. S7: Extract the full-band reflectance information of the target pixels corresponding to each target viewpoint, and generate a hyperspectral 3D point cloud with accurate spectral and geometric registration.

2. The multi-view hyperspectral three-dimensional reconstruction and spectral mapping method according to claim 1, characterized in that: In step S2, after the target is reconstructed in three dimensions based on the pseudo-color image set, redundant points and outliers are removed from the original three-dimensional point cloud obtained by the three-dimensional reconstruction to obtain the global three-dimensional point cloud of the target surface.

3. The multi-view hyperspectral three-dimensional reconstruction and spectral mapping method according to claim 1, characterized in that: In step S3, the calculation formula used to project the spatial points contained in the global 3D point cloud in the world coordinate system onto the image plane of the hyperspectral image from the corresponding viewpoint is as follows: ； ； ； in, and These are the equivalent focal lengths of the camera in the horizontal and vertical directions of the image plane of the hyperspectral image at the corresponding viewpoint. , () represents the principal point coordinates of the image plane of the hyperspectral image from the corresponding viewpoint. K For the camera intrinsic parameter matrix, R Let be the orthogonal rotation matrix of the camera coordinate system relative to the world coordinate system, t be the spatial offset vector of the origin of the camera coordinate system relative to the world coordinate system, and (x, y, z) be the three-dimensional coordinates of the point in the space in the world coordinate system. , () represents the homogeneous coordinate components of a point in the camera coordinate system. This represents the depth value of a point in the spatial coordinate system. , , , , , , , and All are matrix elements. Let x be the translation of the camera in the x-direction of the world coordinate system. Let be the translation of the camera in the y-direction of the world coordinate system. This represents the camera's translation in the z-direction of the world coordinate system. By normalizing the homogeneous coordinate components of the spatial points in the camera coordinate system, we obtain the two-dimensional pixel coordinates of the spatial points projected onto the image plane of the hyperspectral image from the corresponding viewpoint. This completes the construction of the mapping model of the global 3D point cloud projected onto the image plane of the hyperspectral image from the corresponding viewpoint. ； in,( , The coordinates are two-dimensional pixel coordinates obtained by projecting a spatial point in the world coordinate system onto the image plane of a hyperspectral image at the corresponding viewpoint.

4. The multi-view hyperspectral three-dimensional reconstruction and spectral mapping method according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41: Read the mask image of the current viewpoint, use the mask image to binarize the hyperspectral image corresponding to the current viewpoint, divide the target area and background area, generate a binary mask of the target area, and complete the mask preprocessing; S42: Based on the camera's intrinsic and extrinsic parameters corresponding to the current viewpoint, project the global 3D point cloud onto the image plane of the hyperspectral image of the current viewpoint, and remove invalid spatial points located behind the camera and outside the frame of the hyperspectral image corresponding to the current viewpoint. S43: Use a binary mask to process the pixels corresponding to the image plane of the hyperspectral image projected onto the current viewpoint from the global 3D point cloud, and remove the spatial points corresponding to the pixels falling in the background area. S44: From the current perspective, remove isolated noise points with abnormal neighborhood distance distribution within the target area; S45: Replace the mask image of the current view with the mask image of the next view. Repeat steps S41-S44 to remove all invalid spatial points in the global 3D point cloud and complete the selection of valid spatial points in the global 3D point cloud.

5. The multi-view hyperspectral three-dimensional reconstruction and spectral mapping method according to claim 1, characterized in that: Step S5 specifically includes: S51: Select effective spatial points in the normalized point cloud of the target surface, and search for neighborhood spatial points of the effective spatial points with a fixed radius to obtain a local neighborhood spatial point set, so as to construct the covariance matrix of the current effective spatial points: ； in, Let i be the covariance matrix of the current valid spatial points, and i be the index number of the neighboring spatial points. P i Let be the three-dimensional coordinates of the i-th neighborhood point. The three-dimensional coordinates of the current valid spatial point are given, N is the total number of neighborhood spatial points contained in the local neighborhood spatial point set, and T is the transpose. S52: Perform eigenvalue decomposition on the covariance matrix of the current valid space point, extract the smallest eigenvalue among the three eigenvalues ​​of the covariance matrix, and use the eigenvector corresponding to the smallest eigenvalue as the surface normal vector n of the current valid space point; S53: Normalize the surface normal vector of the current valid space point to obtain the unit normal vector of the surface of the current valid space point: ； in, The surface normal vector of the current valid spatial point. Let be the magnitude of the surface normal vector of the current valid spatial point. This is the unit normal vector of the surface of the currently valid spatial point; S54: Replace the current valid spatial point with the next valid spatial point, repeat steps S51-S53, obtain the unit normal vector of the surface of each valid spatial point, and establish the corresponding KDTree spatial index according to the valid spatial points corresponding to the hyperspectral images from different perspectives.

6. The multi-view hyperspectral three-dimensional reconstruction and spectral mapping method according to claim 1, characterized in that: In step S6, the specific steps for calculating the target viewpoint include: S6A1: Select the spatial point to be processed in the normalized point cloud of the target surface, and calculate the unit incident ray vector from the camera optical center to the current spatial point to be processed from the current viewpoint: ； in, Let be the unit incident ray vector pointing from the camera's optical center to the current point in space to be processed. The three-dimensional coordinates of the point to be processed. The three-dimensional coordinates of the camera's optical center from the current perspective; S6A2: Calculate the angle between the unit incident ray vector from the camera's optical center to the current point in space and the unit normal vector to the surface of the current point in space. : ； ； in, Let r be the two-dimensional projection vector of the unit normal vector of the surface of the current point to be processed onto the XOZ plane of the world coordinate system. xoz Let be the two-dimensional projection vector of the unit incident ray vector from the camera's optical center to the current point in space onto the XOZ plane of the world coordinate system. and These are the X-axis and Z-axis components of the unit incident ray vector from the camera's optical center to the current point in space, respectively, in the world coordinate system. and These are the X-axis and Z-axis components of the unit normal vector of the surface of the current point to be processed in the world coordinate system, respectively. S6A3: Replace the current view with the next view. Repeat steps S6A1-S6A2 to calculate the angle between the unit incident ray vector of the camera optical center pointing to the current spatial point and the unit normal vector of the surface of the current spatial point under different view. Take the view with the smallest angle as the target view corresponding to the current spatial point. S6A4: Replace the current spatial point to be processed with the next spatial point to be processed, and repeat steps S6A1-S6A3 until the target viewpoints corresponding to all spatial points to be processed are obtained.

7. The multi-view hyperspectral three-dimensional reconstruction and spectral mapping method according to claim 1, characterized in that: In step S6, the specific steps for calculating the target pixel points corresponding to the target viewpoint include: S6B1: Based on the KDTree spatial index corresponding to the current target viewpoint, retrieve and extract the candidate point cloud corresponding to each pixel of the image plane of each hyperspectral image under the current target viewpoint. S6B2: Calculate the Euclidean distance between the candidate spatial points contained in the candidate point cloud corresponding to the current pixel to be processed under the current target view and the camera optical center corresponding to the current target view, and take the candidate spatial point with the smallest Euclidean distance to the camera optical center as the target spatial point. S6B3: Replace the current pixel with the next pixel to be processed and repeat step S6B2 to obtain all target spatial points under the current target view. S6B4: Replace the current target view with the next target view, and repeat steps S6B1-S6B3 until all target spatial points under all target views are obtained. S6B5: Based on the standardized point cloud of the target surface in the world coordinate system, calculate the voxel size of each dimension according to the preset voxel grid number: ； in, , These represent the voxel dimensions of the target space point along the x-axis, y-axis, and z-axis, respectively; that is, the side length of each voxel along the corresponding coordinate axis. , and These represent the maximum x-axis, maximum y-axis, and maximum z-axis coordinates of the target point in the world coordinate system, respectively. , and Let x be the minimum x-axis, y-axis, and z-axis coordinates of the target point in the world coordinate system. , and The preset number of voxel grids for the target space point in the x-axis, y-axis, and z-axis directions of the world coordinate system; S6B6: Based on the calculation results of step S6B5, voxelize the standardized point cloud and calculate the voxel index corresponding to each target spatial point. , , ): ； in,( , , () represents the three-dimensional coordinates of the target point in space; S6B7: Calculate the ray vectors of the camera optical center pointing to the corresponding target spatial points under different target viewpoints, count the number of voxels occupied by each ray vector, and take the pixels corresponding to the target spatial points whose voxel count does not exceed the threshold as target pixels.

8. The multi-view hyperspectral three-dimensional reconstruction and spectral mapping method according to claim 1, characterized in that: In step S7, the full-band reflectivity information of the target pixels is extracted from each target viewpoint: ； in, This is the full-band reflectance spectral vector. From the perspective of the target, These are the pixel coordinates of the target pixel from the target's perspective.

Images