Fruit tree canopy three-dimensional model reconstruction method and reconstruction system based on multi-modal data

Abstract

The invention provides a fruit tree canopy three-dimensional model reconstruction method and a fruit tree canopy three-dimensional model reconstruction system based on multi-modal data, and relates to the technical field of fruit tree canopy three-dimensional model reconstruction based on multi-modal data. Sub-pixel-level space registration is achieved by means of ground control points, voxelization division is carried out based on the range of a point cloud bounding box, and the number of point clouds is screened to construct an initial seed set; and adopting a structure-spectrum collaborative growth strategy, taking the adjacent voxels which are not accessed as candidate voxels, calculating a spectrum similarity mean value of the candidate voxels and directly adjacent voxels in the seed set, stopping growth when no new voxels are added in continuous three times of iteration, and outputting a final seed set. Meanwhile, the set is converted into a three-dimensional canopy model fusing space coordinates and full-wave-band spectral reflectivity, voxel-level pathological labeling is achieved based on the chlorophyll sensitivity index and the blue-red slope, and the whole process of reconstruction and analysis is completed.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional model reconstruction of fruit tree canopies from multi-modal data, and specifically provides a method and a system for reconstructing a three-dimensional model of a fruit tree canopy based on multi-modal data. Background Art

[0002] The structural and physiological state information of fruit tree canopies is of great significance for precision agriculture management and early pest and disease identification. Traditional three-dimensional canopy modeling methods mostly rely on single-modal data, such as lidar point clouds or visible light images. Although they have their own advantages in spatial structure modeling or species classification, due to the lack of information support in spectral dimensions such as hyperspectral, they cannot effectively express the spectral reflection characteristics of vegetation, especially in the spatial identification of canopy growth status, physiological stress or disease distribution. In addition, existing point cloud and image registration methods often only achieve rough spatial alignment, lacking the processing of natural plant disturbances such as wind-induced jitter and sub-pixel mapping accuracy, resulting in large registration errors and affecting the accuracy of the modeling results.

[0003] Existing technologies have tried to project hyperspectral data onto point cloud models, but due to insufficient spatial registration accuracy and wind-induced foliage displacement, there is a spatial misalignment between spectral and structural data, and it is impossible to establish a sub-voxel-level mapping relationship; while pure spectral analysis methods can detect the physiological state of leaves, but due to the lack of three-dimensional spatial positioning ability, it is difficult to distinguish the disease distribution between the inner and outer layers of the canopy. More critically, existing canopy segmentation algorithms, such as the region growing method, only rely on a single criterion of spatial continuity or spectral similarity, which is prone to overgrowth in complex fruit tree scenarios, such as mis-incorporating the background sky, or growth interruption, such as missing detection of occluded areas. For example, the morphological method by Chen et al. misjudges dead branches as healthy canopies due to ignoring spectral consistency.

[0004] The above information disclosed in the background art section is only used to enhance the understanding of the background of the present disclosure, and thus it may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and a system for reconstructing a three-dimensional model of a fruit tree canopy based on multi-modal data, so as to solve the problems raised in the above background art.

[0006] To achieve the above purpose, the present invention provides the following technical solutions: A method for reconstructing a three-dimensional model of a fruit tree canopy based on multi-modal data, the specific steps include: Step 1: Obtain the three-dimensional structure information and spectral information of the fruit tree canopy in the area to be monitored. The three-dimensional structure information is the point cloud data obtained by lidar scanning, and the spectral information is the hyperspectral image corresponding to the canopy. Unify the point cloud and the image to the world coordinate system through ground control points, and establish an accurate mapping relationship between the point cloud coordinates and the hyperspectral pixels; Step 2: Based on the bounding box range of the registered point cloud data, equally divide this range at a preset voxel scale to generate voxel units covering the entire canopy space to construct the current voxel set. Count the number of point clouds contained in each voxel unit, and determine whether it is occupied according to the preset occupancy threshold. Compose the voxel units that meet the conditions into a seed voxel set; Step 3: Take the seed voxel set as the growth starting point, and use the structure-spectral collaborative growth strategy for iterative expansion. In each round of iteration, select candidate voxels that are directly adjacent in space to any voxel in the seed voxel set from the current voxel set. For this candidate voxel, calculate the spectral feature similarity between it and the voxels adjacent to it in space in the seed voxel set. Only when the spectral feature similarity of the candidate voxel meets the preset consistency threshold, include it in the seed voxel set. If no new voxels are included in the current voxel set for three consecutive iterations, it is determined that the growth process ends, and the final seed voxel set is output; Step 4: Convert the final seed voxel set into a three-dimensional canopy model that integrates spatial structure and spectral information. In this model, each voxel unit contains its three-dimensional spatial coordinates and its corresponding spectral reflectance data. Based on the spectral reflectance characteristics of the voxel units, automatically annotate the vegetation physiological state they represent, and complete the three-dimensional reconstruction and physiological state analysis of the canopy.

[0007] Furthermore, the logic for obtaining the three-dimensional structure information and spectral information of the fruit tree canopy in the area to be monitored is as follows: Lay out no less than 6 high-reflectivity ground control points in the monitoring area, and use lidar scanning to obtain the canopy point cloud data and the three-dimensional coordinates of the control points with millimeter-level accuracy; Synchronously obtain the hyperspectral image covering the canopy area, and extract the pixel coordinates of each control point in the image; Calculate the spatial transformation matrix between the point cloud and the hyperspectral image based on the bidirectional least squares matching algorithm, and apply thin plate spline deformation compensation to the canopy edge area to eliminate geometric distortion caused by wind-induced foliage displacement; Establish a sub-pixel-level accuracy mapping relationship between the point cloud and the hyperspectral pixels, ensuring that the registration error does not exceed 0.2 pixels.

[0008] Furthermore, the logic for equally dividing the range at a preset voxel scale based on the bounding box range of the registered point cloud data is as follows: Based on the three-dimensional spatial bounding box of the registered point cloud data, at a preset voxel size Taking the basic unit, equally spaced division is carried out in the three coordinate axes of X, Y, and Z to generate three-dimensional voxel units covering the entire canopy space; The range of the three-dimensional space bounding box is determined by the minimum value and the maximum value of the point cloud coordinates, ensuring that all point clouds are included in the voxel units; Only when the number of point clouds in a voxel unit is greater than or equal to 3, and the standard deviation of the Euclidean distances between all point pairs in this voxel meets , it is determined that this voxel unit meets the occupancy condition and is included in the initial seed voxel set. After being included in the set, atmospheric scattering correction and solar altitude angle-related radiation compensation processing are performed on the hyperspectral data mapped to this voxel unit to improve the consistency and comparability of the spectral data.

[0009] Furthermore, taking the seed voxel set as the growth starting point, the logic of the iterative structure-spectral collaborative growth strategy is as follows: The iterative process of the structure-spectral collaborative voxel growth strategy specifically includes: Starting from the seed voxel set for iteration, mark all voxels in the intersection of the seed voxel set and the current voxel set as the visited state, Let it be the seed voxel set for the t-th iteration. Traverse the 26 spatial adjacent positions of each visited voxel unit in the current voxel set. If there is an unvisited voxel unit in the adjacent position, mark it as the candidate voxel c, and confirm that the candidate voxel c is spatially adjacent to at least one voxel unit in the seed voxel set; For each candidate voxel c, calculate its spectral angle similarity with all voxels in the seed voxel set. The formula is: ; Among them, is the wavelength variable, with a range of 400 nanometers to 2500 nanometers, is the reflectance of the candidate voxel at wavelength , is the average reflectance of the candidate voxel, is the reflectance of the i-th seed voxel at , is the average reflectance of the i-th seed voxel, is the number of voxels in the seed voxel set, and i is the index of the seed voxels in the seed voxel set; If is satisfied, it is considered that there is consistency in the spectral space between this candidate voxel and the seed voxel set , allowing it to be added, and update the seed voxel set.

[0010] Furthermore, the candidate voxels are added to the seed voxel set Another constraint is that the spatial connectivity satisfies the following conditions: ; where is the total volume of the void voxels between the candidate voxels and the seed voxel set in the t-th iteration. The void voxels refer to the voxel units that are located between the candidate voxels and the seed voxel set but have not been occupied by any voxels, is the volume of the seed voxel set, is the number of voxels within is the voxel volume, is the central coordinate of the candidate voxel, is the central coordinate of the voxel with the closest Euclidean distance to the candidate voxel in represents the Euclidean distance.

[0011] Furthermore, the growth termination needs to satisfy both: 1) The number of newly added voxels in three consecutive iterations does not exceed 0.1% of the total number of the seed voxel set in the previous iteration round; 2) The spectral information entropy converges. Specifically: ; where represents the Shannon entropy of the reflectance of the seed voxel set in the 680 - 750 nm band at the t-th iteration, is the index of the reflectance histogram partition, is the number of reflectance histogram partitions, is the probability that the reflectance value falls into the k-th interval at the t-th iteration; When there are no newly added voxels in three consecutive iterations and the number of newly added voxels in three consecutive iterations does not exceed 0.1% of the total number of the seed voxel set in the previous iteration round, the growth terminates and the final seed voxel set is output .

[0012] Furthermore, each voxel unit in the final seed voxel set is resolved into a three-dimensional spatial entity, and its attributes include: geometric attributes and spectral attributes: The geometric attributes are the central coordinate of the voxel and the preset voxel size , and the spectral attribute is the hyperspectral reflectance vector mapped by the voxel; A three-dimensional canopy model integrating structure and spectrum is constructed based on the geometric and spectral attributes. This model is stored in the form of voxel units, and each unit contains coordinate and reflectance data; The logic for automatically annotating its vegetation physiological state based on its spectral characteristics is: Calculate the chlorophyll sensitive index of the voxel unit in the three-dimensional canopy model at 680 nm and 750 nm: When then it is determined that the voxel is in a stressed state, otherwise it is labeled as healthy; are the reflectances of the voxel at 750 nm and 680 nm respectively; For the voxels determined to be stressed, further calculate the reflectance slope in the blue band If then it is identified as a bacterial disease; the reflectance slope in the red band If then it is determined as a fungal disease; if neither of the two conditions is met, it is labeled as an unknown stress; are the reflectances of the voxel at 490 nm, 530 nm, and 700 nm bands respectively; Output the pathological label of each voxel and its spatial coordinates to obtain a three-dimensional canopy pathological spatial distribution map for vegetation disease localization and health status assessment.

[0013] The present invention further provides a three-dimensional model reconstruction system for a fruit tree canopy based on multi-modal data, and the system is used to execute the above-mentioned method for reconstructing a three-dimensional model of a fruit tree canopy based on multi-modal data, including: A data acquisition module for acquiring the three-dimensional structure information and spectral information of the fruit tree canopy in the area to be monitored. The three-dimensional structure information is the point cloud data obtained by lidar scanning, and the spectral information is the hyperspectral image corresponding to the canopy. The point cloud and the image are unified to the world coordinate system through ground control points to establish an accurate mapping relationship between the point cloud coordinates and the hyperspectral pixels; A set determination module for equally dividing the range based on the bounding box range of the registered point cloud data at a preset voxel scale to generate voxel units covering the entire canopy space to construct the current voxel set, counting the number of point clouds contained in each voxel unit, and determining whether it is occupied according to a preset occupancy threshold, and forming a seed voxel set from the voxel units that meet the conditions; An iterative update module for using the seed voxel set as the growth starting point and performing iterative expansion using a structure-spectral co-growth strategy. In each iteration, select candidate voxels that are directly adjacent in space to any voxel in the seed voxel set from the current voxel set. For this candidate voxel, calculate the spectral feature similarity between it and the voxel adjacent to it in space in the seed voxel set. Only when the spectral feature similarity of the candidate voxel meets the preset consistency threshold, include it in the seed voxel set. If no new voxels are included in the current voxel set for three consecutive iterations, it is determined that the growth process ends, and the final seed voxel set is output; The model analysis module is used to convert the final set of seed voxels into a three-dimensional canopy model that integrates spatial structure and spectral information. In this model, each voxel unit contains its three-dimensional spatial coordinates and the corresponding spectral reflectance data. Based on the spectral reflectance characteristics of the voxel units, the physiological state of the vegetation they represent is automatically labeled, completing the three-dimensional reconstruction and physiological state analysis of the canopy.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: By constructing an accurate mapping relationship between the point cloud based on ground control points and the hyperspectral image, the present invention unifies the three-dimensional spatial structure information and the spectral reflectance information into the same coordinate system, effectively solving the problem of insufficient registration accuracy of multimodal data in the prior art; using this accurate registration result, the corresponding structural points in the hyperspectral image can be accurately located at the voxel division stage, improving the basic accuracy of the subsequent joint modeling of the canopy structure and physiological state; further, by performing spatial division and occupancy determination on the point cloud data to generate a set of seed voxels, the data in redundant or non-canopy regions can be filtered out, thus significantly improving the reliability and representativeness of the voxel growth starting point, and avoiding misgrowth and model drift caused by improper selection of the initial region.

[0015] The present invention introduces a structure-spectral collaborative growth strategy during the voxel expansion process: in each iteration, it is strictly specified that the candidate voxels must be spatially adjacent to the current voxel set, and it is required that their spectral characteristics meet the consistency threshold with the candidate voxels. Only the voxels that satisfy both the spatial connectivity and spectral similarity conditions can be incorporated into the growth model; this strategy effectively avoids the pseudo-expansion problem caused by structural noise or spectral outliers, ensuring the natural closure of the model boundary and the coherence of spectral expression.

[0016] The present invention adopts a dynamic convergence mechanism of "terminating growth when there are no new voxels added for three consecutive rounds", which can automatically determine whether the model reaches a stable state, avoid over-expansion or falling into local abnormal fluctuations, and improve the robustness and intelligence level of canopy boundary determination. Through the synergistic effect of the above technical features, the present invention can achieve three-dimensional canopy modeling with high structural restoration degree and high spectral expression accuracy, significantly superior to the modeling effects of existing single-modal or regular expansion methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the overall method flow of the present invention; Figure 2 It is a flow chart of the overall system module of the present invention; Figure 3 It is a curve graph of the spectral angle similarity - average reflectance difference of the present invention; Figure 4 It is a broken line graph of the average reflectance of candidate voxels - average reflectance of seed voxels of the present invention; Figure 5 This is the graph of the slope of the reflectance in the red band of the chlorophyll sensitivity index of the present invention; Figure 6 This is the broken line graph of the slope of the reflectance in the blue band of the chlorophyll sensitivity index of the present invention. Detailed implementation manners

[0018] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with specific embodiments.

[0019] It should be noted that unless otherwise defined, the technical terms or scientific terms used in the present invention should have the ordinary meanings understood by those with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. The terms such as "including" or "comprising" mean that the elements or objects appearing before this term cover the elements or objects listed after this term and their equivalents, without excluding other elements or objects. The terms such as "connected" or "linked" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left" and "right" are only used to represent relative positional relationships, and when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] Embodiment: Please refer to Figures 1 - 6 , the present invention provides a technical solution: A method for reconstructing a three-dimensional model of a fruit tree canopy based on multi-modal data, the specific steps include: Step 1: Obtain the three-dimensional structure information and spectral information of the fruit tree canopy in the area to be monitored. The three-dimensional structure information is the point cloud data obtained by lidar scanning, and the spectral information is the hyperspectral image corresponding to the canopy. The point cloud and the image are unified to the world coordinate system through ground control points, and an accurate mapping relationship between the point cloud coordinates and the hyperspectral pixels is established; The logic for obtaining the three-dimensional structure information and spectral information of the fruit tree canopy in the area to be monitored is: Deploy no less than 6 high-reflectivity ground control points in the monitoring area, and use lidar scanning to obtain the canopy point cloud data and the three-dimensional coordinates of the control points with millimeter-level accuracy; synchronously obtain the hyperspectral image covering the canopy area, and extract the pixel coordinates of each control point in the image; calculate the spatial transformation matrix between the point cloud and the hyperspectral image based on the bidirectional least squares matching algorithm, and apply thin plate spline deformation compensation to the canopy edge area to eliminate geometric distortion caused by wind-induced foliage displacement; Establish a sub-pixel accuracy mapping relationship between the point cloud and hyperspectral pixels, ensuring that the registration error does not exceed 0.2 pixels; To achieve spatial registration between the lidar point cloud and hyperspectral image, a set of ground control points with observable three-dimensional spatial coordinates and image pixel coordinates must be deployed within the monitoring area. At least 6 control points should be deployed. The reasons include: Three-dimensional affine / projection transformation fitting usually requires at least 6 control points to ensure the uniqueness and robustness of the solution. The control points should be evenly distributed to cover the entire canopy range, avoiding the expansion of registration errors in the edge area. High-reflectivity materials such as painted metal balls and white PVC are used for the control points to ensure that they are clearly identifiable in both lidar and hyperspectral images; The lidar scans with laser beams to obtain three-dimensional structure information and can output high-density point clouds including the canopy surface, tree trunk structure, and branch and leaf distribution. At the same time, during the scanning process, the reflection intensity of the control points is significantly higher than that of the background vegetation, facilitating the extraction of the control point positions from the point cloud. To ensure the registration accuracy, the coordinate accuracy of the control points is required to reach the millimeter level, such as within ±2mm, which is often obtained by combining high-precision RTK-GNSS and total station; Synchronous acquisition refers to using a hyperspectral imaging system, such as a visible-near infrared push-broom hyperspectral camera, to collect spectral images of the canopy area within a time window as synchronous as possible with the lidar scanning. The hyperspectral image is a three-dimensional data cube , where represents the band dimension. Due to their special color or reflectivity characteristics, the control points can also appear as obvious feature points in the hyperspectral image, such as bright circular spots. Their two-dimensional pixel coordinates are obtained through image processing methods such as brightness thresholding plus template matching; The bidirectional least squares matching algorithm is a geometric registration method that can achieve the optimal alignment between the point cloud control point coordinates and the image pixel coordinates. The core idea of the algorithm is: simultaneously minimize the projection error of the point cloud projected onto the image plane and the reconstruction error of the image pixels back-projected into three-dimensional space, so that the transformation matrix obtains the minimum residual in both spaces; the output result is a spatial transformation matrix, which includes rotation, translation, and scale change parameters and is used to accurately map the three-dimensional coordinates to the image coordinate system; In the actual monitoring of the fruit tree canopy, due to wind disturbances, the leaves and branches will jitter in position at different times, resulting in a non-rigid offset between the point cloud and the image. The thin plate spline is a smooth non-rigid deformation model that can model the local non-linear differences between point sets. Based on the control point registration, this invention applies TPS deformation compensation to the canopy edge area (most affected by the wind), making the registration relationship between the point cloud and the image locally continuous and physically consistent, and significantly improving the registration quality; After completing the transformation matrix calculation and deformation compensation, each point in the three-dimensional point cloud is projected onto the image coordinate system to obtain its corresponding pixel coordinates. Since pixel coordinates are floating-point data, significant errors may be introduced if only integer pixels are taken. Therefore, methods such as bilinear interpolation or cubic interpolation are used to map the three-dimensional points to sub-pixel positions on the image. Finally, a one-to-one correspondence table between the point cloud points and the hyperspectral pixels is constructed to ensure that each point cloud can obtain a hyperspectral vector. To ensure accuracy, the maximum allowable projection residual in this scheme is set to 0.2 pixels, and the error is further reduced through residual rejection and iterative optimization.

[0021] Step 2: Based on the bounding box range of the registered point cloud data, equally spaced division is performed on this range at a preset voxel scale to generate voxel units covering the entire canopy space to construct the current voxel set. Count the number of point clouds contained in each voxel unit, and determine whether it is occupied according to the preset occupancy threshold. The voxel units that meet the conditions are grouped into a seed voxel set; The logic of equally spaced division of the range based on the bounding box range of the registered point cloud data is as follows: Based on the three-dimensional space bounding box of the registered point cloud data, with the preset voxel size as the basic unit, equally spaced division is performed in the X, Y, and Z coordinate axes directions to generate three-dimensional voxel units covering the entire canopy space; The registered point cloud data refers to the three-dimensional point set that has been registered through lidar point cloud and hyperspectral image in the previous stage, and has been unified into a common world coordinate system to ensure the one-to-one correspondence of structural and spectral information. The three-dimensional space bounding box is the smallest cuboid space boundary box used to completely contain all point clouds. The bounding box can be constructed by statistically calculating the minimum / maximum values of all point cloud points in each axis direction; The preset voxel size is the side length of a single cubic voxel, which determines the spatial resolution of the three-dimensional grid. The smaller the resolution, the finer the model. Equally spaced division means that in the X, Y, and Z directions, the space coordinates are divided from the minimum value to the maximum value at the step size, and the entire space is divided into a three-dimensional voxel grid, forming a regular spatial grid structure; The range of the three-dimensional space bounding box is determined by the minimum value and the maximum value of the point cloud coordinates, ensuring that all point clouds are included in the voxel units; The number of voxels divided in each coordinate direction is respectively: ; Only when the number of point clouds in a voxel unit is greater than or equal to 3, and the standard deviation of the Euclidean distances between all points in the voxel meets When it is determined that the voxel unit meets the occupancy condition, it is included in the seed voxel set. After inclusion, the hyperspectral data mapped to the voxel unit is subjected to atmospheric scattering correction and solar elevation angle-related radiation compensation processing to improve the consistency and comparability of the spectral data; At least 3 points are required to form the smallest geometric structure to prevent the introduction of noise points or isolated points. Calculate the Euclidean distance between any two points in the voxel, and then calculate the standard deviation of the distances of all point pairs. , represents the maximum possible distance between any two points in the voxel, is the threshold for the allowed spatial compactness; The established sub-pixel mapping relationship between the point cloud and the hyperspectral image can be used to reverse-lookup the spectral pixel positions corresponding to all points in each voxel, extract its hyperspectral data, and eliminate the spectral distortion caused by atmospheric molecular / aerosol scattering. For example, correction is performed using MODTRAN / 6S or the empirical linear method. Considering the influence of the solar angle on the radiation intensity during shooting, the cosine law is used to adjust the spectral reflectance value so that the images acquired at different times / angles are comparable.

[0022] Step 3: Use the seed voxel set as the growth starting point and adopt a structure-spectral collaborative growth strategy for iterative expansion. In each iteration, select candidate voxels that are spatially directly adjacent to any voxel in the seed voxel set from the current voxel set. For this candidate voxel, calculate the spectral feature similarity between it and the voxels in the seed voxel set that are spatially adjacent to it. Only when the spectral feature similarity of the candidate voxel meets the preset consistency threshold, it is included in the seed voxel set. If no new voxels are included in the current voxel set for three consecutive iterations, it is determined that the growth process ends, and the final seed voxel set is output; Using the seed voxel set as the growth starting point and adopting a structure-spectral collaborative growth strategy for iteration, the logic is as follows: The iterative process of the structure-spectral collaborative voxel growth strategy specifically includes: Start iteration with the seed voxel set and mark all voxels in the intersection of the seed voxel set and the current voxel set as visited. Let \(S_t\) be the seed voxel set in the \(t\) -th iteration. Traverse the 26 spatial adjacent positions of each voxel unit in the current voxel set. If there is an unvisited voxel unit at the adjacent position, mark it as candidate voxel \(c\), and confirm that candidate voxel \(c\) is spatially adjacent to at least one voxel unit in the seed voxel set. Each voxel has 26 direct adjacent voxels (excluding itself), called the three-dimensional Moore neighborhood. Candidate voxels are adjacent voxels that have not participated in iterative calculations before. Only candidate voxels adjacent to at least one voxel in the seed voxel set are retained to ensure structural connectivity; The candidate voxels are added to the seed voxel set Another constraint for spatial connectivity is that it satisfies the following conditions: ; Among them, is the total volume of the void voxels between the candidate voxels and the existing voxels in the t-th iteration, is the volume of the seed voxel set, is the number of voxels inside, is the voxel volume, is the central coordinate of the candidate voxel, is the central coordinate of the voxel with the closest Euclidean distance to the candidate voxel in represents the Euclidean distance; is the total volume of the spatial void voxels between the candidate voxels and the seed voxel set in the t-th iteration. Void voxels refer to voxel units that are located between the candidate voxels and the seed voxel set but have not been occupied by any voxels; A strict threshold is set, which means that there is almost no interrupted voxel jump space between the candidate voxels and the existing model, and they are very closely connected spatially; The larger is, the greater the spatial jump between the candidate voxels and the existing model, which may be unreasonable fracture growth, such as suddenly growing through the air. If it is smaller, it means the structure is tight and the spatial connectivity is good, allowing growth; The smaller is, the closer the candidate voxels are to the existing model structure, and the more compact the spatial structure is. The larger it is, the more serious the structural fracture is, and access is not allowed even if the spectra are consistent; For each candidate voxel c, calculate the spectral angle similarity between it and all voxels in the seed voxel set The formula is: ; where is the wavelength variable, ranging from 400 nanometers to 2500 nanometers, is the reflectance of the candidate voxel at wavelength , is the average reflectance of the candidate voxel, is the reflectance of the i-th seed voxel at , is the average reflectance of the i-th seed voxel, is the number of voxels in the seed voxel set, and i is the index of the seed voxels in the seed voxel set; is the average spectral angle similarity of the candidate voxel c, which measures the average similarity degree of the candidate voxel with all adjacent voxels in the spectral space. The smaller it is, the more similar. An angle of 0 means complete consistency; the wavelength represents the band of hyperspectral reflectance, and the integration process covers all bands of interest, reflects the plant tissue state of this voxel, such as moisture, chlorophyll, etc., , is used to remove the influence of spectral intensity differences and focus on spectral shape matching; represents the spectral co-deviation of two voxels, similar to covariance, which measures the consistency of two voxels at each wavelength after deviating from their respective averages; and are the energy or standard deviation of the spectra of two voxels respectively, used for normalization to cancel the absolute amplitude difference and only focus on shape matching; is the cosine similarity, which maps the similarity to an angle. The smaller it is, the better. A small angle represents more consistent spectral directions; Specific data of some sample numbers and spectral angle similarities are shown in Table 1.

[0023]

[0024] By analyzing the above 15 groups of voxel spectral similarity data, a positive correlation was observed between the average reflectance difference between voxels and the spectral angle similarity. For example, the average reflectances of the candidate voxel and the seed voxel of sample number 1 were 0.31 and 0.29 respectively, with a corresponding difference of 0.02, and the number of seed voxels was 5. The calculated spectral angle similarity was only 0.153, indicating that under the conditions of small reflectance difference and moderate adjacency number, the spectral shapes between voxels were highly consistent. Similarly, for sample number 6, the difference was smaller (0.01) and the number of seed voxels was larger (8), and its spectral angle similarity further decreased to 0.134, indicating that more spatial neighborhood metrics could improve the robustness of similarity calculation.

[0025] For most samples (such as numbers 3, 7, 9, 13), when the reflectance difference was maintained between 0.02 - 0.04, the spectral angle similarity remained at a relatively low level of 0.16 - 0.21. This indicates that as long as the reflectance difference between the candidate voxel and the seed voxel does not exceed 0.05, and the number of seed voxels is not less than 4, the spectral shapes of the voxels have almost no significant deviation and can be smoothly incorporated into the canopy model. Taking sample number 8 as an example, its reflectance difference was 0.04, the number of seed voxels was 6, and the corresponding similarity was 0.210. Although slightly higher than 0.153 of number 1, it was still lower than the commonly used growth threshold of about 0.55 (corresponding cosine similarity of 0.85).

[0026] In the case where the difference was close to the upper limit of the threshold (such as numbers 4 and 10, with differences both being 0.055), the similarity increased to above 0.25, reflecting an increase in spectral shape deviation and being closer to the rejection boundary for growth criteria. The spectral angle similarity of number 4 was 0.251 and that of number 10 was 0.256, both significantly higher than the results when the difference was 0.02 - 0.04, indicating that when the reflectance difference exceeded 0.05, even if the number of seed voxels was relatively large (4 for number 10), there would be a large spectral angle deviation, and it was necessary to comprehensively judge by combining spatial connectivity and other criteria. Generally speaking, this data distribution verified the mathematical logic between "reflectance shape difference" and "spectral similarity angle" in the formula, providing an empirical basis for setting the voxel growth threshold.

[0027] If is satisfied, then it is considered that the candidate voxel and the seed voxel set are consistent in spectral space, allowing addition, and updating the seed voxel set; The growth termination needs to simultaneously meet: 1) The number of newly added voxels in three consecutive iterations does not exceed 0.1% of the total number of the seed voxel set in the previous iteration round; 2) The spectral information entropy converges, specifically: ; Among them, Denote the Shannon entropy of the reflectance of the seed voxel set at the t-th iteration in the 680 - 750 nm band, is the index of the reflectance histogram partition, is the number of reflectance histogram partitions, is the probability that the reflectance value at the t-th iteration falls into the k-th interval; When there are no new voxels in three consecutive iterations and the number of newly added voxels in three consecutive rounds of iteration does not exceed 0.1% of the total number of the seed voxel set in the previous iteration round, the growth terminates and the final seed voxel set is output ; Divide the band reflectance into intervals, extract all seed voxel sets inside, for each voxel, extract the reflectance curve within the range of 680 - 750 nm, count the distribution of these reflectance data, and construct a normalized histogram: the total number of samples is N, and there are values falling into the k-th interval, then the probability is , substitute it into the entropy formula for calculation; Entropy measures the spectral uncertainty or diversity of the seed voxel set. If the reflectance distribution is more dispersed, that is, the reflectances in each band have large differences, or the differences between voxels are large, then the entropy value is high; if the reflectance is concentrated in certain values, that is, the spectral consistency of voxels is enhanced, the entropy value is low; If is more average, then the distribution is more dispersed and the entropy is higher. If is more concentrated, then the distribution is more concentrated and the entropy is lower. If the number of voxels N increases but the distribution remains unchanged, it does not affect the entropy and the entropy remains the same. If is more, then the partition is refined and the entropy may increase and be more sensitive; Denote that the change amplitude of the current information entropy compared to the previous round drops to within 5%, indicating that the spectral distribution of the seed voxel set has been stable and the newly added voxels have not significantly changed the spectral characteristics of the model; if the entropy value changes are less than the threshold in three consecutive rounds (which can be extended to a sliding window judgment), the spectral structure convergence can be judged.

[0028] Step 4: Convert the final seed voxel set into a three-dimensional canopy model that integrates spatial structure and spectral information. In this model, each voxel unit contains its three-dimensional spatial coordinates and its corresponding spectral reflectance data. Based on the spectral reflectance characteristics of the voxel unit, automatically label the vegetation physiological state it represents, and complete the three-dimensional reconstruction and physiological state analysis of the canopy; Parse each voxel unit in the final seed voxel set into a three-dimensional spatial entity, and its attributes include: geometric attributes and spectral attributes: This is a structured definition for voxel unit modeling. Each voxel is a three-dimensional cube, and its position and attributes are described by the following two types of information: Geometric attributes refer to the position of the voxel in three-dimensional space (i.e., its central coordinates X, Y, Z and the side length of the cube , which are used to reconstruct its physical form and spatial relationship; Spectral attributes are the hyperspectral reflectance vectors corresponding to the voxels, representing the spectral reflectance of each band in the range of 400–2500 nm for the area mapped by the voxel; The geometric attributes are the central coordinates of the voxel and the preset voxel size , and the spectral attributes are the hyperspectral reflectance vectors mapped by the voxel; A three-dimensional canopy model integrating structure and spectrum is constructed based on the geometric attributes and spectral attributes. This model is stored in the form of voxel units, and each unit contains coordinate and reflectance data; This step completes the three-dimensional digital modeling process of the canopy. The goal is to establish a unified expression of spatial structure and spectral characteristics: The voxel grid constitutes the spatial framework of the entire canopy. Each voxel contains its central position and a set of multi-dimensional spectral characteristics, which is a coupling body of structure and function. The storage method is the voxel grid structure, which is convenient for spatial traversal, adjacency judgment, and subsequent voxel-level diagnosis; The logic for automatically annotating the vegetation physiological state based on its spectral characteristics is as follows: Calculate the chlorophyll sensitivity index of the voxel units in the three-dimensional canopy model at 680 nm and 750 nm: , when , then it is determined that the voxel is in a stressed state, otherwise it is labeled as healthy; are the reflectances of the voxel at 750 nm (near-infrared reflection band) and 680 nm (chlorophyll absorption band), is the chlorophyll sensitivity index, which measures the difference between red light absorption and near-infrared reflection; The larger it is, the healthier the vegetation and the more active the photosynthesis. The smaller it is, the more limited the photosynthesis, and there may be diseases or nutrient stress; is the near-infrared band, which refers to the reflection peak of the internal structure of healthy leaves and is an index for the integrity of cell structure, is the chlorophyll absorption valley, which refers to the sensitive area of chlorophyll concentration, that is, the stronger the absorption, the lower the reflection; If , then , the vegetation is healthy. If , then , the vegetation state is suspicious. If , then is negative, which is an atypical or incorrect match; For the voxels determined to be stressed, further calculate the reflectance slope of the blue band , if , it is identified as a bacterial disease; the slope of the reflectance in the red band , if , it is determined as a fungal disease; if neither of the two conditions is met, it is labeled as an unknown stress; reflects the characteristics of the rapid change region of chlorophyll, representing the rising speed of the reflectance within the blue edge band (490 - 530 nm). The rapid rise of the blue edge reflectance is related to the changes in the leaf cell structure and the alteration of bacterial osmotic pressure. Bacterial diseases usually lead to an accelerated rise in the blue edge reflectance, and the value increases significantly; is the ratio of the reflectance at 680 nm to that at 700 nm, reflecting the relative depth of the red valley. Healthy vegetation has the strongest absorption at 680 nm (red valley), and the reflectance rapidly rises at 700 nm. Fungal diseases can damage the cell structure, resulting in a shallower red valley (an increase in the reflectance at 680 nm), and the and are located at the lower and upper ends of the typical blue edge band respectively. The blue edge interval (about 490 - 530 nm) is particularly sensitive to chlorophyll and secondary pigments; is the green light band, referring to the reflectance peak of carotenoids, is the blue - green transition band, which is the absorption region of lutein, is the starting point of the red edge, reflecting the change from the absorption valley of the leaf to the near - infrared reflectance region; The specific data of the sample numbers and chlorophyll sensitivity indices for some samples are shown in Table 2.

[0029]

[0030] Through the analysis of the spectral indices of the first 15 voxel samples, the chlorophyll sensitivity index (CSI) can effectively distinguish between healthy and stressed states. For example, the CSI of sample number 1 is 0.382 (lower than 0.45), so it is determined as a stressed state; its blue edge slope BES is 0.028 (lower than 0.03), and the red valley ratio RVI is 1.045 (lower than 1.1). Neither of them reaches the criteria for bacterial or fungal diseases, so it is labeled as "unknown stress". In contrast, the CSI of sample number 2 is 0.512 (higher than 0.45), and it is directly determined as a healthy state without further disease classification. This shows that CSI has a high discriminative power in the preliminary screening of the health degree of voxels; In stress samples with CSI < 0.45, BES and RVI can respectively indicate different disease types. For example, the CSI of sample No. 3 is 0.421 and the BES is 0.042 (> 0.03), which meets the criteria for bacterial diseases; the CSI of sample No. 7 is only 0.215, while the RVI is as high as 1.150 (> 1.1), so it is labeled as a fungal disease; another example is that samples No. 4 and 11 (CSI are 0.305 and 0.278 respectively) both show relatively high BES (0.050 and 0.040) and excessive RVI (1.035 and 1.200), but due to the priority of the BES threshold for judgment, they are both labeled as bacterial diseases. This indicates that among the stress voxels screened by CSI, BES is particularly sensitive to the identification of bacterial diseases, while RVI provides an effective supplementary criterion for fungal diseases. Through this dual-index combination, it is possible to achieve a refined pathological state positioning of the fruit tree canopy voxels.

[0031] Output the three-dimensional canopy pathological spatial distribution map by matching the pathological label of each voxel with its spatial coordinates, which is used for vegetation disease positioning and health status assessment.

[0032] Please refer to Figure 2 , the present invention further provides a three-dimensional model reconstruction system for a fruit tree canopy based on multi-modal data, which is used to execute the above-mentioned method for reconstructing a three-dimensional model of a fruit tree canopy based on multi-modal data, including: A data acquisition module, which is used to acquire the three-dimensional structure information and spectral information of the fruit tree canopy in the area to be monitored. The three-dimensional structure information is the point cloud data obtained by lidar scanning, and the spectral information is the hyperspectral image corresponding to the canopy. The point cloud and the image are unified to the world coordinate system through ground control points, and an accurate mapping relationship between the point cloud coordinates and the hyperspectral pixels is established; A set determination module, which is used to equally divide the range based on the bounding box range of the registered point cloud data at a preset voxel scale, generate voxel units covering the entire canopy space to construct the current voxel set, count the number of point clouds contained in each voxel unit, and determine whether it is occupied according to a preset occupancy threshold, and form a seed voxel set by combining the voxel units that meet the conditions; An iterative update module, which is used to use the seed voxel set as the growth starting point and perform iterative expansion using a structure-spectral co-growth strategy. In each iteration, select candidate voxels that are directly adjacent in space to any voxel in the seed voxel set from the current voxel set. For this candidate voxel, calculate the spectral feature similarity between it and the voxel adjacent to it in space in the seed voxel set. Only when the spectral feature similarity of the candidate voxel meets the preset consistency threshold, include it in the seed voxel set. If no new voxels are included in the current voxel set for three consecutive iterations, it is determined that the growth process ends, and the final seed voxel set is output; The model analysis module is used to convert the final set of seed voxels into a three-dimensional canopy model that integrates spatial structure and spectral information. In this model, each voxel unit contains its three-dimensional spatial coordinates and the corresponding spectral reflectance data. Based on the spectral reflectance characteristics of the voxel units, the physiological state of the vegetation they represent is automatically annotated, and the three-dimensional reconstruction and physiological state analysis of the canopy are completed.

[0033] The above formulas are all dimensionless and take their numerical values for calculation. The formulas are obtained by collecting a large amount of data for software simulation to get a formula that is closest to the actual situation. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0034] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware or the combination of computer software and electronic hardware. Whether these functions are executed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0035] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units. They can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0036] The above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application, and all should be covered within the protection scope of this application.

Claims

1. A method for reconstructing a three-dimensional model of a fruit tree canopy based on multimodal data, characterized in that: The specific steps include: Step 1: Obtain the three-dimensional structural information and spectral information of the fruit tree canopy in the monitored area. The three-dimensional structural information is point cloud data obtained by laser radar scanning, and the spectral information is a hyperspectral image corresponding to the canopy. Use ground control points to unify the point cloud and image into a world coordinate system, and establish a precise mapping relationship between the point cloud coordinates and the hyperspectral pixels. Step 2: Based on the bounding box range of the registered point cloud data, the range is divided into equal intervals at a preset voxel scale to generate voxel units covering the entire canopy space to construct the current voxel set. The number of point clouds contained in each voxel unit is counted, and whether it is occupied is determined according to the preset occupancy threshold. The voxel units that meet the conditions are formed into a seed voxel set; Step 3: Using the seed voxel set as the starting point for growth, the structure-spectrum co-growth strategy is used for iterative expansion. In each round of iteration, a candidate voxel that is directly spatially adjacent to any voxel in the seed voxel set is selected from the current voxel set. For this candidate voxel, the spectral feature similarity between it and the spatially adjacent voxel in the seed voxel set is calculated. Only when the spectral feature similarity of the candidate voxel meets the preset consistency threshold is it included in the seed voxel set. If no new voxel is included in the current voxel set for three consecutive iterations, the growth process is considered to be terminated and the final seed voxel set is output. Step 4: Convert the final seed voxel set into a three-dimensional canopy model that integrates spatial structure and spectral information. In this model, each voxel unit contains its three-dimensional spatial coordinates and its corresponding spectral reflectance data. Based on the spectral reflectance characteristics of the voxel unit, the physiological state of the vegetation it represents is automatically labeled to complete the three-dimensional reconstruction and physiological state analysis of the canopy.

2. The method for reconstructing a three-dimensional model of a fruit tree canopy based on multimodal data according to claim 1, characterized in that: The logic for obtaining the three-dimensional structural information and spectral information of the fruit tree canopy in the monitored area is as follows: At least six high-reflectivity ground control points are deployed in the monitoring area, and lidar scanning is used to obtain canopy point cloud data and the millimeter-level precision three-dimensional coordinates of the control points. A hyperspectral image covering the canopy area is simultaneously acquired, and the pixel coordinates of each control point in the image are extracted. The spatial transformation matrix between the point cloud and the hyperspectral image is calculated based on a bidirectional least squares matching algorithm, and thin-plate spline deformation compensation is applied to the canopy edge area to eliminate geometric distortion caused by wind-induced branch and leaf displacement. A sub-pixel precision mapping relationship is established between the point cloud and the hyperspectral pixels to ensure that the registration error does not exceed 0.2 pixels.

3. The method for reconstructing a three-dimensional model of a fruit tree canopy based on multimodal data according to claim 2, characterized in that: Based on the bounding box range of the registered point cloud data, the logic of dividing the range into equal intervals at the preset voxel scale is as follows: A 3D bounding box based on the registered point cloud data with a preset voxel size As the basic unit, it is divided into equal intervals in the directions of the X, Y, and Z coordinate axes to generate three-dimensional voxel units covering the entire canopy space; The three-dimensional bounding box range is determined by the minimum value of the point cloud coordinates and maximum value Make sure that all point clouds are contained within the voxel unit; Only when the number of point clouds in a voxel unit is greater than or equal to 3 and the standard deviation of the Euclidean distance between all point pairs in the voxel conform to When the voxel unit is determined to meet the occupancy condition and is included in the initial seed voxel set, after being included in the set, atmospheric scattering correction and radiation compensation related to the solar altitude angle are performed on the hyperspectral data mapped to the voxel unit to improve the consistency and comparability of the spectral data.

4. The method for reconstructing a three-dimensional model of a fruit tree canopy based on multimodal data according to claim 3, characterized in that: The logic of using the seed voxel set as the starting point of growth and adopting the structure-spectrum collaborative growth strategy for iteration is as follows: The iterative process of the structure-spectrum collaborative voxel growth strategy specifically includes: Iterate with the seed voxel set as the starting point of growth and mark the seed voxel set All voxels in the intersection with the current voxel set are in the visited state. For the seed voxel set of iteration t, traverse the 26 spatial adjacent positions of each visited voxel unit in the current voxel set. If there is an unvisited voxel unit in the adjacent position, mark it as a candidate voxel c, and confirm that the candidate voxel c is spatially adjacent to at least one voxel unit in the seed voxel set; For each candidate voxel c, calculate its spectral angle similarity with all voxels in the seed voxel set The formula is: ； in, is a wavelength variable ranging from 400 nm to 2500 nm, For candidate voxels at wavelength The reflectivity, is the average reflectivity of the candidate voxels, is the i-th seed voxel in The reflectivity at is the average reflectivity of the i-th seed voxel, is the number of voxels in the seed voxel set, i is the index of the seed voxel in the seed voxel set; If satisfied , then the candidate voxel is considered to be consistent with the seed voxel set There is consistency between them in the spectral space, allowing them to be joined and the seed voxel set to be updated.

5. The method for reconstructing a three-dimensional model of a fruit tree canopy based on multimodal data according to claim 4, characterized in that: Candidate voxels are added to the seed voxel set Another constraint is that the spatial connectivity satisfies the following conditions: ； in, is the sum of the volumes of candidate voxels and interstitial voxels in the tth iteration, where the interstitial voxel is the spatial position between the candidate voxel and the seed voxel set. The voxel units between them but not yet occupied by any voxel, is the volume of the seed voxel set, for Endosome prime number, is the voxel volume, are the candidate voxel center coordinates, for The coordinates of the voxel center with the closest Euclidean distance to the candidate voxel, represents the Euclidean distance.

6. The method for reconstructing a three-dimensional model of a fruit tree canopy based on multimodal data according to claim 5, characterized in that: The growth termination must simultaneously meet the following requirements: 1) the number of newly added voxels in three consecutive iterations does not exceed 0.1% of the total number of seed voxels in the previous iteration; and 2) the spectral information entropy converges, specifically: ； in, represents the Shannon entropy of the reflectivity of the seed voxel set in the 680-750 nm band at the t-th iteration, is the reflectivity histogram partition index, is the number of reflectance histogram partitions, is the probability that the reflectivity value falls into the kth interval at the tth iteration; When there are no new voxels in three consecutive iterations and the number of new voxels in three consecutive iterations does not exceed 0.1% of the total number of seed voxel sets in the previous iteration, the growth is terminated and the final seed voxel set is output. .

7. The method for reconstructing a three-dimensional model of a fruit tree canopy based on multimodal data according to claim 6, characterized in that: The final seed voxel set Each voxel unit in is resolved into a three-dimensional space entity, whose properties include: geometric properties and spectral properties: The geometric attributes are the voxel center coordinates and preset voxel sizes , the spectral attribute is the hyperspectral reflectance vector mapped by the voxel; a three-dimensional canopy model integrating structure and spectrum is constructed based on the geometric attributes and spectral attributes, and the model is stored in the form of voxel units, each unit containing coordinates and reflectance data; The logic for automatically labeling the physiological status of vegetation based on its spectral characteristics is as follows: Calculate the chlorophyll sensitivity index of voxel cells in the 3D canopy model at 680 nm and 750 nm: ,when , the voxel is judged to be in a stress state, otherwise it is marked as healthy; are the reflectance of the voxel at 750nm and 680nm, respectively; For voxels judged to be stressed, the reflectance slope of the blue band is further calculated. ,like , it is identified as a bacterial disease; the reflectivity slope of the red band ,like , it is determined to be a fungal disease; if neither of the two conditions is met, it is marked as an unknown stress; are the reflectance of the voxel in the 490nm, 530nm, and 700nm bands, respectively; The pathological label of each voxel and its spatial coordinates are used to output a three-dimensional canopy pathology spatial distribution map for vegetation disease location and health status assessment.

8. A fruit tree canopy 3D model reconstruction system based on multimodal data, characterized by: The system is used to execute the method for reconstructing a three-dimensional model of a fruit tree canopy based on multimodal data according to any one of claims 1 to 7, comprising: A data acquisition module is used to obtain the three-dimensional structural information and spectral information of the fruit tree canopy in the monitored area. The three-dimensional structural information is point cloud data obtained by laser radar scanning, and the spectral information is a hyperspectral image corresponding to the canopy. The point cloud and image are unified into a world coordinate system through ground control points to establish a precise mapping relationship between point cloud coordinates and hyperspectral pixels. The set determination module is used to divide the bounding box range of the registered point cloud data into equal intervals at a preset voxel scale, generate voxel units covering the entire canopy space to construct the current voxel set, count the number of point clouds contained in each voxel unit, and determine whether it is occupied according to a preset occupancy threshold. The voxel units that meet the conditions are formed into a seed voxel set; An iterative update module is used to use the seed voxel set as the starting point for growth and iteratively expand using a structural-spectral collaborative growth strategy. In each round of iteration, a candidate voxel that is directly spatially adjacent to any voxel in the seed voxel set is selected from the current voxel set. For each candidate voxel, the spectral feature similarity between it and the spatially adjacent voxels in the seed voxel set is calculated. Only when the spectral feature similarity of the candidate voxel meets a preset consistency threshold is it included in the seed voxel set. If no new voxel is included in the current voxel set for three consecutive iterations, the growth process is determined to be complete and the final seed voxel set is output. The model analysis module is used to convert the final seed voxel set into a three-dimensional canopy model that integrates spatial structure and spectral information. In this model, each voxel unit contains its three-dimensional spatial coordinates and its corresponding spectral reflectance data. Based on the spectral reflectance characteristics of the voxel unit, the physiological state of the vegetation it represents is automatically labeled to complete the three-dimensional reconstruction and physiological state analysis of the canopy.

Images