Method and system for dynamically acquiring growth phenotypes of fruit trees based on multi-modal perception fusion

Abstract

The present application relates to a fruit tree growth phenotype dynamic acquisition method and system based on multi-modal perception fusion. It belongs to the field of fruit tree growth detection. By fusing the structure and growth phenotype data of individual and group fruit trees collected by laser radar, machine vision, remote sensing and other multi-source sensors, a fruit tree full growth period growth dynamic knowledge graph is established, and spatiotemporal modeling and automatic recognition of tree structure, growth state and key growth period are realized. Further combined with knowledge reasoning and decision algorithm, intelligent management services such as fruit tree nutrition diagnosis, pruning suggestion and yield prediction are provided. The present application improves the acquisition, analysis and utilization efficiency of fruit tree phenotype data, realizes fine, intelligent and digital management of fruit trees, and can be widely applied in the fields of modern orchard digital twin and intelligent agriculture.

Description

Technical Field

[0001] This invention belongs to the field of fruit tree growth detection, and more specifically relates to a method and system for dynamically acquiring fruit tree growth phenotypes based on multimodal perception fusion. Background Technology

[0002] With the rapid development of smart agriculture, refined management and intelligent decision-making for fruit trees have become crucial for improving orchard production efficiency and fruit quality. Traditional fruit tree management relies mainly on manual observation, experience-based judgment, and scattered recording, lacking a systematic and digital understanding of the individual structure, growth dynamics, and spatial distribution of fruit trees. This makes it difficult to meet the demands of modern orchards for high-throughput, automated, and intelligent growth monitoring and nutrient regulation. In recent years, advancements in sensor technology, machine vision, remote sensing, 3D reconstruction, and other multimodal data acquisition methods have continuously improved the granularity of fruit tree phenotypic information acquisition. However, existing methods often focus on static feature analysis at local or single time points, lacking the ability to jointly model the complex topological structure, spatiotemporal evolution process, and key growth events of fruit trees. Furthermore, existing fruit tree phenotypic analysis and management systems are mostly fragmented functional modules, making it difficult to achieve cross-modal, multi-timescale data fusion, knowledge extraction, and intelligent reasoning. This results in intelligent management services such as nutrient diagnosis, pruning decisions, and yield prediction being difficult to achieve in a high-efficiency, accurate, and sustainable manner. Therefore, there is an urgent need for a technical solution that can be applied to individual and group fruit trees, integrate multi-source heterogeneous spatiotemporal data, deeply model the phenotypic structure and growth dynamics of fruit trees, and realize intelligent decision-making driven by dynamic knowledge graphs, so as to promote fruit tree production management to a new stage of standardization, intelligence and digital twins. Summary of the Invention

[0003] This invention aims to solve the technical problems in existing fruit tree management, such as the fragmented acquisition and analysis of phenotypic data, the inability to effectively integrate multimodal and multi-temporal information, the difficulty in accurately modeling the dynamic structure and key growth events of fruit trees, and the lack of intelligent reasoning and decision support based on knowledge graphs. In order to achieve efficient phenotypic monitoring, comprehensive growth dynamic modeling, and refined intelligent management of fruit trees throughout their entire life cycle.

[0004] To achieve the above objectives, the present invention employs the following technical solution: the method comprises:

[0005] Synchronous acquisition of multimodal raw data of fruit trees: Using a hyperspectral imager, 3D lidar, thermal infrared camera and visible light camera, based on a spatiotemporal synchronous triggering mechanism, real-time acquisition of multimodal raw data of fruit trees at different growth stages, including hyperspectral, 3D structure, temperature distribution and RGB images;

[0006] Multimodal feature fusion based on improved sparse dictionary learning: By utilizing modality adaptive regularization constraints, the correlation and discriminative power between features are enhanced, and a high-dimensional joint feature representation after fusion is obtained;

[0007] Fruit tree phenotypic structure modeling based on global-local dynamic graph neural network: Based on the fusion of features, an innovative global-local dynamic graph attention neural network is used to dynamically construct a multi-scale graph structure that reflects the spatial structure and growth status of fruit trees.

[0008] Analysis of dynamic changes in fruit tree growth based on cross-modal temporal contrastive learning: Combining historical and current multimodal data, an improved cross-modal temporal contrastive learning algorithm is proposed. By constructing positive and negative sample pairs and modal temporal consistency constraints, the algorithm automatically mines the potential temporal patterns of phenotypic changes during fruit tree growth.

[0009] By utilizing multidimensional phenotypic attributes, spatiotemporal features, and growth event triples, a dynamic knowledge graph of fruit tree growth phenotypes is automatically constructed, and fruit tree nutrition diagnosis, pruning suggestions, and yield prediction are realized based on knowledge reasoning algorithms.

[0010] In one approach, the data synchronization acquisition includes: all sensors automatically recording raw data and marking spatial location information according to timestamps; performing preliminary spatial coordinate transformation on the acquired multi-source data by automatically matching feature points between 3D point clouds and RGB images, combined with a calibration board or a specially designed spatial reference target; and simultaneously performing fine-grained timestamp correction on all raw data using a synchronization clock.

[0011] In one approach, the multimodal feature fusion first employs a specific preprocessing procedure for different modal data. Then, during the feature extraction stage, a multimodal sparse dictionary learning method based on an improved sparse subspace clustering algorithm is used to establish sparse mapping relationships between modalities. Simultaneously, modal adaptive regularization constraints are utilized to enhance the correlation and discriminative power between features, ultimately obtaining a fused high-dimensional joint feature representation.

[0012] In one approach, the fruit tree phenotypic structure modeling specifically includes:

[0013] First, using the obtained high-dimensional joint feature representation, the sampling points of different phenotypic parts of the fruit tree in three-dimensional space are used as nodes of the graph, and the node feature vectors are based on multimodal fusion features.

[0014] When constructing the initial graph structure, a hierarchical K-nearest neighbor strategy and feature correlation are adopted, combining spatial neighborhood distance and semantic similarity, to adaptively generate local connection edges and global jump connection edges, forming a multi-scale and dynamically variable graph structure G = (V, E), where V is the set of nodes and E is the set of edges;

[0015] A global-local dynamic graph attention neural network is adopted, which assigns local and global attention weights to each node to achieve dynamic control of structural modeling and information aggregation;

[0016] By stacking multiple global-local dynamic graph attention neural networks, the network continuously aggregates spatial structure and multimodal semantic features, enabling the segmentation of fruit tree phenotypic components and topological reasoning at different scales.

[0017] In one approach, the analysis of the dynamic changes in fruit tree growth specifically includes: firstly, organizing the multimodal high-dimensional features and node dynamic phenotypic structures into a temporal feature sequence according to individual fruit trees and time periods; and for each fruit tree object, extracting its fused feature vector at a series of consecutive growth time points.

[0018] The structured phenotypic representations obtained from processing all modal data through a global-local dynamic graph neural network are further concatenated to form time-level joint features;

[0019] An improved cross-modal temporal contrastive learning algorithm is proposed, which models dynamic changes by constructing positive and negative sample pairs and introducing temporal consistency constraints.

[0020] In one approach, the improved cross-modal temporal contrastive learning algorithm is specifically as follows:

[0021] Within each training batch, the feature z of a randomly selected fruit tree sample at time t is used. t As anchor points, the feature representations of trees at the same location but in different modalities at adjacent historical or future times are used as positive samples; while the feature representations of fruit trees at the same time point but in different spatial locations, or historical state features over a long time span, are used as negative samples.

[0022] The goal of contrastive learning is to narrow the distance between positive sample pairs in the latent space while widening the distance between negative sample pairs; an improved contrastive loss function is employed, in the form of...

[0023]

[0024] Where sim(x,y) represents the similarity measure between features, and τ is the temperature hyperparameter. For anchor point features, The positive sample features corresponding to this anchor point. These are negative sample features. To further enhance cross-modal temporal consistency, a modal temporal consistency regularization term L is introduced. align The average distance between features of different modalities and adjacent time points is limited by the expression:

[0025]

[0026] Where (m, m') represent different modes, and Δ represents adjacent or specified time intervals. The final total loss is L. total =L CMTCL +λ align L align Where, λalign It is the weighting coefficient that adjusts the contrast loss and consistency constraint.

[0027] In one approach, a dynamic knowledge graph of fruit tree growth phenotypes is automatically constructed. Specifically, based on the analysis results of multimodal joint features, dynamic graph structure modeling, and cross-temporal comparative learning, the structured phenotype states of each fruit tree object at different times and spatial locations are materialized, and multidimensional attributes such as {fruit tree ID, time, spatial partition, fusion features, key growth events, etc.} are extracted.

[0028] The basic triples for constructing a knowledge graph are: (leaf node, "belongs to", branch node), (a certain fruit tree at a certain time, "state is", "flowering"), (branch node, "spatial adjacency", leaf node), and (a certain time period, "appears", "fruit development"), which cover spatiotemporal features, structural hierarchy, phenotypic attributes, and growth events.

[0029] Graph convolutional networks are used to encode the knowledge graph structure to enhance entity and relation representations; for each node v i and its attribute characteristics x i Its embedding representation in the l-th layer of GCN The iterative updates are as follows:

[0030]

[0031] Where N(i) is the set of neighbors of node i, c ij The normalization coefficient is... The relation feature transformation parameters for the \(l\)th layer are... σ represents the self-circulating weights, and σ is the non-linear activation function.

[0032] Combining GCN embeddings and expert rules, specific inference objective functions are defined for tasks such as nutrient diagnosis, pruning suggestions, and yield prediction; let the tree node embedding be h. tree The key nutritional attribute is embedded as h nutrient Through the scoring function:

[0033]

[0034] Where M is a learnable relation matrix, and when score(tree, nutrient) is below the diagnostic threshold, it can be determined as nutrient deficiency; for pruning suggestions, high-density / overgrown branches are inferred and pruning timing and location are output by using rule templates or conditional probability models based on the growth event sequence and spatial topology of nodes within the graph structure; for yield prediction, a regression model is used by combining the spatiotemporal evolution embedding of fruit entities and historical yield data.

[0035]

[0036] in, To predict output, h fruit h tree h env These are the map embeddings for fruit, fruit trees, and environment, respectively, and g(·) is the regression function.

[0037] In one aspect, a dynamic acquisition system for fruit tree growth phenotypes based on multimodal perception fusion is provided, the system being applicable to the method described above, the system comprising:

[0038] The hyperspectral imager, 3D lidar, thermal infrared camera and visible light camera are connected through a spatiotemporal synchronization triggering mechanism to collect hyperspectral data, 3D structural data, temperature distribution data and RGB images of fruit trees at different growth stages in real time.

[0039] The system also includes a data processing module, which performs multimodal feature fusion by improving sparse dictionary learning technology to enhance the correlation and discriminativeness between features and generate high-dimensional joint feature representations;

[0040] Meanwhile, the system uses a global-local dynamic graph neural network to model the fruit tree phenotypic structure of the above-mentioned fused features and constructs a multi-scale graph structure to reflect the spatial structure and growth status of the fruit tree.

[0041] By combining historical and current multimodal data, a cross-temporal comparative learning algorithm is implemented to automatically mine phenotypic changes in the fruit tree growth process. By analyzing multidimensional phenotypic attributes, spatiotemporal features, and growth event triples, a dynamic knowledge graph of fruit tree growth and development phenotypic patterns is constructed, thereby enabling intelligent management services for fruit tree nutrition diagnosis, pruning suggestions, and yield prediction.

[0042] Beneficial effects of this invention:

[0043] This invention automatically constructs a dynamic knowledge graph of fruit tree growth phenotypes by integrating multimodal and spatiotemporal fruit tree phenotypic data. Combined with advanced knowledge reasoning algorithms, it provides intelligent decision support for nutrient diagnosis, pruning suggestions, and yield prediction, significantly improving the digitalization and intelligence of fruit tree growth monitoring and management. This method comprehensively and dynamically reflects the structural and phenotypic changes of individual and group fruit trees at various growth stages, enabling automatic identification and causal correlation analysis of complex fruit tree growth events. It effectively improves orchard production efficiency and fruit quality, reduces labor input and management costs, and provides a solid data foundation and technical support for smart orchard construction and modern agricultural development. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention;

[0045] Figure 2This is a system block diagram of the present invention. Detailed Implementation

[0046] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Typical embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0047] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. To facilitate understanding, the invention will now be described more fully with reference to the accompanying drawings. Typical embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to make the disclosure of the invention more thorough and complete.

[0048] like Figure 1 As shown, the specific implementation steps of the method for dynamically acquiring fruit tree growth phenotypes based on multimodal perception fusion are as follows:

[0049] Step 1. Synchronous acquisition of multimodal raw data of fruit trees: Using a variety of sensors such as hyperspectral imager, 3D LiDAR, thermal infrared camera and visible light camera, based on the spatiotemporal synchronization triggering mechanism, multimodal raw data such as hyperspectral, 3D structure, temperature distribution and RGB images of fruit trees at different growth stages are acquired in real time. Through the self-developed edge data alignment module, sub-second time synchronization and preliminary spatial registration of multi-source data are performed.

[0050] In the detailed implementation of synchronous acquisition of multimodal raw data from fruit trees, the first step is to deploy various sensing devices, such as hyperspectral imagers, 3D LiDAR, thermal infrared cameras, and visible light cameras, on the target fruit trees in the orchard environment. These devices are networked using fixed-point or mobile platforms (such as unmanned mobile robots or track-based acquisition vehicles). Each sensor is cascaded through a self-developed multi-channel spatiotemporal synchronization triggering system to ensure precise frame-level or sub-second-level time synchronization of the multimodal sensors within a specified acquisition window. To effectively address the issues of fruit tree movement caused by external disturbances (such as wind) or inconsistencies in the imaging coordinate systems of different sensors, the system integrates a precise IMU (Inertial Measurement Unit) and positioning module to acquire real-time attitude and spatial position data of the sensing devices, assisting in subsequent data registration.

[0051] In the data acquisition process, the synchronization trigger module first issues acquisition commands uniformly, and all sensors automatically record raw data and mark spatial location information according to timestamps. Considering the differences in format, resolution, and field of view of multimodal data, the edge data alignment module runs locally in real time at each sensor acquisition node. Using lightweight spatial mapping and feature point extraction algorithms, it performs coarse registration of the raw point cloud, image, and spectral data. Specifically, it automatically matches feature points between the 3D point cloud and the RGB image, and combines a calibration board or a specially designed spatial reference target to perform preliminary spatial coordinate transformation on the acquired multi-source data. Simultaneously, a synchronization clock is used to perform fine-grained timestamp correction on all raw data, minimizing temporal and spatial deviations in multimodal information. Finally, all multimodal data that has undergone time synchronization and preliminary spatial registration is packaged and output uniformly, providing a high-quality, highly consistent raw data foundation for subsequent fusion analysis and dynamic phenotypic modeling.

[0052] Step 2. Multimodal feature fusion based on improved sparse dictionary learning: First, specific preprocessing procedures are adopted for different modal data (such as hyperspectral interband correction, 3D point cloud noise filtering, thermal image equalization, etc.). Then, in the feature extraction stage, based on the improved multimodal sparse dictionary learning method of Sparse Subspace Clustering (SSC), sparse mapping relationships between modalities are established. At the same time, modality adaptive regularization constraints are used to enhance the correlation and discriminativeness between features, and finally, a high-dimensional joint feature representation after fusion is obtained.

[0053] In the detailed implementation of multimodal feature fusion based on improved sparse dictionary learning, the following preprocessing methods are first applied to different modalities: interband radiometric correction and noise removal are performed on hyperspectral images to ensure that different bands reflect the true spectral differences; statistical filtering and surface reconstruction are applied to 3D LiDAR point clouds to eliminate scattered points and outliers and improve the representation accuracy of spatial structures; and histogram equalization and pseudo-color conversion are used for thermal infrared images to make the temperature distribution details more obvious. After the above preprocessing, primary features are extracted from the multimodal data, such as principal components of hyperspectral data, geometric shape descriptors of point clouds, local texture statistics of heatmaps, and depth convolution features of RGB images; these features are uniformly organized into a matrix. Where m represents the modality category, d m Let be the feature dimension of this mode. N This represents the number of samples.

[0054] In the feature fusion stage, an improved multi-modal sparse dictionary learning (MSDL) algorithm, based on traditional sparse subspace clustering (SSC), is employed. Specifically, for each modality, a sparse coding matrix A is first learned. (m) Modal-specific dictionary D (m) Its optimization objective function is:

[0055]

[0056] in,‖·‖ F Denotes the Frobenius norm, ||A|| (m) ||1 represents L1 regularization for sparse coding, and λ is the sparse weight coefficient. To achieve sparse relation mapping between multimodalities, a modality adaptive regularization term is introduced to encourage different modalities to have interconnected sparse structures in the joint subspace:

[0057]

[0058] Where, γ ij The inter-modal adaptive correlation weights are dynamically adjusted based on the correlation coefficients obtained from sample statistics. Combining the above objectives, the final optimization problem for multimodal fusion is:

[0059]

[0060] The model optimizes the dictionary D alternately. (m) With sparse coding A (m) This approach not only ensures efficient and sparse representation of different modal features in their respective subspaces but also effectively fuses the correlation and complementarity among multimodal features through adaptive regularization. Finally, the sparse encoding vectors obtained from each modality are concatenated or projected onto a shared high-dimensional joint feature space to obtain the fused phenotypic feature representation, providing a solid data foundation for subsequent dynamic phenotypic modeling of fruit tree growth. Among these, X... (m) Let D be the input feature matrix of the m-th mode. (m) Let A be the dictionary matrix for the m-th mode. (m) Let λ and γ be the sparse coding matrix of the m-th mode. ij These are the sparsity and fusion regularization coefficients, respectively.

[0061] Step 3. Phenotypic structure modeling of fruit trees based on global-local dynamic graph neural network: Based on the fusion features, an innovative global-local dynamic graph attention network (GL-DGAN) is adopted to dynamically construct a multi-scale graph structure that reflects the spatial structure and growth status of fruit trees. This enables intelligent segmentation and topological association modeling of key phenotypic components such as branches, leaves, flowers and fruits, effectively integrating multimodal information and spatiotemporal relationships, and improving the accuracy and robustness of phenotypic extraction.

[0062] In the process of modeling the phenotypic structure of fruit trees based on global-local dynamic graph neural networks, the high-dimensional joint feature representation obtained in step 2 is first used to take the sampling points of different phenotypic parts of the fruit tree (such as branches, leaves, flowers and fruits) in three-dimensional space as nodes of the graph. The node feature vectors are based on multimodal fusion features. When constructing the initial graph structure, a hierarchical K-nearest neighbor (KNN) strategy and feature correlation are adopted, combining spatial neighborhood distance and semantic similarity, to adaptively generate local connection edges and global jump connection edges, forming a multi-scale and dynamically variable graph structure G = (V, E), where V is the set of nodes and E is the set of edges.

[0063] Subsequently, an innovative Global-Local Dynamic Graph Attention Network (GL-DGAN) was designed. Its core lies in assigning local and global attention weights to each node, achieving dynamic control over structural modeling and information aggregation. For the l-th layer of the network, the updated features of node i... Driven by both local and global information, the specific update formula is as follows:

[0064]

[0065] Where, N i Let G represent the set of local neighbors of node i. i Let represent the set of distant nodes that are globally related to node i. and These are the local and global adjacency weights obtained through adaptive learning via an attention mechanism, W, respectively. loc With W glob These are the feature transformation matrices for local and global information, respectively, α loc and α glob Satisfy α loc +α glob =1, used to dynamically balance the proportion of local and global information. The attention weight is determined by the similarity of the fused features and temporal dynamic factors, and its calculation method is as follows:

[0066]

[0067] Where a is a trainable parameter vector, || represents the feature concatenation operation, and LeakyReLU is the activation function.

[0068] By stacking multiple layers of GL-DGAN, the network continuously aggregates spatial structure and multimodal semantic features, achieving segmentation and topological inference of fruit tree phenotypic components at different scales. Finally, through clustering or classification layers, phenotypic category labels for each sampling point are output, enabling intelligent segmentation of key parts such as branches, leaves, flowers, and fruits. Simultaneously, by combining the relationships between nodes and edges in the graph, the self-organizing topology of the fruit tree is automatically extracted, accurately reconstructing its branching, connectivity, and spatial hierarchy, among other growth morphological features. This modeling method significantly enhances the ability to integrate multimodal information and spatiotemporal dynamic relationships, improving the resolution, accuracy, and robustness of fruit tree phenotypic modeling. W represents the feature of node i in the l-th layer. loc With W glob It is the eigenmap matrix, β ij For attention weights, α loc ,α glob The coefficient represents the global-local information fusion coefficient.

[0069] Step 4. Dynamic Change Analysis of Fruit Tree Growth Based on Cross-modal Temporal Contrastive Learning: Combining historical and current multimodal data, an improved Cross-modal Temporal Contrastive Learning (CMTCL) algorithm is proposed. By constructing positive and negative sample pairs and modal temporal consistency constraints, the algorithm automatically mines the potential temporal patterns of phenotypic changes during fruit tree growth, realizing the dynamic identification and trend prediction of key growth states (such as budding, flowering, fruiting, leaf fall, etc.).

[0070] In the analysis of fruit tree growth dynamics based on cross-temporal contrastive learning, the multimodal high-dimensional features obtained in step 2 and the node dynamic phenotypic structure modeled in step 3 are first organized into a temporal feature sequence according to individual fruit trees and time points. For each fruit tree object, at a series of consecutive growth time points {t1,t2,...,t...} T}, extract their fused feature vectors respectively in This represents the multimodal feature representation at time t and mode m. The structured phenotypic representation obtained by processing all modal data through a global-local dynamic graph neural network is further concatenated or pooled to form the time-level joint feature z. tTo learn the phenotypic changes of fruit trees at different growth stages, an improved Cross-modal Temporal Contrastive Learning (CMTCL) algorithm is proposed. This algorithm models dynamic changes by constructing positive and negative sample pairs and introducing temporal consistency constraints.

[0071] Specifically, within each training batch, the feature z of a randomly selected fruit tree sample at the current time t is... t As an anchor point, it represents the features of the same location but different modes at the same time point in the past or future adjacent moments (e.g., t±Δ). As positive samples, fruit trees at the same time point but in different spatial locations, or historical state characteristics over a long time span, are represented as negative samples (e.g., ...). And |t′-t|>δ or n≠m). The goal of contrastive learning is to bring positive sample pairs (i.e., phenotypic features from the same tree, adjacent time points, and / or different modalities) closer together in the latent space, while pushing negative sample pairs further apart. The algorithm employs an improved contrastive loss function, in the form of...

[0072]

[0073] Where sim(x,y) represents the similarity measure between features (such as cosine similarity), and τ is the temperature hyperparameter. For anchor point features, The positive sample features corresponding to this anchor point. These are negative sample features. To further enhance cross-modal temporal consistency, a modal temporal consistency regularization term L is introduced. align To limit the average distance of features between different modalities and adjacent time points, the expression is:

[0074] Where (m, m') represent different modes, and Δ represents adjacent or specified time intervals. The final total loss is

[0075] L total =L CMTCL +λ align L align

[0076] Where, λ align It is the weighting coefficient that adjusts the contrast loss and consistency constraint.

[0077] The improved CMTCL algorithm described above automatically discovers the potential growth phenotypic evolution trajectory of fruit trees over time through comparative stretching of temporally structured feature pairs and multimodal temporal consistency constraints, and can accurately identify important growth nodes such as budding, flowering, fruiting, and leaf fall. Furthermore, by utilizing historical time-series data-driven phenotypic feature streams, it can be combined with regression or classifiers to predict future phenotypic trends, providing effective technical support for long-term, high-precision dynamic analysis of fruit tree growth and digital twin modeling. In the formula, Let be the feature vector at time t and mode m, sim(x,y) be the similarity function, τ be the temperature coefficient, and L be the eigenvector. align For modal timing consistency loss, λ align These are the weight parameters.

[0078] Step 5. Generation of Dynamic Knowledge Graph of Fruit Tree Growth Phenotype and Intelligent Decision Support: Based on the above dynamic phenotypic analysis results, a dynamic knowledge graph of fruit tree growth phenotypic is automatically constructed by innovatively utilizing multidimensional phenotypic attributes, spatiotemporal features, and growth event triples. Based on knowledge reasoning algorithms (as shown in the figure, convolutional networks combined with reasoning rules), intelligent decisions such as fruit tree nutrition diagnosis, pruning suggestions, and yield prediction are realized, providing data support and technical guarantee for the refined management of fruit trees and smart agriculture.

[0079] In the process of generating a dynamic knowledge graph of fruit tree growth phenotypes and implementing intelligent decision support, the analysis results of multimodal joint features, dynamic graph structure modeling, and cross-temporal comparative learning accumulated in steps 2-4 are first used to materialize the structured phenotypic state of each fruit tree object at different times and spatial locations, extracting multidimensional attributes such as {fruit tree ID, time, spatial partition, fusion features, and key growth events (such as "flowering" and "fruiting")}. Based on this, the basic triples of the knowledge graph are innovatively constructed: (entity 1, relation, entity 2), such as (leaf node, "belongs to", branch node), (a certain fruit tree at a certain time, "state is", "flowering"), (branch node, "spatially adjacent", leaf node), (a certain time period, "appears", "fruit development"), etc., covering spatiotemporal features, structural hierarchy, phenotypic attributes, and growth events. All triples are organized into a dynamic graph structure to express the complex dynamic relationships within the entire growth cycle of the fruit tree.

[0080] In knowledge graph representation learning, Graph Convolutional Networks (GCNs) are used to encode the knowledge graph structure to enhance entity and relation representations. For each node (entity) v i and its attribute characteristics x i Its embedding representation in the l-th layer of GCN The iterative updates are as follows:

[0081]

[0082] Where N(i) is the set of neighbors of node i, c ij The normalization coefficient is... The parameter for transforming the relation features at level \(l\) (weights can be set according to different relation types), σ represents the self-circulating weights, and σ is the non-linear activation function. Through a multi-layer GCN, entities and growth events obtain high-dimensional feature embeddings h within their context and structural semantics. i This indicates that it is able to capture the spatiotemporal relationships and multi-level structural reasoning features of fruit tree growth phenotypes.

[0083] In the intelligent decision-making and reasoning stage, combining GCN embedding and expert rules, specific reasoning objective functions are defined for tasks such as nutrient diagnosis, pruning suggestions, and yield prediction. Taking nutrient diagnosis as an example, let the tree node embedding be h. tree The key nutritional attribute is embedded as h nutrient It can be achieved through a scoring function.

[0084]

[0085] Here, M is a learnable relation matrix, and nutrient deficiency can be identified when the score(tree, nutrient) is below the diagnostic threshold. For pruning suggestions, high-density / overgrown branches are inferred using rule templates or conditional probability models based on the growth event sequence and spatial topology of nodes within the graph structure, and the pruning timing and location are output. For yield prediction, a regression model is used, combining the spatiotemporal evolution embedding of fruit entities and historical yield data.

[0086]

[0087] in, To predict output, h fruit h tree h env These are the map embeddings for fruit, fruit trees, and environment, respectively, and g(·) is the regression function (such as a multilayer perceptron MLP or a temporal regression network).

[0088] The dynamic graph update mechanism of the knowledge graph can incorporate new observations and phenotypic events at any time, enabling continuous expansion and intelligent evolution of the knowledge base. This process realizes reasoning, prediction, and decision-making based on dynamic spatiotemporal phenotypic knowledge, providing full-chain intelligent support for refined fruit tree management and smart agriculture scheduling and services.

[0089] like Figure 2 As shown, a system for dynamically acquiring fruit tree growth phenotypes based on multimodal perception fusion is described. This system is applicable to the method described above, and includes:

[0090] The system comprises a hyperspectral imager, a 3D lidar, a thermal infrared camera, and a visible light camera. These sensors are connected via a spatiotemporal synchronization triggering mechanism to synchronously acquire hyperspectral data, 3D structural data, temperature distribution data, and RGB images of fruit trees at different growth stages in real time. The system also includes a data processing module that uses improved sparse dictionary learning technology to fuse multimodal features, enhancing the correlation and discriminative power between features and generating high-dimensional joint feature representations. Simultaneously, the system employs a global-local dynamic graph neural network to model the phenotypic structure of the fruit trees using the fused features, constructing a multi-scale graph structure to reflect the spatial structure and growth status of the fruit trees. Combining historical and current multimodal data, a cross-temporal comparative learning algorithm is implemented to automatically mine phenotypic changes during the fruit tree's growth process. By analyzing multidimensional phenotypic attributes, spatiotemporal features, and growth event triples, a dynamic knowledge graph of fruit tree growth and development phenotypes is constructed, thereby enabling intelligent management services for fruit tree nutrition diagnosis, pruning recommendations, and yield prediction.

[0091] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0092] It should be understood that the above detailed description of the technical solutions of the present invention with reference to preferred embodiments is illustrative and not restrictive. Those skilled in the art can modify the technical solutions described in the embodiments or make equivalent substitutions for some of the technical features based on reading this specification; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamically acquiring fruit tree growth phenotypes based on multimodal perception fusion, characterized in that: The method includes: Synchronous acquisition of multimodal raw data of fruit trees: Using a hyperspectral imager, 3D lidar, thermal infrared camera and visible light camera, based on a spatiotemporal synchronous triggering mechanism, real-time acquisition of multimodal raw data of fruit trees at different growth stages, including hyperspectral, 3D structure, temperature distribution and RGB images; Multimodal feature fusion based on improved sparse dictionary learning: By utilizing modality adaptive regularization constraints, the correlation and discriminative power between features are enhanced, and a high-dimensional joint feature representation after fusion is obtained; Fruit tree phenotypic structure modeling based on global-local dynamic graph neural network: Based on the fusion of features, a global-local dynamic graph attention neural network is used to dynamically construct a multi-scale graph structure that reflects the spatial structure and growth status of fruit trees. Analysis of dynamic changes in fruit tree growth based on cross-modal temporal contrastive learning: Combining historical and current multimodal data, an improved cross-modal temporal contrastive learning algorithm is proposed. By constructing positive and negative sample pairs and modal temporal consistency constraints, the algorithm automatically mines the potential temporal patterns of phenotypic changes during fruit tree growth. By utilizing multidimensional phenotypic attributes, spatiotemporal features, and growth event triples, a dynamic knowledge graph of fruit tree growth and development phenotypic characteristics is automatically constructed, and intelligent management services for fruit tree nutrition diagnosis, pruning suggestions, and yield prediction are realized based on knowledge reasoning algorithms. Automatically constructing a dynamic knowledge graph of fruit tree growth phenotypes, specifically including: Based on the analysis results of multimodal joint features, dynamic graph structure modeling and cross-temporal comparative learning, the structured phenotypic state of each fruit tree object under different time and spatial locations is materialized, and the multidimensional attributes of {fruit tree ID, time, spatial partition, fusion features, and key growth events} are extracted. The basic triples for constructing a knowledge graph are: (leaf node, "belongs to", branch node), (a certain fruit tree at a certain time, "state is", "flowering"), (branch node, "spatial adjacency", leaf node), and (a certain time period, "appears", "fruit development"), which cover spatiotemporal features, structural hierarchy, phenotypic attributes, and growth events. Graph convolutional networks are used to encode the knowledge graph structure to enhance entity and relation representations; for each node and its attributes and characteristics Its embedding representation in the l-th layer of GCN The iterative updates are as follows: ； in, Let i be the set of neighbors of node i. The normalization coefficient is... For the first Layer relational feature transformation parameters For self-circulating weights, It is a non-linear activation function; Combining GCN embeddings and expert rules, specific inference objective functions are defined for tasks such as nutrient diagnosis, pruning suggestions, and yield prediction; let the fruit tree node embedding be... Key nutritional attributes are embedded as Through the scoring function: ； Where M is the learnable relation matrix, when Nutrient deficiency is diagnosed when levels fall below the diagnostic threshold. For pruning recommendations, the timing and location of pruning are output using rule templates or conditional probability models, based on the growth event sequence and spatial topology of nodes within the graph structure. For yield prediction, a regression model is used, incorporating the spatiotemporal evolution embedding of fruit entities and historical yield data. ； in, To predict production, , , The image embeddings are for fruit, fruit trees, and the environment, respectively. This is a regression function.

2. The method for dynamically acquiring fruit tree growth phenotypes based on multimodal perception fusion according to claim 1, characterized in that: The data synchronization acquisition includes: all sensors automatically recording raw data and marking spatial location information according to timestamps; performing preliminary spatial coordinate transformation on the acquired multi-source data by automatically matching feature points between 3D point clouds and RGB images, combined with a calibration board or a specially made spatial reference target; and simultaneously using a synchronization clock to perform fine-grained timestamp correction on all raw data.

3. The method for dynamically acquiring fruit tree growth phenotypes based on multimodal perception fusion according to claim 1, characterized in that: The multimodal feature fusion described above first employs a specific preprocessing procedure for different modal data. Then, in the feature extraction stage, a multimodal sparse dictionary learning method based on the improved sparse subspace clustering algorithm is used to establish sparse mapping relationships between modalities. At the same time, modal adaptive regularization constraints are used to enhance the correlation and discriminativeness between features, ultimately obtaining a fused high-dimensional joint feature representation.

4. The method for dynamically acquiring fruit tree growth phenotypes based on multimodal perception fusion according to claim 1, characterized in that: The aforementioned fruit tree phenotypic structure modeling specifically includes: First, using the obtained high-dimensional joint feature expression, the sampling points of different phenotypic parts of the fruit tree in three-dimensional space are used as nodes of the graph, and the node feature vectors are based on the high-dimensional joint feature expression. When constructing the initial graph structure, a hierarchical K-nearest neighbor strategy and feature relevance are adopted, combining spatial neighborhood distance and semantic similarity, to adaptively generate local and global skip connections, forming a multi-scale and dynamically variable graph structure. ,in For a set of nodes, Let it be the set of edges; A global-local dynamic graph attention neural network is adopted, which assigns local and global attention weights to each node to achieve dynamic control of structural modeling and information aggregation; By stacking multiple global-local dynamic graph attention neural networks, the network continuously aggregates spatial structure and multimodal semantic features, enabling the segmentation of fruit tree phenotypic components and topological reasoning at different scales.

5. The method for dynamically acquiring fruit tree growth phenotypes based on multimodal perception fusion according to claim 1, characterized in that: The aforementioned analysis of the dynamic changes in fruit tree growth specifically includes: firstly, organizing the high-dimensional joint feature representation and fruit tree phenotypic structure into a time-series feature sequence according to individual fruit trees and time points; for each fruit tree object, extracting its fused feature vector at a series of consecutive growth time points. ,in This represents the multimodal feature representation at time t and mode m. The structured phenotypic representations obtained from processing all modal data through a global-local dynamic graph neural network are further concatenated to form time-level joint features. ; An improved cross-modal temporal contrastive learning algorithm is proposed, which models dynamic changes by constructing positive and negative sample pairs and introducing temporal consistency constraints.

6. The method for dynamically acquiring fruit tree growth phenotypes based on multimodal perception fusion according to claim 5, characterized in that: The improved cross-modal temporal contrastive learning algorithm is as follows: Within each training batch, select the features of a random fruit tree sample at the current time t. As anchor points, the feature representations of trees at the same location but in different modalities at adjacent historical or future times are used as positive samples; while the feature representations of fruit trees at the same time point but in different spatial locations, or the historical state features over a long time span, are used as negative samples. The goal of contrastive learning is to narrow the distance between positive sample pairs in the latent space while widening the distance between negative sample pairs; an improved contrastive loss function is employed, in the form of... ； in, This represents a measure of similarity between features. For temperature hyperparameters, For anchor point features, The positive sample features corresponding to this anchor point. For negative sample features; to further enhance cross-modal temporal consistency, a modal temporal consistency regularization term is introduced. The average distance between features of different modalities and adjacent time points is limited by the expression: ； Where (m,m') represent different modes. For adjacent or specified time intervals; the final total loss is in, It is the weighting coefficient that adjusts the contrast loss and consistency constraint.

7. A system for dynamically acquiring fruit tree growth phenotypes based on multimodal perception fusion, wherein the system is applicable to the method as described in any one of claims 1-6, characterized in that: The system includes: The hyperspectral imager, 3D lidar, thermal infrared camera and visible light camera are connected through a spatiotemporal synchronization triggering mechanism to collect hyperspectral data, 3D structural data, temperature distribution data and RGB images of fruit trees at different growth stages in real time. The system also includes a data processing module, which performs multimodal feature fusion by improving sparse dictionary learning technology to enhance the correlation and discriminativeness between features and generate high-dimensional joint feature representations; Meanwhile, the system uses a global-local dynamic graph neural network to model the fruit tree phenotypic structure of the above-mentioned fused features and constructs a multi-scale graph structure to reflect the spatial structure and growth status of the fruit tree. By combining historical and current multimodal data, a cross-temporal comparative learning algorithm is implemented to automatically mine phenotypic changes in the fruit tree growth process. By analyzing multidimensional phenotypic attributes, spatiotemporal features, and growth event triples, a dynamic knowledge graph of fruit tree growth and development phenotypic patterns is constructed, thereby enabling intelligent management services for fruit tree nutrition diagnosis, pruning suggestions, and yield prediction.

Images