Forest industry chain multi-modal data intelligent processing method and system

Abstract

The invention provides an intelligent processing method and system for multi-modal data of an industrial chain of the forest industry, and relates to the technical field of data processing. According to the method, acquisition nodes are arranged in links of seedling raising, planting, tending, felling, transportation, processing, storage and sales, and three types of original data of images, sensors and texts are obtained; forming a time-space aligned image feature, sensor feature and text feature set through unified time correction, space coordinate conversion, anomaly elimination, deletion interpolation and normalization; and further mapping the three types of features to the same feature space by adopting a unified coding mechanism, and constructing a knowledge graph by combining space-time diagram modeling and a forest ontology library to obtain multi-modal fusion feature representation. According to the invention, a high-consistency and extensible data basis is provided for forestry full-life-cycle digital management.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for intelligently processing multimodal data in a forestry industry chain. Background Art

[0002] The forestry industry chain covers seedling cultivation, planting, tending, logging, transportation, processing, warehousing and sales. It has long relied on manual inspections, single sensor monitoring and offline report management. The data sources are scattered and the formats are not unified, resulting in information barriers, delayed decision-making and waste of resources.

[0003] With the rapid development of satellite remote sensing, drones, the Internet of Things (IoT), 5G / 6G edge computing, and artificial intelligence, the real-time acquisition and intelligent analysis of multimodal, full-life cycle data have become important directions for the digital transformation of the forestry industry. Technologies such as cross-modal alignment, deep graph neural networks, and digital twins are gradually being applied to precise forestry management, carbon sink assessment, and supply chain risk control.

[0004] Existing solutions mostly focus on a certain link or a single data source, lacking unified data semantics, dynamically updated knowledge graphs, and cross-modal spatiotemporal fusion models for the entire forestry industry chain; they lack timeliness and have limited prediction accuracy, making it difficult to support collaborative decision-making and closed-loop optimization from forest land to terminal markets. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a method and system for intelligent processing of multimodal data in the forestry industry chain. Through full-link multimodal fusion and unified knowledge-driven intelligent prediction, the real-time and accuracy of forestry production scheduling, risk warning and carbon sink assessment are significantly improved.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A method for intelligently processing multimodal data in a forestry industry chain, comprising:

[0008] Deploy collection nodes in each business link of seedling cultivation, planting, tending, harvesting, transportation, processing, warehousing and sales to obtain image modality data, sensor modality data and text modality data, forming an original data set covering the entire life cycle;

[0009] Performing unified timestamp correction, spatial coordinate conversion, outlier removal, missing value interpolation, and feature normalization on the raw data to obtain a spatiotemporally aligned feature data set; the feature data set includes image features, sensor features, and text features;

[0010] map the image features, the sensor features and the text features to the same feature space using a unified encoding mechanism, and construct a knowledge graph based on a spatio-temporal graph modeling technique and a forestry ontology library to obtain a multi-modal fusion feature representation;

[0011] input the multi-modal fusion feature representation into a pre-trained multi-task learning model to obtain an output result; the output result includes forest yield estimation results, pest and disease early warning results, and carbon sink evaluation results.

[0012] Preferably, the image modal data includes remote sensing images, unmanned aerial vehicle images, and ground camera images; the sensor modal data includes temperature and humidity, soil moisture, pest and disease sound and image, mechanical vibration, and vehicle positioning information; and the text features include operation logs, warehouse documents, order contracts, and carbon emission trading records.

[0013] Preferably, the original data is subjected to unified timestamp correction, spatial coordinate conversion, outlier removal, missing value interpolation, and feature normalization to obtain a spatio-temporally aligned feature data set, including:

[0014] read the time and space markers of the original data of the three types of image, sensor, and text data, and generate an initial index table with a unified time reference and a unified geodetic coordinate system as the key values;

[0015] use the initial index table to perform time compensation adjustment on the original data of each modality, align the collection time to a unified sampling period, and make the spatial coordinates fall into the same reference system, and output the corrected multi-modal data entries;

[0016] For the corrected multi-modal data entries, Z-Score threshold method is used to detect and delete the observation values exceeding the confidence interval to obtain an outlier-removed data set;

[0017] On the outlier-removed data set, linear interpolation is used to complete the time continuous gaps, and spatial interpolation based on the nearest neighbor relationship is used to complete the spatial discrete gaps to generate a missing-completed data set;

[0018] extract image features, sensor features, and text features from the missing-completed data set, and perform minimum-maximum normalization processing on each type of feature to output the spatio-temporally aligned feature data set.

[0019] Preferably, on the outlier-removed data set, linear interpolation is used to complete the time continuous gaps, and spatial interpolation based on the nearest neighbor relationship is used to complete the spatial discrete gaps to generate a missing-completed data set, including:

[0020] scan the unified spatio-temporal index of the outlier-removed data set to mark the start and end positions of the continuous time gaps and the discrete spatial gaps;

[0021] For each continuous time gap, select one valid data record V(t1) before the gap and one valid data record V(t2) after the gap, and use the linear interpolation formula to calculate the estimated value of any time point t m to be interpolated within the gap , wherein t1 is the timestamp of the valid time point before the gap; t2 is the timestamp of the valid time point after the gap; t m is the timestamp of the time point to be interpolated within the gap; V(t) is the observation value at time t; and m is the index number of the time point within the gap.

[0022] For each discrete spatial gap, search for the observed position p k closest in Euclidean distance, and use the nearest neighbor interpolation formula to generate the estimated value of the gap position p m ; wherein p m is the coordinate vector of the spatial position to be interpolated; p k is the coordinate of the observed position closest in Euclidean distance to p m ; d(·) is the Euclidean distance function; and V(p) is the observation value at position p. , wherein j represents the index of the point with the smallest Euclidean distance among all observed points, and the corresponding position is p k .

[0023] Merge and in the original spatio-temporal index order to form the missing completed data set.

[0024] Preferably, the image features, the sensor features, and the text features are mapped to the same feature space using a unified encoding mechanism, and a knowledge graph is constructed based on spatio-temporal graph modeling technology and a forestry ontology library, to obtain a multi-modal fusion feature representation, including:

[0025] The image features are extracted using a convolutional neural network encoding structure to obtain a spatial distribution tensor, the sensor features are encoded using a time series attention network to obtain a dynamic change pattern, to obtain an encoding vector, and the text features are extracted using a pre-trained language model to obtain a context embedding representation, which is uniformly converted into a fixed-dimensional embedding vector.

[0026] The spatial distribution tensor, the encoding vector, and the embedding vector are input into a cross-modal feature alignment module, and a shared embedding layer is used to uniformly align the multi-modal feature vectors, to obtain a normalized joint feature representation.

[0027] Taking a forest land block, a processing node, and a circulation unit as a graph node, taking spatial proximity, supply chain correlation, and operation coordination relationship as an edge, combining a multi-modal feature vector to establish a graph structure, and introducing a unified time index to form a space-time graph model;

[0028] Call the forest ontology library, and embed semantic information of a species type, an operation mode, and an ecological zoning of the graph node into a graph node attribute;

[0029] Use a graph neural network to train the space-time graph model, capture correlation and propagation structure between multi-modal feature vectors, and obtain multi-modal fusion feature representation.

[0030] Preferably, the construction method of the forest ontology library is:

[0031] Extract six high-frequency terms of forest species type, growth stage, soil type, operation mode, pest and disease category, and climate zoning from national forestry industry standards, forestry science and technology journals, and forestry government database, and perform de-redundancy, semantic alignment, and morphological restoration processing on the high-frequency terms to output a standardized term set;

[0032] According to the standardized term set, create four ontology concept classes, namely "forest entity", "ecological factor", "operation measure", and "disaster event", and determine the hierarchical structure and attribute set for each ontology concept class;

[0033] Define the semantic relationship between the concept classes in the form of triplets <C1, R, C2>, including "belongs to relationship", "growth dependence relationship", "management measure relationship", and "disease influence relationship", and generate a relationship set;

[0034] Use an OWL description logic reasoning engine to verify the consistency of the set of ontology concept classes and the relationship set, eliminate circular definitions, attribute conflicts, and concept overlaps, and obtain the consistent forest ontology library;

[0035] Convert the forest ontology library that passes the consistency verification to a vectorized representation, and bind it to the node entities in the space-time graph model.

[0036] Preferably, the graph neural network is used to train the space-time graph model, capture the correlation and propagation structure between multi-modal feature vectors, and obtain the multi-modal fusion feature representation, including:

[0037] Taking each node in the space-time graph model as input, loading corresponding image features, sensor features, and text features, and initializing node vector representation

[0038] In each propagation layer, the output result of message passing of the l-th layer is calculated by weighted aggregation of the neighbor set of node i The representation of each node j in the l-th layer is denoted as

[0039] After repeating the L-layer message passing and nonlinear transformation, the fusion representation of each node in the L-th layer is obtained which is the multi-modal fusion feature representation.

[0040] Preferably, the output result of message passing is calculated by

[0041]

[0042] wherein, α ij represents the adjacency attention weight constructed by the edge feature, reflecting the association strength of node j to node i; W (l) is the trainable weight matrix of the l-th layer; and σ(·) is an activation function for introducing nonlinearity.

[0043] Preferably, the multi-modal fusion feature representation is input into a pre-trained multi-task learning model to obtain an output result, including:

[0044] The corresponding yield monitoring data, pest records, logistics events and carbon sink observation values in the historical forest management samples are preprocessed and standardized, and are aligned with the corresponding spatio-temporal feature index to construct a training sample set with multiple output labels;

[0045] A multi-task learning model including a shared encoder and multiple task branches is constructed, wherein the shared encoder is used to receive and abstract the multi-modal fusion feature representation, and each task branch is used for target prediction of yield estimation, risk warning and carbon sink evaluation, respectively;

[0046] The training sample set is modeled in a supervised learning manner, and the following joint loss function is used for multi-task optimization:

[0047]

[0048] wherein, represents the mean square error loss of yield estimation, represents the cross-entropy loss of risk warning, represents the bias square loss of carbon sink evaluation, and λ1, λ2 and λ3 are task balancing coefficients.

[0049] ​​The multi-task learning model optimized is deployed to an inference environment to input a multi-modal fusion feature representation of a current space-time node, and output results including forest yield estimation, pest and disease and logistics risk early warning and carbon sink evaluation.

[0050] A forest industry chain multi-modal data intelligent processing system comprises:

[0051] A data acquisition unit is configured to arrange acquisition nodes at each business link of seedling raising, planting, tending, felling, transportation, processing, storage and sales, and acquire image modal data, sensor modal data and text modal data respectively to form an original data set covering the whole life cycle;

[0052] A data preprocessing unit is configured to perform unified timestamp correction, spatial coordinate conversion, outlier removal, missing value interpolation and feature normalization on the original data to obtain a feature data set aligned in time and space; the feature data set comprises image features, sensor features and text features;

[0053] A feature fusion and knowledge graph construction unit is configured to map the image features, the sensor features and the text features to the same feature space using a unified coding mechanism, and construct a knowledge graph based on a spatio-temporal graph modeling technology and a forestry ontology library to obtain a multi-modal fusion feature representation;

[0054] A multi-task intelligent prediction unit is configured to input the multi-modal fusion feature representation into a pre-trained multi-task learning model to obtain output results; the output results comprise forest yield estimation, pest and disease and logistics risk early warning and carbon sink evaluation results.

[0055] According to the specific embodiments provided by the present application, the following technical effects are disclosed:

[0056] The present application first unifies the multi-modal data system at the source, then performs spatio-temporal alignment and quality control, eliminating errors caused by sensor asynchrony, image offset and text redundancy; then the knowledge graph is used to connect the dispersed management, ecological and logistics entities in an ontology semantic way, realizing cross-link information interoperability; finally, the multi-task learning model is used to output three key indicators of yield, risk and carbon sink at one time, significantly improving the prediction accuracy and response speed, and providing data-driven support for fine management, risk control and carbon trading. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0058] Figure 1 A method flowchart is provided for the embodiments of the present application.

[0059] Figure 2 A system structure schematic diagram is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0061] The purpose of the present application is to provide a forest industry chain multi-modal data intelligent processing method and system, which significantly improves the real-time and accuracy of forest production scheduling, risk early warning and carbon sink assessment through full-link multi-modal fusion and unified knowledge-driven intelligent prediction.

[0062] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0063] Figure 1 A method flowchart is provided for the embodiments of the present application, as shown in Figure 1 The present application provides a forest industry chain multi-modal data intelligent processing method, which comprises:

[0064] Step 100: laying out collection nodes at each business link of seedling raising, planting, tending, felling, transportation, processing, warehousing and sales, respectively acquiring image modal data, sensor modal data and text modal data, and forming an original data set covering the whole life cycle;

[0065] Step 200: correcting the original data with a unified timestamp, converting the spatial coordinates, removing outliers, interpolating missing values and normalizing the features, to obtain a spatio-temporal aligned feature data set; the feature data set includes image features, sensor features and text features;

[0066] Step 300: using a unified encoding mechanism to map the image features, sensor features and text features to the same feature space, and constructing a knowledge graph based on spatio-temporal graph modeling technology and forestry ontology library, to obtain a multi-modal fusion feature representation;

[0067] Step 400: inputting the multi-modal fusion feature representation into a pre-trained multi-task learning model to obtain an output result; the output result includes: forest yield estimation result, disease and pest hazard and logistics risk early warning result and carbon sink assessment result.

[0068] Preferably, the image modal data includes remote sensing images, unmanned aerial vehicle images, and ground camera images; the sensor modal data includes temperature and humidity, soil moisture, pest and disease sound and image, mechanical vibration, and vehicle positioning information; and the text features include operation logs, warehouse documents, order contracts, and carbon emission trading records.

[0069] In step 100 of the embodiment, the collection node adopts a layered heterogeneous deployment: a replaceable module type sensor bin is configured at the forest land end, an integrated temperature and humidity-soil moisture probe and a pest and disease sound and image array are built-in; a lightweight multi-spectral camera and a pitch adjustable gimbal are loaded at the production link; a vibration-positioning dual-mode module is embedded at the circulation and storage link for monitoring wood load and vehicle path. Through the self-organizing network protocol based on LoRa-UWB, the embodiment enables each node to automatically write the "node ID-business link-geographic coordinates" three metadata when accessing the gateway, thereby realizing the unique indexing and full-link tracing of the three types of data, i.e., image, sensor, and text, which provides indispensable basic support for subsequent spatio-temporal alignment and knowledge graph binding.

[0070] For real-time synchronization and quality control of multi-modal raw data, the embodiment integrates a lightweight clock pulse receiving unit at the node side, and adopts a phase-locked temperature drift combined calibration algorithm to control the collection time error within ±50ms; the image data is directly written into the vector pose at the collection end, the sensor data is marked with ΔZ-coordinate difference value, and the text data uses two-dimensional code-RFID double binding operation work order to realize consistent cross-domain timestamp and coordinate. Through the end-side calibration-marking strategy, the embodiment avoids the information drift caused by traditional post-correction, and provides a verifiable benchmark for missing value interpolation and abnormal value elimination, thereby forming a clear, complete, and implementable raw data set in the whole life cycle of the forest industry, and laying a reliable data foundation for multi-modal fusion and intelligent prediction.

[0071] In step 200 of the embodiment, first, to solve the problem of inconsistent collection time of multi-modal data, the embodiment establishes a unified time reference based on the GPS time signal, and calibrates the local clock of each node to the microsecond level through the phase-locked loop. All collection entries are written into the "four-index" (node ID, business link, geographic coordinates, timestamp) before entering the processing pipeline, and then the time deviation is compensated by the multi-threaded hash mapping program, and the sampling frequency is unified to one second or an integer multiple thereof. The spatial coordinates are projected to the WGS-84 coordinate system: the unmanned aerial vehicle image reads the attitude matrix at the collection moment, the sensor entry is attached with ΔXYZ offset, and the text record is traced back to the static coordinates through the work order two-dimensional code. After completing the space-time correction, the pipeline enters the first stage of quality control: the mean and variance are dynamically calculated using a sliding window, and the observation values exceeding the 99.7% confidence interval are directly removed to ensure that the downstream model is not affected by extreme abnormalities.

[0072] The second stage focuses on missing data repair and feature normalization. For temporally continuous gaps, the embodiment adopts a linear interpolation strategy, but only triggers when the gap width is less than a preset threshold and the volatility of the preceding and subsequent observations is lower than a set threshold, to avoid introducing excessive smoothing errors. For spatially discrete gaps, the KD-Tree is used to accelerate the search for neighboring observed nodes and reuse their observations; if the business scenario or environment label of the neighboring nodes does not match the node to be completed, it is directly marked as “unfillable” and enters the subsequent abnormal report. After the missing data is repaired, the image features are first subjected to adaptive histogram equalization and take the multi-channel gray mean value, the sensor features use five-number quantile truncation smoothing to extract the steady-state interval mean value, and the text features use pre-trained word vector pooling to obtain sentence-level embedding; all three types of features are mapped to the 0-1 interval, forming a set of spatio-temporally aligned and dimensionally consistent feature data, laying a high-confidence data foundation for subsequent feature fusion and knowledge graph binding.

[0073] In the embodiment, first, the unified spatio-temporal index scanning is performed on the multi-modal data after the abnormal data is removed, and the sections with continuous gaps in the time dimension and the positions with discrete gaps in the space dimension are marked. For each temporally continuous gap, the embodiment positions one valid observation record before and after it, and determines whether it meets the interpolation requirements according to the gap length, data volatility and other conditions; under the premise of meeting the interpolation conditions, a plurality of time points to be interpolated in the gap are automatically generated, and linear extrapolation is performed based on the observation values at both ends to complete the interpolation and completion of the continuous time series. After the interpolation is completed, each interpolated time point is marked and updated to the unified spatio-temporal index table for the next stage of spatial completion process.

[0074] For the discrete gaps in the spatial dimension, the embodiment relies on the node position data in the unified geodetic coordinate system, and searches for the nearest observed node to the gap point as the reference object based on the Euclidean distance; if the business link type and environment label of the reference node and the gap node match, the observation value of the reference node is directly copied and filled into the gap position to realize spatial interpolation; if not, the gap position is marked as “unfillable” for subsequent manual or model processing. Finally, the embodiment fuses the time interpolation results and the spatial interpolation results in the original spatio-temporal index order to generate a complete structure, time-continuous, and spatial-consistent missing data set, ensuring that the subsequent feature extraction and normalization modeling have sufficient data support.

[0075] Exemplarily, in the embodiment, the parameters involved in time interpolation include the timestamp t1 of the previous valid time point, the timestamp t2 of the subsequent valid time point, and the timestamp t m, and the corresponding observation values V(t1), V(t2). The above timestamp parameters are derived from a space-time index table under a unified time reference, where t1 and t2 are automatically obtained by scanning the nearest valid records at both ends of the gap, and t m are uniformly divided according to the set interpolation granularity (e.g., 1 second or 10 seconds); the observation values V(t1), V(t2) are derived from the original or processed data records at the corresponding time points. For the coordinates p m of the positions to be interpolated in spatial interpolation, they are given by the position identification of the gap node, and all the reference positions p j come from the spatial information of the observed nodes, and the Euclidean distance function d(p m , p j ) is based on standard geometric calculations, and the nearest neighbor index k is automatically selected by the system from all p j that meet the minimum distance condition.

[0076] Further, in step 300 of the embodiment, first, the image features, sensor features and text features are encoded by using a convolutional neural network, a time sequence attention network and a pre-trained language model respectively, and then in the cross-modal feature alignment module, the three types of features are uniformly mapped and normalized into joint feature representations through a shared embedding layer. Subsequently, the embodiment constructs a space-time graph with a unified time index by taking forest land blocks, processing nodes and circulation units as nodes and taking spatial proximity, supply chain correlation and operation coordination relationship as edges, introduces species type, management mode and ecological zoning semantic information of the forestry ontology library as node attributes, and obtains a multi-modal fusion feature representation through a graph neural network training.

[0077] Still further, in the embodiment, first, original terms are collected from national forestry industry standards, forestry science and technology journals and forestry government database, including six high-frequency vocabularies of forest species type, growth stage, soil type, management mode, pest and disease category and climate zoning. The system performs de-redundancy filtering, semantic alignment and morphological restoration on these terms to form a unified set of standardized terms, and accordingly establishes four ontology concept classes: “forest entity”, “ecological factor”, “management measure” and “disaster event”, and constructs a hierarchical structure and attribute set for each concept class to ensure clear concept boundaries and complete description.

[0078] After the concept system is established, the embodiment defines the semantic relationship between the concept classes in the form of triplets <C1, R, C2>, and the relationship types are fixed as “belonging relationship”, “growth dependence relationship”, “management measure relationship” and “disease influence relationship”, thereby generating a relationship set covering species classification, ecological interaction and management activities. Subsequently, the OWL description logic reasoning engine is called to perform consistency checking on the concept class set and the relationship set, automatically eliminate circular inheritance, attribute conflict and concept overlap, and output a forestry ontology library with consistent structure and reasoning capability.

[0079] Finally, this embodiment vectorizes the consistency-verified ontology model, enabling it to be bound one-to-one to node entities in the spatiotemporal graph model, providing semantically enhanced input for the graph neural network. Here, C1 in the triple represents the head concept class (the starting point of the relationship), R represents the semantic relationship type between the two concept classes, and C2 represents the tail concept class (the end point of the relationship); these three together characterize the knowledge unit of "starting point concept - semantic relationship - end point concept", supporting subsequent multimodal feature fusion and semantic reasoning.

[0080] Preferably, the spatiotemporal graph model is trained using a graph neural network to capture the correlation and propagation structure between multimodal feature vectors to obtain the multimodal fusion feature representation, including:

[0081] Take each node in the spatiotemporal graph model as input, load the corresponding image features, sensor features and text features, and initialize the node vector representation

[0082] In each propagation layer, the neighbor set of node i is calculated based on the spatial adjacency relationship between nodes, supply chain edge weight and operation time window. The representation of each node j in Perform weighted aggregation and calculate the output of the message passing at layer l

[0083] After repeatedly performing L-layer message passing and nonlinear transformation, the fused representation of each node in the L-layer is obtained. That is the multimodal fusion feature representation.

[0084] In this embodiment, the training process of the graph neural network uses a constructed spatiotemporal graph model as input, where each graph node corresponds to a forest plot, processing unit, or logistics entity. Before training begins, this embodiment loads the corresponding image features, sensor features, and text features for each node, and uniformly encodes the three types of features into a fixed-dimensional vector as the initial representation of the node. This representation not only preserves the complementarity of multimodal information but also provides a unified numerical basis for subsequent graph structure propagation.

[0085] In each propagation layer of the graph neural network, the present embodiment takes the spatial neighboring nodes of the current node as the neighbor set, while combining the edge attribute information between the two nodes, including the geographical proximity relationship, the supply chain flow weight, and the overlap degree of the operation time window. The system calculates the attention weight according to these edge features, which is used to measure the importance of each neighbor node to the current node. Then, the vector representation of the neighbor nodes is weighted and aggregated according to the weight, and the aggregation result is combined with the representation of the current node itself, and then a set of trainable weight matrices are used for linear transformation, and an activation function is applied to introduce a nonlinear response, forming the output representation of the layer.

[0086] The present embodiment repeatedly executes the above propagation and transformation mechanism for multiple layers (for example, three to five layers) to expand the information perception domain and enhance the joint expression ability of cross-modal features. The node representation obtained at the last layer is the multi-modal fusion feature representation, which is used as the input of the downstream multi-task prediction model. The attention weight used in the training process is dynamically calculated by the system according to the edge attribute during training; the trainable weight matrix is automatically determined by the initialization parameter and the back propagation optimization; and the activation function adopts a known nonlinear function form, such as the rectified linear unit (ReLU) or its variants.

[0087] In step 400 of the present embodiment, first, the historical forest management samples are uniformly arranged: for each sample, the yield monitoring value, pest and disease occurrence record, logistics abnormal event and carbon sink measured data are extracted, and the missing data is removed, the unit is converted and the normalization processing is performed; then according to the plot number, operation timestamp and logistics node coordinates of the sample, the above labels are aligned with the previously generated spatio-temporal feature index one by one, and a multi-label training dataset containing four types of output indicators is constructed, laying a data foundation for subsequent multi-task learning.

[0088] The multi-task learning network constructed in the present embodiment consists of a shared encoder and three task branches. The shared encoder is responsible for reading the multi-modal fusion feature representation, and compressing redundant information and extracting cross-modal commonality through several layers of nonlinear mapping. The subsequent yield estimation branch adopts a regression structure to output the forest stock volume prediction value; the risk early warning branch adopts a classification structure to judge the outbreak probability of pests and diseases and the delay risk grade of logistics at the same time; the carbon sink evaluation branch adopts a regression structure to calculate the carbon storage and dynamic increment, and the three branches share the front-end feature abstraction ability while maintaining their own task-specific output space.

[0089] In the training stage, the embodiment uses a supervised learning framework to input the multi-label training data into the model and simultaneously calculate the loss of three types of tasks; the system sets a learnable weight coefficient for the loss term, so that the model automatically balances the importance of each task during the training process, and improves the overall prediction accuracy. After the model converges, it is deployed in the inference environment: when new multi-modal fusion feature representations are input in real time, the system can output forest yield estimates, pest and disease risk levels, and carbon sink assessment results at one time, realizing immediate decision support for production scheduling, risk control, and carbon asset management.

[0090] Corresponding to the above method, as shown in Figure 2 The embodiment also provides a forest industry chain multi-modal data intelligent processing system, comprising:

[0091] A data acquisition unit is configured to deploy acquisition nodes at each business link of seedling raising, planting, nurturing, felling, transportation, processing, warehousing and sales, and acquire image modal data, sensor modal data and text modal data respectively to form an original data set covering the whole life cycle;

[0092] A data preprocessing unit is configured to perform unified timestamp correction, spatial coordinate conversion, outlier removal, missing value interpolation and feature normalization on the original data to obtain a spatio-temporally aligned feature data set; the feature data set comprises image features, sensor features and text features;

[0093] A feature fusion and knowledge graph construction unit is configured to map the image features, the sensor features and the text features to the same feature space using a unified encoding mechanism, and construct a knowledge graph based on spatio-temporal graph modeling technology and a forestry ontology library to obtain a multi-modal fusion feature representation;

[0094] A multi-task intelligent prediction unit is configured to input the multi-modal fusion feature representation into a pre-trained multi-task learning model to obtain an output result; the output result comprises forest yield estimation results, pest and disease risk early warning results and carbon sink assessment results.

[0095] The beneficial effects of the present application are as follows:

[0096] (1) The present application deploys heterogeneous acquisition nodes at each key link of the forest industry whole industry chain, establishes a unified spatio-temporal indexing system, and for the first time realizes the structured fusion acquisition and whole life cycle tracking of image modal data, sensor modal data and text modal data, solves the problems of data fragmentation, time misalignment and non-uniform coordinates in traditional forestry information systems, and provides a highly consistent data basis for subsequent intelligent analysis.

[0097] (2) The application introduces a spatio-temporal modeling mechanism based on a graph neural network, combines a forest ontology library to construct a knowledge graph with semantic enhancement capability, so that multi-modal data can not only be fused in the feature dimension, but also form a structure expression that can be inferred in the semantic level, significantly improving the explainability and modeling capability of the relationship between forestry management, ecological elements and logistics.

[0098] (3) The multi-task learning model constructed by the application has end-to-end parallel prediction capability, can complete three types of decision support tasks of forest yield estimation, pest and disease and logistics risk early warning and carbon sink evaluation in a unified model, avoids repeated calculation and collaborative error caused by independent training and data splitting of multiple models, and improves the overall reasoning efficiency and application stability.

[0099] (4) The interpolation completion, abnormality rejection and normalization strategy and feature fusion process adopted by the application are closely connected, avoiding manual intervention and rule fragmentation, forming a closed-loop mechanism from the data source to intelligent reasoning, having the characteristics of high automation and strong engineering portability, and being suitable for intelligent management and carbon sink supervision scenarios in large-scale forest areas.

[0100] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0101] In this paper, specific examples are used to illustrate the principles and implementation modes of the application, and the above embodiment descriptions are only used to help understand the method and core idea of the application; at the same time, for those skilled in the art, according to the idea of the application, the specific implementation mode and application range will be changed. In view of the above, the content of the specification should not be understood as a limitation of the application.

Claims

1. A method for intelligent processing of multimodal data in the forestry industry chain, characterized in that: include: Deploy collection nodes in each business link of seedling cultivation, planting, tending, harvesting, transportation, processing, warehousing and sales to obtain image modality data, sensor modality data and text modality data, forming an original data set covering the entire life cycle; Performing unified timestamp correction, spatial coordinate conversion, outlier removal, missing value interpolation, and feature normalization on the raw data to obtain a spatiotemporally aligned feature data set; the feature data set includes image features, sensor features, and text features; The image features, the sensor features, and the text features are mapped to the same feature space using a unified coding mechanism, and a knowledge graph is constructed based on spatiotemporal graph modeling technology and a forestry ontology library to obtain a multimodal fusion feature representation; Inputting the multimodal fusion feature representation into a pre-trained multi-task learning model to obtain an output result; The output results include: forest yield estimation results, pest and disease and logistics risk warning results and carbon sink assessment results.

2. The method for intelligent processing of multimodal data of the forestry industry chain according to claim 1, characterized in that: The image modal data includes remote sensing images, drone images and ground camera images; the sensor modal data includes temperature and humidity, soil moisture, pest and disease sound and image, mechanical vibration and vehicle positioning information; the text features include operation logs, warehouse documents, order contracts and carbon emission transaction records.

3. The method for intelligent processing of multimodal data of the forestry industry chain according to claim 1, characterized in that: The raw data is subjected to unified timestamp correction, spatial coordinate conversion, outlier removal, missing value interpolation and feature normalization to obtain a spatiotemporally aligned feature data set, including: Reading the time stamps and spatial stamps of the three types of raw data, namely, image, sensor, and text, and generating an initial index table with a unified time base and a unified geodetic coordinate system as key values; Using the initial index table, performing time compensation adjustment on the raw data of each modality so that the acquisition time is aligned to a unified sampling period and the spatial coordinates fall into the same reference system, and outputting corrected multimodal data entries; For the corrected multimodal data entries, the Z-Score threshold method is used to detect and delete observations that exceed the confidence interval to obtain an anomaly elimination data set; On the anomaly elimination dataset, linear interpolation is used to fill time-continuous gaps, and spatial interpolation based on the nearest neighbor relationship is used to fill spatial discrete gaps to generate a missing-filled dataset; Image features, sensor features and text features are extracted from the missing completion data set respectively, minimum-maximum normalization processing is performed on each type of feature, and the feature data set aligned in time and space is output.

4. The method for intelligent processing of multimodal data of the forestry industry chain according to claim 3 is characterized in that: On the anomaly elimination dataset, linear interpolation is used to fill time-continuous gaps, and spatial interpolation based on the nearest neighbor relationship is used to fill spatial discrete gaps to generate a missing completion dataset, including: Scanning the unified spatiotemporal index of the anomaly removal dataset to mark the start and end positions of continuous time gaps and discrete space gaps; For each continuous time gap, select a valid data record V(t1) and V(t2) before and after the continuous time gap, and use the linear interpolation formula Calculate any interpolation time point t within the gap m Estimated value of Among them, t1 is the timestamp of the valid time point before the gap; t2 is the timestamp of the valid time point after the gap; t m is the timestamp of the time point to be interpolated in the gap; V(t) is the observation value at time t; m is the index number of the time point in the gap; For each discrete space gap, search for the observed position p with the closest Euclidean distance k , and use the nearest neighbor interpolation formula Generate gap position p m The estimated value of m is the coordinate vector of the spatial position to be interpolated; p k For p m The coordinates of the observed position with the closest Euclidean distance; d(·) is the Euclidean distance function; V(p) is the observed value at position p; where, Indicates that among all observed points, the point index j with the minimum Euclidean distance corresponds to position p k ; Merge in the order of original time and space index and The missing completion data set is formed.

5. The method for intelligent processing of multimodal data of the forestry industry chain according to claim 1, characterized in that: The image features, sensor features, and text features are mapped to the same feature space using a unified coding mechanism, and a knowledge graph is constructed based on spatiotemporal graph modeling technology and a forestry ontology library to obtain a multimodal fusion feature representation, including: A convolutional neural network encoding structure is used to extract spatial distribution tensors from the image features, a temporal attention network is used to encode dynamic change patterns of the sensor features to obtain encoding vectors, and a pre-trained language model is used to extract contextual embedding representations from the text features, which are uniformly converted into fixed-dimensional embedding vectors. The spatial distribution tensor, the encoding vector, and the embedding vector are fed into a cross-modal feature alignment module, and the multi-modal feature vectors are uniformly aligned based on a shared embedding layer to obtain a normalized joint feature representation; Taking forest plots, processing nodes, and circulation units as graph nodes, and spatial proximity, supply chain relevance, and operational collaboration as edges, a graph structure is established by combining multimodal feature vectors, and a unified time index is introduced to form a spatiotemporal graph model. Calling the forestry ontology library, embedding the semantic information of the species type, management mode, and ecological zoning of the graph node into the graph node attributes; The spatiotemporal graph model is trained using a graph neural network to capture the correlation and propagation structure between multimodal feature vectors and obtain the multimodal fusion feature representation.

6. The method for intelligently processing multimodal data of the forestry industry chain according to claim 5, characterized in that: The construction method of the forestry ontology library is as follows: Extract six high-frequency terms, including forest species, growth stages, soil types, management methods, pest and disease categories, and climate zones, from national forestry industry standards, forestry science and technology journals, and forestry government databases. Perform redundancy removal, semantic alignment, and morphological restoration on these high-frequency terms to output a standardized term set. Based on the standardized terminology, four ontology concept classes were created, namely "forest entity", "ecological factor", "management measure" and "disaster event", and the hierarchical structure and attribute set were determined for each ontology concept class; In triple form<C1,R,C2> Define semantic relationships between concept classes, including "belonging to relationship", "growth dependency relationship", "management measure relationship" and "disease impact relationship", and generate relationship sets; Using the OWL description logic reasoning engine to verify the consistency of the ontology concept class set and the relationship set, eliminating circular definitions, attribute conflicts and concept overlaps, and obtaining the forestry ontology library with consistent structure; The forestry ontology library that has passed consistency verification is converted into a vectorized representation and bound to the node entity in the spatiotemporal graph model.

7. The method for intelligently processing multimodal data of the forestry industry chain according to claim 5, characterized in that: The spatiotemporal graph model is trained using a graph neural network to capture the correlation and propagation structure between multimodal feature vectors, thereby obtaining the multimodal fusion feature representation, including: Take each node in the spatiotemporal graph model as input, load the corresponding image features, sensor features and text features, and initialize the node vector representation In each propagation layer, the neighbor set of node i is calculated based on the spatial adjacency relationship between nodes, supply chain edge weight and operation time window. The representation of each node j in Perform weighted aggregation and calculate the output of the message passing at layer l After repeatedly performing L-layer message passing and nonlinear transformation, the fused representation of each node in the L-layer is obtained. That is the multimodal fusion feature representation.

8. The method for intelligently processing multimodal data of the forestry industry chain according to claim 7, characterized in that: The output of the message passing The calculation formula is: Among them, α ij W represents the adjacent attention weight constructed by edge features, reflecting the association strength of node j to node i; (l) is the trainable weight matrix of the lth layer; σ(·) is the activation function used to introduce nonlinearity.

9. The method for intelligent processing of multimodal data in the forestry industry chain according to claim 1, characterized in that: The multimodal fusion feature representation is input into a pre-trained multi-task learning model to obtain an output result, including: The yield monitoring data, pest and disease records, logistics events, and carbon sink observations corresponding to historical forest management samples were preprocessed and standardized, and aligned with the corresponding spatiotemporal feature indexes to construct a training sample set with multiple output labels. Constructing a multi-task learning model that includes a shared encoder and multiple task branches, wherein the shared encoder is used to receive and abstract multimodal fusion feature representations, and each task branch is used to predict the target of yield estimation, risk warning, and carbon sink assessment respectively; The training sample set is modeled using supervised learning, and the following joint loss function is used for multi-task optimization: in, represents the mean square error loss of yield estimation, represents the cross entropy loss of risk warning, represents the squared deviation loss of carbon sink assessment, λ1, λ2, and λ3 are task balance coefficients; The optimized multi-task learning model is deployed into the inference environment to input the multimodal fusion feature representation of the current spatiotemporal node. The output includes forest yield estimation results, pest and disease and logistics risk warning results, and carbon sequestration assessment results.

10. A multimodal data intelligent processing system for the forestry industry chain, characterized in that: include: The data collection unit is used to deploy collection nodes in each business link of seedling cultivation, planting, tending, felling, transportation, processing, warehousing and sales to obtain image modality data, sensor modality data and text modality data respectively, forming an original data set covering the entire life cycle; A data preprocessing unit, configured to perform unified timestamp correction, spatial coordinate conversion, outlier removal, missing value interpolation, and feature normalization on the raw data to obtain a spatiotemporally aligned feature data set; the feature data set includes image features, sensor features, and text features; A feature fusion and knowledge graph construction unit, configured to map the image features, the sensor features, and the text features into the same feature space using a unified encoding mechanism, and to construct a knowledge graph based on spatiotemporal graph modeling technology and a forestry ontology library to obtain a multimodal fusion feature representation; A multi-task intelligent prediction unit, configured to input the multimodal fusion feature representation into a pre-trained multi-task learning model to obtain an output result; The output results include: forest yield estimation results, pest and disease and logistics risk warning results and carbon sink assessment results.

Images