Palm oil plantation multi-modal decision management method, system, equipment and medium

Abstract

The invention provides a palm oil plantation multi-modal decision management method, system and device and a medium, and belongs to the technical field of Internet of Things in the planting industry, and the method comprises the steps: building a heterogeneous federated learning framework for a palm oil plantation; the edge layer constructs a lightweight model, a planting industry heterogeneous terminal device is used for collecting data for local training, and asynchronous federal aggregation is performed in combination with the fog computing layer until requirements are met; the edge layer is used for extracting local features from data collected by heterogeneous terminal equipment in the planting industry; the cloud computing layer aligns local features of the edge layer by using a cross-modal space-time encoder, constructs a causal graph in combination with a Bayesian network, determines intervention measures, quantifies the influence of the intervention measures by using anti-factual reasoning, generates and executes a decision scheme, and solves layered Nash equilibrium through a distributed Actor-Critic algorithm, so as to improve the performance of the cloud computing layer. And task allocation and path planning are carried out on the planting industry heterogeneous terminal equipment. According to the invention, the utilization rate of planting management data, decision reasonability and equipment collaboration are improved.

Description

Technical Field

[0001] The present application belongs to the technical field of Internet of Things in the planting industry, and specifically relates to a multimodal decision-making management method, system, equipment and medium for palm oil plantations. Background Art

[0002] Traditional palm oil plantation management relies primarily on manual inspections and empirical judgment, which is not only inefficient but also difficult to meet the needs of intelligent management and decision-making. With the development of the Internet of Things in the plantation industry, intelligent technology has begun to be applied in various plantations, but the following problems still exist: First, palm oil plantation management requires comprehensive consideration of multiple data sources, including RGB images captured by drones, LiDAR point clouds scanned by unmanned vehicles, and environmental data collected by soil sensors. These data modalities differ significantly, and existing multimodal data fusion methods primarily rely on simple feature concatenation or weighted averaging, failing to fully utilize the spatiotemporal characteristics of each modality. This results in low fusion accuracy and makes it difficult to meet the needs of precision agriculture. Second, palm oil plantation management requires decision-making based on real-time data, such as for pest control, fertilization, and irrigation. Traditional decision-making methods, which primarily rely on empirical models for correlation analysis, cannot provide explainable decision solutions and are unable to meet the needs of complex agricultural scenarios. Third, intelligent palm oil plantation management requires efficient collaboration between the edge layer, fog computing layer, and cloud computing layer. Traditional federated learning frameworks primarily rely on centralized data processing, making it difficult to achieve cross-device feature alignment and form a global decision-making basis. This leads to large errors in multi-source data fusion and low device collaboration efficiency. Summary of the Invention

[0003] In a first aspect, an embodiment of the present application provides a multimodal decision-making management method for a palm oil plantation, comprising the following steps: S1. Build a heterogeneous federated learning framework for palm oil plantations. This involves deploying an edge layer on various heterogeneous plantation terminal devices, a fog computing layer on local plantation servers, and a cloud computing layer on remote cloud servers. S2. Each edge layer builds a lightweight model and uses data collected from its own heterogeneous agricultural terminal devices for local training. The local model parameters are uploaded to the fog computing layer for asynchronous federated aggregation. The fog computing layer then returns the global model parameters for re-local training until the lightweight model meets the requirements. S3. The edge layer uses the trained lightweight model to extract local features from the time series data collected by heterogeneous terminal devices in the corresponding planting industry and provides them to the cloud computing layer. S4. The cloud computing layer uses a cross-modal spatiotemporal encoder to align local features at each edge layer to generate global features. This global feature is then combined with a Bayesian network to construct a causal graph. Interventions are determined based on the planting phase requirements and the causal graph. Counterfactual reasoning is used to quantify the impact of the interventions and generate a decision plan. S5. The cloud computing layer executes the decision-making plan, solves the hierarchical Nash equilibrium through the distributed actor-critic algorithm, and performs task allocation and path planning for heterogeneous terminal devices in the agricultural industry.

[0004] Furthermore, the heterogeneous terminal devices for the planting industry in step S1 include drones equipped with RGB and thermal imaging cameras, unmanned vehicles equipped with LiDAR scanners, and soil sensors for collecting soil environment data.

[0005] Furthermore, the specific steps of step S2 are as follows: S21. Each edge layer constructs a lightweight model and performs local training using time series data collected by its own heterogeneous agricultural terminal devices. The trained local model parameters are then gradient-encoded and uploaded to the fog computing layer. S22. The fog computing layer receives local model parameter encoding data and measures communication delay , and then according to the communication delay Calculate the proximal term coefficient ;

[0006] S23. The fog computing layer regularly uses the optimized asynchronous federated aggregation algorithm to asynchronously weight the decoded local model parameters to generate global model parameters, and then performs gradient coding on the global model parameters and the proximal term coefficients. Send it to each edge layer together; S24. The edge layer receives the global model parameter encoding and the proximal term coefficients , using the time series data collected by the heterogeneous terminal devices of each planting industry for local training, and using the proximal term coefficient in the local training process Update the local model parameters with the decoded global model parameters; S25. Determine whether the global maximum number of iterations is met or whether the difference between each local model parameter and the global model parameter converges; If yes, go to step S3; If not, upload the local model parameters to the fog computing layer, continue iterating, and return to step S22.

[0007] Furthermore, the specific steps of constructing the lightweight model in step S21 are as follows: Select the YOLOv7 model and replace the standard convolutional layer in the base feature extractor of the YOLOv7 model with a depthwise separable convolutional layer; Determine the width of the YOLOv7 model based on the computing power of the heterogeneous terminal devices and the target planting stage, and optimize the detection head using the NEON instruction set; The improved YOLOv7 model is deployed as the initial lightweight model on the ARM Cortex-A72 platform of the corresponding heterogeneous terminal equipment in the planting industry.

[0008] Furthermore, in step S21, the edge layer i constructs a lightweight model and calculates the local loss function based on the local collected data. ; During the training process, the edge layer i minimizes the local loss function through the optimization algorithm , get the local model parameters θ ; In step S24, the edge layer i calculates the proximal term coefficients returned by the fog computing layer and the decoded global model parameters Calculate the loss function regularization term ; The local loss function and loss function regularization term Superposition to obtain the corrected loss function ; During the training process, the edge layer i minimizes the local loss function through the optimization algorithm , get the updated local model parameters ; The specific steps of performing asynchronous weighted aggregation in step S23 are as follows: The fog computing layer creates a delay counter for each edge layer i , record the consecutive rounds that did not participate in aggregation; The fog computing layer updates the delay counter in the following way before each aggregation ; If edge layer i participates in the current round of aggregation, =0; otherwise = +1; When the fog computing layer performs aggregation, only ≤N edge layer i is aggregated and communication delay is used and the proximal term coefficient Calculate the weights of each local model parameter ;

[0009] in, ; During the aggregation process, each local model parameter Perform weight normalization to calculate global model parameters ;

[0010] Among them, S is satisfied The set of edge layers ≤ N.

[0011] Furthermore, in step S3, the lightweight model of the edge layer to which the UAV belongs extracts RGB image features from the collected RGB image; The lightweight model of the edge layer of the unmanned vehicle extracts LiDAR point cloud features from the collected LiDAR point cloud; The lightweight model of the edge layer to which the soil sensor belongs extracts soil time series features from soil environmental data; The specific steps of step S4 are as follows: S41. The cloud computing layer constructs three branches of the Transformer architecture, including the RGB branch, the LiDAR branch, and the sensor timing branch. S42. The cloud computing layer uses the ViT-Base model in the RGB branch to divide the RGB image features into image blocks of a set number of dimensions and converts them into image vectors through linear projection. S43. The cloud computing layer uses the Point Transformer model in the LiDAR branch to convert the LiDAR point cloud features into voxel vectors of a set unit size; S44. The cloud computing layer uses the TCN temporal convolution layer in the sensor timing branch to convert the soil time series features into a time series vector according to the set convolution kernel size and step size; S45. The cloud computing layer calculates the query parameters, key parameters, and value parameters for each vector based on the preset learnable weights, and calculates the cross-attention scores between the branches, and then fuses the cross-attention scores using a gated fusion network; S46. The cloud computing layer encodes the GPS coordinates and timestamps into a position vector, adds the position vector to the fused cross-attention score, and obtains a fused feature. A fully connected layer is then used to convert the fused feature into a global feature map. S47. Preselect key variables from the global feature map, decode the variable values of the key variables, and convert them into agronomic variable vectors of set dimensions; S48. Construct an initial causal graph. Based on prior knowledge in the agronomy field, determine the domain constraint edges related to key variables and add them to the initial causal graph. First, use the G-test conditional independence test to determine whether there is a correlation between two key variables at different stages of palm oil cultivation based on statistical independence. Obtain a set of candidate edges. Then, use the agronomy constraints to determine the final causal relationship, completing the causal graph construction. S49. Based on the cause-and-effect diagram and in accordance with target requirements, select key variables with an impact higher than the threshold in the corresponding palm oil planting stage as intervention variables. Determine the target value and intervention measures of the intervention variables based on agronomic knowledge and historical data. Use the do-calculus method to quantify the impact of the intervention measures on the target variables and use Monte Carlo sampling to estimate them. When the quantification and estimation results meet the requirements, generate a decision plan and determine the operation type and operation area of the intervention measures.

[0012] Furthermore, the specific steps of step S5 are as follows: S51. The cloud computing layer predefines state vectors for the drone and the unmanned vehicle to characterize their states. The drone's state vector includes its current location, remaining battery power, current mission type, and mission queue length. The unmanned vehicle's state vector includes terrain slope, soil hardness, robotic arm status, and vehicle storage status. S52. Establish an objective function using the drone as the upper layer, with the coverage of crops in the intervention state as the reward term and the number of intersections between the drone and the unmanned vehicle's operation path as the penalty term. Establish a Q-value function using the unmanned vehicle as the lower layer, with the task progress term as the positive incentive and the energy efficiency term and path deviation term as the negative incentives. S53. Build a UAV strategy network with the goal of maximizing the objective function value. Based on the UAV's state vector, solve the access probability distribution of each operating area to guide the UAV's task allocation. Build a measurement network for the unmanned vehicle, aiming to maximize the Q-value function. Calculate the speed or steering angle in the continuous action space based on the vehicle's state vector to guide the vehicle's path planning. S54. Construct a Q network to minimize the value prediction error and calculate the long-term value prediction of the current state action based on the corresponding state vector and the action selected by the policy network; S55. Calculate task priorities based on the crop's intervention level and planting stage, predict path conflicts between the UAV and the unmanned vehicle, and adjust the corresponding paths or timing when a conflict is predicted. S56. Calculate the task completion degree, evaluate the task execution effect based on the completion degree, and then update the strategy network parameters based on the difference between the actual production increase rate and the predicted production increase rate: If the actual production increase rate is higher than the predicted production increase rate, the probability of selecting the original strategy will be increased; If the actual production increase rate is lower than the predicted production increase rate, the probability of selecting the original strategy will be reduced.

[0013] In a second aspect, an embodiment of the present application further provides a palm oil plantation multimodal decision management system, comprising: A federated learning framework building module is used to build a heterogeneous federated learning framework for palm oil plantations, deploying the edge layer on various heterogeneous terminal devices in the plantation, deploying the fog computing layer on the plantation's local servers, and deploying the cloud computing layer on remote cloud servers; The asynchronous federated learning module is used to build lightweight models at each edge layer, collect data from heterogeneous terminal devices in each plant, perform local training, upload local model parameters to the fog computing layer for asynchronous federated aggregation, and then receive the global model parameters returned by the fog computing layer for re-local training until the lightweight model meets the requirements; The feature extraction module is used to extract local features from the time series data collected by heterogeneous terminal devices in the corresponding planting industry using a trained lightweight model at the edge layer and provide it to the cloud computing layer; The decision generation module is used to align local features of each edge layer using a cross-modal spatiotemporal encoder at the cloud computing layer to generate global features. This is then combined with a Bayesian network to construct a causal graph. Intervention measures are determined based on the needs of the planting stage and the causal graph. Counterfactual reasoning is used to quantify the impact of the intervention measures and generate a decision plan. The decision execution module is used to execute decision plans in the cloud computing layer, solve the hierarchical Nash equilibrium through the distributed Actor-Critic algorithm, and perform task allocation and path planning for heterogeneous terminal devices in the planting industry.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the steps of the multimodal decision-making management method for palm oil plantations as described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application further provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal decision-making management method for palm oil plantations as described in the first aspect.

[0016] It can be seen from the above technical solutions that this application has the following advantages: In the multimodal decision-making management method, system, equipment and medium for palm oil plantations provided in this application, a cross-modal spatiotemporal encoder is used to achieve alignment of RGB images, LiDAR point clouds and soil sensor data, reduce multi-source data fusion errors, improve data fusion accuracy, and meet the needs of the planting industry; by combining Bayesian networks and agronomic prior knowledge to construct a causal graph, do-calculus is used to quantify the impact of intervention measures on target variables, generate intervention strategies that conform to agronomic laws, and improve the decision rationality verification pass rate; through the distributed Actor-Critic algorithm to solve the hierarchical Nash equilibrium, efficient collaborative operation of drones and unmanned vehicles is achieved, the energy consumption of equipment collaborative operation is reduced, and operational efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 Schematic diagram of the process of the multimodal decision-making management method for palm oil plantations of the present invention.

[0019] Figure 2 Schematic diagram of the process of the multimodal decision management system for palm oil plantations of the present invention. DETAILED DESCRIPTION

[0020] The various embodiments of the present disclosure will be described more fully below in detail in the specific steps of the multimodal decision-making management method for palm oil plantations. The present disclosure can have various embodiments, and modifications and variations can be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but rather that the present disclosure is to encompass all modifications, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of the present disclosure.

[0021] For example, in the traditional palm oil plantation management model, the limitations of manual inspections and experience-based decision-making are becoming increasingly prominent, and their inefficiency and lack of intelligence need to be addressed urgently. Although the Internet of Things technology in the planting industry has been gradually applied to plantation scenarios, existing technologies still face the following problems: First, palm oil plantation management requires the integration of multi-source heterogeneous information such as drone RGB images, unmanned vehicle LiDAR point clouds, and soil sensor environmental data. However, current multimodal data fusion technologies generally use simple processing methods such as feature splicing or weighted averaging, which fail to deeply capture the spatiotemporal correlation characteristics of data of different modalities. Secondly, existing decision-making mechanisms mostly construct empirical models based on statistical correlation, and lack systematic modeling of causal relationships in the planting process. The inherent defects of this correlation analysis lead to a lack of explainability and scientific basis for decision-making plans, making it difficult to meet the dynamic intervention needs in complex planting scenarios. Thirdly, intelligent management places higher demands on the collaborative operation of the edge layer, fog computing layer, and cloud computing layer. However, existing federated learning frameworks mostly adopt a centralized data processing mode, which has two problems: on the one hand, the cross-device feature alignment mechanism is missing, and the local features of each edge terminal are difficult to form a globally consistent representation in the fog computing layer, resulting in the accumulation of multi-source data fusion errors; on the other hand, device collaboration relies on unified scheduling in the cloud. In scenarios with high network latency or device offline, problems such as unbalanced task allocation and path planning conflicts are prone to occur, which cannot meet the real-time management needs of high-density plantations.

[0022] The above problems seriously restrict the depth and breadth of the intelligent upgrade of palm oil plantations, and there is an urgent need for a management paradigm that integrates multimodal deep modeling, causal reasoning and hierarchical collaborative optimization.

[0023] To address the above issues, this embodiment provides a multimodal decision-making management method for palm oil plantations. By interactively coupling a heterogeneous federated learning framework, multimodal deep fusion, causal reasoning modeling, and a hierarchical collaborative optimization algorithm, it overcomes the limitations of inefficient data utilization, insufficient decision rationality, and poor equipment coordination in traditional plantation management.

[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0025] See also Figure 1 FIG. 1 is a flowchart of a multimodal decision-making management method for a palm oil plantation in a specific embodiment, wherein the method comprises the following steps: S1. Build a heterogeneous federated learning framework for palm oil plantations. This involves deploying an edge layer on various heterogeneous plantation terminal devices, a fog computing layer on local plantation servers, and a cloud computing layer on remote cloud servers. It should be noted that by deploying the edge layer, fog computing layer, and cloud layer in layers, the edge layer processes raw data locally, reducing data transmission volume, privacy leakage risks, and communication costs. The fog computing layer is responsible for model aggregation and primary decision-making, which can alleviate the pressure on the cloud and reduce the delay of real-time tasks. Global optimization and causal reasoning are performed by the cloud computing layer, which can achieve in-depth calculations for complex decisions. S2. Each edge layer builds a lightweight model and uses data collected from its own heterogeneous agricultural terminal devices for local training. The local model parameters are uploaded to the fog computing layer for asynchronous federated aggregation. The fog computing layer then returns the global model parameters for re-local training until the lightweight model meets the requirements. It should be noted that through lightweight model local training and asynchronous aggregation of the fog computing layer, local training can be achieved without cross-domain data transmission, eliminating the need for data conversion and meeting the data requirements of the agricultural industry. Asynchronous aggregation allows breakpoint resumption after the device is offline, improving the efficiency of model training interruption recovery. Gradient coding compression reduces parameter transmission volume and adapts to low-bandwidth networks. S3. The edge layer uses the trained lightweight model to extract local features from the time series data collected by heterogeneous terminal devices in the corresponding planting industry and provides them to the cloud computing layer. It should be noted that by extracting local features through lightweight models, feature dimensionality reduction is achieved on the edge side, the load transmission volume is reduced, and invalid data interference is reduced through pre-screening of time series features, thereby improving the feature signal-to-noise ratio; S4. The cloud computing layer uses a cross-modal spatiotemporal encoder to align local features at each edge layer to generate global features. This global feature is then combined with a Bayesian network to construct a causal graph. Interventions are determined based on the planting phase requirements and the causal graph. Counterfactual reasoning is used to quantify the impact of the interventions and generate a decision plan. It should be noted that by aligning features with the Bayesian network through the Transformer encoder, the causal graph is constructed, thereby improving the feature dimension through cross-modal fusion and enhancing the semantic representation ability. The intervention effect is quantified through the causal graph, which improves the confidence of the decision. Counterfactual reasoning supports multiple scenario simulations and reduces the cost of trial and error in the field. S5. The cloud computing layer executes the decision-making plan, solving the hierarchical Nash equilibrium using a distributed actor-critic algorithm, and performing task allocation and path planning for heterogeneous terminal devices in the agricultural industry. It should be noted that the Actor-Critic algorithm is a reinforcement learning algorithm that combines policy gradient and value function. It is achieved through collaborative learning between two neural networks. The Actor network is responsible for generating the action strategy of the intelligent agent, that is, deciding what action to take in a given state; the Critic network is responsible for evaluating the quality of the strategy generated by the Actor network, and providing feedback signals to the Actor network by estimating the state value function or action value function; the distributed Actor-Critic algorithm extends this architecture to a distributed computing environment, using multiple Actors and Critic to learn in parallel to accelerate the training process. Actors and Critic on different heterogeneous terminal devices can interact with the environment independently and share experience data, thereby improving sample efficiency and training speed. Through the distributed Actor-Critic algorithm, the collaborative efficiency of equipment is improved and energy consumption is reduced.

[0026] This embodiment realizes distributed processing and global decision-making of multi-source data by constructing a heterogeneous federated learning framework between the edge layer, fog computing layer and cloud computing layer. Through the asynchronous federated aggregation algorithm, the training efficiency of the lightweight model is improved and the dynamic update of the global model parameters is realized, thereby improving the training efficiency and convergence speed of the model. Local features are extracted by the lightweight model, which reduces the amount of data transmission and improves the efficiency and real-time performance of feature extraction. Through cross-modal spatiotemporal encoders and causal reasoning, the alignment of multi-source data and reasonable decision-making are realized, thereby improving the rationality and explainability of the decision. Through the distributed Actor-Critic algorithm, dynamic task allocation and path planning between devices are realized, thereby improving the efficiency of device collaboration and reducing energy consumption.

[0027] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another multimodal decision-making management method for a palm oil plantation is provided, which includes the following steps: S1. Build a heterogeneous federated learning framework for palm oil plantations. This involves deploying an edge layer on various heterogeneous plantation terminal devices, a fog computing layer on local plantation servers, and a cloud computing layer on remote cloud servers. The heterogeneous terminal devices for the planting industry in step S1 include drones equipped with RGB and thermal imaging cameras, unmanned vehicles equipped with LiDAR scanners, and soil sensors for collecting soil environmental data; It should be noted that for the coordinated pest and disease control scenario, drones equipped with RGB and thermal imaging cameras cover crop phenotypes, and unmanned vehicles equipped with LiDAR scanners cover canopy structure. For fruit maturity harvesting scenarios, drones equipped with RGB and thermal imaging cameras identify fruit maturity, unmanned vehicles equipped with LiDAR scanners monitor tree trunk structure, and soil sensors collect soil environment data to achieve full-dimensional data collection. Compared with single-sensor solutions, this improves data coverage and provides a foundation for multimodal fusion. S2. Each edge layer constructs a lightweight model and uses data collected from its own heterogeneous agricultural terminal devices for local training. The local model parameters are uploaded to the fog computing layer for asynchronous federated aggregation. The fog computing layer then receives the global model parameters returned by the layer and retrains the model locally until the lightweight model meets the requirements. The specific steps of step S2 are as follows: S21. Each edge layer constructs a lightweight model and performs local training using time series data collected by its own heterogeneous agricultural terminal devices. The trained local model parameters are then gradient-encoded and uploaded to the fog computing layer. For example, the gradient encoding process uses 8-bit fixed-point quantization in the range of [-127, 127], and the encoding uses Huffman coding, thereby reducing the amount of communication data in the federated aggregation; The specific steps of constructing the lightweight model in step S21 are as follows: Select the YOLOv7 model and replace the standard convolutional layer in the base feature extractor of the YOLOv7 model with a depthwise separable convolutional layer; Determine the width of the YOLOv7 model based on the computing power of the heterogeneous terminal devices and the target planting stage, and optimize the detection head using the NEON instruction set; Dynamic pruning through adjustable YOLOv7 model width; The improved YOLOv7 model is deployed as the initial lightweight model on the ARM Cortex-A72 platform of the corresponding heterogeneous terminal equipment in the planting industry; In pest control scenarios, RGB images collected by drones are used to detect disease spots, LiDAR point clouds collected by unmanned vehicles are used to detect tree trunk health, and soil sensor data is used to monitor soil moisture and nutrients. In fruit ripening monitoring scenarios, RGB images collected by drones are used to detect fruit color, LiDAR point clouds collected by unmanned vehicles are used to detect fruit location, and soil sensor data is used to monitor soil moisture. It should be noted that by improving the selection of the YOLOv7 model, using depthwise separable convolution and optimizing the NEON instruction set to reduce the number of model parameters, the inference speed of heterogeneous terminal devices in the edge layer of the planting industry on the ARM Cortex-A72 platform was improved to meet the needs of real-time detection. The inference latency fluctuation of the model on different devices was reduced through dynamic adaptation of computing power. In step S21, the edge layer i constructs a lightweight model and calculates the local loss function based on the local collected data ; During the training process, the edge layer i minimizes the local loss function through the optimization algorithm , get the local model parameters θ ; S22. The fog computing layer receives local model parameter encoding data and measures communication delay , and then according to the communication delay Calculate the proximal term coefficient ;

[0028] For example, if the communication delay between the UAV and the fog computing layer is 15ms, the calculated proximal term coefficient is 0.13; S23. The fog computing layer regularly uses the optimized asynchronous federated aggregation algorithm to asynchronously weight the decoded local model parameters to generate global model parameters, and then performs gradient coding on the global model parameters and the proximal term coefficients. Send it to each edge layer together; The specific steps of performing asynchronous weighted aggregation in step S23 are as follows: The fog computing layer creates a delay counter for each edge layer i , record the consecutive rounds that did not participate in aggregation; The fog computing layer updates the delay counter in the following way before each aggregation ; If edge layer i participates in the current round of aggregation, then =0; otherwise = +1; When the fog computing layer performs aggregation, only ≤N edge layer i is aggregated and communication delay is used and the proximal term coefficient Calculate the weights of each local model parameter ;

[0029] in, ; During the aggregation process, each local model parameter Perform weight normalization to calculate global model parameters ;

[0030] Among them, S is satisfied The set of edge layers ≤ N; In addition, for >N edge layers, temporarily suspend their parameter aggregation, and resume training from the latest global model after they are reconnected; For example, it is possible to choose to delay the edge layer for 5 rounds to participate in the aggregation; S24. The edge layer receives the global model parameter encoding and the proximal term coefficients , using the time series data collected by the heterogeneous terminal devices of each planting industry for local training, and using the proximal term coefficient in the local training process Update the local model parameters with the decoded global model parameters; In step S24, the edge layer i calculates the proximal term coefficients returned by the fog computing layer and the decoded global model parameters Calculate the loss function regularization term ; The local loss function and loss function regularization term Superposition to obtain the corrected loss function ; During the training process, the edge layer i minimizes the local loss function through the optimization algorithm , get the updated local model parameters ; For example, in the coordinated pest and disease control scenario, the updated model parameters of the drone edge layer are used to more accurately detect disease spots, while in the fruit maturity monitoring and harvest scheduling scenario, the updated model parameters of the drone edge layer are used to more accurately detect fruit color. It should be noted that the stability of asynchronous aggregation is improved through the loss function regularization term and delay-sensitive weight distribution. Specifically, the regularization term suppresses device model overfitting and reduces the variance of local training loss. By prioritizing low-latency devices during weighted aggregation, aggregation time is shortened and network resource utilization is improved. S25. Determine whether the global maximum number of iterations is met or whether the difference between each local model parameter and the global model parameter converges; If yes, go to step S3; If not, upload the local model parameters to the fog computing layer, continue iteration, and return to step S22; For example, the parameter difference between the current local model and the global model is calculated:

[0031] in, are local model parameters, is the global model parameter; If the average difference of all devices is lower than the preset threshold, for example, the preset value is 1×10 −4 Or when the maximum number of iterations is reached, such as 50 rounds, the training is terminated; Otherwise, return to step S21 to continue the next round of federated training; It should be noted that by introducing the proximal term coefficient and delay counter for communication delay awareness, the asynchronous aggregation strategy is optimized. Specifically, the proximal term coefficient is used to reduce the risk of the device model deviating from the global optimal solution, improving the model convergence accuracy. The delay counter is used to filter out long-term offline devices, improving the aggregation effectiveness and adapting to the high mobility and low connection stability characteristics of agricultural equipment. S3. The edge layer uses the trained lightweight model to extract local features from the time series data collected by heterogeneous terminal devices in the corresponding planting industry and provides them to the cloud computing layer. In step S3, the lightweight model of the edge layer to which the UAV belongs extracts RGB image features from the collected RGB image; The lightweight model of the edge layer of the unmanned vehicle extracts LiDAR point cloud features from the collected LiDAR point cloud; The lightweight model of the edge layer to which the soil sensor belongs extracts soil time series features from soil environmental data; For example, in a coordinated pest and disease control scenario, drones extract disease spot features from RGB images, unmanned vehicles extract tree trunk health features from LiDAR point clouds, and soil sensors extract moisture and nutrient features from soil data. The disease spot features extracted by drones are used to assess disease severity, the trunk features extracted by unmanned vehicles are used to assess tree health, and the soil sensor data is used to assess the soil environment. In the fruit maturity monitoring and harvesting scenario, drones extract fruit color features from RGB images, unmanned vehicles extract fruit location features from LiDAR point clouds, and soil sensors extract moisture features from soil data. The fruit color features extracted by drones are used to assess fruit maturity, the fruit location features extracted by unmanned vehicles are used to plan harvesting routes, and soil sensor data is used to assess soil conditions. S4. The cloud computing layer uses a cross-modal spatiotemporal encoder to align local features at each edge layer to generate global features. This global feature is then combined with a Bayesian network to construct a causal graph. Interventions are determined based on the planting phase requirements and the causal graph. Counterfactual reasoning is used to quantify the impact of the interventions and generate a decision plan. The specific steps of step S4 are as follows: S41. The cloud computing layer constructs three branches of the Transformer architecture, including the RGB branch, the LiDAR branch, and the sensor timing branch. S42. The cloud computing layer uses the ViT-Base model in the RGB branch to divide the RGB image features into image blocks of a set number of dimensions and converts them into image vectors through linear projection. Among them, the patch size is set to 16×16 in the ViT-Base model; S43. The cloud computing layer uses the Point Transformer model in the LiDAR branch to convert the LiDAR point cloud features into voxel vectors of a set unit size; For example, the unit size is set to a voxel size of 0.5 m³; S44. The cloud computing layer uses the TCN temporal convolution layer in the sensor timing branch to convert the soil time series features into a time series vector according to the set convolution kernel size and step size; For example, the convolution kernel size is selected as 5 and the stride is 2; S45. The cloud computing layer calculates the query parameters, key parameters, and value parameters for each vector based on the preset learnable weights, and calculates the cross-attention scores between the branches, and then fuses the cross-attention scores using a gated fusion network; Specifically, for each of the three vectors, the focus points can be selected, and the query parameters, key parameters, and value parameters can be calculated separately for each focus point as a head. For example, 8 focus points can be selected from the three vectors. For example, the query parameter is calculated for the RGB image branch , key parameters and value parameters as follows:

[0032]

[0033]

[0034] in, is the image vector; Query parameters for LiDAR point cloud branch calculation , key parameters and value parameters as follows:

[0035]

[0036]

[0037] in, is the voxel vector; Calculate query parameters for sensor timing branches , key parameters and value parameters as follows:

[0038]

[0039]

[0040] in, is the time series vector, W Q , W K , W V is a learnable weight matrix; The cross attention between the RGB image branch and the LiDAR point cloud branch is as follows:

[0041] The cross attention between the RGB image branch and the sensor timing branch is as follows:

[0042] The cross attention between the LiDAR point cloud branch and the sensor timing branch is as follows:

[0043] in, is the dimension of the key vector K, which takes a value of 64; Use the gated fusion network to fuse the cross-attention scores: First calculate the gate weight

[0044] in, W g is a learnable weight matrix, σ is the sigmoid activation function; Apply gating weights for feature fusion:

[0045] Among them, ⊙ represents element-by-element multiplication; S46. The cloud computing layer encodes the GPS coordinates and timestamps into a position vector, adds the position vector to the fused cross-attention score, and obtains a fused feature. A fully connected layer is then used to convert the fused feature into a global feature map. Encode GPS coordinates and timestamp into a position vector P ; Adding the position vector to the fused cross attention score, the fused feature is obtained as follows:

[0046] Use a fully connected layer to convert the fused features into a global feature map:

[0047] in, W fc is the weight matrix of the fully connected layer, F global is the global feature map; S47. Preselect key variables from the global feature map, decode the variable values of the key variables, and convert them into agronomic variable vectors of set dimensions; For example, based on ≥5 years of historical planting data and domain knowledge, 32 key variables were selected, including soil electrical conductivity, chlorophyll content (SPAD value), soil pH, etc. These variables were determined through data analysis and advice from agronomists, specifically those that have a significant impact on the growth and pest and disease management of palm oil plantations. Decode specific variables: Soil pH:

[0048] It should be noted that since soil pH has a direct impact on plant growth and pests and diseases, it was used as a key variable and mapped to the range of 5.0–7.0; Chlorophyll content (SPAD value):

[0049] It should be noted that, since chlorophyll content is an important indicator for assessing plant health status, it was used as a key variable and mapped to the range of 30–80; Among them, W ph and W chl It is a learnable parameter, pre-trained by historical data; S48. Construct an initial causal graph. Based on prior knowledge in the agronomy field, determine the domain constraint edges related to key variables and add them to the initial causal graph. First, use the G-test conditional independence test to determine whether there is a correlation between two key variables at different stages of palm oil cultivation based on statistical independence. Obtain a set of candidate edges. Then, use the agronomy constraints to determine the final causal relationship, completing the causal graph construction. It should be noted that the domain constraint edge is added to the initial causal graph, for example, adding the constraint edge soil pH → root activity. The constraint edge is based on prior knowledge in the field of agronomy. For example, soil pH has a direct impact on root activity, and acidic soil may inhibit root growth. G-test is a statistical test method used to evaluate the independence between two variables. In causal diagram construction, G-test is used to determine whether there is a causal relationship between variables. G-test determines the relationship between variables through statistical independence, and its null hypothesis is (X and Y are independent given Z). If the p-value is less than α (e.g., 0.01), the null hypothesis is rejected and an association is considered to exist. However, there may be the following problems in actual situations: False correlation: non-causal statistical dependence may be detected (e.g. "fertilizer application" and "pests and diseases" are both affected by "rainfall"); Lack of directionality: G-test cannot determine the direction of cause and effect (is it fertilization → disease, or disease → need for fertilization?); Small sample bias: Planting industry data usually have limited samples (e.g., only 5 years of data), which reduces the reliability of independence tests; By forcibly inserting causal edges recognized in the field, such as "soil pH directly affects root activity" and "temperature determines insect pest reproduction rate", even if the G-test does not detect these relationships, they must be retained; Use agronomic constraints to prohibit edges that do not conform to biological logic, such as "leaf color does not affect rainfall" and "drone speed is unrelated to soil moisture." Even if the G-test gives a significant p-value, they are forcibly deleted. Fuzzy relationships are oriented according to agronomic knowledge through agronomic constraints. For example, G-test may give a bidirectional association of "fertilizer amount ← → disease", but because fertilization is a human controllable intervention, the agronomic constraint is forcibly changed to "fertilizer amount → disease"; Use G-test (α=0.01) to learn candidate edge sets from the data

[0050] Check the agronomic literature for the remaining edges: If the literature supports X→Y, keep it If Y→X is supported, reverse; If there is no clear evidence, maintain the original direction of the G-test; Specifically, in the scenario of coordinated pest and disease control, the relationship between "nitrogen application rate" and "leaf spot disease" is analyzed as follows: G-test results: p-value = 0.003 (significant) But it cannot distinguish: Excessive nitrogen application → leaf spot (true cause and effect) Severe leaf spot → Increased nitrogen fertilizer application required (false association) And through agronomic constraints intervention: Since the literature indicates that excessive nitrogen can reduce leaf disease resistance (i.e., nitrogen application rate → leaf spot → nitrogen application rate → leaf spot), the reverse edge "leaf spot → nitrogen application rate → leaf spot → nitrogen application rate" is deleted forcibly. S49. Based on the cause-and-effect diagram and target requirements, select key variables with impacts above the threshold at the corresponding palm oil planting stage as intervention variables. Determine the target values and intervention measures for the intervention variables based on agronomic knowledge and historical data. Quantify the impact of the intervention measures on the target variables using the do-calculus method and estimate them using Monte Carlo sampling. When the quantification and estimation results meet the requirements, generate a decision plan and determine the type and area of intervention operations. Based on the causal diagram, key variables that have a significant impact on palm oil plantation pest and disease control, environmental factors, and related measures are selected as intervention variables. These variables are usually key nodes in the causal diagram, such as fertilizer application, irrigation volume, and pest and disease incidence in the coordinated pest and disease control scenario. The intervention variables were selected as follows: Intervention variable X: nitrogen application rate Intervention variable Y: disease incidence Based on recommendations from the agronomy community and historical data, target values for intervention variables are set; for example, a target could be to increase nitrogen application by 20%.

[0051] Example: X0 = 1.2 × current nitrogen application rate Quantify the effect of an intervention on a target variable using do-calculus, a mathematical tool for causal inference that simulates the effects of an intervention by performing do operations.

[0052] in, X is the intervention variable, Y is the target variable, Z are other relevant variables; Use Monte Carlo sampling (1000 times) to estimate the potential outcomes Y;

[0053] in, are randomly sampled values of other variables (such as temperature, humidity), f (⋅) is the structural equation model It should be noted that the three-branch Transformer-based architecture achieves multimodal feature alignment and combines G-test with agronomic constraints to construct a causal graph. The cross-attention mechanism reduces the spatial alignment error between RGB images and LiDAR point clouds, as well as the temporal alignment error between time series data and spatial features. The causal graph, through the dual constraints of statistical tests and domain knowledge, improves the accuracy of causal relationship identification by 10% compared to simple data-driven methods. S5. The cloud computing layer executes the decision plan, solves the hierarchical Nash equilibrium using a distributed actor-critic algorithm, and performs task allocation and path planning for heterogeneous terminal devices in the agricultural industry. The specific steps of step S5 are as follows: S51. The cloud computing layer predefines state vectors for the drone and the unmanned vehicle to characterize their states. The drone's state vector includes its current location, remaining battery power, current mission type, and mission queue length. The unmanned vehicle's state vector includes terrain slope, soil hardness, robotic arm status, and vehicle storage status. It should be noted that in the coordinated pest control scenario, the vehicle storage status is the remaining amount of medicine in the medicine box, while in the fruit ripening monitoring and harvest scheduling scenario, the vehicle storage status is the remaining capacity of the harvest box; S52. Establish an objective function using the drone as the upper layer, with the coverage of crops in the intervention state as the reward term and the number of intersections between the drone and the unmanned vehicle's operation path as the penalty term. Establish a Q-value function using the unmanned vehicle as the lower layer, with the task progress term as the positive incentive and the energy efficiency term and path deviation term as the negative incentives. Taking the collaborative pest control scenario as an example, the upper-level objective function is as follows:

[0054] If the current lesion coverage = 85% and the number of conflicts = 2, then R = 0.535 is calculated; The lower Q value function is as follows:

[0055] If the application completion rate = 90%, energy consumption = 0.8 kWh, and path deviation = 1.2, then Q = 0.282; Taking the fruit harvesting scenario as an example, the upper-level objective function is as follows:

[0056] If the recognition rate = 95% and the missed detection rate = 8%, then R = 0.81 is calculated; The lower Q value function is as follows:

[0057] If harvested amount = 42 kg, damage = 2, and steering fluctuation = 15°, then Q = 0.488; S53. Build a UAV strategy network with the goal of maximizing the objective function value. Based on the UAV's state vector, the access probability distribution of each operating area is solved to guide the UAV's task allocation. Build a measurement network for the unmanned vehicle, aiming to maximize the Q-value function. Calculate the speed or steering angle in the continuous action space based on the vehicle's state vector to guide the vehicle's path planning. For example, take the scenario of collaborative pest control as an example: The drone policy network output is as follows: State input: [longitude = 120.5, latitude = 3.2, battery = 65%, current task = monitoring, queue length = 3] Action probability: [Area A=0.7,Area B=0.25,Area C=0.05] → Select Area A The unmanned vehicle action output is as follows: State input: [Slope = 8°, Soil Hardness = 180kPa, Robotic Arm = Ready, Drug Level = 45%] Q network decision: speed = 1.1m / s, spraying interval = 1.8m Take the fruit harvesting scene as an example: The drone policy network output is as follows: Status input: [Longitude = 121.1, Latitude = 3.3, Battery = 80%, Current Task = Scan, Queue Length = 2] Action probability: [Northeast region = 0.9, Southwest region = 0.1] → Mark the Northeast region Unmanned vehicle action output: State input: [Slope = 5°, Handle strength = 12N, Robotic arm = Working, Carrier capacity = 60%] Q network decision: Clamping force = 14N, Harvesting speed = 1.2 ears / minute S54. Construct a Q network to minimize the value prediction error and calculate the long-term value prediction of the current state action based on the corresponding state vector and the action selected by the policy network; Specifically, the Q network objective function is as follows:

[0058] in, Indicates the current Q value, represents the target Q value, Indicates actual reward; For example, if the current Q value = 0.6, the actual reward = 0.8, the next state Q value = 0.7 Then we calculate δ=0.865; If the mean square error loss =0.748, then back propagate to update the weight; S55. Calculate task priorities based on the crop's intervention level and planting stage, predict path conflicts between the UAV and the unmanned vehicle, and adjust the corresponding paths or timing when a conflict is predicted. For priority calculation, take the coordinated pest control scenario as an example:

[0059] If the lesion density = 30%, the spread speed = 0.2m / day, then the priority p = 0.26; Take the fruit harvesting scene as an example:

[0060] If maturity = 0.9 and rainfall occurs 6 hours later, the priority p = 0.855 is calculated; For spatial conflict detection:

[0061] If the row spacing = 6m, then the safety distance = 3m; When the distance between the UAV (e.g. x=50, y=60) and the UGV (e.g. x=52, y=58) is 2.8m<3m, the UAV will ascend 2m to avoid the UGV, and the UGV will decelerate to 0.5m / s. S56. Calculate the task completion degree, evaluate the task execution effect based on the completion degree, and then update the strategy network parameters based on the difference between the actual production increase rate and the predicted production increase rate: If the actual production increase rate is higher than the predicted production increase rate, the probability of selecting the original strategy will be increased; If the actual production increase rate is lower than the predicted production increase rate, the probability of selecting the original strategy will be reduced; Exemplarily, the pest control effect is verified as follows: If the disease is predicted to decrease by 20% and actually decreases by 25%, then the probability of this strategy is increased by 5%; If the forecast decreases by 20% and the actual decrease is 15%, the probability of the strategy is reduced by 3%; It should be noted that by solving the hierarchical Nash equilibrium through the Actor-Critic algorithm, drones are responsible for wide-area inspections, improving coverage, unmanned vehicles are responsible for operations, and positioning accuracy is improved. This improves the rationality of task division, and dynamic path planning shortens the average operating distance of equipment, reduces operation time, and increases the daily task processing volume. The online optimization mechanism of reinforcement learning enables the system to adapt to seasonal changes and enhance decision-making generalization capabilities. It should be noted that the collaborative control of drones and unmanned vehicles is achieved through the Actor-Critic algorithm based on hierarchical Nash equilibrium; the response time in high-risk areas is shortened through dynamic calculation of task priorities; and the repeated operations caused by equipment collisions are reduced through path conflict prediction, thereby improving energy efficiency and operation coverage.

[0062] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0063] This application is applicable to high-density palm oil plantations with row spacing ≤ 8 meters and plant density ≥ 120 plants / hectare, and can improve the intelligent management level of the plantation.

[0064] like Figure 2 As shown, the following is an embodiment of the multimodal decision management system for palm oil plantations provided by the embodiments of the present disclosure. This system and the multimodal decision management method for palm oil plantations of the above-mentioned embodiments belong to the same inventive concept. For details not fully described in the embodiments of the multimodal decision management system for palm oil plantations, please refer to the embodiments of the multimodal decision management method for palm oil plantations mentioned above.

[0065] The system includes: A federated learning framework building module is used to build a heterogeneous federated learning framework for palm oil plantations, deploying the edge layer on various heterogeneous terminal devices in the plantation, deploying the fog computing layer on the plantation's local servers, and deploying the cloud computing layer on remote cloud servers; The asynchronous federated learning module is used to build lightweight models at each edge layer, collect data from heterogeneous terminal devices in each plant, perform local training, upload local model parameters to the fog computing layer for asynchronous federated aggregation, and then receive the global model parameters returned by the fog computing layer for re-local training until the lightweight model meets the requirements; The feature extraction module is used to extract local features from the time series data collected by heterogeneous terminal devices in the corresponding planting industry using a trained lightweight model at the edge layer and provide it to the cloud computing layer; The decision generation module is used to align local features of each edge layer using a cross-modal spatiotemporal encoder at the cloud computing layer to generate global features. This is then combined with a Bayesian network to construct a causal graph. Intervention measures are determined based on the needs of the planting stage and the causal graph. Counterfactual reasoning is used to quantify the impact of the intervention measures and generate a decision plan. The decision execution module is used to execute decision plans in the cloud computing layer, solve the hierarchical Nash equilibrium through the distributed Actor-Critic algorithm, and perform task allocation and path planning for heterogeneous terminal devices in the planting industry.

[0066] This embodiment achieves the decoupling of federated learning, feature extraction, decision generation, and execution control through the interactive collaboration of the federated learning framework building module, asynchronous federated learning module, feature extraction module, decision generation module, and decision execution module, breaking through the limitations of inefficient data utilization, insufficient decision rationality, and poor equipment coordination in traditional planting management.

[0067] The multimodal decision management method for palm oil plantations provided in the embodiments of the present application can be applied to electronic devices. Those skilled in the art will appreciate that the electronic device structure involved in the embodiments of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange components differently. In the embodiments of the present invention, electronic devices include but are not limited to laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0068] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, a button, a camera, a display, and a SIM card interface, etc.

[0069] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0070] A processor may include one or more processing units, such as a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0071] The processor can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on the instruction opcode and timing signal to complete the control of instruction fetching and execution.

[0072] The processor may also include a memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or is reusing. If the processor needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0073] The above-mentioned electronic device realizes the multimodal decision-making management method of palm oil plantations in the present application, builds a heterogeneous federated learning framework for palm oil plantations, deploys edge layers on various heterogeneous terminal devices in the plantation, deploys fog computing layers on local servers in the plantation, and deploys cloud computing layers on remote cloud servers; each edge layer constructs a lightweight model respectively, and uses its own heterogeneous terminal devices in the plantation to collect data for local training, uploads local model parameters to the fog computing layer for asynchronous federated aggregation, and then receives the global model parameters returned by the fog computing layer for re-local training until the lightweight model meets the requirements; the edge layer uses the trained lightweight model to collect data from the corresponding heterogeneous terminal devices in the plantation Local features are extracted from time series data and provided to the cloud computing layer; the cloud computing layer uses a cross-modal spatiotemporal encoder to align the local features of each edge layer to generate global features, and then combines the Bayesian network to construct a causal graph. Intervention measures are determined based on the needs of the planting stage and the causal graph, and counterfactual reasoning is used to quantify the impact of the intervention measures to generate a decision plan; the cloud computing layer executes the decision plan, solves the hierarchical Nash equilibrium through the distributed Actor-Critic algorithm, and performs task allocation and path planning for heterogeneous terminal devices in the planting industry, achieving the beneficial effect of breaking through the limitations of inefficient data utilization, insufficient decision rationality, and poor equipment coordination in traditional planting management.

[0074] The storage medium provided in this application stores a program product that can implement a multimodal decision-making management method for palm oil plantations.

[0075] A multimodal decision-making management method for palm oil plantations includes: building a heterogeneous federated learning framework for palm oil plantations, deploying an edge layer on various heterogeneous plantation terminal devices, deploying a fog computing layer on local plantation servers, and deploying a cloud computing layer on remote cloud servers; each edge layer constructs a lightweight model and uses data collected by its own heterogeneous plantation terminal devices for local training, uploading the local model parameters to the fog computing layer for asynchronous federated aggregation, and then receiving the global model parameters returned by the fog computing layer for re-local training until the lightweight model meets the requirements; the edge layer uses the trained lightweight model to extract local features from the time series data collected by the corresponding heterogeneous plantation terminal devices and provides them to the cloud computing layer; the cloud computing layer uses a cross-modal spatiotemporal encoder to align the local features of each edge layer to generate global features, and then combines them with a Bayesian network to construct a causal graph. Intervention measures are determined based on the planting stage requirements and the causal graph, and the impact of the intervention measures is quantified using counterfactual reasoning to generate a decision plan; the cloud computing layer executes the decision plan, solving a hierarchical Nash equilibrium using a distributed actor-critic algorithm, and performing task allocation and path planning for the heterogeneous plantation terminal devices.

[0076] In some possible embodiments, the multimodal decision-making management method for palm oil plantations disclosed herein can be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0077] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0078] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal decision-making management method for a palm oil plantation, characterized in that: The steps include: S1. Build a heterogeneous federated learning framework for palm oil plantations. This involves deploying an edge layer on various heterogeneous plantation terminal devices, a fog computing layer on local plantation servers, and a cloud computing layer on remote cloud servers. S2. Each edge layer builds a lightweight model and uses data collected from its own heterogeneous agricultural terminal devices for local training. The local model parameters are uploaded to the fog computing layer for asynchronous federated aggregation. The fog computing layer then returns the global model parameters for re-local training until the lightweight model meets the requirements. S3. The edge layer uses the trained lightweight model to extract local features from the time series data collected by heterogeneous terminal devices in the corresponding planting industry and provides them to the cloud computing layer. S4. The cloud computing layer uses a cross-modal spatiotemporal encoder to align local features at each edge layer to generate global features. This global feature is then combined with a Bayesian network to construct a causal graph. Interventions are determined based on the planting phase requirements and the causal graph. Counterfactual reasoning is used to quantify the impact of the interventions and generate a decision plan. S5. The cloud computing layer executes the decision-making plan, solves the hierarchical Nash equilibrium through the distributed actor-critic algorithm, and performs task allocation and path planning for heterogeneous terminal devices in the agricultural industry.

2. The multimodal decision-making management method for palm oil plantations according to claim 1, characterized in that: The heterogeneous terminal devices for the planting industry in step S1 include drones equipped with RGB and thermal imaging cameras, unmanned vehicles equipped with LiDAR scanners, and soil sensors for collecting soil environment data.

3. The multimodal decision-making management method for palm oil plantations according to claim 2, characterized in that: The specific steps of step S2 are as follows: S21. Each edge layer constructs a lightweight model and performs local training using time series data collected by its own heterogeneous agricultural terminal devices. The trained local model parameters are then gradient-encoded and uploaded to the fog computing layer. S22. The fog computing layer receives local model parameter encoding data and measures communication delay , and then according to the communication delay Calculate the proximal term coefficient ; S23. The fog computing layer regularly uses the optimized asynchronous federated aggregation algorithm to asynchronously weight the decoded local model parameters to generate global model parameters, and then performs gradient coding on the global model parameters and the proximal term coefficients. Send it to each edge layer together; S24. The edge layer receives the global model parameter encoding and the proximal term coefficients , using the time series data collected by the heterogeneous terminal devices of each planting industry for local training, and using the proximal term coefficient in the local training process Update the local model parameters with the decoded global model parameters; S25. Determine whether the global maximum number of iterations is met or whether the difference between each local model parameter and the global model parameter converges; If yes, go to step S3; If not, upload the local model parameters to the fog computing layer, continue iterating, and return to step S22.

4. The multimodal decision-making management method for palm oil plantations according to claim 3, characterized in that: The specific steps of constructing the lightweight model in step S21 are as follows: Select the YOLOv7 model and replace the standard convolutional layer in the base feature extractor of the YOLOv7 model with a depthwise separable convolutional layer; Determine the width of the YOLOv7 model based on the computing power of the heterogeneous terminal devices and the target planting stage, and optimize the detection head using the NEON instruction set; The improved YOLOv7 model is deployed as the initial lightweight model on the ARM Cortex-A72 platform of the corresponding heterogeneous terminal equipment in the planting industry.

5. The multimodal decision-making management method for palm oil plantations according to claim 3, characterized in that: In step S21, the edge layer i constructs a lightweight model and calculates the local loss function based on the local collected data ; During the training process, the edge layer i minimizes the local loss function through the optimization algorithm , get the local model parameters θ ; In step S24, the edge layer i calculates the proximal term coefficients returned by the fog computing layer and the decoded global model parameters Calculate the loss function regularization term ; The local loss function and loss function regularization term Superposition to obtain the corrected loss function ; During the training process, the edge layer i minimizes the local loss function through the optimization algorithm , get the updated local model parameters ; The specific steps of performing asynchronous weighted aggregation in step S23 are as follows: The fog computing layer creates a delay counter for each edge layer i , record the consecutive rounds that did not participate in aggregation; The fog computing layer updates the delay counter in the following way before each aggregation ; If edge layer i participates in the current round of aggregation, then =0; otherwise = +1; When the fog computing layer performs aggregation, only ≤N edge layer i is aggregated and communication delay is used and the proximal term coefficient Calculate the weights of each local model parameter ; in, ; During the aggregation process, each local model parameter Perform weight normalization to calculate global model parameters ; Among them, S is satisfied The set of edge layers ≤ N.

6. The multimodal decision-making management method for palm oil plantations according to claim 3, characterized in that: In step S3, the lightweight model of the edge layer to which the UAV belongs extracts RGB image features from the collected RGB image; The lightweight model of the edge layer of the unmanned vehicle extracts LiDAR point cloud features from the collected LiDAR point cloud; The lightweight model of the edge layer to which the soil sensor belongs extracts soil time series features from soil environmental data; The specific steps of step S4 are as follows: S41. The cloud computing layer constructs three branches of the Transformer architecture, including the RGB branch, the LiDAR branch, and the sensor timing branch. S42. The cloud computing layer uses the ViT-Base model in the RGB branch to divide the RGB image features into image blocks of a set number of dimensions and converts them into image vectors through linear projection. S43. The cloud computing layer uses the Point Transformer model in the LiDAR branch to convert the LiDAR point cloud features into voxel vectors of a set unit size; S44. The cloud computing layer uses the TCN temporal convolution layer in the sensor timing branch to convert the soil time series features into a time series vector according to the set convolution kernel size and step size; S45. The cloud computing layer calculates the query parameters, key parameters, and value parameters for each vector based on the preset learnable weights, and calculates the cross-attention scores between the branches, and then fuses the cross-attention scores using a gated fusion network; S46. The cloud computing layer encodes the GPS coordinates and timestamps into a position vector, adds the position vector to the fused cross-attention score, and obtains a fused feature. A fully connected layer is then used to convert the fused feature into a global feature map. S47. Preselect key variables from the global feature map, decode the variable values of the key variables, and convert them into agronomic variable vectors of set dimensions; S48. Construct an initial causal graph. Based on prior knowledge in the agronomy field, determine the domain constraint edges related to key variables and add them to the initial causal graph. First, use the G-test conditional independence test to determine whether there is a correlation between two key variables at different stages of palm oil cultivation based on statistical independence. Obtain a set of candidate edges. Then, use the agronomy constraints to determine the final causal relationship, completing the causal graph construction. S49. Based on the cause-and-effect diagram and in accordance with target requirements, select key variables with an impact higher than the threshold in the corresponding palm oil planting stage as intervention variables. Determine the target value and intervention measures of the intervention variables based on agronomic knowledge and historical data. Use the do-calculus method to quantify the impact of the intervention measures on the target variables and use Monte Carlo sampling to estimate them. When the quantification and estimation results meet the requirements, generate a decision plan and determine the operation type and operation area of the intervention measures.

7. The multimodal decision-making management method for palm oil plantations according to claim 6, characterized in that: The specific steps of step S5 are as follows: S51. The cloud computing layer predefines state vectors for the drone and the unmanned vehicle to characterize their states. The drone's state vector includes its current location, remaining battery power, current mission type, and mission queue length. The unmanned vehicle's state vector includes terrain slope, soil hardness, robotic arm status, and vehicle storage status. S52. Establish an objective function using the drone as the upper layer, with the coverage of crops in the intervention state as the reward term and the number of intersections between the drone and the unmanned vehicle's operation path as the penalty term. Establish a Q-value function using the unmanned vehicle as the lower layer, with the task progress term as the positive incentive and the energy efficiency term and path deviation term as the negative incentives. S53. Build a UAV strategy network with the goal of maximizing the objective function value. Based on the UAV's state vector, the access probability distribution of each operating area is solved to guide the UAV's task allocation. Build a measurement network for the unmanned vehicle, aiming to maximize the Q-value function. Calculate the speed or steering angle in the continuous action space based on the vehicle's state vector to guide the vehicle's path planning. S54. Construct a Q network to minimize the value prediction error and calculate the long-term value prediction of the current state action based on the corresponding state vector and the action selected by the policy network; S55. Calculate task priorities based on the crop's intervention level and planting stage, predict path conflicts between the UAV and the unmanned vehicle, and adjust the corresponding paths or timing when a conflict is predicted. S56. Calculate the task completion degree, evaluate the task execution effect based on the completion degree, and then update the strategy network parameters based on the difference between the actual production increase rate and the predicted production increase rate: If the actual production increase rate is higher than the predicted production increase rate, the probability of selecting the original strategy will be increased; If the actual production increase rate is lower than the predicted production increase rate, the probability of selecting the original strategy will be reduced.

8. A palm oil plantation multimodal decision management system, characterized in that: include: A federated learning framework building module is used to build a heterogeneous federated learning framework for palm oil plantations, deploying the edge layer on various heterogeneous terminal devices in the plantation, deploying the fog computing layer on the plantation's local servers, and deploying the cloud computing layer on remote cloud servers; The asynchronous federated learning module is used to build lightweight models at each edge layer, collect data from heterogeneous terminal devices in each plant, perform local training, upload local model parameters to the fog computing layer for asynchronous federated aggregation, and then receive the global model parameters returned by the fog computing layer for re-local training until the lightweight model meets the requirements; The feature extraction module is used to extract local features from the time series data collected by heterogeneous terminal devices in the corresponding planting industry using a trained lightweight model at the edge layer and provide it to the cloud computing layer; The decision generation module is used to align local features of each edge layer using a cross-modal spatiotemporal encoder at the cloud computing layer to generate global features. This is then combined with a Bayesian network to construct a causal graph. Intervention measures are determined based on the needs of the planting stage and the causal graph. Counterfactual reasoning is used to quantify the impact of the intervention measures and generate a decision plan. The decision execution module is used to execute decision plans in the cloud computing layer, solve the hierarchical Nash equilibrium through the distributed Actor-Critic algorithm, and perform task allocation and path planning for heterogeneous terminal devices in the planting industry.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method implements the steps of the multimodal decision-making management method for a palm oil plantation as claimed in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal decision-making management method for palm oil plantations as claimed in any one of claims 1 to 7 are implemented.

Images