An automatic driving and agricultural AI coordination method and system based on a heterogeneous model

Abstract

This invention provides a method and system for collaborative autonomous driving and agricultural AI based on heterogeneous models. The method includes processing collected autonomous driving perception data and agricultural data separately to obtain single-modal feature data with a unified spatiotemporal reference; constructing a cross-domain parameter adapter to establish a parameter mapping channel between the autonomous driving model and the agricultural AI model, enabling mutual transfer of low-level feature extraction capabilities; performing hierarchical decision fusion on the single-modal feature data based on the model parameters after bidirectional knowledge transfer; constructing a three-dimensional voxel feature map based on the obtained fused features to achieve joint decision output from the perception layer and the decision layer; dynamically allocating computing resources in response to resource requirement parameters established during the hierarchical decision fusion process, and implementing encryption protection when data is shared between cross-domain nodes. This invention achieves resource sharing and knowledge transfer between the autonomous driving model and the agricultural AI model, significantly improving task execution efficiency and resource utilization in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for collaborative autonomous driving and agricultural AI based on heterogeneous models. Background Technology

[0002] With the rapid development of artificial intelligence technology, autonomous driving and agricultural AI, as two important vertical application areas, have each accumulated a large amount of sensor data and specialized models, and achieved significant results in specific scenarios. However, there are significant differences between the two in terms of model architecture, data modality, and task objectives, making it difficult for traditional knowledge transfer methods to directly achieve efficient cross-domain collaboration.

[0003] In the field of autonomous driving, existing technologies such as patent CN119678160A primarily focus on optimizing the vehicle's own perception, decision-making, and control, lacking interaction mechanisms with models from other domains. Similarly, in the field of agricultural AI, technologies like patent CN119450391A are limited to IoT monitoring and data analysis within the agricultural environment, without considering collaboration with external intelligent systems. None of these technological solutions have broken through domain barriers, making it difficult to achieve cross-domain knowledge sharing and resource integration.

[0004] In fact, autonomous driving and agricultural AI share a high degree of similarity in their core challenges: both must cope with complex and ever-changing physical environments, rely on multi-sensor fusion (such as vision, radar, and time-series data), and their decision-making processes exhibit significant spatiotemporal continuity. More importantly, they possess a natural complementarity. For example, the unstructured roads, complex terrain, and variable obstacles prevalent in agricultural scenarios are precisely the main challenges currently faced by autonomous driving systems; while the real-time obstacle avoidance capabilities and high-precision path planning capabilities of autonomous driving in high-speed dynamic environments can significantly improve the operational efficiency of agricultural robots (such as unmanned agricultural machinery). This potential for mutual complementarity is difficult to replace with other cross-domain combinations.

[0005] Traditional knowledge transfer methods (such as fine-tuning or simple feature alignment) typically require the source and target domains to have similar network architectures or homogeneous data distributions. When applied to heterogeneous models such as autonomous driving (focusing on real-time point cloud and image perception and control) and agricultural AI (focusing on time-series meteorology and multispectral image analysis and prediction), traditional methods suffer from the following drawbacks: (1) Parameters cannot be directly transferred across heterogeneous network architectures. Traditional fine-tuning mechanisms rely heavily on the isomorphism of the source and target models in terms of network layers and dimensions. However, there is a huge gap in the underlying architecture between autonomous driving models (which often use PointNet to process 3D point clouds or CNN to process high frame rate vision) and agricultural AI models (which often use LSTM to process time series data or ViT to process multispectral data), causing traditional hard parameter sharing or direct copying methods to completely fail in terms of mathematical dimensions.

[0006] (2) It ignores the huge differences in spatiotemporal references of multimodal data, which can easily lead to "negative transfer". Autonomous driving perception data is characterized by high frequency, real-time and based on the vehicle's local coordinate system; while agricultural data (such as meteorology and soil) is often low frequency, long period and based on the global geographic coordinate system. If traditional feature alignment methods do not perform strict spatiotemporal standardization and forcibly map misaligned data to the same feature space, they will not only fail to extract effective commonalities, but will also interfere with the original decision-making logic of the model, resulting in performance degradation.

[0007] (3) Traditional one-way transfer models cannot achieve mutual promotion between heterogeneous domains. Existing knowledge distillation or transfer learning mostly adopt a one-way "teacher-student" guidance model. However, in the combination of autonomous driving and agricultural AI, the two complement each other: autonomous driving requires agricultural models to have prior understanding of complex unstructured terrain, while agricultural robots require autonomous driving models to have the ability to dynamically avoid obstacles and plan in real time. One-way transfer cannot tap into the potential value of this two-way collaboration.

[0008] (4) Lack of computing power scheduling and privacy protection mechanisms for cross-domain joint computing. Joint training and inference of heterogeneous models will generate huge and dynamically fluctuating computing power requirements. Traditional algorithms only focus on model accuracy and ignore the resource allocation bottleneck when data is exchanged between vehicle edge computing nodes and agricultural cloud nodes; at the same time, the cross-domain sharing of raw data without encryption measures is very likely to lead to the leakage of core agricultural data or vehicle trajectory privacy.

[0009] In summary, existing technologies have significant shortcomings in knowledge sharing and resource optimization for heterogeneous models, making it difficult to fully leverage the combined potential of data and models, resulting in resource waste and inefficiency. Therefore, how to achieve efficient knowledge transfer, multimodal fusion, and resource collaboration between autonomous driving and agricultural AI has become an urgent technical problem to be solved. Summary of the Invention

[0010] In view of this, the present invention provides a method and system for collaborative autonomous driving and agricultural AI based on heterogeneous models, aiming to overcome the architectural barriers between heterogeneous models, realize bidirectional knowledge transfer and resource sharing between autonomous driving and agricultural AI, and improve the generalization ability and resource utilization of the models.

[0011] One aspect of this invention provides a collaborative method for autonomous driving and agricultural AI based on heterogeneous models, comprising: processing collected autonomous driving perception data and agricultural data separately to obtain single-modal feature data with a unified spatiotemporal reference; constructing a cross-domain parameter adapter, wherein the cross-domain parameter adapter is used to establish a parameter mapping channel between the autonomous driving model and the agricultural AI model to realize the mutual transfer of underlying feature extraction capabilities, specifically including: reducing the dimensionality of the parameter matrices of the autonomous driving model and the agricultural AI model through low-rank decomposition, and extracting cross-domain common features from the dimensionality-reduced parameter matrices using an attention mechanism; constructing a domain adversarial loss function based on the parameter mapping channel and the cross-domain common features to enhance knowledge transfer between the autonomous driving model and the agricultural AI model, and obtaining model parameters after bidirectional knowledge transfer; performing hierarchical decision fusion on the single-modal feature data based on the model parameters after bidirectional knowledge transfer, constructing a three-dimensional voxel feature map based on the obtained fusion features, and realizing joint decision output of the perception layer and the decision layer; dynamically allocating computing resources in response to the resource requirement parameters established during the hierarchical decision fusion process, and implementing encryption protection when data is shared between cross-domain nodes.

[0012] Optionally, the step of processing the collected autonomous driving perception data and agricultural data separately to obtain single-modal feature data with a unified spatiotemporal reference includes: performing spatiotemporal standardization processing on the autonomous driving perception data, wherein the spatiotemporal standardization processing includes spatial coordinate system alignment and timestamp hardware synchronization to eliminate reference differences between modes and obtain spatiotemporally aligned autonomous driving feature data; and performing sliding window normalization, radiometric correction and feature extraction on the agricultural data to obtain spatiotemporally aligned agricultural feature data.

[0013] Optionally, the low-rank decomposition is achieved through truncated singular value decomposition, retaining the first r singular values, satisfying the energy percentage threshold:

[0014] Generate low-rank approximate matrices , Represents the left singular value eigenmap matrix. Denotes the right singular value eigenmap matrix, where , , , Indicates the first A singular value, This indicates the number of rows in the parameter matrix being decomposed. This indicates the number of columns in the parameter matrix being decomposed. Represents the space of real numbers.

[0015] Optionally, the step of extracting cross-domain common features from the dimensionality-reduced parameter matrix using an attention mechanism includes: using a multi-head attention mechanism to calculate unified autonomous driving features in the dimensionality-reduced parameter matrix. With unified agricultural characteristics The interaction weights are determined, and cross-domain common features are extracted based on these interaction weights:

[0016] in, , Q Represents the query matrix. K Represents the key matrix. V Represents a value matrix, Represents the feature dimension of the key matrix. Represents the weight matrix. F Indicates input features, This represents the bias used to calculate position perception bias in multi-head attention mechanisms. This represents the position-aware attention bias matrix. This represents the matrix transpose operation.

[0017] Optionally, the domain adversarial loss function is:

[0018] in, Represents the domain adversarial loss function. Represents the mathematical expectation operator. This represents the input sample data. This represents the data distribution for autonomous driving. G represents the data distribution in the agricultural domain, G is the generator, which is configured to generate cross-domain shared features, and D is the domain discriminator, which is configured to distinguish whether features come from the autonomous driving domain or the agricultural domain.

[0019] Optionally, adversarial training based on the domain adversarial loss function employs progressive transfer scheduling to achieve a smooth transition:

[0020] in, t This indicates the current training iteration step number. This represents the maximum number of total training iterations. This represents a hyperparameter that controls the smooth transition rate during migration. This represents the knowledge transfer weight coefficient for the current training step, which increases with the number of iterations. It represents the base of the natural logarithm.

[0021] Optionally, the hierarchical decision fusion of the single-modal feature data based on the model parameters after the bidirectional knowledge transfer includes: using a cascaded Transformer architecture to fuse spatiotemporally aligned autonomous driving feature data and spatiotemporally aligned agricultural feature data based on the model parameters after the bidirectional knowledge transfer to obtain the fused features; constructing a three-dimensional voxel feature map based on the fused features to achieve joint decision output between the perception layer and the decision layer; and establishing a multi-objective optimization problem regarding task execution accuracy, computational resource consumption, and decision latency based on the joint decision process, and outputting the resource requirement parameters of the multi-objective optimization problem.

[0022] Optionally, the step of constructing a three-dimensional voxel feature map based on the fused features to achieve joint decision output between the perception layer and the decision layer includes: dividing the farmland physical area into discrete voxel grids, filling the fused features into the corresponding voxels according to their spatial coordinates to construct a three-dimensional voxel feature map; simultaneously inputting the three-dimensional voxel feature map into the autonomous driving model unit and the agricultural AI model unit, and outputting a joint decision instruction, which is used to execute agricultural operation tasks while ensuring the safety of autonomous driving.

[0023] Optionally, the dynamic allocation of computing resources includes: responding to the multi-objective optimization problem and the resource requirement parameters, performing multi-objective optimization allocation of computing resources based on the improved NSGA-III algorithm, solving the multi-objective optimization problem, and obtaining the optimal computing power and storage allocation strategy.

[0024] Another aspect of the present invention provides a heterogeneous model-based autonomous driving and agricultural AI collaborative system, comprising: a heterogeneous model collaborative module, including an autonomous driving model unit and an agricultural AI model unit, wherein: the autonomous driving model unit integrates a lidar point cloud parser, a multi-camera visual recognition network, and an inertial navigation compensation algorithm, for real-time processing of multimodal sensor data streams and generating vehicle control commands; the agricultural AI model unit includes a soil conductivity analysis layer, a multispectral feature extraction network, and a meteorological time-series prediction model, for processing soil moisture sensor arrays, UAV multispectral images, and meteorological station observation data, and outputting crop management decisions; and a cross-domain parameter adapter, connecting... The autonomous driving model and the agricultural AI model undergo bidirectional knowledge transfer. A multimodal fusion module processes the collected autonomous driving perception data and agricultural data separately to obtain single-modal feature data with a unified spatiotemporal reference and model parameters based on bidirectional knowledge transfer. The single-modal feature data is then fused using a hierarchical decision-making mechanism. A three-dimensional voxel feature map is constructed based on the obtained fused features to achieve joint decision-making output between the perception layer and the decision-making layer. A resource sharing platform, responding to resource requirement parameters established during the hierarchical decision-making fusion process, manages and allocates computing resources, storage resources, and data resources to achieve resource sharing between the autonomous driving model and the agricultural AI model.

[0025] Compared with the prior art, the present invention has the following beneficial effects: (1) Logical progressive collaborative mechanism: This invention provides a benchmark for subsequent processing through data spatiotemporal alignment; through bidirectional knowledge transfer at the model parameter level, it realizes the alignment of the underlying feature extraction capabilities of autonomous driving models and agricultural AI models (such as autonomous driving learning the complex terrain understanding of agricultural models, and agricultural models learning the dynamic obstacle avoidance of autonomous driving), reducing semantic conflicts caused by direct feature fusion and improving the accuracy of fusion decision.

[0026] (2) Deep multimodal fusion: This invention uses spatiotemporal alignment and cascaded attention mechanism to deeply fuse multi-source data and construct a three-dimensional voxel feature map, which enables the system to exhibit higher decision accuracy and environmental adaptability, while improving robustness to interference.

[0027] (3) Efficient and secure resource scheduling: This invention optimizes the efficient utilization and secure sharing of computing resources and data through elastic resource scheduling and privacy protection technologies. It dynamically allocates computing resources based on a multi-objective optimization algorithm, improving resource utilization and reducing operation and maintenance costs. Combining fully homomorphic encryption and differential privacy technologies ensures data security during the sharing process and reduces the risk of data leakage.

[0028] In summary, this invention improves the overall performance of autonomous driving and agricultural AI systems through a heterogeneous model collaborative architecture and cross-domain optimization mechanisms. The system employs intelligent parameter adaptation and multimodal data fusion technologies to solve the problem that traditional knowledge transfer cannot overcome heterogeneous architectures. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. By reading the detailed description of the embodiments below, the advantages and benefits of the solutions will become clear to those skilled in the art. The accompanying drawings are only for illustrating preferred embodiments and are not intended to limit the present invention. In the accompanying drawings: Figure 1 This is a flowchart of the steps of the method of the present invention.

[0030] Figure 2 This is a schematic diagram of the cascaded Transformer architecture for multimodal data fusion in this invention.

[0031] Figure 3 This is a schematic diagram of the overall architecture of the autonomous driving and agricultural AI collaborative system based on heterogeneous models of the present invention.

[0032] Figure 4 This is a pseudocode diagram illustrating the entire process of this invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0034] See Figure 1 The present invention provides a method for collaborative autonomous driving and agricultural AI based on heterogeneous models, comprising: Step S101: Process the collected autonomous driving perception data and agricultural data separately to obtain single-modal feature data with a unified spatiotemporal reference. Step S102: Construct a cross-domain parameter adapter. The cross-domain parameter adapter is used to establish a parameter mapping channel between the autonomous driving model and the agricultural AI model, realizing the mutual transfer of underlying feature extraction capabilities. Specifically, it includes: reducing the dimensionality of the parameter matrices of the autonomous driving model and the agricultural AI model through low-rank decomposition, and extracting cross-domain common features from the dimensionality-reduced parameter matrices using an attention mechanism; based on the parameter mapping channel and the cross-domain common features, constructing a domain adversarial loss function to enhance the knowledge transfer between the autonomous driving model and the agricultural AI model, and obtaining the model parameters after bidirectional knowledge transfer. Step S103: Based on the model parameters after the bidirectional knowledge transfer, perform hierarchical decision fusion on the single-modal feature data, construct a three-dimensional voxel feature map based on the obtained fusion features, and realize the joint decision output of the perception layer and the decision layer. Step S104: In response to the resource requirement parameters established during the hierarchical decision fusion process, dynamically allocate computing resources and implement encryption protection when data is shared across domain nodes.

[0035] Optionally, the step of processing the collected autonomous driving perception data and agricultural data separately to obtain single-modal feature data with a unified spatiotemporal reference includes: performing spatiotemporal standardization processing on the autonomous driving perception data, wherein the spatiotemporal standardization processing includes spatial coordinate system alignment and timestamp hardware synchronization to eliminate reference differences between modes and obtain spatiotemporally aligned autonomous driving feature data; and performing sliding window normalization, radiometric correction and feature extraction on the agricultural data to obtain spatiotemporally aligned agricultural feature data.

[0036] Optionally, the low-rank decomposition is achieved through truncated singular value decomposition, retaining the first r singular values, satisfying the energy percentage threshold:

[0037] Generate low-rank approximate matrices , Represents the left singular value eigenmap matrix. Denotes the right singular value eigenmap matrix, where , , , Indicates the first A singular value, This indicates the number of rows in the parameter matrix being decomposed. This indicates the number of columns in the parameter matrix being decomposed. Represents the space of real numbers.

[0038] Optionally, the step of extracting cross-domain common features from the dimensionality-reduced parameter matrix using an attention mechanism includes: using a multi-head attention mechanism to calculate unified autonomous driving features in the dimensionality-reduced parameter matrix. With unified agricultural characteristics The interaction weights are determined, and cross-domain common features are extracted based on these interaction weights:

[0039] in, , Q Represents the query matrix. K Represents the key matrix. V Represents a value matrix, Represents the feature dimension of the key matrix. Represents the weight matrix. F Indicates input features, This represents the bias used to calculate position perception bias in multi-head attention mechanisms. This represents the position-aware attention bias matrix. This represents the matrix transpose operation.

[0040] Optionally, the domain adversarial loss function is:

[0041] in, Represents the domain adversarial loss function. Represents the mathematical expectation operator. This represents the input sample data. This represents the data distribution for autonomous driving. G represents the data distribution in the agricultural domain, G is the generator, which is configured to generate cross-domain shared features, and D is the domain discriminator, which is configured to distinguish whether features come from the autonomous driving domain or the agricultural domain.

[0042] Optionally, adversarial training based on the domain adversarial loss function employs progressive transfer scheduling to achieve a smooth transition:

[0043] in, t This indicates the current training iteration step number. This represents the maximum number of total training iterations. This represents a hyperparameter that controls the smooth transition rate during migration. This represents the knowledge transfer weight coefficient for the current training step, which increases with the number of iterations. It represents the base of the natural logarithm.

[0044] Optionally, the hierarchical decision fusion of the single-modal feature data based on the model parameters after the bidirectional knowledge transfer includes: using a cascaded Transformer architecture to fuse spatiotemporally aligned autonomous driving feature data and spatiotemporally aligned agricultural feature data based on the model parameters after the bidirectional knowledge transfer to obtain the fused features; constructing a three-dimensional voxel feature map based on the fused features to achieve joint decision output between the perception layer and the decision layer; and establishing a multi-objective optimization problem regarding task execution accuracy, computational resource consumption, and decision latency based on the joint decision process, and outputting the resource requirement parameters of the multi-objective optimization problem.

[0045] Optionally, the step of constructing a three-dimensional voxel feature map based on the fused features to achieve joint decision output between the perception layer and the decision layer includes: dividing the farmland physical area into discrete voxel grids, filling the fused features into the corresponding voxels according to their spatial coordinates to construct a three-dimensional voxel feature map; simultaneously inputting the three-dimensional voxel feature map into the autonomous driving model unit and the agricultural AI model unit, and outputting a joint decision instruction, which is used to execute agricultural operation tasks while ensuring the safety of autonomous driving.

[0046] Optionally, the dynamic allocation of computing resources includes: responding to the multi-objective optimization problem and the resource requirement parameters, performing multi-objective optimization allocation of computing resources based on the improved NSGA-III algorithm, solving the multi-objective optimization problem, and obtaining the optimal computing power and storage allocation strategy.

[0047] Another aspect of the present invention provides a collaborative system for autonomous driving and agricultural AI based on heterogeneous models, comprising: The heterogeneous model collaboration module includes an autonomous driving model unit and an agricultural AI model unit, wherein: The autonomous driving model unit integrates a lidar point cloud parser, a multi-camera visual recognition network, and an inertial navigation compensation algorithm to process multimodal sensor data streams in real time and generate vehicle control commands. The agricultural AI model unit includes a soil conductivity analysis layer, a multispectral feature extraction network, and a meteorological time series prediction model, which are used to process soil moisture sensor arrays, UAV multispectral images, and meteorological station observation data to output crop management decisions. Cross-domain parameter adapter connects autonomous driving models and agricultural AI models and enables bidirectional knowledge transfer; The multimodal fusion module is used to process the collected autonomous driving perception data and agricultural data separately to obtain single-modal feature data with a unified spatiotemporal reference and model parameters based on bidirectional knowledge transfer. The single-modal feature data is then subjected to hierarchical decision fusion, and a three-dimensional voxel feature map is constructed based on the obtained fusion features to realize the joint decision output of the perception layer and the decision layer. The resource sharing platform is used to manage and allocate computing resources, storage resources and data resources in response to the resource demand parameters established during the hierarchical decision fusion process, so as to realize resource sharing between the autonomous driving model and the agricultural AI model.

[0048] It should be understood that the system of the present invention is used to implement the corresponding steps in the foregoing multiple method examples and has the beneficial effects of the corresponding method examples.

[0049] Specifically, the solution of the present invention is further described according to the following embodiments: Example 1 See Figure 4 This is a pseudocode diagram illustrating the entire process of this invention.

[0050] Step S1. Cross-domain data synchronization and preprocessing implementation To address the issue of inconsistent spatiotemporal references in heterogeneous data, the collected autonomous driving perception data and agricultural data are processed separately to obtain single-modal feature data with a unified spatiotemporal reference, providing a unified physical world representation for subsequent multimodal fusion. This step further includes: S1.1 Data Acquisition: Configure a multimodal sensor array to acquire heterogeneous raw data.

[0051] Specifically, for autonomous driving, LiDAR point clouds containing three-dimensional spatial geometric information are acquired through onboard sensors. (where N is the number of points) and RGB images containing rich texture features (Where H and W represent the image height and width); For the agricultural sector, meteorological time-series data reflecting dynamic environmental changes are collected through IoT nodes and drones deployed in farmland. Multispectral images reflecting crop growth physicochemical indicators (where k is the number of spectral channels), Represents the space of real numbers. The characteristic channel dimension of meteorological time series data.

[0052] S1.2 Spatiotemporal Standardization Processing (Spatiotemporal Alignment): Since autonomous driving perception data is based on the vehicle's local coordinate system and has high-frequency characteristics, while agricultural data is based on the global geographic coordinate system and has low-frequency characteristics, this step performs strict spatiotemporal alignment.

[0053] In terms of spatial alignment: Autonomous driving perception data obtains the vehicle's latitude and longitude coordinates through an onboard GPS receiver. ,in For vehicle longitude, The vehicle's latitude is used as the reference coordinate, and its pose in the local coordinate system is calculated in real time using IMU data. Agricultural data uses the WGS-84 standard, and both UAV multispectral images and weather station data are labeled with geographic coordinates. ,in Longitude of farmland The latitude of the farmland is given. Select at least three common geographic control points and solve for the affine transformation matrix using the least squares method. To achieve the mapping from the local coordinate system to the global coordinate system, satisfying:

[0054] in, It represents the horizontal axis coordinates (usually corresponding to the longitude direction) in the global geographic coordinate system of farmland. This represents the vertical coordinate of the farmland in the global geographic coordinate system (usually corresponding to the latitude direction). This represents the horizontal axis coordinate in the local coordinate system of the autonomous vehicle. This represents the ordinate of the vertical axis in the local coordinate system of the autonomous vehicle. This represents the matrix transpose operation. This represents a multispectral image reflecting the physicochemical indicators of crop growth.

[0055] Regarding time synchronization: Sensor data is synchronized with a hardware clock via PTP / NTP protocol or GPS PPS signal to ensure that the time deviation between LiDAR point cloud, camera image and IMU data is strictly controlled within less than 1 millisecond.

[0056] S1.3 Feature Extraction: A specific deep learning network is used to perform preliminary feature extraction on the standardized data. The autonomous driving perception data is standardized as follows: , ,in, This represents the lidar point cloud features after zero-mean, unit-variance standardization. This represents the visual image features after pixel value normalization. This represents the spatial position vector of the original LiDAR point cloud. This represents the mean spatial coordinates of the input point cloud dataset. This represents the standard deviation of the spatial coordinates of the input point cloud dataset; subsequently, PointNet++ is used to process the point cloud data, and ResNet is used to process the image data to extract preliminary features. Agricultural data, after being normalized by a sliding window and radiometrically corrected, is then processed using LSTM to extract preliminary features from the time-series data. At this point, we obtained single-modal feature data with a unified spatiotemporal reference, namely, spatiotemporally aligned and initially encoded autonomous driving feature data and agricultural feature data.

[0057] Step S2. Implementation of Two-Way Knowledge Transfer Before multimodal data feature fusion, a cross-domain parameter adapter is constructed to establish a parameter mapping channel between heterogeneous models. Due to the huge parameter space of heterogeneous models, direct fusion will lead to the curse of dimensionality and semantic conflicts. This step achieves implicit alignment of the underlying feature extraction capabilities of autonomous driving models and agricultural AI models through bidirectional knowledge transfer.

[0058] S2.1 Low-rank decomposition: To reduce the computational complexity of cross-domain mapping, the parameter matrices of the autonomous driving model and the agricultural AI model, i.e., the source model parameter matrices, are decomposed. Perform truncated singular value decomposition (SVD) to extract a low-rank subspace representing the core knowledge. This decomposes the subspace into low-rank matrices. , Describes a left singular matrix. This represents the transpose of the right singular matrix, retaining the first r singular values, satisfying the energy percentage threshold:

[0059] Generate low-rank approximate matrices , This represents the left singular value eigenmap matrix, used to project the original high-dimensional parameters onto a low-rank subspace. Denotes the right singular value eigenmap matrix, used to recover the adapted parametric features from the low-rank subspace, where , Indicates the first A singular value, This indicates the number of rows in the parameter matrix being decomposed. This indicates the number of columns in the parameter matrix being decomposed. Represents the space of real numbers.

[0060] S2.2 Attention Mechanism: To select the most valuable prior knowledge for the target domain, a multi-head attention mechanism is used to compute unified autonomous driving features in the dimensionality-reduced parameter matrix. With unified agricultural characteristics Interaction weights:

[0061] in, , Q Represents the query matrix. K Represents the key matrix. V Represents a value matrix, Represents the feature dimension of the key matrix. Represents the weight matrix. F Indicates input features, This represents the bias used to calculate position perception bias in multi-head attention mechanisms. This represents the position-aware attention bias matrix, used to adjust the feature attention weights between different modalities. This represents the matrix transpose operation.

[0062] This mechanism enables, for example, autonomous driving models to "pay attention" to terrain constraint information provided by agricultural models when processing features.

[0063] Subsequently, based on the hypernetwork architecture , Represents a low-rank approximate matrix. This represents the model parameters at the current time. The generated target network parameters are represented by the low-rank matrix [B; A] dynamically generated through nonlinear mapping to adapt the network parameters to the target model structure. , Represents the projection weight matrix. This represents the bias used when generating target network parameters through nonlinear mapping of the hypernetwork.

[0064] S2.3 Enhanced Dual-Model Knowledge Transfer (Adversarial Training): To further eliminate the differences in feature distribution between heterogeneous domains, a domain adversarial training mechanism is introduced.

[0065] Constructing a domain-adversarial loss function:

[0066] in, This represents the data distribution for autonomous driving. This represents the data distribution in the agricultural domain. G is the generator (i.e., the feature extraction backbone), configured to generate cross-domain shared features; D is the domain discriminator, configured to distinguish whether features come from the autonomous driving domain (label 1) or the agricultural domain (label 0).

[0067] The underlying mechanism for enhancing bi-model knowledge transfer is as follows: by inserting a gradient inversion layer (GRL) between the generator G and the discriminator D, the gradient is multiplied by the gradient during backpropagation. Coefficients. This forces the generator G to update its parameters in a way that "deceives" the discriminator D, ultimately extracting "cross-domain invariant features" that are difficult to distinguish between domains. In optimizing the domain-specific task loss ( At the same time, this mechanism significantly improves the generalization ability of heterogeneous models in complex scenarios.

[0068] To prevent the model from collapsing due to excessive adversarial loss in the early stages of training, progressive transfer scheduling is introduced, with transfer weights increasing over time: To achieve a smooth transition.

[0069] Step S3. Hierarchical decision fusion and multi-objective problem establishment and implementation Based on the model parameters after bidirectional knowledge transfer, deep hierarchical decision fusion is performed on spatiotemporally aligned single-modal feature data.

[0070] S3.1 Cascaded Transformer Fusion: To fully exploit intra-modal and cross-modal context dependencies, a two-level cascaded Transformer architecture was designed.

[0071] Level 1 (Intramodal Interaction): Input unified autonomous driving features With unified agricultural characteristics Each modality's spatiotemporal dependencies are extracted independently through a self-attention layer, preserving domain-specific attributes. Level 2 (Cross-modal interaction): Employs a cross-attention mechanism to encode autonomous driving features. As a query, based on agricultural coding features As the key and value, obtain the fusion features:

[0072] After adding the fused features to the original input features, the final fused features, which include the global context, are obtained through layer normalization and stable training. .

[0073] S3.2 Construction and Joint Decision-Making of 3D Voxel Feature Maps: The farmland physical region is divided into discrete voxel grids (resolution 0.5m×0.5m×0.2m), and the final fused features are... The voxel feature map is filled into the corresponding voxel based on its spatial coordinates (after spatiotemporal alignment in step S1). This voxel feature map provides a unified spatial representation. The three-dimensional voxel feature map is simultaneously input into the autonomous driving model unit and the agricultural AI model unit, and a joint decision instruction is output. The joint decision instruction is used to perform agricultural operation tasks while ensuring the safety of autonomous driving (e.g., the planning algorithm can identify and perform precise spraying tasks in specific vulnerable crop areas while avoiding dynamic obstacles).

[0074] S3.3 Establishing a Multi-Objective Optimization Problem: The joint decision-making and massive voxel computation described above will generate a huge computing bottleneck on edge devices. Therefore, when integrating path planning and crop growth prediction at the decision layer, a multi-objective optimization problem must be established: the objective function is configured to maximize task completion rate, minimize computational latency, and minimize total system energy consumption. The state space of this problem (including real-time resource requirements and task priorities) will serve as the trigger input for step S4.

[0075] Specifically, see Figure 2The cascaded Transformer architecture achieves intra-modal and cross-modal feature extraction through the collaborative work of the encoder and decoder. Intramodal feature extraction stage: The encoder performs N operations in a loop to embed and encode the input single-modal feature data, and then feeds it into a multi-head attention mechanism. In conjunction with residual connections, layer normalization and feedforward network, it extracts the fine spatiotemporal dependencies within the modality. Cross-modal semantic fusion stage: The decoder performs N operations in a loop, processes the output embedding using a masked multi-head attention mechanism, and achieves deep interaction with the encoder output features through a specific multi-head attention mechanism layer, that is, performs the aforementioned cross-attention fusion, thereby obtaining cross-domain fusion features containing global context. Decision support output stage: The fused features undergo dimensionality transformation through linear mapping, and finally output the predicted probability distribution through the Softmax layer, providing probabilistic support for subsequent construction of 3D voxel feature maps and execution of joint decisions.

[0076] Step S4. Dynamic Resource Scheduling and Privacy Protection In response to the multi-objective optimization problem and resource requirements established by S3, dynamic resource scheduling and cross-domain privacy protection are implemented.

[0077] S4.1 Resource Scheduling Based on Improved NSGA-III and DQN: To address the limited computing power of edge computing nodes, a scheduling strategy combining heuristic algorithms and reinforcement learning is adopted. First, the resource requirements from S3 are received, and multi-objective optimization (including hierarchical non-dominated sorting solution set and adaptive adjustment of crossover and mutation probabilities) is performed based on the improved NSGA-III algorithm to generate a static Pareto optimal solution set. Subsequently, in the context of a dynamically fluctuating real-time environment, a Deep Q-Network (DQN) is used to select the optimal action from the Pareto optimal solution set.

[0078] Define the state space(s): including GPU utilization, memory usage, and task priority weights; define the action space(a): including GPU computing power allocation ratio and memory allocation size. The overall reward function is designed as follows:

[0079] in, This represents the instant reward function. Indicates the current state. Indicates taking action. This represents a hyperparameter that controls the smooth transition rate during migration. This represents a performance bonus item that reflects the accuracy of task execution. This represents a cost penalty item that reflects the system's computing power overhead and energy consumption level. This represents the weighting factor of the energy consumption reward term in multi-objective optimization.

[0080] The optimization goal of DQN is to maximize long-term cumulative rewards.

[0081] in, Represents the state-action value function. Represents the mathematical expectation operator. Indicates an immediate reward. This represents the reward discount factor. This represents the operation of finding the maximum value among all possible next actions. Indicates the next state. Indicates the next action.

[0082] S4.2 Data Sharing Protocol Based on Fully Homomorphic Encryption (FHE): When sharing voxel features of S3 or migration parameters of S2 between autonomous vehicles (edge ​​nodes) and the agricultural cloud (central node), fully homomorphic encryption (FHE) is used to prevent the leakage of core agricultural data or vehicle trajectory privacy. Encryption is performed to ensure that aggregation calculations can be performed directly in the encrypted domain on the cloud. The encryption process is defined as follows:

[0083] in, This represents the chain multiplication and aggregation operation in the ciphertext field. This represents the local model parameters or feature vectors that the i-th heterogeneous node participates in sharing. Represents a fully homomorphic encryption mapping function. This represents modulo operations, used to limit the range of values ​​in the encryption space. This represents the square of the modulus generated based on large prime numbers in encryption schemes such as Paillier.

[0084] To defend against inference attacks, a differential privacy (DP) noise addition module is used to add Laplace noise to the shared data between the autonomous driving model and the agricultural AI model:

[0085] in, This indicates shared features resistant to inference attacks after injecting Laplacian noise. ε represents the global sensitivity function value in differential privacy, and ε is the privacy budget. By controlling the noise level, the anonymity and security of the data are further guaranteed while ensuring the accuracy of joint training of the model.

[0086] Example 2 System Architecture and Module Implementation See Figure 3 The present invention provides a collaborative system for autonomous driving and agricultural AI based on heterogeneous models, which can be divided into four levels in terms of overall logical architecture: (1) Infrastructure layer: It covers the collection and transmission of heterogeneous data, including vehicle sensors (LiDAR, millimeter-wave radar, GPS-RTK, etc.) on the autonomous driving side, farmland dynamic databases and sensors (weather station data, soil sensors, multispectral cameras, etc.) on the agricultural side, as well as edge computing and distributed storage sharing networks that connect cross-domain nodes.

[0087] (2) Understanding and cognition layer: Corresponding to the single-modal feature processing stage, spatiotemporal dimension co-construction and preliminary feature extraction are performed on the input autonomous driving multimodal data and agricultural multi-source data respectively.

[0088] (3) Decision reasoning layer: Corresponding to the cross-domain parameter adapter and multimodal fusion module, through the knowledge-aware Transformer (KAT) and temporal fusion Transformer (TFT) architectures, model knowledge transfer (cross-domain reuse, model distillation) and cross-modal cross-attention fusion are realized.

[0089] (4) Control Application Layer: The joint decision output of the corresponding system includes the physical control of the unmanned agricultural machinery and the comprehensive management decision of agricultural big data such as pest and disease identification and agricultural product price prediction.

[0090] The steps of the method described above, which fully maps autonomous driving and agricultural AI collaborative system based on heterogeneous models, are as follows: 1. Heterogeneous Model Collaboration Module: Includes an autonomous driving model unit (configured with LiDAR point cloud resolver PointNet++, visual recognition ResNet-50, and inertial navigation compensation) and an agricultural AI model unit (configured with convolutional LSTM and VisionTransformer multispectral extraction).

[0091] The autonomous driving model unit is used to process multimodal sensor data streams in real time and generate vehicle control commands that include steering angle control values, braking pressure thresholds, and acceleration planning values.

[0092] The agricultural AI model unit processes soil moisture sensor arrays, drone multispectral images, and meteorological station observation data to output crop management decisions that include irrigation recommendations, fertilizer ratio schemes, and pest and disease warning levels.

[0093] 2. Cross-domain parameter adapter: Execute step S2 to connect the autonomous driving model and the agricultural AI model and perform bidirectional knowledge transfer. It includes a low-rank decomposition module, an attention knowledge distillation module, and a dynamic parameter generation module.

[0094] The low-rank decomposition module is used to reduce the dimensionality of the parameter matrices of the autonomous driving model and the agricultural AI model, i.e., the parameter matrices of the source model. The attention-based knowledge distillation module employs a cross-attention mechanism to extract common features across domains. The dynamic parameter generation module generates adaptive parameters based on the target model structure.

[0095] LAMB optimizer configuration:

[0096] in, This represents the updated model parameters. This represents the model parameters at the current time. The learning rate represents the step size for updating the control parameters. The scale alignment mapping function represents the gradient of the parameters. This represents the gradient value of the joint loss of multiple tasks with respect to the parameters. This represents the exponential moving average of the squared gradient (second-order momentum). This refers to a privacy budget.

[0097] Joint training of multi-task loss functions:

[0098] in, This represents the weighted multi-task total loss during system training. This represents the loss weight for the autonomous driving task. This represents the loss weight of agricultural decision-making tasks. This represents the loss weights for the knowledge transfer task. This represents the loss in path planning and target perception on the autonomous driving side. This indicates losses related to operational decisions and environmental identification on the agricultural side. This represents the KL divergence knowledge distillation loss used to align heterogeneous model distributions.

[0099] 3. Multimodal Fusion Module: Performs steps S1 and S3. Includes a data preprocessing and spatiotemporal alignment unit (performs spatiotemporal normalization), feature extraction, and fusion decision unit (performs cascaded Transformer and voxel construction).

[0100] The data preprocessing and spatiotemporal alignment unit is used to unify the spatiotemporal reference of autonomous driving perception data and agricultural data; The feature extraction and fusion decision unit uses a cascaded attention mechanism and a recurrent neural network to achieve cross-modal interaction and construct a three-dimensional voxel feature map.

[0101] 4. Resource Sharing Platform: Executes step S4. Used to manage and allocate computing, storage, and data resources, enabling resource sharing between autonomous driving models and agricultural AI models. Includes a resource management module, a resource scheduling module (executing NSGA-III and DQN), and a privacy-preserving data sharing unit (executing FHE and differential privacy), achieving encrypted domain data aggregation.

[0102] Compared with the prior art, the present invention has the following beneficial effects: (1) Logical progressive collaborative mechanism: This invention provides a benchmark for subsequent processing through data spatiotemporal alignment; through bidirectional knowledge transfer at the model parameter level, it realizes the alignment of the underlying feature extraction capabilities of autonomous driving models and agricultural AI models (such as autonomous driving learning the complex terrain understanding of agricultural models, and agricultural models learning the dynamic obstacle avoidance of autonomous driving), reducing semantic conflicts caused by direct feature fusion and improving the accuracy of fusion decision.

[0103] (2) Deep multimodal fusion: This invention uses spatiotemporal alignment and cascaded attention mechanism to deeply fuse multi-source data and construct a three-dimensional voxel feature map, which enables the system to exhibit higher decision accuracy and environmental adaptability, while improving robustness to interference.

[0104] (3) Efficient and secure resource scheduling: This invention optimizes the efficient utilization and secure sharing of computing resources and data through elastic resource scheduling and privacy protection technologies. It dynamically allocates computing resources based on a multi-objective optimization algorithm, improving resource utilization and reducing operation and maintenance costs. Combining fully homomorphic encryption and differential privacy technologies ensures data security during the sharing process and reduces the risk of data leakage.

[0105] In summary, this invention improves the overall performance of autonomous driving and agricultural AI systems through a heterogeneous model collaborative architecture and cross-domain optimization mechanisms. The system employs intelligent parameter adaptation and multimodal data fusion technologies to solve the problem that traditional knowledge transfer cannot overcome heterogeneous architectures.

[0106] Specific embodiments of the present invention have now been described. Other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result.

[0107] It should be noted that all directional indications (such as up, down, left, right, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship between the components in a certain order (as shown in the figure). If the specific order changes, the directional indication will also change accordingly.

[0108] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0109] It should be noted that although specific embodiments of the present invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of the present invention. Various modifications and variations that can be made by those skilled in the art without inventive effort within the scope described in the claims still fall within the scope of protection of the present invention.

[0110] The examples of the embodiments of the present invention are intended to concisely illustrate the technical features of the embodiments of the present invention, so that those skilled in the art can intuitively understand the technical features of the embodiments of the present invention, and are not intended to be an improper limitation of the embodiments of the present invention.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for collaborative autonomous driving and agricultural AI based on heterogeneous models, characterized in that, include: The collected autonomous driving perception data and agricultural data are processed separately to obtain single-modal feature data with a unified spatiotemporal reference. A cross-domain parameter adapter is constructed to establish a parameter mapping channel between an autonomous driving model and an agricultural AI model, enabling mutual transfer of underlying feature extraction capabilities. Specifically, this includes: reducing the dimensionality of the parameter matrices of the autonomous driving model and the agricultural AI model through low-rank decomposition, and extracting cross-domain common features from the dimensionality-reduced parameter matrices using an attention mechanism; based on the parameter mapping channel and the cross-domain common features, constructing a domain adversarial loss function to enhance knowledge transfer between the autonomous driving model and the agricultural AI model, thereby obtaining model parameters after bidirectional knowledge transfer. Based on the model parameters after the bidirectional knowledge transfer, hierarchical decision fusion is performed on the single-modal feature data. A three-dimensional voxel feature map is constructed based on the obtained fused features to realize the joint decision output of the perception layer and the decision layer. In response to the resource requirement parameters established during the hierarchical decision fusion process, computing resources are dynamically allocated, and encryption protection is implemented when data is shared across domain nodes.

2. The method according to claim 1, characterized in that, The collected autonomous driving perception data and agricultural data are processed separately to obtain single-modal feature data with a unified spatiotemporal reference, including: Spatiotemporal standardization processing is performed on autonomous driving perception data. The spatiotemporal standardization processing includes spatial coordinate system alignment and timestamp hardware synchronization to eliminate intermodal reference differences and obtain spatiotemporally aligned autonomous driving feature data. Agricultural data is subjected to sliding window normalization, radiometric correction, and feature extraction to obtain spatiotemporally aligned agricultural feature data.

3. The method according to claim 1, characterized in that, The low-rank decomposition is achieved through truncated singular value decomposition, retaining the first r singular values, which satisfy the energy percentage threshold: Generate low-rank approximate matrices , Represents the left singular value eigenmap matrix. Denotes the right singular value eigenmap matrix, where , , , Indicates the first A singular value, This indicates the number of rows in the parameter matrix being decomposed. This indicates the number of columns in the parameter matrix being decomposed. Represents the space of real numbers.

4. The method according to claim 2, characterized in that, The method of extracting cross-domain common features from the dimensionality-reduced parameter matrix using an attention mechanism includes: A multi-head attention mechanism is used to compute unified autonomous driving features in the dimensionality-reduced parameter matrix. With unified agricultural characteristics The interaction weights are determined, and cross-domain common features are extracted based on these interaction weights: in, , Q Represents the query matrix. K Represents the key matrix. V Represents a value matrix, Represents the characteristic dimension of the key matrix. Represents the weight matrix. F Indicates input features, This represents the bias used to calculate position perception bias in multi-head attention mechanisms. This represents the position-aware attention bias matrix. This represents the matrix transpose operation.

5. The method according to claim 1, characterized in that, The domain adversarial loss function is: in, Represents the domain adversarial loss function. Represents the mathematical expectation operator. This represents the input sample data. This represents the data distribution for autonomous driving. G represents the data distribution in the agricultural domain, G is the generator, which is configured to generate cross-domain shared features, and D is the domain discriminator, which is configured to distinguish whether features come from the autonomous driving domain or the agricultural domain.

6. The method according to claim 5, characterized in that, Adversarial training based on the aforementioned domain adversarial loss function employs progressive transfer scheduling to achieve a smooth transition: in, t This indicates the current training iteration step number. This represents the maximum number of total training iterations. This represents a hyperparameter that controls the smooth transition rate during migration. This represents the knowledge transfer weight coefficient for the current training step, which increases with the number of iterations. It represents the base of the natural logarithm.

7. The method according to claim 2, characterized in that, The hierarchical decision fusion of the single-modal feature data based on the model parameters after the bidirectional knowledge transfer includes: Based on the model parameters after the bidirectional knowledge transfer, a cascaded Transformer architecture is adopted to fuse spatiotemporally aligned autonomous driving feature data and spatiotemporally aligned agricultural feature data to obtain the fused features. Based on the fusion features, a three-dimensional voxel feature map is constructed to realize the joint decision output of the perception layer and the decision layer; Based on the joint decision-making process, a multi-objective optimization problem is established concerning task execution accuracy, computational resource consumption, and decision delay, and the resource requirement parameters of the multi-objective optimization problem are output.

8. The method according to claim 7, characterized in that, The construction of a three-dimensional voxel feature map based on the fused features to achieve joint decision output of the perception layer and the decision layer includes: The physical region of farmland is divided into discrete voxel grids, and the fused features are filled into the corresponding voxels according to their spatial coordinates to construct a three-dimensional voxel feature map. The three-dimensional voxel feature map is simultaneously input into the autonomous driving model unit and the agricultural AI model unit, and a joint decision command is output. The joint decision command is used to execute agricultural operation tasks while ensuring the safety of autonomous driving.

9. The method according to claim 7, characterized in that, The dynamically allocated computing resources include: In response to the multi-objective optimization problem and the resource requirement parameters, a multi-objective optimization allocation of computing resources is performed based on the improved NSGA-III algorithm to solve the multi-objective optimization problem and obtain the optimal computing power and storage allocation strategy.

10. A collaborative system for autonomous driving and agricultural AI based on heterogeneous models, characterized in that, include: The heterogeneous model collaboration module includes an autonomous driving model unit and an agricultural AI model unit, wherein: The autonomous driving model unit integrates a lidar point cloud parser, a multi-camera visual recognition network, and an inertial navigation compensation algorithm to process multimodal sensor data streams in real time and generate vehicle control commands. The agricultural AI model unit includes a soil conductivity analysis layer, a multispectral feature extraction network, and a meteorological time series prediction model, which are used to process soil moisture sensor arrays, UAV multispectral images, and meteorological station observation data to output crop management decisions. Cross-domain parameter adapter connects autonomous driving models and agricultural AI models and enables bidirectional knowledge transfer; The multimodal fusion module is used to process the collected autonomous driving perception data and agricultural data separately to obtain single-modal feature data with a unified spatiotemporal reference and model parameters based on bidirectional knowledge transfer. The single-modal feature data is then subjected to hierarchical decision fusion, and a three-dimensional voxel feature map is constructed based on the obtained fusion features to realize the joint decision output of the perception layer and the decision layer. The resource sharing platform is used to manage and allocate computing resources, storage resources and data resources in response to the resource demand parameters established during the hierarchical decision fusion process, so as to realize resource sharing between the autonomous driving model and the agricultural AI model.

Images