A multi-modal trajectory prediction method and device for intelligent agricultural machine path planning

Abstract

A multi-modal trajectory prediction method and device for intelligent agricultural machine path planning belong to the field of intelligent agricultural machine path planning, acquire scene images and mobile target state data of a work area, construct a historical trajectory sequence for explicit diffusion and conditional denoising to obtain historical trajectory preprocessing features; a multi-modal node interaction feature extraction module with parameter sharing is used to extract multi-modal interaction features from an interaction graph structure constructed based on the historical trajectory sequence and the historical trajectory preprocessing features, respectively, and to fuse them; trajectory latent features and explicit intention latent features are generated from future trajectory multi-modal features, the fused features are jointly guided, future trajectory distribution parameters are regressed, and multiple future trajectories and corresponding modal probabilities are obtained for intelligent agricultural machine path planning and driving decisions. The trajectory prediction process can utilize high-level semantic information in the agricultural machine work scene, take into account dynamic target interaction and scene structure constraints, and improve the expression ability of multi-modal future behavior.

Description

Technical Field

[0001] This invention belongs to the field of intelligent agricultural machinery path planning, and specifically relates to a multimodal trajectory prediction method and device for intelligent agricultural machinery path planning. Background Technology

[0002] In intelligent agricultural machinery path planning, trajectory prediction of moving targets within the work area is a crucial intermediate link connecting environmental perception, target tracking, path planning, behavioral decision-making, and motion control in the autonomous operation system of intelligent agricultural machinery. When intelligent tractors perform tasks such as tillage, sowing, fertilization, plant protection, harvest assistance, and inter-plot transportation, the host intelligent agricultural machinery, as the performer of autonomous tasks, not only needs to acquire the positions and categories of other agricultural machinery, field transport vehicles, operators, and other dynamic obstacles within the work area, but also needs to predict the various possible trajectories and probabilities of the aforementioned moving targets within future time intervals to provide conflict judgment and avoidance decision-making basis for the path planning module or safety decision-making module.

[0003] Compared to urban road traffic scenarios, intelligent agricultural machinery operation scenarios exhibit distinct industry characteristics. On the one hand, during the straight-line operation phase within a pure field, dynamic obstacles are relatively few, and the target movement exhibits strong constraints imposed by the work row. On the other hand, in scenarios such as turning at the edge of the field, entering and exiting the field, switching between work rows, inter-field transfers, multi-person and multi-machine collaborative operations, and temporary obstacles entering the work area, the target's movement behavior exhibits significant uncertainty, multimodality, and scenario dependence. These scenarios are simultaneously constrained by plot boundaries, crop row direction, implement width, operational direction rules, surface attachment status, and traffic safety boundaries. Traditional methods based solely on uniform speed, uniform acceleration, or simple kinematic extrapolation are insufficient to meet the accuracy and robustness requirements.

[0004] Existing trajectory prediction technologies can be broadly categorized into three types. The first type is extrapolation methods based on kinematic models. These methods are simple to implement and computationally inexpensive, but they are less adaptable to complex scenarios such as turning, intersection avoidance, and transition switching. The second type is deep learning methods based on interaction modeling. These methods improve prediction accuracy by encoding the historical trajectories of multiple targets and learning the interaction relationships between them, but they do not adequately consider the constraints of plot structure, work row semantics, and implement constraints in agricultural machinery scenarios. The third type is multimodal prediction methods based on diffusion or generative models. These methods have advantages in representing multimodal distributions, but their model inference chains are long and computationally complex. Without explicit modeling of historical trajectory errors and the specific semantics of agricultural machinery scenarios, they are difficult to deploy stably on vehicle-mounted platforms.

[0005] Furthermore, the existing technology has at least the following problems: 1. Historical trajectory input usually comes directly from the perception and tracking results. Trajectory errors caused by positioning drift, visual occlusion, radar noise, field bumps and dust lead to unstable input quality. 2. Most methods rely primarily on geometric trajectories for scene understanding, lacking the integration and utilization of high-level semantic information such as weather, time, plot type, crop row direction, field area status, and transition road status; 3. Insufficient utilization of agricultural machinery-specific scenario structural constraints such as plot boundaries, work row centerlines, field entrances and exits, restricted areas, and machinery passage constraints, which makes the prediction results prone to deviating from the actual passable areas; 4. There is insufficient explicit expression of the intention to perform tasks, the intention to transition between tasks, and the intention to switch paths, making it difficult to distinguish between targets with similar short-term historical trajectories but different future behaviors; 5. Existing methods typically leave trajectory prediction results at the model output layer, lacking an engineering data link description that provides multiple future trajectories and their probabilities to the host intelligent agricultural machinery path planning module or safety decision module, which is not conducive to direct application in autonomous operation systems. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal trajectory prediction method and device for intelligent agricultural machinery path planning, which solves the problems that the historical trajectory of the moving target to be predicted is disturbed by positioning error, environmental perception error and target tracking error, insufficient utilization of the interaction relationship between moving targets and between moving targets and the structure of the operation scene, insufficient fusion of scene language information and traffic constraint information, and difficulty in effectively jointly expressing the multimodal features of the future trajectory and the motion intention.

[0007] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a multimodal trajectory prediction method for intelligent agricultural machinery path planning, which acquires scene images and state data of moving targets within the working area through environmental sensing components, and performs path planning and driving decisions based on the predicted future trajectory of the moving targets, including the following steps: S1. Obtain historical motion state data of the moving target to be predicted and nearby moving targets, as well as scene images, scene structure information and traffic constraint information of the work area; S2. Construct a historical trajectory sequence from the historical motion state data of the target to be predicted and neighboring moving targets, and perform explicit diffusion and conditional denoising on the trajectory error in the historical trajectory sequence to obtain the historical trajectory preprocessing features; S3. Construct corresponding target-target interaction graph structures based on historical trajectory sequences and historical trajectory preprocessing features respectively; construct corresponding target-scene structure interaction graph structures based on historical trajectory sequences and historical trajectory preprocessing features, combined with comprehensive scene structure information including scene structure information and traffic constraint information; extract scene language features based on the scene images obtained in step S1 and the preset scene descriptive word set; S4. Using a parameter-sharing multimodal node interaction feature extraction module, multimodal interaction features are extracted from the interaction graph structure constructed based on historical trajectory sequences and the interaction graph structure constructed based on preprocessed features of historical trajectories, respectively. The extracted multimodal interaction features are then fused to obtain historical trajectory-scene structure interaction features. Spatial interaction weight matrix and scene language weight matrix are generated based on the historical trajectory-scene structure interaction features and the scene language features extracted in step S3, respectively. The obtained historical trajectory-scene structure interaction features are then fused with language weighting to obtain scene-constrained fusion interaction features. S5. Initial noise is collected from the preset noise distribution, and the original historical trajectory and conditional information are used as conditional inputs. The future trajectory multimodal features are generated through conditional denoising. The future trajectory multimodal features are used to generate trajectory latent features and explicit intention latent features. The scene constraint fusion interaction features, trajectory latent features and explicit intention latent features are jointly fused to regress the future trajectory distribution parameters. S6. Output multiple future trajectories and corresponding modal probabilities of the moving target to be predicted based on the future trajectory distribution parameters, and use the multiple future trajectories and corresponding modal probabilities for path planning and driving decisions of intelligent agricultural machinery.

[0008] In step S2, the trajectory error in the historical trajectory sequence is explicitly diffused by dynamic noise scheduling, adaptive step sampling based on the trajectory state change rate, and regularization constraints are applied.

[0009] In step S2, the diffused noise trajectory and condition information are input into the linear trajectory noise feature extraction module for conditional denoising to obtain historical trajectory preprocessing features; the condition information includes the historical trajectory and timestamp information of the moving target to be predicted, as well as one or more of the relative position, relative velocity, relative acceleration and relative heading information between the moving target to be predicted and neighboring moving targets.

[0010] The linear trajectory noise feature extraction module includes a forward sequence feature extraction branch and a reverse sequence feature extraction branch; the forward sequence feature extraction branch and the reverse sequence feature extraction branch respectively adopt a sequence feature extraction unit composed of a self-attention unit, a residual connection, a feedforward neural network and a layer normalization to extract forward time series features and reverse time series features.

[0011] In step S3, based on the historical trajectory sequence, the historical trajectory information of the target to be predicted and its neighboring targets is abstracted into historical trajectory nodes, and the interaction relationship between the target to be predicted and its neighboring targets is abstracted into target-target interaction edges, thus constructing a target-target interaction graph structure based on the historical trajectory sequence; the target to be predicted and scene structure elements are abstracted into target-scene structure nodes, and the interaction relationship between the target to be predicted and scene structure elements is abstracted into target-scene structure interaction edges, thus constructing a target-scene structure interaction graph structure based on the historical trajectory sequence; using the same method as the historical trajectory sequence, a target-target interaction graph structure and a target-scene structure interaction graph structure based on the historical trajectory preprocessing features are constructed.

[0012] In step S3, the scene image obtained in step S1 is image encoded to obtain a visual feature vector, the preset scene descriptive words are text encoded to obtain a text embedding vector, the scene descriptive words are selected according to the similarity between the scene image and the scene descriptive words, and after combination, the scene language features are extracted.

[0013] In step S4, the multimodal node interaction feature extraction module extracts interaction features based on the target-target interaction graph structure and the target-scene interaction graph structure, respectively. The modules for extracting interaction features have the same structure, both including a nearest neighbor search graph convolution branch and a graph self-attention branch. The features extracted by the two branches are added together and linearly transformed to obtain the node interaction output features. The node interaction output features extracted based on the target-target interaction graph structure and the target-scene interaction graph structure are fused into multimodal interaction features. The multimodal interaction features corresponding to the historical trajectory sequence and the historical trajectory preprocessing features are concatenated to obtain the historical trajectory-scene structure interaction features.

[0014] In step S4, after the historical trajectory-scene structure interaction features and scene language features are initially fused, features are extracted from the time dimension and the spatial dimension respectively, and fused into spatial interaction enhancement features. The spatial interaction weight matrix generated based on the spatial interaction enhancement features is added to the scene language weight matrix generated based on the scene language features to obtain a mixed weight matrix. The mixed weight matrix is ​​multiplied element-wise with the spatial interaction enhancement features to obtain a weighted fusion feature, which is then further fused with the scene language features to obtain scene constraint fusion interaction features.

[0015] In step S5, the multimodal features of the future trajectory are input into a multilayer perceptron to obtain the trajectory latent features; the multimodal features of the future trajectory are decomposed into horizontal and vertical components, and horizontal and vertical clustering features are obtained through clustering; the horizontal and vertical clustering features are concatenated and then extracted from the time and spatial dimensions respectively, and fused to obtain the explicit intent latent features; the trajectory latent features and the explicit intent latent features are concatenated along the sequence direction, and then concatenated with the scene constraint fusion interaction features along the channel direction to obtain the trajectory guidance features; the trajectory guidance features are concatenated with the intent guidance features obtained by linear transformation of the explicit intent latent features along the sequence direction to obtain the joint guidance features, and then the multilayer perceptron regresses the future trajectory distribution parameters.

[0016] A multimodal trajectory prediction device for intelligent agricultural machinery path planning includes an environmental sensing component, a data acquisition interface, a multimodal trajectory prediction calculation unit, and an output interface. The environmental sensing component is used to acquire scene images within the work area and state data of the moving target to be predicted and nearby moving targets. The data acquisition interface is used to acquire positioning and target tracking data, historical motion state data, work scene structure information, and traffic constraint information. The multimodal trajectory prediction calculation unit is used to execute the prediction method and output multiple future trajectories of the moving target to be predicted and their corresponding modal probabilities. The output interface is used to send the multiple future trajectories of the moving target to be predicted and their corresponding modal probabilities to the path planning module or safety decision module of the intelligent agricultural machinery.

[0017] The beneficial effects of this invention are: 1. This invention, through explicit modeling and conditional denoising of trajectory errors in the historical motion state data of the moving target to be predicted, can reduce the impact of positioning errors, environmental perception errors, target tracking errors, and factors such as field bumps, dust, and occlusion on the quality of historical trajectories, thereby improving the stability and reliability of input trajectory features. Furthermore, the original historical trajectory and the preprocessed historical trajectory features obtained through diffusion and denoising have complementary effects. The original historical trajectory can preserve target motion changes and local motion details, while the preprocessed historical trajectory features can reduce the influence of positioning, perception, and target tracking errors. By extracting the interaction features of both separately and then fusing them, motion information can be preserved simultaneously while reducing the impact of input errors on the prediction results.

[0018] 2. This invention further constructs a historical trajectory error modeling process through dynamic noise scheduling, adaptive step sampling, and noise regularization, and combines it with a linear trajectory noise feature extraction module for conditional denoising, which helps to improve the preprocessing effect of historical trajectories and takes into account the computational efficiency when deploying on the vehicle platform.

[0019] 3. This invention extracts scene language features such as weather conditions, time information, plot type, crop row direction, field area status, operation density, surface attachment status, and transfer road status by calculating the image-text similarity between scene images and a preset set of scene descriptive words. This enables the trajectory prediction process to utilize high-level semantic information in agricultural machinery operation scenarios.

[0020] 4. This invention abstracts the moving target to be predicted, neighboring moving targets, and structural elements of the work scene into nodes, and the target-target interaction relationship and the target-scene structure interaction relationship into edges. It employs a parallel extraction of interaction features using nearest neighbor search graph convolutional branches and graph self-attention branches. This allows it to simultaneously express the dynamic interaction relationships between moving targets and the constraints between moving targets and scene structures such as plot boundaries, work row centerlines, field turning areas, field entrances / exits, and restricted areas. Extracting features from the two types of graph structures separately before fusing them avoids information interference caused by directly mixing different types of interaction relationships. Simultaneously, it enables the prediction model to consider both dynamic target interactions and scene structure constraints. This solves the problems of insufficient utilization of interaction relationships between moving targets and between moving targets and the work scene structure, as well as the inadequate fusion of scene language information and traffic constraint information.

[0021] 5. This invention uses language-guided fusion of historical trajectory-scene structure interaction features containing traffic constraint information and scene language features to enable prediction results to better conform to equipment passage constraints, operation direction rules, safety distance thresholds, and operation line constraints, thereby reducing the risk of predicted trajectories deviating from the actual passable area.

[0022] Furthermore, scene language features can be fused in stages. The first stage initially fuses scene language features with historical trajectory-scene structure interaction features, so that the interaction features contain the semantic information of the current scene. The second stage generates scene language weights and spatial interaction weights respectively, and adaptively weights the spatial interaction enhancement features through mixed weights to enhance the interaction information related to the current scene and suppress features with low relevance. The third stage introduces scene language residual features again to preserve the original scene semantic information and reduce the impact of language information attenuation during multi-level feature transformation and weighting.

[0023] 6. This invention, through the joint guidance of trajectory latent features and explicit intention latent features, can express different future movement possibilities such as maintaining the current work path, switching to adjacent work paths, turning at the end of the field, moving between sites, and working at a constant speed, accelerating, decelerating, and stopping, thereby improving the ability to express multimodal future behaviors. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the structure and method flow of the prediction device of the present invention.

[0025] Figure 2 This is a schematic diagram of the linear trajectory noise feature extraction module of the present invention.

[0026] Figure 3 This is a schematic diagram of the linear feature extraction module that considers spatial interaction in this invention.

[0027] Figure 4 This is a schematic diagram of the multimodal node interaction feature extraction module of the present invention.

[0028] Figure 5 This is a schematic diagram of the language-historical trajectory-scene structure interaction feature fusion module of this invention.

[0029] The diagram is labeled as follows: 100, Autonomous Operation System; 101, Environmental Sensing Component; 102, Data Acquisition Interface; 103, Multimodal Trajectory Prediction and Calculation Unit; 104, Path Planning Module; 105, Safety Decision Module; 106, Autonomous Operation Control Module. Detailed Implementation

[0030] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. The specific contents listed in the embodiments below are not limited to the technical features necessary to solve the technical problem. Furthermore, the listed embodiments are only a part of the present invention, and not all of the embodiments.

[0031] like Figure 1 As shown, the multimodal trajectory prediction device of the present invention is integrated into the autonomous operation system 100 of the host intelligent agricultural machinery, including an environmental perception component 101, a data acquisition interface 102, a multimodal trajectory prediction calculation unit 103 and an output interface, and provides future trajectory prediction results for decision control to the path planning module 104, the safety decision module 105 and the autonomous operation control module 106 in the autonomous operation system 100.

[0032] Unless otherwise specified, the host intelligent agricultural machinery refers to intelligent agricultural machinery equipped with the multimodal trajectory prediction method or device of the present invention, and using the trajectory prediction results for path planning, safety decision-making or autonomous operation control. It can be an intelligent tractor, harvester, plant protection machine, field transport vehicle or other unmanned agricultural machinery platform.

[0033] The environmental perception component 101 is used to acquire scene images of the work area, as well as target status data such as the position, speed, type, and size of the moving target to be predicted and nearby moving targets. The environmental perception component 101 may include a camera, lidar, millimeter-wave radar, integrated navigation device, or other sensors capable of acquiring environmental information and target status information of the work area.

[0034] The moving target to be predicted refers to a moving object located within the operating area of ​​the host intelligent agricultural machinery whose future trajectory needs to be predicted. This object can be other agricultural machinery, field transport vehicles, workers, animals, or other dynamic obstacles. The neighboring moving target refers to a moving object that has a spatial proximity relationship, a relative motion relationship, or a potential interaction relationship with the moving target to be predicted.

[0035] The data acquisition interface 102 is used to acquire positioning and target tracking data, historical motion state data of the moving target to be predicted and nearby moving targets, operational scene structure information, and traffic constraint information. The data acquisition interface 102 may include one or more data input channels, one type of data input channel is used to acquire positioning and target tracking data and historical motion state data, and another type of data input channel is used to acquire operational scene structure information and traffic constraint information.

[0036] The operational scenario structure information describes scenario structure elements such as plot boundaries, work row centerlines, field turning areas, field entrances and exits, field roads, ditch boundaries, obstacle areas, and restricted areas. The traffic constraint information describes constraints related to agricultural machinery traffic and operations, such as machinery passage width, minimum turning radius, work direction rules, work zone overlap constraints, and safety distance thresholds.

[0037] The multimodal trajectory prediction calculation unit 103 executes the prediction method-related program and outputs multiple future trajectories of the moving target to be predicted and their corresponding modal probabilities. The output interface sends the multiple future trajectories of the moving target to be predicted and their corresponding modal probabilities to the path planning module or safety decision module of the intelligent agricultural machinery. The path planning module 104 performs local path adjustment, obstacle avoidance path generation, or speed planning based on the multiple future trajectories and their corresponding modal probabilities. The safety decision module 105 performs risk assessment based on the multiple future trajectories and their corresponding modal probabilities and generates warning, speed limit, avoidance, or stopping decisions. The autonomous operation control module 106 generates control commands based on the output of the path planning module 104 or the safety decision module 105 and executes autonomous operation.

[0038] The multimodal trajectory prediction method for intelligent agricultural machinery path planning of the present invention includes the following steps: S1. Obtain historical motion state data of the moving target to be predicted and nearby moving targets, as well as scene images of the work area, work scene structure information, and traffic constraint information.

[0039] The historical motion state data includes at least one or more of the following: position, velocity, acceleration, heading angle, timestamp, vehicle dimensions, dimensions of associated equipment, and target category. All acquired data is standardized to the same time reference and transformed to the same coordinate system.

[0040] In a preferred embodiment, the historical motion state data of the target to be predicted and neighboring targets in the host intelligent agricultural machinery coordinate system or the unified coordinate system of the working area are represented as follows:

[0041] in, This represents the set of historical motion state data for all targets. This represents the historical motion state data of the i-th target. Let represent the position, velocity, acceleration, and heading angle of the i-th moving target at historical time step t, respectively. This indicates the dimensions of the vehicle body and associated equipment. Indicates the target category, Indicates the corresponding time identifier. Indicates the target quantity. Indicates the length of the historical time window.

[0042] S2. Construct a historical trajectory sequence from the historical motion state data of the target to be predicted and neighboring targets. Perform explicit diffusion and conditional denoising on the trajectory errors in the historical trajectory sequence. Add noise to the historical trajectory sequence through explicit diffusion and remove noise through conditional denoising guided by conditional information. Reduce the impact of factors such as positioning drift and field bumps on the quality of historical trajectories and obtain the preprocessed features of historical trajectories.

[0043] In the explicit diffusion process, dynamic noise scheduling is used to dynamically set the noise addition ratio, adaptive step sampling is performed according to the trajectory state change rate, and regularization constraints are applied to explicitly diffuse the trajectory error in the historical trajectory sequence.

[0044] The conditional denoising process is completed using a linear trajectory noise feature extraction module. In one embodiment, the linear trajectory noise feature extraction module includes a forward sequence feature extraction branch and a reverse sequence feature extraction branch; the forward sequence feature extraction branch and the reverse sequence feature extraction branch respectively employ a sequence feature extraction unit composed of a self-attention unit, residual connection, feedforward neural network, and layer normalization to extract forward time series features and reverse time series features. The diffused noise trajectory and conditional information are respectively input into the linear trajectory noise feature extraction module, which extracts features from the noise trajectory and conditional information respectively. The features extracted from the two paths are concatenated, and the preprocessed historical trajectory features after conditional denoising are obtained through a multilayer perceptron.

[0045] in, This represents the noise trajectory after diffusion. The first part represents the historical trajectory preprocessing features, the second part represents the linear trajectory noise feature extraction module, the third part represents the feature concatenation, and the fourth part represents the multilayer perceptron.

[0046] S3. Construct corresponding target-target interaction graph structures based on historical trajectory sequences and historical trajectory preprocessing features respectively; construct corresponding target-scene structure interaction graph structures based on historical trajectory sequences and historical trajectory preprocessing features, combined with comprehensive scene structure information including scene structure information and traffic constraint information; extract scene language features based on the scene images obtained in step S1 and the preset scene descriptive word set.

[0047] Construction of the interaction graph structure: To illustrate how the interaction graph structure is constructed, the spatial interaction relationship between the moving target to be predicted and its neighboring targets is represented as follows:

[0048] in, Indicates relative position, Represents relative velocity. N represents relative acceleration. neighbors Indicates the number of nearby targets.

[0049] In intelligent agricultural machinery scenarios, scene structure information and traffic constraint information are used to represent plots of land, work rows, and traffic rules. The comprehensive scene structure information can be represented as follows:

[0050] in, It represents geometric structure information such as plot boundaries, work row centerlines, field edges, or field roads. Indicates the structure number. It indicates traffic rules such as working direction, equipment passage constraints, restricted areas, and safe distances.

[0051] The interaction features between the moving target to be predicted and the scene structure can be represented as:

[0052] in, This represents the distance between the moving target to be predicted and the scene structure nodes. Indicates the deviation in heading angle. Indicates lateral deviation. This indicates the job row or traffic rule to which the moving target to be predicted belongs.

[0053] Taking historical trajectory sequences as an example, the construction method of their interaction graph structure is as follows: For target-target interaction graph structure The historical trajectory information of the target to be predicted and its neighboring targets is abstracted into historical trajectory nodes. The interaction relationships between the target to be predicted and its neighboring targets, such as relative position, relative velocity, and relative acceleration, are abstracted into target-target interaction edges. Based on these interaction relationships, a corresponding cost matrix is ​​constructed to form a target-target interaction graph structure of the historical trajectory sequence. .

[0054] For the target-scene structure interaction graph structure The predicted moving target and scene structural elements are abstracted into target-scene structural nodes. The interaction relationships between the predicted moving target and scene structural elements such as plot boundaries, work row centerlines, field turning areas, field entrances and exits, field roads, obstacle areas, and restricted areas are abstracted into target-scene structural interaction edges. Based on the interaction relationships, a corresponding cost matrix is ​​constructed to form a target-scene structural interaction graph structure of the historical trajectory sequence. .

[0055] For the historical trajectory preprocessing features after explicit error modeling and conditional denoising, the same method as for the historical trajectory sequence is used to construct the corresponding target-target interaction graph structure and target-scene interaction graph structure. .

[0056] Scene language features: These are features obtained through image-text similarity between scene images and preset scene descriptors, used to characterize scene semantic attributes such as weather conditions, time information, plot type, crop row direction, field area status, operation density, surface adhesion status, and transition road status. Scene language features are obtained based on image-text pair encoding. The preset scene descriptor set includes different attribute categories such as weather conditions, time information, plot type, crop row direction, field area status, operation density, surface adhesion status, and transition road status. The scene images obtained in step S1 are image-encoded using an image encoding network to obtain visual feature vectors; the preset scene descriptors are text-encoded using a text encoding network to obtain text embedding vectors for each scene descriptor. Scene descriptors are selected based on the similarity between the scene image and the scene descriptor. The similarity between the scene image and the i-th scene descriptor can be expressed as:

[0057] Among them, T i Let I represent the text embedding vector of the i-th scene descriptor. j A visual feature vector representing a scene image.

[0058] Scene descriptive words that meet preset similarity conditions are selected from different scene attribute categories to obtain R scene descriptive word embedding vectors E1, E2, ..., ER The scene descriptor embedding vectors are concatenated to obtain a two-dimensional sequence, and high-dimensional scene language features are extracted from it. :

[0059] SBiS6 is a feature extraction module. In a preferred embodiment, SBiS6 employs a linear feature extraction module that considers spatial interactions, extracting features from both dimensions of the two-dimensional sequence. For the two-dimensional sequence of scene descriptor embedding vectors, it extracts the contextual relationships between different scene attribute categories from the first dimension, and simultaneously extracts the interaction relationships between descriptor feature groups along the second dimension, then fuses the two sets of features. This method generates richer and more expressive high-dimensional scene language features. .

[0060] S4. A parameter-sharing multimodal node interaction feature extraction module is used to extract multimodal interaction features from the interaction graph structure constructed based on historical trajectory sequences and the interaction graph structure constructed based on preprocessed features of historical trajectories, respectively. The extracted multimodal interaction features are then fused to obtain the historical trajectory-scene structure interaction features. Taking historical trajectory sequences as an example, the multimodal node interaction feature extraction module extracts features from both the target-target interaction graph structure and the target-scene structure interaction graph structure, and then fuses the features extracted from the two interaction graphs.

[0061] like Figure 4 As shown, This represents the spatial interaction relationship between the target and its neighboring targets, where x0 represents the set of historical motion state data. This represents comprehensive scene structure information. The interaction features between the target and the scene structure are represented by SI-Edge, Cost Matrix, HT-Node, TM-Node, MI-Edge, and ASIM (Multimodal Node Interaction Feature Extraction Module). A target-target interaction graph structure is constructed from historical trajectory nodes and target-target interaction edges, and then fed into a multimodal node interaction feature extraction module for feature extraction. A target-scene structure interaction graph structure constructed from target-scene structure nodes and target-scene structure interaction edges is fed into another multimodal node interaction feature extraction module for feature extraction. Both interaction graph structures employ a dual-branch feature extraction module.

[0062] The dual-branch feature extraction module includes a nearest neighbor search graph convolution branch and a graph self-attention branch. The nearest neighbor search graph convolution branch determines the neighboring nodes of each node based on the K-nearest neighbor (KNN) search results, and sequentially inputs the node features into two graph convolution units and batch normalization and activation units to obtain the first node interaction feature F. NO1 :

[0063] in, Representing node characteristics, Let GCN represent the node adjacency matrix obtained through K-nearest neighbor search, BN represent graph convolutional network, BN represent batch normalization, and ReLU represent nonlinear activation function.

[0064] The graph attention branch sequentially inputs node features into the edge convolution unit, the graph attention unit, and the batch normalization and activation unit to obtain the second node interaction feature F. NO2 .

[0065]

[0066] Where GAT stands for Graph Attention Network and EdgeConv stands for Edge Convolution.

[0067] For the target-target interaction graph structure, the two node interaction features extracted by the dual-branch feature extraction module are added together and subjected to a linear transformation to obtain the node interaction output feature F. G1 :

[0068] For the target-scene interaction graph structure, the same structure is used to obtain the interaction graph structure features F. G2 .

[0069] Output feature F from node interactions G1 and interaction graph structural features F G2 By fusing the data, we can obtain the multimodal interaction features F corresponding to the historical trajectory sequences. GC1 :

[0070] For the preprocessed features of historical trajectories after explicit error modeling and conditional denoising, a multimodal node interaction feature extraction module that shares parameters with the historical trajectory sequence is used for feature extraction, and the constructed interaction graph structure is used. Input the same multimodal node interaction feature extraction module to obtain the multimodal interaction features F corresponding to the historical trajectory preprocessing features. GC2 .

[0071] The multimodal interaction features F corresponding to the original historical trajectory sequenceGC1 The multimodal interaction features F corresponding to the historical trajectory preprocessing features GC2 By stitching the data together, we can obtain the historical trajectory-scene structure interaction features. :

[0072] Then, the historical trajectory-scene structure interaction features and scene language features are input into the language-historical trajectory-scene structure interaction feature fusion module. For example... Figure 5 As shown, in this module, the interaction features of historical trajectory and scene structure are first analyzed. and scene language features Perform layer normalization and linear transformation, and then concatenate the transformed two feature streams to obtain the preliminary fused feature F. Concat :

[0073] Then, the feature extraction module is used to process the initial fused features F. Concat Feature extraction is performed. The structure of the feature extraction module can be the same as that of the SBiS6 linear feature extraction module that considers spatial interaction, used when extracting scene language features. The SBiS6 linear feature extraction module considering spatial interaction is used to extract F... Concat Further feature extraction is performed from both the temporal and spatial dimensions to obtain spatial interaction enhancement features.

[0074] Furthermore, the spatial interaction enhancement feature is sequentially input into linear transformation, layer normalization, and activation units to obtain the spatial interaction weight matrix. .

[0075] Scene language features By sequentially inputting the linear transformation unit and the normalization unit, the scene language weight matrix is ​​obtained. .

[0076] The above spatial interaction weight matrix and scene language weight matrix The weights are added together to obtain a hybrid weight matrix; the hybrid weight matrix is ​​then combined with the spatial interaction enhancement feature. Perform element-wise multiplication to obtain the weighted fusion feature. ,in, This indicates an element-wise multiplication operation.

[0077] Furthermore, a linear transformation is performed on the scene language features to obtain scene language residual features; the scene language residual features are added to the weighted fusion features, and a linear transformation is then performed to obtain the scene constraint fusion interaction features F.S2 :

[0078] S5. For different prediction modes, initial noise is collected from a preset noise distribution as noise input, and the original historical trajectory and conditional information are used as conditional input. After conditional denoising, multimodal features of the future trajectory are generated. :

[0079] in, Indicates noise input, x0 represents the original historical trajectory, c d This indicates the conditional information. BiS6 is the linear trajectory noise feature extraction module mentioned above, and SBiS6 is the linear feature extraction module that considers spatial interaction mentioned above.

[0080] Multimodal features of future trajectories The input trajectory-intent joint guidance module enhances guidance through trajectory features and intent features, integrating them with scene constraints and interactive features.

[0081] First, the multimodal features of future trajectories Inputting the data into a multilayer perceptron and mapping it to the trajectory latent space yields the trajectory latent features. .

[0082] Furthermore, the multimodal characteristics of future trajectories The components are decomposed into vertical components (parallel to the target's motion direction) and horizontal components (perpendicular to the target's motion direction). K-Means clustering is then performed on each component to obtain the horizontal clustering features. and vertical clustering features The horizontal and vertical clustering features are concatenated along the channel dimension, and then features are extracted from the time and spatial dimensions respectively. The feature extraction module used can be the aforementioned linear feature extraction module SBiS6, which considers spatial interaction. The explicit intent latent features are obtained through the linear feature extraction module SBiS6, which considers spatial interaction. Among them, the lateral intention includes at least one of maintaining the current work path, switching to an adjacent work path, turning at the end of the field, and moving to another site; the longitudinal intention includes at least one of maintaining a constant speed, accelerating, decelerating, and stopping.

[0083] The trajectory-intent joint guidance module includes a trajectory feature enhancement guidance branch and an intent feature enhancement guidance branch. The trajectory feature enhancement guidance branch concatenates the latent trajectory features and the latent explicit intent features along the sequence direction and generates trajectory-intent joint guidance features through a linear transformation. Where Secat represents feature concatenation along the sequence dimension. The scene constraints are fused with the interaction features F. S2 Trajectory-Intent Joint Guidance Features By splicing along the channel direction, the trajectory guidance feature F is obtained. S3 :

[0084] Intent feature enhancement guiding branch uses linear transformation to enhance the latent features of explicit intent. Adjust the dimensions to obtain intent-guided features. ; guide the trajectory feature F S3 and intention-guided features By splicing along the sequence direction, the joint guiding feature F is obtained. S4 :

[0085] Subsequently, the joint guiding features are sequentially input into the linear transformation unit, two linear trajectory noise feature extraction modules (BiS6), and the multilayer perceptron to regress the future trajectory distribution parameters of the Gaussian mixture model:

[0086] in, It is mode k m At time step t hp The mean vector represents the center of the trajectory; It is mode k m The covariance matrix; It is mode k m The mixed weights represent modal probabilities and satisfy normalization constraints.

[0087] S6. Output multiple future trajectories and corresponding modal probabilities of the moving target to be predicted based on the future trajectory distribution parameters, and use the multiple future trajectories and corresponding modal probabilities for path planning and driving decisions of intelligent agricultural machinery.

[0088] To facilitate direct use by the path planning module and the safety decision module, for the k-th... m For each prediction mode, the mean vector of each prediction time step is used to construct a deterministic representative trajectory for that prediction mode. :

[0089] The final output of the multimodal future trajectory and its probability set can be represented as:

[0090] The multimodal future trajectory and its probability set are sent to the path planning module or the safety decision module, and path planning and driving decisions are made based on the predicted future trajectory of the moving target.

[0091] Example 1:

[0092] This embodiment provides a detailed description of the explicit diffusion and conditional denoising process in step S2.

[0093] 1. Explicit diffusion modeling The propagation process of historical trajectory errors can be represented as:

[0094] in, This represents the trajectory data after the t-th diffusion step. This represents the trajectory data from the previous diffusion step. Used to control the noise ratio, where I is the unit covariance matrix. Representing a Gaussian distribution, the trajectory data includes the target's position, velocity, acceleration, timestamp, size, category, number, etc.

[0095] To improve the efficiency of trajectory error propagation modeling, dynamic noise scheduling and adaptive step sampling can be used: Dynamic noise scheduling can be expressed as:

[0096] in, Represents the dynamic noise figure at step t. This represents the total number of diffusion steps. The nonlinear fluctuation intensity is represented by Linear(·), which represents the linear growth term. The proportion of noise added in each diffusion step is dynamically adjusted through dynamic noise scheduling.

[0097] Adaptive step sampling determines the step size of the diffusion process based on the trajectory state change rate, which is expressed as:

[0098] in, This is used to prevent the denominator from being zero.

[0099] Then update the step size based on this rate of change:

[0100] in, and These represent the minimum and maximum step sizes, respectively. It is a dynamic adjustment coefficient; It is a reference threshold for changes in trajectory state.

[0101] To reduce the impact of noise on spatial location and modal distribution, a noise regularization constraint is introduced:

[0102] in, and These represent the theoretical mean and theoretical covariance, respectively.

[0103] By using dynamic noise scheduling, adaptive step sampling based on the trajectory state change rate, and applying regularization constraints, the trajectory error in the historical trajectory sequence is explicitly diffused.

[0104] 2. Conditional noise reduction The denoising process can be viewed as the inverse of the diffusion process. The conditional denoising process, after introducing conditional information, can be represented as:

[0105] in, and Let represent the conditional mean and conditional covariance matrix predicted by the conditional denoising model at diffusion time step t, respectively. This indicates conditional information, including the historical trajectory of the moving target to be predicted. Timestamp information Relative state information between the moving target to be predicted and neighboring targets Information such as relative position, relative speed, relative acceleration, and relative heading.

[0106] This invention utilizes a linear trajectory noise feature extraction module to extract features from the diffused noise trajectory and conditional information, and uses a multilayer perceptron to predict the denoised trajectory to obtain preprocessed features of historical trajectories.

[0107] Example 2:

[0108] In this embodiment, the linear trajectory noise feature extraction module BiS6 is based on a scaled dot product self-attention mechanism, extracting sequence features in both forward and reverse directions, and combining residual connections, feedforward neural networks, and layer normalization mechanisms to achieve feature fusion. The construction of the linear trajectory noise feature extraction module is described in detail below: First, a forward sequence feature extraction unit S6 is constructed. Let the input sequence be... Where L represents the sequence length and d represents the feature dimension. The query matrix (Q), key matrix (K), and value matrix (V) are obtained through linear mapping, as follows: Therefore, the scaled dot product self-attention output can be expressed as: .in, These represent linear mapping matrices for the query, key, and value, respectively. The key vector dimension is represented by Att(·), which represents the scaled dot product self-attention operation.

[0109] Based on the self-attention output, residual connections, feedforward neural networks, and layer normalization are further introduced to obtain the positive sequence feature extraction unit S6:

[0110]

[0111] Here, FFN represents a feedforward neural network, and LN represents layer normalization.

[0112] Inverse sequence feature extraction unit S6 - It adopts the same structure as the forward sequence feature extraction unit S6, but performs feature extraction in the opposite sequence direction. For input feature F, the reverse sequence feature extraction process can be represented as: S6 - (F) = Reverse(S6(Reverse(F))). Here, Reverse indicates that the sequence is reversed.

[0113] like Figure 2 As shown, the linear trajectory noise feature extraction module first processes the input features... Depthwise separable convolution and SiLU activation are performed, and the processed features are input into the forward sequence feature extraction branch. The features then pass through the forward sequence feature extraction unit, layer normalization, and linear transformation, and are finally fused with the input features using residual fusion to obtain the forward sequence features. :

[0114] in, This represents depthwise separable convolution. For activation function, This represents a forward sequence feature extraction unit. Representation layer normalization, It represents a linear transformation.

[0115] Subsequently, the positive sequence features The input is processed in the reverse sequence feature extraction branch in the opposite sequence direction, and then sequentially passes through SiLU activation and the reverse sequence feature extraction unit S6. - Layer normalization and linear transformation, and their correlation with positive sequence features. Perform residual fusion to obtain inversely enhanced sequence features. :

[0116] Finally, input features and reverse enhancement sequence features Feature concatenation is performed, followed by layer normalization and linear transformation to obtain the output features of the linear trajectory noise feature extraction module. As shown below:

[0117] Concat is a feature concatenation operation along the channel dimension.

[0118] Example 3:

[0119] In this embodiment, the linear feature extraction module SBiS6, which considers spatial interaction, is built based on the aforementioned linear trajectory noise feature extraction module BiS6. It includes two-dimensional feature extraction branches, the meaning of which differs depending on the input data. Using historical trajectory data... For example, the two dimensional branches are the time dimension feature extraction branch and the spatial dimension feature extraction branch. Figure 3 As shown, input historical trajectory data. First, we expand along the time dimension to obtain temporal features. Simultaneously, the spatial interaction characteristics are obtained by expanding along the dimension of the number of moving targets. .

[0120] Finally, the temporal and spatial interaction features are added and fused together, and the fused features are then combined with the input features through channel concatenation, layer normalization, and linear transformation to obtain fused features that take into account spatiotemporal information. As shown below:

[0121] Example 4:

[0122] Regarding the training, deployment, and application of the aforementioned models, the training samples are collected by intelligent agricultural machinery in scenarios such as straight-line operations, turning at the edge of the field, entering and exiting the field, switching between work rows, inter-field transfer, and multi-person / multi-machine collaborative operations. After time synchronization, coordinate unification, target labeling, and trajectory smoothing, the training data forms historical trajectory sequences, scene images, scene structure information, traffic constraint information, and future real trajectory labels.

[0123] The future real-world trajectory is not used as input during the inference phase, but only for supervision and loss calculation during the training phase. The training phase can utilize historical trajectories for preprocessing the loss. Trajectory regression loss Modal probability loss And intention classification loss The total loss function can be expressed as: Wherein, λ1, λ2, λ3, and λ4 represent the weight coefficients of the historical trajectory preprocessing loss, trajectory regression loss, modality probability loss, and intent classification loss, respectively. These weight coefficients can be pre-set or adaptively adjusted based on the training task, sample distribution, and model convergence.

[0124] The historical trajectory preprocessing loss measures the difference between the preprocessed historical trajectory obtained after conditional denoising and the corresponding reference historical trajectory, constraining the historical trajectory reconstruction results during the conditional denoising process. The trajectory regression loss first selects the mode with the smallest distance from the true future trajectory from multiple predicted modes, then uses Huber loss to calculate the regression error between this mode and the true trajectory, constraining the deviation between the predicted trajectory and the future true trajectory. The modality probability loss uses the negative log-likelihood of a Gaussian mixture model, calculated based on the likelihood of the true future trajectory under the probability distribution corresponding to each predicted mode, constraining the probability distribution corresponding to different predicted modes. The intent classification loss measures the difference between the motion intent corresponding to the predicted trajectory and the motion intent corresponding to the true future trajectory, constraining the classification results of lateral and longitudinal intents. All loss terms are weighted and summed according to preset weights to obtain the total model training loss.

[0125] During online deployment, the multimodal trajectory prediction calculation unit 103 receives sensor data and target status update information provided by the environmental perception component 101 and the data acquisition interface 102 at fixed intervals. It then sequentially performs historical trajectory construction, explicit error diffusion modeling, conditional denoising, scene language feature extraction, multimodal node interaction feature fusion, language guidance fusion, trajectory-intention joint guidance, and multimodal trajectory regression. The output results are sent to the path planning module 104 or the safety decision module 105, and the autonomous operation control module 106 executes the corresponding control commands.

[0126] In field turning scenarios, when a nearby moving target approaches the turning area, the field area and work row nodes in the scene structure nodes enhance the constraint on trajectory regression, and the explicit intent latent features tend to produce a combination of field turning and deceleration avoidance. In field entrance / exit or inter-field transfer scenarios, entrance / exit boundaries, field road conditions, restricted areas, and transfer driving intentions have a stronger impact on trajectory mode probabilities.

[0127] Example 5:

[0128] This invention also provides a multimodal trajectory prediction device for intelligent agricultural machinery path planning. The device can be installed in a host intelligent agricultural machinery autonomous operation system 100, and includes an environmental perception component 101, a data acquisition interface 102, a multimodal trajectory prediction calculation unit 103, and an output interface. The data acquisition interface 102 is used to acquire positioning and target tracking data, historical motion state data, operation scene structure information, and traffic constraint information. The data acquisition interface 102 can be connected to the host intelligent agricultural machinery's environmental perception module, positioning and target tracking module, map service module, or operation management platform via a vehicle communication bus, Ethernet, serial communication interface, wireless communication interface, or shared storage interface.

[0129] The multimodal trajectory prediction calculation unit 103 includes a trajectory sequence construction module, a trajectory sequence preprocessing module, a scene language feature extraction module, a multimodal node interaction feature extraction module, a language-historical trajectory-scene structure interaction feature fusion module, a trajectory-intent joint guidance module, and a trajectory output module.

[0130] The trajectory sequence construction module is used to construct historical trajectory sequences of the moving target to be predicted and nearby moving targets based on positioning and target tracking data.

[0131] The trajectory sequence preprocessing module is used to perform explicit diffusion modeling of trajectory errors, conditional denoising, and linear trajectory noise feature extraction to obtain historical trajectory preprocessing features.

[0132] The scene language feature extraction module is used to calculate the similarity between the visual features of the scene image and the text embedding vector of the preset scene descriptive words, and to obtain scene language features that characterize the semantic attributes of the current work scene.

[0133] The multimodal node interaction feature extraction module is used to construct the target-target interaction graph structure and the target-scene interaction graph structure, and extracts the historical trajectory-scene structure interaction features corresponding to the original historical trajectory and the historical trajectory preprocessing features through the nearest neighbor search graph convolution branch and the graph self-attention branch.

[0134] The language-historical trajectory-scene structure interaction feature fusion module is used to generate a scene language weight matrix and a spatial interaction weight matrix based on scene language features and historical trajectory-scene structure interaction features, and to obtain scene constraint fusion interaction features.

[0135] The trajectory-intent joint guidance module is used to generate trajectory latent features and explicit intent latent features, and to jointly fuse scene constraint fusion interaction features, trajectory latent features and explicit intent latent features.

[0136] The trajectory output module is used to regress the future trajectory distribution parameters corresponding to multiple prediction modes, and output multiple future trajectories of the moving target to be predicted and their corresponding mode probabilities.

[0137] The output interface is used to send the multiple future trajectories and their corresponding modal probabilities to the path planning module 104 or the safety decision module 105. The path planning module 104 and the safety decision module 105 generate path planning results or safety decision results based on the trajectory prediction results, and provide the corresponding results to the autonomous operation control module 106.

[0138] To verify the effectiveness of the method, ablation experiments were conducted on the publicly available NGSIM vehicle trajectory dataset. Under the same dataset and evaluation conditions, ablation experiments were performed by removing different functional modules from the present invention. These experiments are used to verify the technical effects of the core data processing procedures in this invention, such as historical trajectory error, multimodal interaction features, scene language feature fusion, and trajectory-intent joint guidance, and are not intended to limit the specific application scenarios of this invention.

[0139] The experimental results are shown in Table 1. minADE and minFDE represent the minimum average displacement error and minimum final displacement error in multimodal prediction, respectively, and minMR represents the miss rate of multimodal prediction.

[0140]

[0141] The process includes several modules: **Track Sequence Removal Module:** This module removes historical trajectory diffusion modeling and conditional denoising, directly using the historical trajectory sequence for subsequent processing. **Scene Language Feature Removal Module:** This module removes operations related to scene language feature fusion, retaining all other operations. **Multimodal Node Interaction Feature Extraction Module:** This module removes interaction graph construction and multimodal interaction feature extraction operations, using only a simple channel summation method to fuse the basic features of the historical trajectory sequence and its preprocessed features before feeding them into subsequent modules. **Track-Intent Joint Guidance Module:** This module removes the operation of acquiring potential trajectory features and fusing them with explicit intent potential features, decomposing the future trajectory multimodal features to obtain horizontal and vertical explicit intent potential features, then fusing these with scene constraint interaction features to regress the future trajectory. The indicators corresponding to the ablation experiment rows represent relative change values.

[0142] The above description of specific embodiments is only for the purpose of helping to understand the technical concept and core idea of ​​the present invention. Although specific preferred embodiments have been used to describe and illustrate the technical solutions, they should not be construed as limiting the present invention itself. Those skilled in the art can make various changes in form and detail without departing from the technical concept of the present invention. These easily conceived changes or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A multimodal trajectory prediction method for intelligent agricultural machinery path planning, which acquires scene images and state data of moving targets within the work area through environmental sensing components, and performs path planning and driving decisions based on the predicted future trajectory of the moving targets, characterized in that: Includes the following steps: S1. Obtain historical motion state data of the moving target to be predicted and nearby moving targets, as well as scene images, scene structure information and traffic constraint information of the work area; S2. Construct a historical trajectory sequence from the historical motion state data of the target to be predicted and neighboring moving targets, and perform explicit diffusion and conditional denoising on the trajectory error in the historical trajectory sequence to obtain the historical trajectory preprocessing features; S3. Construct corresponding target-target interaction graph structures based on historical trajectory sequences and historical trajectory preprocessing features respectively; construct corresponding target-scene structure interaction graph structures based on historical trajectory sequences and historical trajectory preprocessing features, combined with comprehensive scene structure information including scene structure information and traffic constraint information; extract scene language features based on the scene images obtained in step S1 and the preset scene descriptive word set; S4. A parameter-sharing multimodal node interaction feature extraction module is used to extract multimodal interaction features from the interaction graph structure constructed based on historical trajectory sequences and the interaction graph structure constructed based on historical trajectory preprocessing features, respectively. The extracted multimodal interaction features are then fused to obtain historical trajectory-scene structure interaction features. Based on the historical trajectory-scene structure interaction features and the scene language features extracted in step S3, a spatial interaction weight matrix and a scene language weight matrix are generated respectively. The obtained historical trajectory-scene structure interaction features are then fused with language weighting to obtain scene constraint fusion interaction features. S5. Collect initial noise from the preset noise distribution, and use the original historical trajectory and conditional information as conditional inputs to generate multimodal features of the future trajectory through conditional denoising; The future trajectory multimodal features are used to generate trajectory latent features and explicit intent latent features. The scene constraint fusion interaction features, trajectory latent features and explicit intent latent features are jointly fused to regress the future trajectory distribution parameters. S6. Output multiple future trajectories and corresponding modal probabilities of the moving target to be predicted based on the future trajectory distribution parameters, and use the multiple future trajectories and corresponding modal probabilities for path planning and driving decisions of intelligent agricultural machinery.

2. The multimodal trajectory prediction method for intelligent agricultural machinery path planning as described in claim 1, characterized in that: In step S2, the trajectory error in the historical trajectory sequence is explicitly diffused by dynamic noise scheduling, adaptive step sampling based on the trajectory state change rate, and regularization constraints are applied.

3. The multimodal trajectory prediction method for intelligent agricultural machinery path planning as described in claim 1, characterized in that: In step S2, the diffused noise trajectory and condition information are input into the linear trajectory noise feature extraction module for conditional denoising to obtain historical trajectory preprocessing features; the condition information includes the historical trajectory and timestamp information of the moving target to be predicted, as well as one or more of the relative position, relative velocity, relative acceleration and relative heading information between the moving target to be predicted and neighboring moving targets.

4. The multimodal trajectory prediction method for intelligent agricultural machinery path planning as described in claim 3, characterized in that: The linear trajectory noise feature extraction module includes a forward sequence feature extraction branch and a reverse sequence feature extraction branch; The forward sequence feature extraction branch and the reverse sequence feature extraction branch respectively employ a sequence feature extraction unit composed of a self-attention unit, a residual connection, a feedforward neural network, and a layer normalization to extract forward time series features and reverse time series features.

5. The multimodal trajectory prediction method for intelligent agricultural machinery path planning as described in claim 1, characterized in that: In step S3, based on the historical trajectory sequence, the historical trajectory information of the target to be predicted and the neighboring targets is abstracted into historical trajectory nodes, and the interaction relationship between the target to be predicted and the neighboring targets is abstracted into target-target interaction edges, thus constructing a target-target interaction graph structure based on the historical trajectory sequence. The moving target to be predicted and the scene structure elements are abstracted into target-scene structure nodes, and the interaction relationship between the moving target to be predicted and the scene structure elements is abstracted into target-scene structure interaction edges. A target-scene structure interaction graph structure based on historical trajectory sequences is constructed. Using the same method as historical trajectory sequences, we construct target-target interaction graph structures and target-scene interaction graph structures based on the preprocessed features of historical trajectories.

6. The multimodal trajectory prediction method for intelligent agricultural machinery path planning as described in claim 1, characterized in that: In step S3, the scene image obtained in step S1 is image encoded to obtain a visual feature vector, the preset scene descriptive words are text encoded to obtain a text embedding vector, the scene descriptive words are selected according to the similarity between the scene image and the scene descriptive words, and after combination, the scene language features are extracted.

7. The multimodal trajectory prediction method for intelligent agricultural machinery path planning as described in claim 1, characterized in that: In step S4, the multimodal node interaction feature extraction module extracts interaction features based on the target-target interaction graph structure and the target-scene interaction graph structure, respectively. The module structure for extracting interaction features is the same, including a nearest neighbor search graph convolution branch and a graph self-attention branch. The features extracted by the two branches are added together and then linearly transformed to obtain the node interaction output features. The node interaction output features extracted from the target-target interaction graph structure and the target-scene interaction graph structure are fused into multimodal interaction features; the multimodal interaction features corresponding to the historical trajectory sequence and the historical trajectory preprocessing features are concatenated to obtain the historical trajectory-scene structure interaction features.

8. The multimodal trajectory prediction method for intelligent agricultural machinery path planning as described in claim 1, characterized in that: In step S4, after the historical trajectory-scene structure interaction features and scene language features are initially fused, features are extracted from the time dimension and the spatial dimension respectively, and then fused into spatial interaction enhancement features. The spatial interaction weight matrix generated based on the spatial interaction enhancement features is added to the scene language weight matrix generated based on the scene language features to obtain a hybrid weight matrix. The hybrid weight matrix is ​​then multiplied element-wise with the spatial interaction enhancement features to obtain a weighted fusion feature, which is then further fused with the scene language features to obtain the scene constraint fusion interaction feature.

9. The multimodal trajectory prediction method for intelligent agricultural machinery path planning as described in claim 1, characterized in that: In step S5, the multimodal features of the future trajectory are input into a multilayer perceptron to obtain the trajectory latent features; the multimodal features of the future trajectory are decomposed into horizontal and vertical components, and horizontal and vertical clustering features are obtained through clustering; after the horizontal and vertical clustering features are concatenated, features are extracted from the time and spatial dimensions respectively, and then fused to obtain the explicit intent latent features. After concatenating the trajectory latent features and explicit intent latent features along the sequence direction, they are concatenated with the scene constraint fusion interaction features along the channel direction to obtain trajectory guidance features. The trajectory guidance features are then concatenated with the intent guidance features obtained by linearly transforming the explicit intent latent features along the sequence direction to obtain joint guidance features. Finally, the future trajectory distribution parameters are regressed by a multilayer perceptron.

10. A multimodal trajectory prediction device for intelligent agricultural machinery path planning, executing the prediction method according to any one of claims 1-9, characterized in that: The system includes an environmental perception component, a data acquisition interface, a multimodal trajectory prediction calculation unit, and an output interface. The environmental perception component is used to acquire scene images within the work area, as well as state data of the moving target to be predicted and nearby moving targets. The data acquisition interface is used to acquire positioning and target tracking data, historical motion state data, work scene structure information, and traffic constraint information. The multimodal trajectory prediction calculation unit is used to execute the prediction method and output multiple future trajectories of the moving target to be predicted and their corresponding modal probabilities. The output interface is used to send the multiple future trajectories of the moving target to be predicted and their corresponding modal probabilities to the path planning module or safety decision module of the intelligent agricultural machinery.

Images