Agricultural machine track classification method based on visual-temporal bimodal cross attention fusion

Abstract

This invention relates to the field of agricultural machinery trajectory classification, and discloses a method for agricultural machinery trajectory classification based on visual-temporal dual-modal cross-attention fusion. The method includes: acquiring agricultural machinery trajectory data and segmenting and filtering it; acquiring remote sensing images based on the geographical range of the trajectory and generating grayscale fused images; extracting visual features using EfficientNet-B0 combined with CBAM and FPN; calculating 18-dimensional temporal features containing 3-dimensional trailing features, and outputting temporal features after aggregation by a Transformer encoder and end-point weighted pooling; achieving modal depth interaction through bidirectional cross-attention and gating adaptive fusion; and employing a unified 6-class classifier and an auxiliary coarse classification hierarchical constraint strategy to output six categories of results: parking noise, road driving, multi-plot operation, normal operation, high-noise operation, and trailing operation. This invention can accurately identify fine-grained categories such as trailing operations and is suitable for deployment of agricultural machinery edge equipment.

Description

Technical Field

[0001] This invention relates to the field of agricultural machinery trajectory classification, and in particular to an agricultural machinery trajectory classification method based on visual-temporal dual-modal cross-attention fusion. Background Technology

[0002] Currently, GNSS-based monitoring of agricultural machinery operation trajectories has become an important tool for agricultural production management. However, the following technical challenges exist in practical applications: (1) The trajectory types are complex and diverse: Non-operation parking noise trajectory: Positioning drift forms a divergent distribution; Road driving trajectory: Field road or highway transfer; Multi-plot operation trajectory: A single trajectory segment contains multiple discontinuous farmlands; Normal operation trajectory: Standard operation mode; High noise operation trajectory: Antenna obstruction and signal interference cause positioning deviation; Trailing operation trajectory: Normal operation, but there is obvious trailing phenomenon when turning at the end of the field or at the end of the operation. This is usually caused by the operation device not being shut down in time, turning not in time, or non-standard operation, or the trajectory preprocessing basic method is not perfect, which is manifested as irregular extension line segments at the end of the trajectory. (2) Existing methods have limitations: Traditional rule-based methods rely on manually set thresholds (speed, steering angle, density, etc.), have poor generalization ability, are difficult to cope with complex scenarios, are not effective in recognizing high-noise trajectories and trailing trajectories, and cannot distinguish the subtle differences between normal operations and trailing operations. Single-modal deep learning methods: only use trajectory time series data, lack spatial semantic information, cannot utilize the land cover type features provided by remote sensing images, have limited ability to recognize spatial complex patterns such as multi-plot operations, and have difficulty capturing the visual features of trailing phenomena. Simple multimodal fusion methods employ heavyweight networks such as ResNet, resulting in a large number of parameters (>10M). When LSTM processes long sequences, the gradient vanishing problem is obvious. Simple feature splicing does not fully exploit the complementary information between modes. The inference speed is slow, making it difficult to deploy on agricultural machinery edge devices. It also lacks the ability to distinguish fine-grained categories such as trailing operations.

[0003] Therefore, there is an urgent need for an intelligent, lightweight, and high-precision method for classifying agricultural machinery trajectories. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method for classifying agricultural machinery trajectories based on visual-temporal dual-modal cross-attention fusion, thereby solving the technical problem that existing technologies cannot achieve high-precision, high-efficiency, and strong generalization of automatic classification of agricultural machinery trajectories.

[0005] Specifically, this invention provides a method for classifying agricultural machinery trajectories based on visual-temporal dual-modal cross-attention fusion, the method comprising the following steps: S1. Obtain agricultural machinery operation trajectory data containing latitude and longitude, timestamp, speed and heading angle, perform segmentation and quality screening to obtain trajectory segments to be classified; S2. Obtain the corresponding remote sensing image based on the geographical range of the trajectory segment, convert it into a grayscale panchromatic image, overlay and draw a high-contrast color trajectory line and endpoint direction markers, and generate a fused image of fixed size. S3. Using the EfficientNet-B0 backbone network, combined with the CBAM attention module and FPN multi-scale feature pyramid, extract the visual features of the fused image and output the visual feature vector. S4. Calculate the 18-dimensional enhanced temporal features of the trajectory segment. The 18-dimensional enhanced temporal features include 3-dimensional trailing features. The temporal features are extracted using a Transformer encoder and then aggregated through multiple pooling methods, including at least end-weighted pooling, to output a temporal feature vector. S5. Achieve bidirectional interaction between the visual feature vector and the temporal feature vector through a cross-attention fusion mechanism, and adaptively adjust the modal weights using a gating mechanism to output fused features; S6. End-to-end joint training is performed using a unified classifier and an auxiliary hierarchical constraint strategy: The unified 6-class classifier outputs six probabilities: parking noise, road driving, multi-plot operation, normal operation, high-noise operation, and tailing operation; the auxiliary coarse classifier outputs two probabilities: operation / non-operation, as a hierarchical regularization constraint; during inference, the incompatible class probabilities in the main classifier are masked based on the coarse classification prediction results and then re-normalized to obtain the final classification result.

[0006] A storage medium storing instructions and data for implementing an agricultural machinery trajectory classification method based on visual-temporal dual-modal cross-attention fusion.

[0007] A device for classifying agricultural machinery trajectories based on visual-temporal bimodal cross-attention fusion includes: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a method for classifying agricultural machinery trajectories based on visual-temporal bimodal cross-attention fusion.

[0008] The beneficial effects provided by this invention are: 1. Significantly improved classification accuracy: For the first time, six classifications of agricultural machinery trajectories (parking noise, road driving, multi-plot operation, normal operation, high-noise operation, and trailing operation) were achieved. The overall accuracy of the test set reached 90.9%, with the F1 score for trailing operation reaching 0.902. The accuracy of coarse classification (operation / non-operation) reached 96.8%, effectively solving the problem that existing methods have difficulty distinguishing between normal operation and trailing operation, as well as high-noise operation.

[0009] 2. Lightweight model: The total number of parameters is only 5.09M, and the model file is about 25.2MB, which reduces the number of parameters by 62% compared with the ResNet-18+LSTM solution, making it suitable for deployment on mobile devices and agricultural machinery edge devices.

[0010] 3. Fast inference speed: GPU single-sample inference takes about 22.5ms, CPU processing 1000 trajectory points takes about 27ms and 5000 trajectory points takes about 143ms, meeting the needs of real-time job monitoring.

[0011] 4. Multimodal deep fusion: It adopts bidirectional cross-attention + gated adaptive fusion to achieve deep interaction between visual features and temporal features of remote sensing images, dynamically balance the differences in modal quality, and has strong robustness.

[0012] 5. Unique trail detection capability: The specially designed 3D trail features, end-weighted pooling, image trail identification, and trail data enhancement enable the model to achieve a recall rate of 95.17% for trailing operations, which is significantly better than existing methods.

[0013] 6. Strong generalization ability: Gray-scale conversion eliminates seasonal interference, and the accuracy loss in cross-seasonal testing is less than 1%; rich data augmentation strategies make the model insensitive to noise, rotation, and scale changes, and still maintain an accuracy of over 90% in unseen areas.

[0014] 7. High engineering practicality: End-to-end single-stage training, eliminating the need for complex multi-stage processes; supports ONNX export for cross-platform deployment; provides confidence assessment and manual review suggestions for easy application in actual supervision. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the remote sensing image-trajectory fusion visualization enhancement processing workflow; Figure 3 This is a detailed structural diagram of the visual feature extraction branch of EfficientNet-CBAM-FPN; Figure 4 This is the overall architecture diagram of EfficientNet-Transformer's dual-branch flow processing; Figure 5 This is a detailed structural diagram of the feature extraction branch of the Transformer temporal encoder; Figure 6 These are typical samples of six trajectory types; Figure 7 It is a comparison of grayscale conversions of remote sensing images from different seasons; Figure 8 This is a diagram of a confusion matrix; Figure 9This is a schematic diagram of the hardware device operation according to an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0017] Before formally describing the present invention, a general description of the solution of the present invention will be given first to facilitate understanding.

[0018] Example 1 Please refer to Figure 1 This invention provides a method for classifying agricultural machinery trajectories based on visual-temporal dual-modal cross-attention fusion, comprising the following steps: S1. Obtain agricultural machinery operation trajectory data containing latitude and longitude, timestamp, speed and heading angle, perform segmentation and quality screening to obtain trajectory segments to be classified; As one example, the acquisition of raw trajectory data for agricultural machinery operations includes: trajectory point latitude and longitude coordinates, timestamp (accuracy ≤ 1 second), instantaneous speed (km / h), heading angle (0-360°), and optional auxiliary information (altitude, HDOP, number of satellites, etc.).

[0019] For trajectory segmentation, a segmentation algorithm based on speed and time interval is adopted to divide the continuous trajectory into several trajectory segments to be classified: time interval threshold: Δt>300 seconds, distance jump threshold: Δd>500 meters, speed change threshold: from v>2km / h to v<0.5km / h for more than 60 seconds.

[0020] Finally, data quality screening is performed. Unqualified trajectory segments are removed: those with <10 or >10000 trajectory points, total duration <30 seconds, and abnormal HDOP values ​​(>5).

[0021] S2. Obtain the corresponding remote sensing image based on the geographical range of the trajectory segment, convert it into a grayscale panchromatic image, overlay and draw a high-contrast color trajectory line and endpoint direction markers, and generate a fused image of fixed size. As one embodiment, step S2 is specifically as follows: (1) Remote sensing image acquisition and preprocessing Based on the geographical extent (latitude and longitude bounding boxes) of the trajectory segment, acquire high-resolution remote sensing imagery of the corresponding area: preferred data source: Google Earth (sub-meter resolution). Regarding image resolution, referencing the minimum spacing of normal operational trajectories calculated using the current equipment width, it is approximately 2.2 meters; therefore, sub-meter resolution imagery needs to be considered. Image temporal phase: prioritize imagery from within one year of the operational period. Cloud cover filtering: cloud cover <20%. (2) Grayscale conversion eliminates seasonal interference Please refer to Figure 7 To eliminate the interference of seasonal vegetation color changes on model recognition, the RGB remote sensing images were converted into grayscale panchromatic images:

[0022] Alternatively, weighted grayscale conversion can be used (to enhance texture information):

[0023] Then, histogram equalization is performed to enhance contrast.

[0024] (3) Trajectory line overlay and visualization enhancement Please refer to Figure 2 Draw trajectory lines overlaid on the grayscale image: Color selection: Use high-contrast color, recommended red RGB(255, 0, 0) or orange-red RGB(255, 69, 0); Line width setting: Adaptively adjust according to image resolution, 1-2 pixels is recommended; Endpoint marking: Mark the starting point with a green dot and the ending point with a blue dot; Direction of motion: Draw arrows every 20 points to indicate the direction of motion; Trailing indicator: Highlight possible trailing segments (the last 10-20% of the trajectory) with a semi-transparent yellow overlay. (4) Image normalization and standardization Generate a fixed-size fused image as input for visual feature extraction: resolution: 224×224 pixels (compliant with EfficientNet input requirements); normalization: pixel values ​​scaled to [0, 1]; standardization: subtract mean and divide by standard deviation, using ImageNet statistical parameters; S3. Using the EfficientNet-B0 backbone network, combined with the CBAM attention module and FPN multi-scale feature pyramid, extract the visual features of the fused image and output the visual feature vector. It should be noted that in step S3, the CBAM attention module is inserted after the 3rd MBConv block, the 5th MBConv block, and the last convolutional layer; the FPN feature pyramid extracts features from layers C2, C3, C4, and C5, and then splices them together through top-down fusion and global average pooling to obtain multi-scale fused features.

[0025] Please refer to Figure 3 As one example, EfficientNet-B0 is used as the backbone network for visual feature extraction, which has the following advantages over ResNet-18: Number of parameters: 5.3M (ResNet-18 is 11.2M), a reduction of 52%; Inference speed: improved by 40%; Accuracy: improved by 3-5% on ImageNet; The EfficientNet-B0 network structure includes: MBConv module (Mobile Inverted Bottleneck Convolution), Squeeze-and-Excitation attention, and a composite scaling strategy (joint optimization of depth, width, and resolution).

[0026] (2) Insertion of CBAM attention module Inserting CBAM (Convolutional Block Attention Module) after the key layers of EfficientNet enhances its ability to focus on trajectory lines: Channel Attention:

[0027]

[0028] Spatial Attention:

[0029]

[0030] Where: F is the input feature map, For the Sigmoid activation function, For element-wise multiplication, the MLP is a multilayer perceptron (hidden layer dimensionality reduction ratio 16:1). CBAM insertion positions: insert the first CBAM after the 3rd MBConv block, insert the second CBAM after the 5th MBConv block, and insert the third CBAM after the last convolutional layer.

[0031] (3) Multi-scale feature fusion (FPN structure) To capture trajectory pattern features at different scales, a Feature Pyramid Network (FPN) structure is introduced: features are extracted from different stages of EfficientNet. 56×56×24 (low-level features, rich in detail, capturing trailing details) : 28×28×40 (Mid-level features, local pattern) : 14×14×112 (high-level features, semantic information) : 7×7×320 (top-level feature, global mode); Top-down fusion:

[0032] +Upsample( ) +Upsample( ) +Upsample( ) Multi-scale feature aggregation:

[0033] (4) Feature dimensionality reduction and output The multi-scale fused features are reduced to a suitable dimension using a fully connected layer: ,use Prevent overfitting.

[0034] S4. Calculate the 18-dimensional enhanced temporal features of the trajectory segment. The 18-dimensional enhanced temporal features include 3-dimensional trailing features. The temporal features are extracted using a Transformer encoder and then aggregated through multiple pooling methods, including at least end-weighted pooling, to output a temporal feature vector. It should be noted that in step S4, the 3D trailing features include: position normalization index, terminal velocity anomaly degree, and terminal direction deviation degree; the multiple pooling methods include average pooling, max pooling, attention-weighted pooling, and terminal weighted pooling. The weight calculation formula for the terminal weighted pooling is: weight_i = 1 + 4 × max(0, (i - 0.8n) / 0.2n), where n is the total number of trajectory points and i is the index of the current point.

[0035] Please refer to Figures 4-5 As one embodiment, step S4 is specifically as follows: (1) Enhanced Temporal Feature Engineering For each point in the trajectory segment, an 18-dimensional enhanced feature vector is extracted, specifically including the following: Basic spatial features (6 dimensions): 1. Normalized longitude: Longitude offset relative to the center point of the trajectory segment, normalized to [-1, 1]; 2. Normalized latitude: Latitude offset relative to the center point of the trajectory segment, normalized to [-1, 1]; 3. Normalized velocity: The velocity value is normalized to [0, 1], using the following formula:

[0036] 4. Distance between adjacent points: Calculate the geodetic distance between point i and point i-1. (Meters), normalized; 5. Time interval between adjacent points: (seconds), normalized; 6. Rate of change of heading angle: Normalized to [-1, 1]; Kinematic characteristics (4 dimensions): 7. Acceleration: (m / s²); 8. Lateral acceleration: calculated based on changes in heading angle; 9. Longitudinal acceleration: calculated based on velocity changes; 10. Curvature: This characterizes the degree of trajectory curvature; Statistical characteristics (3 dimensions): 11. Local velocity variance: The standard deviation of velocity at 5 points before and after the current point; 12. Local density: The number of trajectory points within a radius of 10 meters; 13. Distance to center point: The distance from the current point to the geometric center of the trajectory segment; Frequency domain characteristics (2D): 14. FFT Low-Frequency Energy: Perform a Fast Fourier Transform on the velocity sequence to extract the low-frequency component energy; 15. Regularity scoring: Calculating the periodicity of the trajectory based on the autocorrelation function; Trailing feature (3D): 16. Position normalization index: The relative position of the current point in the trajectory (0 represents the starting point, 1 represents the ending point), used to identify the trailing edge at the end; 17. Final velocity anomaly: The deviation between the current velocity and the average velocity of the trajectory segment, normalized; 18. Final direction deviation: The deviation angle between the current heading and the main operating direction, normalized to [0, 1]; Feature matrix form:

[0037] Where n is the number of trajectory points.

[0038] (2) Positional Encoding To provide position information for the Transformer, sine-cosine position encoding is used:

[0039]

[0040] in: The position of the trajectory point in the sequence (from 0 to n-1). For feature dimension indexing, For model dimensions (128); Input embedding:

[0041] (3) Transformer encoder structure Constructing a multi-layer Transformer encoder (layers 2-4): The single-layer Transformer structure is as follows: Multi-head self-attention:

[0042]

[0043] in

[0044] Parameter settings: Number of attention heads h = 4, Dimensions of each head Overall model dimensions ; Feed-Forward Network:

[0045] The hidden layer has a dimension of 512 and the activation function is ReLU; Layer normalization and residual connectivity:

[0046]

[0047] Stacked structure:

[0048] The recommended number of layers is L=2 (to balance performance and efficiency) or L=4 (to pursue the highest accuracy).

[0049] (4) Multi-scale temporal modeling To capture operation patterns at different time scales (including trailing phenomena), multi-resolution input is employed: Short-time mode (original resolution): Input: complete trajectory sequence X; Capture: local details, instantaneous changes, trailing details; Medium-time mode (2x downsampling): Input: samples taken at every 1 point. Capture: Job cycle mode; Long-term mode (4x downsampling): Input: Sampled every 3 points Capture: Global trend; Processed separately by independent Transformer encoders and then fused:

[0050] (5) Time series feature aggregation and output The sequence information is aggregated through global pooling, in the following form: Average pooling: ; Max pooling: ; Attention-weighted pooling: ;

[0051] End-point weighted pooling: assign higher weights to the last 20% of points in the trajectory;

[0052]

[0053] Final output: .

[0054] S5. Achieve bidirectional interaction between the visual feature vector and the temporal feature vector through a cross-attention fusion mechanism, and adaptively adjust the modal weights using a gating mechanism to output fused features; It should be noted that, in step S5, the cross-attention fusion mechanism specifically includes: calculating visual-temporal attention with temporal features as queries and visual features as keys and values, and using residual connections to enhance temporal features; calculating temporal-visual attention with visual features as queries and temporal features as keys and values, and using residual connections to enhance visual features; then, using a gating mechanism to adaptively generate weight vectors to perform weighted fusion of the two enhanced features, and finally outputting the fused features after residual connections and layer normalization.

[0055] As one example, traditional feature concatenation methods ignore complementary information between modalities. This invention proposes a cross-attention fusion mechanism, as follows: (1) Intermodal cross attention Visual → Temporal Attention: Temporal features are used as the Query, and visual features are used as the Key and Value.

[0056] Enhanced temporal features:

[0057] Temporal to Visual Attention: Visual features serve as the Query, and temporal features serve as the Key and Value.

[0058] Enhanced visual features:

[0059] (2) Gated adaptive fusion The contribution weights of the two modes are dynamically adjusted through a gating mechanism:

[0060]

[0061] in: For the Sigmoid activation function, For element-wise multiplication, This is a gated vector, where each dimension corresponds to the weight of a feature channel; (3) Residual connectivity and layer normalization To preserve original feature information and ensure stable training:

[0062] Final fusion feature dimensions: 512 dimensions (256+256 or 256 dimensions after compression).

[0063] S6. End-to-end joint training is performed using a unified classifier and an auxiliary hierarchical constraint strategy: The unified 6-class classifier outputs six probabilities: parking noise, road driving, multi-plot operation, normal operation, high-noise operation, and tailing operation; the auxiliary coarse classifier outputs two probabilities: operation / non-operation, as a hierarchical regularization constraint; during inference, the incompatible class probabilities in the main classifier are masked based on the coarse classification prediction results and then re-normalized to obtain the final classification result.

[0064] It should be noted that in step S6, the loss function uses Focal Loss combined with Label Smoothing, where the Focal Loss focusing parameter γ=2, the class weights are set to [1.5, 1.3, 1.5, 0.8, 1.3, 1.8], and the Label Smoothing smoothing parameter ε=0.05; a differentiated learning rate strategy is used during training: the first 70% of the EfficientNet-B0 layers are frozen, the last 30% are fine-tuned using 1 / 10 of the base learning rate, and the remaining modules use the base learning rate e. -4 It employs warmup in conjunction with cosine annealing scheduling, and uses LayerNorm for network normalization.

[0065] As one embodiment, step S6 is specifically as follows: (1) Unified classifier and auxiliary hierarchical constraint strategy Please refer to Figure 6 To improve accuracy, a unified classification and auxiliary hierarchical constraint strategy is adopted based on the characteristics of the six types of trajectories. Unified Master Classifier:

[0066]

[0067]

[0068] Six-category output: Category 0: Parking noise trajectory, Category 1: Road driving trajectory, Category 2: Multi-plot operation trajectory, Category 3: Normal operation trajectory, Category 4: High noise operation trajectory, Category 5: Trailing operation trajectory; The auxiliary coarse classifier (providing hierarchical regularization constraints) is as follows:

[0069]

[0070] Binary constraints: Category 0: Non-operation (parking noise, road driving), Category 1: Operation (multi-plot operation, normal operation, high-noise operation, trailing operation); (2) The role of assisting in coarse classification The coarse classifier does not participate in inference routing decisions, but rather serves as an auxiliary training task to provide hierarchical regularization constraints: During training: The coarse classification loss is used as an auxiliary loss term to guide the learning of the macroscopic semantics of "task / non-task" from the fused features. During inference: coarse classification prediction is used to constrain the main classification result; non-task samples mask the probability of the task class; task samples mask the probability of the non-task class. (3) End-to-end joint training, with differentiated learning rates to protect pre-trained knowledge Define the joint loss function:

[0071] in: Focal Loss for 6 main classification classes (including label smoothing to prevent overfitting) Cross-entropy loss for coarse classification of two classes (auxiliary task) L2 regularization loss for fusion features, This is the coarse classification weight (recommended value 0.3). This is the regularization weight (recommended value 0.05).

[0072] (4) Focal Loss processing class imbalance To address the potential imbalance of the five classes of samples, Focal Loss is employed.

[0073] in: For the predicted probability of the true category, Class weights (set according to sample distribution and classification difficulty) To focus on the parameter (recommended value 2), reduce the weight of easily distinguishable samples and set the category weight (determined based on sample proportion and classification difficulty): Parking noise: = 1.5 (Non-task-related vehicles require increased attention), Road driving: = 1.3 (Non-task-related tasks require increased attention), multiple plots of land for work: = 1.5 (easily confused with trailing operations, needs enhancement), Normal operation: = 0.8 (most samples), high-noise operation: = 1.3 (difficult to distinguish), trailing operation: = 1.8 (less and easily confused with multi-site operations).

[0074] (5) Label Smoothing prevents overfitting To prevent the model from becoming overconfident in the training labels and thus reducing its generalization ability, label smoothing is introduced in Focal Loss:

[0075] in =0.1, =6 represents the number of categories. Label smoothing forces the model to output a softer probability distribution, effectively mitigating the performance difference between the training and validation sets.

[0076] (6) Detailed structure of classification network Main classifier (6 classes):

[0077]

[0078] Auxiliary coarse classifier (2 categories: assignment / non-assignment):

[0079]

[0080] Using LayerNorm instead of BatchNorm ensures consistent behavior during training and inference phases, and avoids abnormal validation loss caused by untimely updates to BN running statistics.

[0081] Finally, the training strategy for end-to-end joint training is as follows: (1) Initialization of transfer learning EfficientNet-B0: Uses ImageNet pre-trained weights; Transformer encoder: Trained from scratch (randomly initialized); CBAM module: Randomly initialized; (2) End-to-end joint training, with differentiated learning rates to protect pre-trained knowledge strategies A single-stage end-to-end joint training method is adopted, in which all modules participate in training simultaneously from the first epoch, and pre-trained knowledge is protected by differentiated learning rates: Differential learning rate strategy: The first 70% of the EfficientNet-B0 layers are frozen (preserving ImageNet low-level features), and the last 30% of the layers are fine-tuned using 1 / 10 of the base learning rate; the Transformer encoder, CBAM, FPN, fusion layer, and classifier use the base learning rate. Learning rate scheduling: Base learning rate: e -4 Backbone learning rate: e -5 (1 / 10 of the base learning rate), Warmup strategy: linearly increase the learning rate for the first 10 epochs (from e... -6 to e -4 Cosine annealing scheduling:

[0082] (3) Data augmentation strategy Visual branch enhancement (performed within the pixel value range [0,1], before ImageNet normalization): Random rotation: ±15 degrees, random flip: horizontal / vertical, brightness adjustment: ±20%, contrast adjustment: ±20%; Gaussian noise: σ=0.01, random erasure: erase 15% of the region with a 15% probability; Enhanced trailing: Randomly shifts the direction of the end of the normal working trajectory by 0-15° to simulate the trailing phenomenon; Temporal branch enhancement: Time warp: Randomly stretch / compress the time axis (±10%), Gaussian noise: Add noise of σ=5 meters to the latitude and longitude coordinates, speed jitter: ±2km / h, temporal truncation: Randomly truncate 80-100% of the trajectory segments, Dropout: Randomly discard 8% of the trajectory points, End perturbation: Add direction and speed perturbations to the last 20% of the points of the trajectory to enhance the trail detection capability.

[0083] (4) The training hyperparameters are set as follows: Batch Size: 32 (or adjust according to GPU memory), Total Epochs: 120, Early Stopping Strategy: Stop if the validation set loss does not decrease for 25 consecutive epochs, Weight Decay: 5e -3 (L2 regularization), gradient clipping: maximum gradient norm 1.0, mixed precision training: accelerated training using FP16 (PyTorch AMP).

[0084] (5) For loss function design, please refer to the pseudocode section of the table below: Table 1. Pseudocode for Loss Function Design

[0085] Category weights: α = [1.5, 1.3, 1.5, 0.8, 1.3, 1.8] It should be noted that the model reasoning and post-processing process of this invention is as follows: (1) Reasoning process Input: Trajectory segment data + corresponding area remote sensing imagery; 1. Generate the fused image → 224×224×3 2. EfficientNet-CBAM-FPN extracts visual features → 256 dimensions; 3. Transformer encoder extracts temporal features → 256 dimensions; 4. Cross-attention fusion → 256 dimensions; 5. Main classifier prediction → (6 types of probability); 6. Auxiliary coarse classifier prediction → (Homework / Non-homework); 7. Post-processing of hierarchical constraints: Mask out incompatible category probabilities based on the coarse classification results; 8. Output the final category and confidence level; (2) Hierarchical constraint reasoning During inference, an auxiliary coarse classifier is used to constrain the main classification result, as shown in Table 2 below: Table 2. Pseudocode for Hierarchical Constraint Reasoning

[0086] (2) Confidence assessment

[0087] if Marked as "Uncertain", manual review is recommended.

[0088] For trailing operations, due to their high similarity to normal operations, additional checks are required, see Table 3: Table 3 Pseudocode for checking trailing operations

[0089] (3) Timing smoothing post-processing Temporal smoothing of classification results for continuous trajectory segments: Sliding window voting is used: for trajectory segment i, consider two trajectory segments before and after it (a total of five); final category = Only votes with a confidence level greater than 0.8 to avoid mispropagation.

[0090] Example 2 Regarding the agricultural machinery trajectory classification network based on visual-temporal dual-modal cross-attention fusion in this invention, this embodiment provides a standard configuration scheme that balances performance and efficiency.

[0091] 1. Hardware environment Training environment: NVIDIA Tesla 4070 GPU (12GB VRAM) or higher; Inference environment: Intel i5-13600KF CPU; Memory requirement: ≥32GB RAM; Storage requirement: ≥10GB (data + model + logs).

[0092] 2. Software Environment Operating System: Windows 11; Python: 3.11+; Deep Learning Framework: PyTorch 2.9.0+cu126; Dependencies include: torchvision 0.24.0+cu126, efficientnet-pytorch 0.7.1, timm 1.0.22 (PyTorch Image Models), numpy, pandas, scikit-learn, opencv-python, pillow (image processing), rasterio, geopandas (remote sensing image processing), matplotlib, seaborn (visualization).

[0093] 3. Data Preparation Training dataset construction: Agricultural machinery operation trajectory data were collected in Hubei Province from 2022 to 2025. Total sample size: 9170 trajectory segments.

[0094] The statistical results of the sample data are as follows: Table 4. Statistical Analysis of Sample Data

[0095] Image data acquisition: Acquire Google remote sensing imagery of the corresponding area (level 15-18 imagery data, resolution approximately 4.8-0.6 meters).

[0096] Dataset partitioning: Training set samples (7333 samples): Parking noise: 1179, Road driving: 1010, Multi-site operation: 1276, Normal operation: 1596, High-noise operation: 1121, Trailing operation: 1151; Validation set samples (914 samples): Parking noise: 147, Road driving: 126, Multi-site operation: 159, Normal operation: 199, High-noise operation: 140, Trailing operation: 143; Test set samples (923 samples): Parking noise: 148, Road driving: 127, Multi-site operation: 161, Normal operation: 201, High-noise operation: 141, Trailing operation: 145; The dataset is randomly divided to ensure that the geographical and temporal distribution of each set is uniform.

[0097] 4. Experimental Results Performance evaluation: Performance evaluation on the test set (923 samples): Table 5 Experimental Results

[0098] Performance in each category: Table 6 Performance Demonstration

[0099] The accuracy rate was 0.909.

[0100] Performance statistics: Table 7 Performance Statistics

[0101] For confusion matrix analysis, please refer to Figure 8 .from Figure 8 It can be seen that the model achieves an accuracy rate of over 95% for categories with distinct characteristics (parking, road, and trailing). The main confusions are concentrated in scenarios with blurred boundaries between operation categories—high noise versus parking noise, normal operation versus multi-site operation, and multi-site operation versus trailing operation. These three pairs of confusions account for 68% (57 / 84) of the total misclassifications. This is because these categories have gradual transitions in trajectory morphology, with no clear boundaries. The accuracy rate for coarse classification (operation / non-operation) is 96.75%.

[0102] Example 3: Please see Figure 9 , Figure 9 This is a schematic diagram of the hardware device in operation according to an embodiment of the present invention. The hardware device specifically includes: an agricultural machinery trajectory classification device 401 based on visual-temporal dual-modal cross-attention fusion, a processor 402, and a storage medium 403.

[0103] A type of agricultural machinery trajectory classification device 401 based on visual-temporal bimodal cross-attention fusion: The agricultural machinery trajectory classification device 401 based on visual-temporal bimodal cross-attention fusion implements the agricultural machinery trajectory classification method based on visual-temporal bimodal cross-attention fusion.

[0104] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the agricultural machinery trajectory classification method based on visual-temporal bimodal cross-attention fusion.

[0105] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the agricultural machinery trajectory classification method based on visual-temporal bimodal cross-attention fusion.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for agricultural machinery trajectory classification based on visual-temporal dual-modal cross attention fusion, characterized in that: Includes the following steps: S1. Obtain agricultural machinery operation trajectory data containing latitude and longitude, timestamp, speed and heading angle, perform segmentation and quality screening to obtain trajectory segments to be classified; S2. Obtain the corresponding remote sensing image based on the geographical range of the trajectory segment, convert it into a grayscale panchromatic image, overlay and draw a high-contrast color trajectory line and endpoint direction markers, and generate a fused image of fixed size. S3. Using the EfficientNet-B0 backbone network, combined with the CBAM attention module and FPN multi-scale feature pyramid, extract the visual features of the fused image and output the visual feature vector. S4. Calculate the 18-dimensional enhanced temporal features of the trajectory segment. The 18-dimensional enhanced temporal features include 3-dimensional trailing features. The temporal features are extracted using a Transformer encoder and then aggregated through multiple pooling methods, including at least end-weighted pooling, to output a temporal feature vector. S5. Achieve bidirectional interaction between the visual feature vector and the temporal feature vector through a cross-attention fusion mechanism, and adaptively adjust the modal weights using a gating mechanism to output fused features; S6. End-to-end joint training is performed using a unified classifier and auxiliary hierarchical constraint strategy: the unified 6-class classifier outputs six probabilities: parking noise, road driving, multi-plot operation, normal operation, high-noise operation and tailing operation. The auxiliary coarse classifier outputs the binary probabilities of job / non-job as a hierarchical regularization constraint; during inference, the incompatible class probabilities in the main classifier are masked based on the coarse classification prediction results and then re-normalized to obtain the final classification result.

2. The agricultural machine trajectory classification method based on visual-temporal dual-modal cross attention fusion of claim 1, wherein: In step S4, the 3D trailing features include: position normalization index, terminal velocity anomaly degree, and terminal direction deviation degree; the multiple pooling methods include average pooling, max pooling, attention-weighted pooling, and terminal weighted pooling. The weight calculation formula for the terminal weighted pooling is: weight_i = 1 + 4 × max(0, (i - 0.8n) / 0.2n), where n is the total number of trajectory points and i is the index of the current point.

3. The agricultural machine trajectory classification method based on visual-temporal dual-modal cross attention fusion of claim 1, wherein: In step S3, the CBAM attention module is inserted after the 3rd MBConv block, the 5th MBConv block, and the last convolutional layer; the FPN feature pyramid extracts features from layers C2, C3, C4, and C5, and then splices them together through top-down fusion and global average pooling to obtain multi-scale fused features.

4. The method of claim 1, wherein the method is based on a visual-temporal dual-modal cross-attention fusion for agricultural machine trajectory classification. In step S5, the cross-attention fusion mechanism specifically includes: calculating visual-temporal attention and reinforcing temporal features with residual connections using temporal features as queries and visual features as keys and values; calculating temporal-visual attention and reinforcing visual features with residual connections using visual features as queries and temporal features as keys and values; then adaptively generating weight vectors through a gating mechanism to perform weighted fusion of the two enhanced features, and finally outputting the fused features after residual connections and layer normalization.

5. The method of claim 1, wherein the method is based on a visual-temporal dual-modal cross-attention fusion for agricultural machine trajectory classification. In step S6, the loss function uses Focal Loss combined with Label Smoothing, where the Focal Loss focusing parameter γ=2, the class weights are set to [1.5, 1.3, 1.5, 0.8, 1.3, 1.8], and the Label Smoothing smoothing parameter ε=0.

05. During training, a differentiated learning rate strategy is used: the first 70% of the EfficientNet-B0 layers are frozen, the last 30% are fine-tuned using 1 / 10 of the base learning rate, and the remaining modules use the base learning rate e. -4 It employs warmup in conjunction with cosine annealing scheduling, and uses LayerNorm for network normalization.

6. The method of claim 1, wherein the method is based on a visual-temporal dual-modal cross-attention fusion for agricultural machine trajectory classification. In step S2, the gray scale conversion adopts the weighted formula Gray = 0.2126xR + 0.7152xG + 0.0722x B, and histogram equalization is performed; the trajectory line adopts red or orange red, and the line width is 1 pixel; the starting point is marked with a green dot, and the ending point is marked with a blue dot; every 20 points draw an arrow to indicate the direction of movement; the last 10-20% of the trajectory is overlaid with a translucent yellow color.

7. A storage medium characterized by: The storage medium stores instructions and data for implementing the agricultural machine trajectory classification method based on visual-temporal dual-modal cross attention fusion according to any one of claims 1-6.

8. A device for agricultural machinery trajectory classification based on visual-temporal dual-modal cross attention fusion, characterized in that: Comprise: A processor and a storage medium; The processor loads and executes the instructions and data in the storage medium to implement the agricultural machine trajectory classification method based on visual-temporal dual-modal cross attention fusion according to any one of claims 1-6.

Images