Forklift trajectory prediction method and system based on improved visual perception and space-time modeling
By improving the YOLOv8 and LSTM forklift trajectory prediction framework and combining lightweight visual perception and spatiotemporal modeling, the problems of high computational cost, insufficient feature fusion, and overfitting in counterbalanced forklift trajectory prediction are solved, achieving high-precision and robust trajectory prediction and supporting real-time safe operation in industrial logistics scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WUHAN UNIV OF TECH
- Filing Date
- 2026-02-04
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies for predicting the trajectory of counterbalance forklifts in industrial logistics scenarios suffer from problems such as high computational cost, insufficient feature fusion, regression loss function dependence on anchor boxes, and overfitting in time-series predictions, resulting in insufficient detection accuracy and robustness.
We employ a lightweight object detection model MobileNetV3, a progressive feature fusion network AFPN, a pixel-level intersection-union loss function PIOU, and an LSTM network with dropout layers, combining visual perception with spatiotemporal modeling to improve detection efficiency and prediction accuracy.
It achieves high-precision and robust forklift trajectory prediction, significantly improving detection efficiency and anti-interference capabilities, and providing reliable collision warning and operational safety support.
Smart Images
Figure CN122156246A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical fields of computer vision, deep learning and intelligent transportation, and specifically to a forklift trajectory prediction method and system based on improved visual perception and spatiotemporal modeling. Background Technology
[0002] In industrial logistics scenarios such as ports, shipyards, and large warehouses, counterbalance forklifts have become key transportation tools due to their flexibility. However, their operating environment often features mixed traffic of people and vehicles, nonlinear trajectories, and frequent starts, stops, and turns, resulting in a high risk of collisions. Traditional trajectory prediction methods based on rules or simple physical models are ineffective in such unstructured environments.
[0003] The existing technology has the following main drawbacks: 1. Bulky Perception Module: Mainstream object detection models are computationally intensive, making them difficult to run in real-time on edge computing devices and unable to meet the low-latency requirements of industrial scenarios. For example, the YOLOv8 model, compared to traditional Anchor-Based detection methods, has higher detection accuracy and faster detection speed. The principle of YOLOv8 can be divided into two parts: feature extraction and object detection. The feature extraction part uses a network structure called CSPDarknet, which is an improved version based on Darknet. CSPDarknet adopts a Cross Stage Partial Network (CSP) structure, dividing the network into two parts, each containing multiple residual blocks.
[0004] 2. Insufficient Feature Fusion: Traditional feature pyramid networks exhibit significant semantic gaps between non-adjacent layers when performing multi-scale feature fusion, leading to decreased detection accuracy for small, occluded, or distant targets. For example, PAFPN (Path Aggregation Feature Pyramid Network) is an improved feature pyramid network structure that enhances the traditional FPN (Feature Pyramid Network) to improve the model's detection performance.
[0005] 3. Regression loss function depends on anchor box: The penalty term of commonly used loss functions such as CIoU depends on the anchor box size. When the target scale changes greatly or there is severe occlusion, it will lead to low anchor box regression efficiency and slow convergence.
[0006] 4. Time series prediction is prone to overfitting: LSTM-based prediction models are prone to overfitting training data when dealing with long sequences, are sensitive to noise, and lead to amplification of the cumulative error of long-term predictions and insufficient generalization ability.
[0007] Therefore, there is an urgent need in this field for a forklift trajectory prediction solution that can balance real-time performance, accuracy, and robustness. Summary of the Invention
[0008] The technical problem to be solved by this invention is to provide a forklift trajectory prediction method and system based on improved visual perception and spatiotemporal modeling, which achieves high-precision and robust trajectory prediction for counterbalance forklifts by integrating lightweight target detection, adaptive multi-scale feature fusion and long-short-term time series modeling techniques.
[0009] The embodiments of this application are implemented as follows: This application provides a forklift trajectory prediction method based on improved visual perception and spatiotemporal modeling, characterized by the following steps: Step S1, improve the visual perception of the object detection model: Step S 11 The backbone network of the object detection model adopts a depthwise separable convolution and inverted residual structure, and introduces a lightweight attention mechanism and activation function to extract features; Step S 12 The asymptotic pyramid network AFPN is adopted, which allows features from non-adjacent levels to interact directly. The network learns autonomously to assign weights to each pixel in the spatial location of feature maps from different levels, thereby enhancing the features. Step S 13 The pixel-level intersection-union ratio (PIoU) loss function is used to minimize the normalized distance between the four boundaries of the predicted box and the ground truth box, guiding the model to perform bounding box regression along a more efficient path. Step S2, Trajectory Sequence Extraction and Data Preprocessing: Step S 21 The improved target detection model is used to perform frame-by-frame target detection and tracking on the acquired forklift operation video, and the pixel coordinates of the center point of the forklift bounding box in each frame are extracted to form an initial trajectory sequence. Step S 22 Based on the spatial mapping relationship between camera calibration parameters and scene, pixel coordinates are converted into actual physical coordinates, and the instantaneous velocity and total acceleration of the forklift are calculated. Step S 23 The original displacement and velocity sequences are smoothed using a filter. Step S3, Time Series Trajectory Prediction: Step S 31 Construct a multi-layer LSTM network model, determine the input and output feature vectors of the model, and embed a Dropout layer between the hidden layers of the LSTM to randomly drop a portion of neurons with a preset probability p, so that their output is zero. Step S32 The processed trajectory dataset is divided into training and validation sets according to the proportions for model training, and the loss function and evaluation metric are determined.
[0010] In some alternative implementations, step S 11 It includes the following specific content: The object detection model is YOLOv8, with MobileNetV3 as the backbone. A lightweight attention mechanism, SENet, is introduced. After the feature maps pass through 3×3 deep convolutional layers, the output features undergo global average pooling, followed by a fully connected layer. The h-swish activation function is used to generate feature vectors and assign weights to each channel. The weighted feature vectors then yield the output features. The activation function uses a rectified linear unit (ReLU6) to crop the input values. The h-swish activation function is as follows:
[0011]
[0012] In the formula, x is the input value of the current layer.
[0013] In some alternative implementations, step S 12 It includes the following specific content: During feature fusion, an adaptive spatial feature fusion mechanism is adopted, and the calculation formula is as follows:
[0014]
[0015] In the formula, This represents the spatial importance weights of three different levels of feature mappings. These represent the output and the output feature vector, respectively. Indicates adjustment to the number layer.
[0016] In some alternative implementations, step S 31 It includes the following specific content: An LSTM network consists of a forget gate, an input gate, and an output gate. During continuous data transmission, the current state x... t and the output h of the previous sequence t-1 The combination is the input vector (h) at time t. t-1 x t The input vector passes through each control gate sequentially and updates the cell state C. t Finally, calculate the output h. t The specific calculation formula is as follows:
[0017] In the formula, σ is the activation function, and f t i t and o t Let W be the output of the forget gate, the output gate, and the output of the output gate at time t. f W i W c and W o These are the weight matrices for each control gate, b f b i b c and b o These are the deviation matrices for each control gate, and * represents the convolution operation.
[0018] In some optional implementations, the input feature vector of the model includes lateral displacement, longitudinal displacement, lateral velocity, longitudinal velocity, and total acceleration; the output feature vector of the model includes lateral displacement and longitudinal displacement at several future time steps.
[0019] In some alternative implementations, step S 32 It includes the following specific content: The stochastic gradient descent optimizer is used with an initial learning rate of 0.1 and a learning rate decay strategy. As the training epochs increase, the learning rate is gradually reduced according to a predetermined rule, using either exponential decay or epoch-based decay. This allows the model to approach the optimal solution region with large steps in the early stages of training and adjust the model parameters with small steps in the later stages of training, avoiding oscillations around the optimal value. This improves the model's generalization ability and enables it to achieve more robust and accurate trajectory prediction performance on the validation set.
[0020] In some alternative implementations, the loss function is the root mean square error (RMSE) and the mean absolute error (MAE) between the predicted and actual displacements:
[0021] In the formula, n represents the total number of samples used to calculate the error, specifically the total number of trajectory points; i represents the index, the i-th sample; This represents the true value of the i-th sample, specifically the true lateral or longitudinal displacement of the forklift position at a time step. This represents the model prediction value for the i-th sample, specifically the forklift displacement predicted by the LSTM model at a future time step.
[0022] In some optional implementation schemes, the specific evaluation method for the evaluation indicators is as follows: The model outputs the lateral and longitudinal displacements for future time steps, calculates the root mean square error (RMSE) and mean absolute error (MAE) between these displacements and the true values, and determines the accuracy by directly comparing their values. At the same time, by observing the trend of these error values as the prediction time domain extends, the robustness of the model in different time periods is analyzed. Pixel errors are converted into actual physical distances through spatial mapping relationships to intuitively reflect the practical accuracy of the prediction results. Furthermore, a visual comparison between the predicted trajectory and the actual trajectory is used to qualitatively evaluate the consistency of their morphology.
[0023] In some alternative implementations, step S 23 The filter described is the Savitzky-Golay filter.
[0024] A forklift trajectory prediction system based on improved visual perception and spatiotemporal modeling, characterized in that it includes: The visual perception module, based on the YOLOv8 model, uses a deep separable convolutional and inverted residual structure in the backbone network MobileNetV3, and introduces a lightweight attention mechanism and h-swish activation function for feature extraction. It employs an asymptotic pyramid network AFPN to enhance effective features through a progressive fusion strategy. In the regression branch, the PIOU loss function is used to minimize the normalized distance between the four boundaries of the predicted box and the ground truth box, guiding the model to perform bounding box regression along a more efficient path. The trajectory sequence extraction and data preprocessing module uses the improved YOLOv8 model to perform frame-by-frame target detection and tracking on the acquired forklift operation video, extracting the pixel coordinates of the center point of the forklift bounding box in each frame to form an initial trajectory sequence; based on the spatial mapping relationship between camera calibration parameters and the scene, the pixel coordinates are converted into actual physical coordinates, and the kinematic characteristics of the forklift's instantaneous velocity and total acceleration are calculated; the Savitzky-Golay filter is used to smooth the original displacement and velocity sequences. The time series prediction module constructs a multi-layer LSTM network as the core of time series prediction, embedding Dropout layers between the hidden layers of the LSTM; the processed trajectory dataset is divided into training and validation sets according to the proportion, and the root mean square error (RMSE) and mean absolute error (MAE) between the predicted displacement and the actual displacement are used as the loss function and evaluation index.
[0025] The beneficial effects of this application are as follows: This application provides a forklift trajectory prediction method and system based on improved visual perception and spatiotemporal modeling, which integrates an improved YOLOv8 and LSTM trajectory prediction framework; in the visual perception part, by introducing a lightweight MobileNetV3 backbone network, progressive feature pyramid and pixel-level intersection-union loss, the detection efficiency and anti-interference ability are significantly improved; in the temporal prediction part, the LSTM network with dropout layer is used to effectively suppress overfitting; it exhibits high accuracy and strong robustness, providing reliable support for collision warning and operational safety. Attached Figure Description
[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0027] Figure 1 This is a structural diagram of the MobileNetV3 model according to an embodiment of this application; Figure 2 This is a diagram of the inverted residual structure according to an embodiment of this application; Figure 3 This is a diagram of the AFPN network structure according to an embodiment of this application; Figure 4 This is a diagram of the improved YOLOv8 network structure according to an embodiment of this application; Figure 5 This is a diagram illustrating the effect of the Savitzky-Golay filter according to an embodiment of this application. Figure 6 This is a diagram of the LSTM structure according to an embodiment of this application; Figure 7a This is a diagram showing the predicted lateral displacement of an embodiment of this application. Figure 7b This is a diagram showing the predicted longitudinal displacement of an embodiment of this application. Figure 8 This is a displacement evaluation index diagram of an embodiment of this application; Figure 9 This is a straight-line trajectory prediction diagram according to an embodiment of this application; Figure 10a , 10b 10c represents the turning motion path of the forklift in this application embodiment. Figure 11 This is a turning trajectory prediction diagram for an embodiment of this application. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0029] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0030] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0031] The features and performance of this application will be further described in detail below with reference to the embodiments.
[0032] The present invention aims to provide a forklift trajectory prediction scheme, the core of which lies in building a deep fusion framework of an "improved visual perception module" and an "enhanced temporal prediction module".
[0033] Step 1: Based on the improved YOLOv8 (see...) Figure 4 Lightweight high-precision visual perception 1. Lightweight backbone network replacement: The original CSPDarknet backbone of the YOLOv8 model was replaced with MobileNetV3. MobileNetV3 adopts a depthwise separable convolutional and inverse residual structure, and introduces a lightweight attention mechanism and h-swish activation function, which greatly reduces the number of parameters and computational cost while maintaining powerful feature extraction capabilities.
[0034] The MobileNetV3 model structure diagram is as follows: Figure 1 As shown, SENet, a lightweight attention mechanism, is introduced on top of MobileNetV2: After the feature maps pass through a 3×3 deep convolutional layer, the output features undergo global average pooling, followed by a fully connected layer. ReLU and activation functions are used to generate feature vectors, and weights are generated for each channel. The feature vectors are then weighted to obtain the output features (see...). Figure 2The formula for the nonlinear activation function is shown in (1). The formula uses a rectified linear unit (ReLU6) to avoid the risk of numerical overflow by pruning the input value.
[0035] (1) (2) In the formula, x is the input value of the current layer, usually the output of the previous layer, such as the output of a convolutional layer or a fully connected layer. min refers to the lower bound limit value, usually 0, to prevent negative numbers from being propagated to the next layer and to maintain the non-negativity of activation. max refers to the upper bound limit value, which is 6 in ReLU6. Input values greater than 6 are pruned to 6 to prevent excessively large values, especially to avoid overflow problems in low-precision calculations, and also to help the model's stability and convergence.
[0036] 2. Progressive Feature Pyramid Network (AFPN): Use AFPN (see) Figure 3 The original PAPPN in the Neck section is replaced. AFPN uses a progressive fusion strategy to allow features from non-adjacent layers to interact directly, avoiding the loss of high-level semantic information during multiple transmissions.
[0037] During the fusion process, an adaptive spatial feature fusion mechanism is adopted. The network learns autonomously to assign appropriate weights to each pixel in the spatial location of the feature maps at different levels, thereby suppressing conflicting information and enhancing effective features.
[0038] Figure 3 The black arrows in the middle represent convolution, and the blue arrows represent adaptive spatial feature fusion (ASFF), the calculation formula of which is shown in (3).
[0039] (3) (4) In the formula, This represents the spatial importance weights of three different levels of feature mappings. These represent the output and the output feature vector, respectively. Indicates adjustment to the number layer.
[0040] AFPN dynamically learns the weights of features at different levels in spatial location through an adaptive spatial weighting mechanism. It suppresses multi-objective conflict information and retains effective features by weighted summation.
[0041] 3. Pixel-level Intersection over Union (IoU) loss function: In the regression branch, the PioU loss function is used instead of the original CIoU loss. The penalty term of PioU is based solely on the true size of the target box, completely eliminating the dependence on the preset anchor box size. By directly minimizing the normalized distance between the four boundaries of the predicted box and the true box, it guides the model to perform bounding box regression along a more efficient path, which is especially suitable for scenes with large scale variations and severe occlusion.
[0042] Step 2: Trajectory Sequence Extraction and Data Preprocessing 1. Using the improved YOLOv8 model described above, perform frame-by-frame target detection and tracking on the acquired forklift operation video, extract the center point pixel coordinates of the forklift bounding box in each frame, and form an initial trajectory sequence.
[0043] 2. Based on the spatial mapping relationship between camera calibration parameters and scene, pixel coordinates are converted into actual physical coordinates, and kinematic characteristics such as instantaneous velocity and total acceleration of the forklift are calculated.
[0044] 3. The original displacement and velocity sequences were smoothed using a Savitzky-Golay filter. This filter, based on local polynomial fitting, effectively removes noise while preserving the true trend and peak characteristics of the signal. The filtering effect is shown in [see figure]. Figure 5 .
[0045] Step 3: Temporal trajectory prediction based on Dropout-LSTM 1. Model Building: like Figure 6 As shown, the LSTM network consists of a forget gate, an input gate, and an output gate.
[0046] During continuous data transmission, the current state x t and the output h of the previous sequence t-1 The combination is the input vector (h) at time t. t-1 x t The input vector passes through each control gate sequentially and updates the cell state C. t Finally, calculate the output h. t The specific calculation formula is as follows: (5) (6) (7) (8) (9) In the formula, σ is the activation function, and f t i t and o tLet W be the output of the forget gate, the output gate, and the output of the output gate at time t. f W i W c and W o These are the weight matrices for each control gate, b f b i b c and b o These are the deviation matrices for each control gate, and * represents the convolution operation.
[0047] A multi-layer LSTM network is constructed as the core of the time series prediction. The input feature vector of the model includes: lateral displacement, longitudinal displacement, lateral velocity, longitudinal velocity, and total acceleration. The output is the lateral and longitudinal displacements for several future time steps.
[0048] A Dropout layer is embedded between the hidden layers of an LSTM. During training, this layer randomly "drops out" a subset of neurons with a preset probability p, causing their output to be zero. This regularization technique forces the network to learn more robust features without relying on any specific neuron, effectively suppressing overfitting.
[0049] 2. Model Training: The processed trajectory dataset was divided into training and validation sets proportionally. A stochastic gradient descent optimizer was used with an initial learning rate of 0.1 and a learning rate decay strategy.
[0050] During model training, a stochastic gradient descent optimizer is used for parameter updates, with an initial learning rate set to 0.1. Choosing a relatively high initial learning rate aims to quickly move the model weights in the direction of the decreasing loss function during the early stages of training, accelerating the convergence process. To further improve training stability and final performance, a learning rate decay strategy is introduced: as the training epochs increase, the learning rate gradually decreases according to a predetermined rule. This can be achieved through exponential decay or epoch-based decay, allowing for rapid approximation of the optimal solution region with large steps in the early stages of training, followed by fine-tuning of model parameters with small steps in the later stages. This avoids oscillations around the optimal value, thereby improving the model's generalization ability and enabling it to achieve more robust and accurate trajectory prediction performance on the validation set. This method, combined with the regularization effect of the Dropout layer in the aforementioned LSTM network, jointly suppresses overfitting and enhances the robustness of the time-series prediction module.
[0051] The root mean square error (RMSE) and mean absolute error (MAE) between the predicted displacement and the actual displacement are used as the loss function and evaluation index.
[0052] The specific formula is as follows: (10) (11) In the formula, n represents the total number of samples used to calculate the error, specifically the total number of trajectory points; i represents the index, the i-th sample; This represents the true value of the i-th sample, specifically the true lateral or longitudinal displacement of the forklift position at a time step. This represents the model prediction value for the i-th sample, specifically the forklift displacement predicted by the LSTM model at a future time step.
[0053] During evaluation, the model outputs the lateral and longitudinal displacements for future time steps, calculating their RMSE and MAE compared to the true values. Accuracy is determined by directly comparing these values; for example, in a turning scenario, the lateral displacement RMSE is 4.0 pixels, and the longitudinal displacement is 1.1 pixels. Simultaneously, the robustness of the model across different time periods is analyzed by observing the trends of these error values as the prediction time domain lengthens. Furthermore, pixel errors are converted into actual physical distances through spatial mapping to intuitively reflect the practical accuracy of the prediction results. This is supplemented by a visual comparison of the predicted and actual trajectories to qualitatively assess their morphological consistency. This evaluation system integrates quantitative calculation and qualitative analysis, jointly verifying the accuracy of the prediction model in the short term and its adaptability to complex nonlinear motion.
[0054] Example 1 In this embodiment, the improved YOLOv8 model reduces the number of parameters by 74.9%, the model size by 74.3%, and the detection speed is significantly improved to 145 FPS, providing a solid guarantee for real-time deployment on embedded devices. Specific parameters are shown in Table 1 below: Table 1. Parameters of the Improved YOLOv8 Model
[0055] like Figure 7a , Figure 7b As shown, quantitative analysis reveals that the root mean square errors of the predicted trajectory in lateral and longitudinal displacements are 4.0 pixels and 1.1 pixels, respectively, while the mean absolute errors are 3.2 pixels and 0.9 pixels, respectively. The spatial distribution of the predicted trajectory maintains a high degree of consistency with the actual trajectory (see...). Figure 8 The fact that there was no significant deviation between the starting point of the turn and the area of path curvature change indicates that this model can predict the image space motion position in a certain time domain in the future based on the current image space trajectory of the forklift.
[0056] The LSTM model with Dropout regularization performs well when the forklift is moving straight or turning (see...). Figure 10a , 10b It exhibits stable predictive performance in complex motion modes such as 10c and swing (see [reference]). Figure 9 , Figure 11 ).
Claims
1. A forklift trajectory prediction method based on improved visual perception and spatiotemporal modeling, characterized in that, Includes the following steps: Step S1, improve the visual perception of the object detection model: Step S 11 The backbone network of the object detection model adopts a depthwise separable convolutional and inverted residual structure, and introduces a lightweight attention mechanism and activation function to extract features; Step S 12 The asymptotic pyramid network AFPN is adopted, which allows features from non-adjacent levels to interact directly. The network learns autonomously to assign weights to each pixel in the spatial location of feature maps from different levels, thereby enhancing the features. step S 13 The pixel-level intersection-union ratio (PIoU) loss function is used to minimize the normalized distance between the four boundaries of the predicted box and the ground truth box, guiding the model to perform bounding box regression along a more efficient path. Step S2, Trajectory Sequence Extraction and Data Preprocessing: Step S 21 The improved target detection model is used to perform frame-by-frame target detection and tracking on the acquired forklift operation video, and the pixel coordinates of the center point of the forklift bounding box in each frame are extracted to form an initial trajectory sequence. Step S 22 Based on the spatial mapping relationship between camera calibration parameters and scene, pixel coordinates are converted into actual physical coordinates, and the instantaneous velocity and total acceleration of the forklift are calculated. Step S 23 The original displacement and velocity sequences are smoothed using a filter. Step S3, Time Series Trajectory Prediction: Step S 31 Construct a multi-layer LSTM network model, determine the input and output feature vectors of the model, and embed a Dropout layer between the hidden layers of the LSTM to randomly drop a portion of neurons with a preset probability p, so that their output is zero. Step S 32 The processed trajectory dataset is divided into training and validation sets according to the proportions for model training, and the loss function and evaluation metric are determined.
2. The forklift trajectory prediction method based on improved visual perception and spatiotemporal modeling according to claim 1, characterized in that, Step S 11 It includes the following specific content: The object detection model is YOLOv8, with MobileNetV3 as the backbone. A lightweight attention mechanism, SENet, is introduced. After the feature maps pass through 3×3 deep convolutional layers, the output features undergo global average pooling, followed by a fully connected layer. The h-swish activation function is used to generate feature vectors and assign weights to each channel. The weighted feature vectors then yield the output features. The activation function uses a rectified linear unit (ReLU6) to crop the input values. The h-swish activation function is as follows: In the formula, x is the input value of the current layer.
3. The forklift trajectory prediction method based on improved visual perception and spatiotemporal modeling according to claim 2, characterized in that, Step S 12 It includes the following specific content: During feature fusion, an adaptive spatial feature fusion mechanism is adopted, and the calculation formula is as follows: In the formula, This represents the spatial importance weights of three different levels of feature mappings. These represent the output and the output feature vector, respectively. Indicates adjustment to the number layer.
4. The forklift trajectory prediction method based on improved visual perception and spatiotemporal modeling according to claim 3, characterized in that, Step S 31 It includes the following specific content: An LSTM network consists of a forget gate, an input gate, and an output gate. During continuous data transmission, the current state x... t and the output h of the previous sequence t-1 The combination is the input vector (h) at time t. t-1 x t The input vector passes through each control gate sequentially and updates the cell state C. t Finally, calculate the output h. t The specific calculation formula is as follows: In the formula, σ is the activation function, and f t i t and o t Let W be the output of the forget gate, the output gate, and the output of the output gate at time t. f W i W c and W o These are the weight matrices for each control gate, b f b i b c and b o These are the deviation matrices for each control gate, and * represents the convolution operation.
5. The forklift trajectory prediction method based on improved visual perception and spatiotemporal modeling according to claim 4, characterized in that, The input feature vector of the model includes lateral displacement, longitudinal displacement, lateral velocity, longitudinal velocity, and total acceleration; the output feature vector of the model includes lateral displacement and longitudinal displacement at several future time steps.
6. The forklift trajectory prediction method based on improved visual perception and spatiotemporal modeling according to claim 5, characterized in that, Step S 32 It includes the following specific content: The stochastic gradient descent optimizer is used with an initial learning rate of 0.1 and a learning rate decay strategy. As the training epochs increase, the learning rate is gradually reduced according to a predetermined rule, using either exponential decay or epoch-based decay. This allows the model to approach the optimal solution region with large steps in the early stages of training and adjust the model parameters with small steps in the later stages of training, avoiding oscillations around the optimal value. This improves the model's generalization ability and enables it to achieve more robust and accurate trajectory prediction performance on the validation set.
7. The forklift trajectory prediction method based on improved visual perception and spatiotemporal modeling according to claim 6, characterized in that, The loss function is the root mean square error (RMSE) and the mean absolute error (MAE) between the predicted displacement and the actual displacement. In the formula, n represents the total number of samples used to calculate the error, specifically the total number of trajectory points; i represents the index, the i-th sample; This represents the true value of the i-th sample, specifically the true lateral or longitudinal displacement of the forklift position at a time step. This represents the model prediction value for the i-th sample, specifically the forklift displacement predicted by the LSTM model at a future time step.
8. The forklift trajectory prediction method based on improved visual perception and spatiotemporal modeling according to claim 7, characterized in that, The specific evaluation method for the aforementioned evaluation indicators is as follows: The model outputs the lateral and longitudinal displacements for future time steps, calculates the root mean square error (RMSE) and mean absolute error (MAE) between these displacements and the true values, and determines the accuracy by directly comparing their values. At the same time, by observing the trend of these error values as the prediction time domain extends, the robustness of the model in different time periods is analyzed. Pixel errors are converted into actual physical distances through spatial mapping relationships to intuitively reflect the practical accuracy of the prediction results. Furthermore, a visual comparison between the predicted trajectory and the actual trajectory is used to qualitatively evaluate the consistency of their morphology.
9. The forklift trajectory prediction method based on improved visual perception and spatiotemporal modeling according to claim 1 or 8, characterized in that, Step S 23 The filter described is the Savitzky-Golay filter.
10. A forklift trajectory prediction system based on improved visual perception and spatiotemporal modeling, characterized in that, include: The visual perception module, based on the YOLOv8 model, uses a depthwise separable convolutional and inverted residual structure in the backbone network MobileNetV3, and introduces a lightweight attention mechanism and h-swish activation function for feature extraction. An asymptotic pyramid network AFPN is adopted to enhance effective features through a progressive fusion strategy; In the regression branch, the PIOU loss function is used to minimize the normalized distance between the predicted box and the four boundaries of the ground truth box, guiding the model to perform bounding box regression along a more efficient path. The trajectory sequence extraction and data preprocessing module uses the improved YOLOv8 model to perform frame-by-frame target detection and tracking on the acquired forklift operation video, extracting the pixel coordinates of the center point of the forklift bounding box in each frame to form an initial trajectory sequence. Based on the spatial mapping relationship between camera calibration parameters and scene, pixel coordinates are converted into actual physical coordinates, and the kinematic characteristics of the forklift's instantaneous velocity and total acceleration are calculated. The original displacement and velocity sequences were smoothed using a Savitzky-Golay filter. The time series prediction module constructs a multi-layer LSTM network as the core of time series prediction, embedding Dropout layers between the hidden layers of the LSTM; the processed trajectory dataset is divided into training and validation sets according to the proportion, and the root mean square error (RMSE) and mean absolute error (MAE) between the predicted displacement and the actual displacement are used as the loss function and evaluation index.