An auxiliary positioning processing method based on forklift image data acquisition

Abstract

The present application discloses an auxiliary positioning processing method based on forklift image data acquisition, the method comprising: using a camera to scan the storage line on the ground to obtain image data of the forklift's surrounding environment; inputting the image data into a processing model, the processing model comprising an integrated model of binary spikes and Sigma Delta neural networks (SDNNs); the processing model utilizes a spatiotemporal constraint learning mechanism to extract time series features and spatial structure information from the image data; based on the time series features and spatial structure information, generating the position information of the forklift relative to the storage line; and utilizing the position information to assist in positioning the forklift to improve the positioning accuracy of the forklift in the warehouse.

Description

Technical Field

[0001] The present invention belongs to the technical field of forklift automation, and in particular relates to an auxiliary positioning processing method and system based on forklift image data acquisition, which is used to improve the positioning accuracy of a forklift in a warehouse. Background Art

[0002] With the rapid development of the logistics industry, forklifts, as important warehousing and handling equipment, are becoming increasingly automated and intelligent. Accurate positioning is one of the key factors in achieving automated forklift operations. At present, forklift positioning mainly relies on technologies such as GPS, LiDAR or visual navigation. However, these technologies still face some challenges in complex warehouse environments:

[0003] GPS signals are easily interfered with in indoor environments, making it difficult to provide high-precision positioning results;

[0004] Although LiDAR can provide accurate distance measurement, the equipment is expensive and errors occur in dynamic environments;

[0005] Traditional visual navigation methods usually only use single spatial information and are unable to cope with complex situations such as lighting changes and occlusions. Summary of the invention

[0006] In view of this, the present application provides an auxiliary positioning processing method and system based on forklift image data acquisition, which solves the problem of insufficient forklift positioning accuracy in the prior art.

[0007] The embodiment of the present application provides an auxiliary positioning processing method based on forklift image data acquisition, including:

[0008] Use the camera to scan the storage location lines on the ground to obtain image data of the forklift's surroundings;

[0009] Inputting the image data into a processing model, the processing model comprising an integrated model of binary spikes and SigmaDelta neural networks (SDNNs);

[0010] The processing model utilizes a spatiotemporal constraint learning mechanism to extract time series features and spatial structure information from the image data;

[0011] Based on the time series characteristics and the spatial structure information, generating the position information of the forklift relative to the storage location line;

[0012] The position information is used to assist in positioning the forklift to improve the positioning accuracy of the forklift in the warehouse.

[0013] The processing model utilizes a spatiotemporal constraint learning mechanism to extract time series features and spatial structure information from the image data, including:

[0014] Perform preliminary feature extraction on image data using binary spiking networks;

[0015] Inputting the extracted preliminary features into the Sigma Delta neural network for spatiotemporal feature learning;

[0016] By means of a spatiotemporal constraint learning mechanism, the temporal continuity and spatial correlation of the image data are simultaneously considered;

[0017] The time series features and the spatial structure information are output.

[0018] Before inputting the image data into the processing model, the method further comprises the following steps:

[0019] applying Taylor expansion to the feature extractor of the processing model to recast the position metric as an input-dependent squared error of the Jacobian matrix;

[0020] Based on the recast positional metrics, local features of the image data are calculated block by block;

[0021] The Jacobian matrix is ​​approximated using random dimensionality reduction techniques to reduce computational complexity.

[0022] The block-by-block calculation of local features of the image data includes:

[0023] Dividing the input image data into a plurality of overlapping local blocks;

[0024] Apply a lightweight feature extraction network to each local block;

[0025] Calculate the feature representation and corresponding Jacobian matrix of each local block;

[0026] Use the weighted average method to fuse the features and Jacobian matrices of all local blocks;

[0027] Based on the fusion results, a globally consistent forklift position estimate is generated.

[0028] The processing model also includes a feature fusion network based on an attention mechanism, which generates the position information of the forklift relative to the storage location line based on the time series features and spatial structure information, including:

[0029] The feature fusion network based on the attention mechanism receives the time series features and spatial structure information;

[0030] Performing multi-scale decomposition on the time series features and spatial structure information to obtain feature representations at different scales;

[0031] Using the attention mechanism, the importance weights of features of different scales for forklift positioning are calculated. The importance weights are used to highlight the spatiotemporal feature information that is most critical to forklift positioning and suppress irrelevant or redundant feature information. The spatiotemporal feature information that is most critical to forklift positioning includes the geometric structure of the storage line and the temporal pattern of forklift movement, and the irrelevant or redundant feature information includes background noise or non-critical objects.

[0032] According to the calculated importance weight, adaptively weighted fusion of the time series features and spatial structure information is performed;

[0033] Based on the weighted fused features, construct the first spatiotemporal feature representation;

[0034] Using the first spatiotemporal feature representation, generating the position coordinates or relative position description of the forklift through a regression or classification model;

[0035] Output the generated location information.

[0036] The method further comprises:

[0037] Preprocessing the input image data using spectral diffusion MRI technology to enhance image contrast and details;

[0038] Extracting the time series features and spatial structure information of the enhanced image data, and inputting them into the attention-based feature fusion network;

[0039] Focus on key areas enhanced by spectral diffusion through the attention mechanism;

[0040] Fusing image information from multiple frequency bands to generate a second spatiotemporal feature representation;

[0041] The second spatiotemporal feature representation is used to generate the position coordinates or relative position description of the forklift through a regression or classification model to improve the accuracy of the forklift position estimation.

[0042] The preprocessing of the input image data using the spectral diffusion MRI technology includes:

[0043] Collect multispectral image data of warehouse environment;

[0044] Denoising and enhancing multispectral images using diffusion models;

[0045] Construct a spectral-spatial joint representation to capture the correlation between different spectral channels;

[0046] The enhanced multispectral data is input into the feature fusion network;

[0047] Using the attention mechanism, the spectral information that is most beneficial to forklift positioning is adaptively selected;

[0048] Based on the selected spectral information, a position feature representation is generated.

[0049] Utilizing the position information to assist in positioning the forklift includes:

[0050] Use the laser radar on the top of the forklift to obtain three-dimensional environmental data;

[0051] registering the three-dimensional environment data with the location information;

[0052] Using a multimodal data fusion algorithm to fuse the three-dimensional point cloud data and the position information;

[0053] Based on the fused data, an accurate 3D model of the forklift’s surroundings is constructed;

[0054] The three-dimensional model is used to locate and navigate the forklift.

[0055] The spatiotemporal constraint learning mechanism includes:

[0056] Design a temporal convolution module to capture short-term and long-term temporal dependencies in image sequences;

[0057] Build a spatial attention module to identify the spatial regions in the image that are most critical for positioning;

[0058] Use the spatiotemporal coupling layer to establish the association between temporal features and spatial features;

[0059] Train the network in an end-to-end manner to optimize the extraction of temporal and spatial features;

[0060] Regularization constraints are introduced to ensure that the learned features have good spatiotemporal consistency.

[0061] The method further comprises:

[0062] Based on historical positioning data, a forklift motion prediction model is established;

[0063] Combine the current image data processing results with the motion prediction to generate a short-term trajectory estimate of the forklift;

[0064] Using the trajectory estimation, the motion control strategy of the forklift is optimized to avoid collision with surrounding cargo.

[0065] The present application also provides a computer device, the computer device comprising:

[0066] at least one processor; and,

[0067] a memory communicatively connected to the at least one processor; wherein,

[0068] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned auxiliary positioning processing method based on forklift image data acquisition.

[0069] An embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned auxiliary positioning processing method based on forklift image data acquisition.

[0070] An embodiment of the present application further provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the above-mentioned auxiliary positioning processing method based on forklift image data acquisition.

[0071] This application has the following technical effects:

[0072] 1. By combining the binary spike network and the Sigma Delta neural network, efficient feature extraction and spatiotemporal feature learning of image data are achieved, improving the accuracy and robustness of forklift positioning;

[0073] 2. The spatiotemporal constraint learning mechanism is adopted, which takes into account the temporal continuity and spatial correlation of image data, effectively captures the dynamic characteristics of forklift movement, and improves the accuracy of positioning;

[0074] 3. Introducing a feature fusion network based on the attention mechanism to adaptively select and fuse spatiotemporal features of different scales, highlighting the most critical information for forklift positioning, suppressing irrelevant or redundant features, and further improving positioning accuracy;

[0075] 4. The image data is preprocessed using spectral diffusion MRI technology to enhance the contrast and details of the image. The position feature representation is enriched through the fusion of multi-spectral information, which improves the adaptability of the system in complex environments.

[0076] 5. Combined with the 3D environmental data obtained by LiDAR, multi-modal data fusion is achieved, and an accurate 3D environmental model is constructed, providing more comprehensive information support for the positioning and navigation of forklifts;

[0077] 6. Through motion prediction model and short-term trajectory estimation, the motion control strategy of the forklift is optimized, improving the safety and efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 A flowchart of an auxiliary positioning processing method based on forklift image data acquisition provided by an embodiment of the present invention;

[0080] Figure 2 A schematic diagram of the structure of a processing model in an embodiment of the present invention;

[0081] Figure 3 Schematic diagram of a spatiotemporal constraint learning mechanism in an embodiment of the present invention;

[0082] Figure 4 It is a structural diagram of a feature fusion network based on an attention mechanism in an embodiment of the present invention;

[0083] Figure 5 This is a flow chart of the preprocessing of the spectral diffusion MRI technology in an embodiment of the present invention;

[0084] Figure 6 Schematic diagram of multimodal data fusion in an embodiment of the present invention;

[0085] Figure 7 A schematic diagram of forklift motion prediction and trajectory estimation in an embodiment of the present invention;

[0086] Figure 8 It is a schematic diagram of forklift auxiliary positioning using image acquisition in an embodiment of the present invention. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0088] Figure 1 FIG. 1 is a flow chart of an auxiliary positioning processing method based on forklift image data acquisition in an embodiment of the present invention. Figure 1 As shown, the embodiment of the present invention includes the following steps S1-S5:

[0089] S1: Use the camera to scan the storage location lines on the ground to obtain image data of the forklift's surrounding environment;

[0090] S2: inputting the image data into a processing model, wherein the processing model includes an integrated model of binary spikes and SigmaDelta neural networks SDNNs;

[0091] Before inputting the image data into the processing model, the method further comprises the following steps:

[0092] A1: applying Taylor expansion to the feature extractor of the processing model to recast the position metric as the input-dependent squared error of the Jacobian matrix;

[0093] A2: Based on the recast position metric, local features of the image data are calculated block by block;

[0094] A2.1: Divide the input image data into a plurality of overlapping local blocks;

[0095] A2.2: Apply a lightweight feature extraction network to each local block;

[0096] A2.3: Calculate the feature representation and corresponding Jacobian matrix of each local block;

[0097] A2.4: Use the weighted average method to fuse the features and Jacobian matrices of all local blocks;

[0098] A2.5: Generate a globally consistent forklift position estimate based on the fusion results.

[0099] A3: Use random dimensionality reduction techniques to approximate the Jacobian matrix and reduce computational complexity.

[0100] S3: The processing model uses a spatiotemporal constraint learning mechanism to extract time series features and spatial structure information from the image data;

[0101] S3.1: Preliminary feature extraction of image data using binary spiking networks;

[0102] S3.2: Inputting the extracted preliminary features into the Sigma Delta neural network for spatiotemporal feature learning;

[0103] S3.3: Considering both the temporal continuity and spatial correlation of the image data through a spatiotemporal constraint learning mechanism;

[0104] Wherein, the spatiotemporal constraint learning mechanism includes:

[0105] S3.3.1: Design a temporal convolution module to capture short-term and long-term temporal dependencies in image sequences;

[0106] S3.3.2: Build a spatial attention module to identify the spatial regions in the image that are most critical for localization;

[0107] S3.3.3: Use the spatiotemporal coupling layer to establish the association between temporal features and spatial features;

[0108] S3.3.4: Train the network in an end-to-end manner while optimizing the extraction of temporal and spatial features;

[0109] S3.3.5: Introduce regularization constraints to ensure that the learned features have good spatiotemporal consistency.

[0110] S3.4: Output the time series features and the spatial structure information.

[0111] Optionally, the processing model further includes a feature fusion network based on an attention mechanism, and S3 generates the position information of the forklift relative to the storage location line based on the time series features and the spatial structure information, including:

[0112] B1: The feature fusion network based on the attention mechanism receives the time series features and spatial structure information;

[0113] B2: Perform multi-scale decomposition on the time series features and spatial structure information to obtain feature representations of different scales;

[0114] B3: Use the attention mechanism to calculate the importance weights of features of different scales for forklift positioning;

[0115] B4: According to the calculated importance weight, adaptively weighted fusion of the time series features and spatial structure information is performed;

[0116] B5: Based on the weighted fusion features, construct the first spatiotemporal feature representation;

[0117] B6: using the first spatiotemporal feature representation, generating the position coordinates or relative position description of the forklift through a regression or classification model;

[0118] B7: Output the generated position information.

[0119] In another embodiment, the feature fusion network based on the attention mechanism can also be combined with MRI technology. Compared with the position information output of B1-B7, the C1-C5 method can also be used to increase the accuracy of the position information output:

[0120] C1: preprocessing the input image data using spectral diffusion MRI technology to enhance the contrast and details of the image;

[0121] C1.1: Collect multispectral image data of the warehouse environment;

[0122] C1.2: Denoising and enhancing multispectral images using diffusion models;

[0123] C1.3: Construct a spectral-spatial joint representation to capture the correlation between different spectral channels;

[0124] C1.4: Input the enhanced multispectral data into the feature fusion network;

[0125] C1.5: Using the attention mechanism, adaptively select the spectral information that is most beneficial to forklift positioning;

[0126] C1.6: Generate positional feature representation based on the selected spectral information.

[0127] C2: extracting the time series features and spatial structure information of the enhanced image data, and inputting them into the attention-based feature fusion network;

[0128] C3: Focus on the key areas enhanced by spectral diffusion through the attention mechanism;

[0129] C4: Fuse image information from multiple frequency bands to generate the second spatiotemporal feature representation;

[0130] C5: Using the second spatiotemporal feature representation, the position coordinates or relative position description of the forklift is generated through a regression or classification model to improve the accuracy of the forklift position estimation.

[0131] S4: generating position information of the forklift relative to the storage location line based on the time series characteristics and the spatial structure information;

[0132] S5: Utilize the position information to assist in positioning the forklift to improve the positioning accuracy of the forklift in the warehouse.

[0133] Wherein, using the position information to assist in positioning the forklift includes:

[0134] S5.1: Use the lidar on the top of the forklift to obtain three-dimensional environmental data;

[0135] S5.2: aligning the three-dimensional environment data with the location information;

[0136] S5.3: Using a multimodal data fusion algorithm to fuse the three-dimensional point cloud data and the position information;

[0137] S5.4: Based on the fused data, construct an accurate three-dimensional model of the forklift's surroundings;

[0138] S5.5: Use the three-dimensional model to locate and navigate the forklift.

[0139] In addition, the method further comprises:

[0140] D1: Build a forklift motion prediction model based on historical positioning data;

[0141] D2: Combine the current image data processing results and motion prediction to generate a short-term trajectory estimate of the forklift;

[0142] D3: Using the trajectory estimation, optimize the motion control strategy of the forklift to avoid collision with surrounding cargo.

[0143] Each step in the above embodiment is described in detail:

[0144] S1: Use the camera to scan the storage location lines on the ground to obtain image data of the forklift's surroundings.

[0145] In this step, the embodiment of the present application uses a camera installed on a forklift to scan the storage line on the ground and obtain image data of the forklift's surrounding environment. The purpose of this step is to provide raw data input for subsequent positioning processing.

[0146] When implementing this, a high-resolution industrial camera with an appropriate wide-angle lens can be used to ensure that a sufficiently wide field of view can be captured. The camera should be installed in an appropriate position on the forklift, usually at the front or bottom of the forklift, so that the storage line on the ground can be clearly captured.

[0147] The acquired image data is usually in RGB format, and the resolution can be selected according to specific needs, such as 1920x1080 or higher. These image data will serve as the basis for subsequent processing to extract the characteristics of the forklift's surroundings, especially the location and shape information of the ground storage line.

[0148] S2: Inputting the image data into a processing model, the processing model comprising an integrated model of binary spikes and SigmaDelta neural networks SDNNs.

[0149] The purpose of this step is to input the acquired image data into a specially designed processing model for efficient feature extraction and spatiotemporal information learning. This processing model is an innovative integrated model such as Figure 2 As shown in Figure 3, the proposed model combines the advantages of binary spiking networks and Sigma Delta Neural Networks (SDNNs).

[0150] Among them, the characteristics of binary spike networks are:

[0151] 1. Simulate the pulse emission mechanism of biological neurons and be able to process timing information;

[0152] 2. It has the characteristics of low power consumption and fast response, suitable for real-time processing;

[0153] 3. Perform sparse coding on input data to improve information transmission efficiency.

[0154] The characteristics of Sigma Delta Neural Networks (SDNNs):

[0155] 1. Good at processing continuous time series data;

[0156] 2. Have good anti-noise ability and signal reconstruction ability;

[0157] 3. The quantization accuracy can be adjusted adaptively to balance accuracy and computational complexity.

[0158] The implementation steps of the integrated model are:

[0159] 1. First, preprocess the image data into a format suitable for binary spike network input, such as normalizing pixel values ​​to the [0, 1] interval;

[0160] 2. Design the topology of the binary spike network, including the input layer, hidden layer, and output layer. The number of neurons in each layer is determined according to the specific task requirements;

[0161] 3. Implement the Sigma Delta neural network and design its network structure, including the Sigma-Delta modulator and digital filter;

[0162] 4. Establish connections between the binary spike network and the SDNNs so that the output of the binary spike network serves as the input of the SDNNs;

[0163] 5. Design a joint training strategy to optimize the parameters of both networks simultaneously.

[0164] The advantage of this integrated model is that it can not only efficiently process spatial information using binary spike networks, but also accurately capture time series features through SDNNs, thus providing a more comprehensive and stable feature representation for forklift positioning.

[0165] In addition, in the embodiment of the present application, before the image data is input into the processing model, the following steps are also included:

[0166] A1: Apply Taylor expansion to the feature extractor of the processing model to recast the position metric as the input-dependent squared error of the Jacobian matrix.

[0167] The goal of this sub-step is to optimize the feature extraction process to make it more suitable for the location estimation task.

[0168] Implementation methods include:

[0169] A1.1. Perform Taylor expansion on the feature extractor f(x) around the current input x_0:

[0170] f(x) ≈ f(x_0) + J(x_0)(x x_0), where J(x_0) is the Jacobian matrix;

[0171] A1.2. Re-express the location metric L(y, f(x)) as a function of the input:

[0172] L(y, f(x)) ≈ L(y, f(x_0) + J(x_0)(x x_0));

[0173] A1.3. Further approximation is the input-dependent squared error form:

[0174] L(y, f(x)) ≈ ||yf(x_0) J(x_0)(x x_0)|| 2 ;

[0175] A1.4. Implement efficient Jacobian matrix calculation methods, such as automatic differentiation techniques;

[0176] This approach is able to transform the position estimation problem into a more tractable form while preserving the local linear relationship between input and output.

[0177] A2: Based on the recast position metric, local features of the image data are calculated block by block;

[0178] This sub-step aims to improve the efficiency and accuracy of feature extraction through block processing.

[0179] A2.1: Divide the input image data into a plurality of overlapping local blocks;

[0180] Determine an appropriate block size, such as 64x64 or 128x128 pixels;

[0181] Use a sliding window technique that moves across the image with a certain step size (e.g., half the block size);

[0182] For image edges, padding techniques can be used to ensure complete coverage.

[0183] A2.2: Apply a lightweight feature extraction network to each local block;

[0184] Design a small convolutional neural network (CNN), such as a 3-5 layer structure; use techniques such as depthwise separable convolution to reduce the amount of computation; consider using pre-trained lightweight models, such as some layers of MobileNetV2.

[0185] A2.3: Calculate the feature representation and corresponding Jacobian matrix of each local block;

[0186] For each local block, a feature vector is obtained through a feature extraction network;

[0187] Compute the Jacobian matrix for each block using automatic differentiation;

[0188] Store the eigenvectors and Jacobian matrices in preparation for subsequent fusion.

[0189] A2.4: Use the weighted average method to fuse the features and Jacobian matrices of all local blocks;

[0190] Design a position-based weighting scheme, such as Gaussian weights, to perform weighted averaging of features and Jacobian matrices in overlapping regions;

[0191] Handle edge cases to ensure smooth transitions.

[0192] A2.5: Generate a globally consistent forklift position estimate based on the fusion results;

[0193] 1. Use the fused features and Jacobian matrix to construct the global position estimation problem;

[0194] 2. Use optimization methods (such as gradient descent or Newton's method) to solve for the optimal position estimate;

[0195] 3. Apply constraints to ensure that the estimated results are within a reasonable range.

[0196] A3: Use random dimensionality reduction technology to approximate the Jacobian matrix to reduce computational complexity;

[0197] The purpose of this step is to reduce computational complexity while maintaining accuracy.

[0198] 1. Choose an appropriate random dimensionality reduction technique, such as random projection or Johnson-Lindenstrauss transform;

[0199] 2. Apply random dimensionality reduction to the Jacobian matrix to map the high-dimensional space to a low-dimensional space;

[0200] 3. Perform subsequent calculations in the reduced-dimensional space, such as feature fusion and position optimization;

[0201] 4. If necessary, map the results back to the original high-dimensional space.

[0202] Through these preprocessing steps, the embodiment of the present application can process a large amount of image data more efficiently while maintaining the accuracy of position estimation, which lays a good foundation for the subsequent main processing flow.

[0203] Exemplarily, it is assumed that the embodiment of the present invention has an image of a warehouse environment, which includes a forklift and surrounding shelves, aisles, etc. The goal of the embodiment of the present invention is to estimate the position of the forklift based on this image.

[0204] A1. Application of Taylor expansion:

[0205] Assume that the Taylor expansion of the feature extractor f(x) in the embodiment of the present invention around the current input x_0 is:

[0206] f(x) ≈ f(x_0) + J(x_0)(x - x_0);

[0207] where J(x_0) is the Jacobian matrix.

[0208] The position metric L(y, f(x)) can be rewritten as:

[0209] L(y, f(x)) ≈ ||y - f(x_0) - J(x_0)(x - x_0)||²;

[0210] A2. Calculate local features block by block:

[0211] A2.1. Image segmentation:

[0212] The 1920x1080 image is divided into a 12x9 grid, each local block size is 160x120 pixels, and there is 50% overlap between adjacent blocks.

[0213] A2.2. Feature Extraction:

[0214] A 5-layer lightweight CNN is applied to each 160x120 block, outputting a 128-dimensional feature vector.

[0215] A2.3. Calculate the characteristic and Jacobian matrices:

[0216] For each block, a 128-dimensional feature vector and a 128x(160*120) Jacobian matrix are obtained.

[0217] A2.4. Weighted average fusion:

[0218] Using Gaussian weights, the features and Jacobian matrices of the overlapping areas are weighted averaged.

[0219] A2.5. Generate global position estimate:

[0220] Based on the fused features and Jacobian matrix, an optimization algorithm (such as gradient descent) is used to solve the optimal forklift position.

[0221] A3. Random Dimensionality Reduction:

[0222] Random projection is used to reduce the Jacobian matrix of 128x(160*120) to 128x1000, which greatly reduces the amount of calculation.

[0223] This process combines local feature extraction with global optimization, which not only ensures computational efficiency but also obtains accurate global position estimation.

[0224] S3: The processing model uses a spatiotemporal constraint learning mechanism to extract time series features and spatial structure information from the image data;

[0225] This step is the core of the processing model, which aims to extract both time series features and spatial structure information from the image data to fully capture the forklift movement and environmental characteristics. The introduction of the spatiotemporal constraint learning mechanism ensures that the extracted features take into account both temporal continuity and spatial correlation.

[0226] S3.1: Preliminary feature extraction of image data using binary spiking networks;

[0227] In this sub-step, a binary spiking network is used to perform preliminary feature extraction on the input image data.

[0228] Specific implementation:

[0229] 1. Convert the image data into a pulse train, for example using rate encoding or time encoding methods;

[0230] 2. Design a multi-layer binary spiking neural network, where each layer contains multiple neuron groups;

[0231] 3. Define the dynamic equations of neurons, such as the Leaky Integrate-and-Fire (LIF) model;

[0232] 4. Implement synaptic plasticity rules, such as Spike-Timing-Dependent Plasticity (STDP);

[0233] 5. Propagate the spike signal through the network and collect the firing patterns of the neurons in the output layer.

[0234] The advantage of this method is that it can efficiently extract the key features of the image while retaining the temporal information, laying the foundation for subsequent spatiotemporal feature learning.

[0235] S3.2: Inputting the extracted preliminary features into the Sigma Delta neural network for spatiotemporal feature learning;

[0236] In this sub-step, the embodiment of the present application inputs the preliminary features extracted by the binary spike network into the SigmaDelta neural network for more in-depth spatiotemporal feature learning.

[0237] Sigma Delta Neural Network Implementation Steps:

[0238] 1. Design a Sigma-Delta modulator to convert the input signal into a bit stream;

[0239] 2. Implement digital filters to reconstruct and process signals;

[0240] 3. Construct a multi-layer SDNNs structure, each layer contains multiple Sigma-Delta neurons;

[0241] 4. Define forward propagation and back propagation algorithms for network training;

[0242] 5. Implement an adaptive quantization mechanism to dynamically adjust the quantization accuracy according to the characteristics of the input signal.

[0243] SDNNs can effectively learn and represent complex patterns in continuous time series and are particularly suitable for processing the dynamic characteristics of forklift motion.

[0244] S3.3: Considering both the temporal continuity and spatial correlation of the image data through a spatiotemporal constraint learning mechanism;

[0245] like Figure 3 As shown in Figure 1, the spatiotemporal constraint learning mechanism is to ensure that the extracted features not only preserve the temporal continuity of the image sequence, but also capture the spatial structure information. This is crucial for accurately locating and predicting the movement of forklifts. It includes:

[0246] 1. Design a temporal convolution module and use one-dimensional convolution operations to capture dependencies at different time scales;

[0247] 2. Build a spatial attention module to dynamically focus on key areas in the image related to positioning;

[0248] 3. Implement the spatiotemporal coupling layer to integrate temporal and spatial features;

[0249] 4. Introduce regularization constraints, such as spatiotemporal consistency loss, to ensure that the learned features are coherent in both time and space;

[0250] 5. Design an end-to-end training strategy that optimizes both temporal and spatial feature extraction.

[0251] This mechanism can effectively capture the dynamic characteristics of the forklift motion and the static structure of the environment, improving the accuracy and robustness of positioning.

[0252] Next, the embodiments of the present application will delve into the implementation details of the spatiotemporal constraint learning mechanism, which is a crucial part of the entire system. It ensures that the features extracted by the embodiments of the present application both retain temporal continuity and capture spatial correlation.

[0253] The core of the spatiotemporal constraint learning mechanism is to design a network structure that can process temporal and spatial information simultaneously. The embodiment of the present application first designs a temporal convolution module to capture short-term and long-term temporal dependencies in image sequences. This module can be built based on dilated convolutions, which can expand the receptive field without increasing the number of parameters. By stacking convolutional layers with different dilation rates, the embodiment of the present application can effectively capture dependencies at different time scales. For example, the first layer focuses on the relationship between adjacent frames, while deeper layers capture patterns over longer time spans.

[0254] At the same time, the embodiment of the present application constructs a spatial attention module for identifying the spatial areas in the image that are most critical for positioning. This module can use a self-attention mechanism to allow the model to dynamically focus on different spatial locations. Specifically, the embodiment of the present application can convert the input feature map into three representations of query, key, and value, and then generate attention weights by calculating the similarity between the query and the key. In this way, the model can adaptively adjust the attention to different spatial areas according to the current context.

[0255] In order to establish the association between temporal features and spatial features, the embodiment of the present application introduces a spatiotemporal coupling layer. The design inspiration of this layer can come from ConvLSTM or 3D convolutional network. It allows information to flow simultaneously in time and space dimensions, thereby capturing complex spatiotemporal correlations. For example, the embodiment of the present application can design a special gating mechanism to regulate the flow of temporal information according to the change of spatial features, and vice versa.

[0256] The training of the entire network adopts an end-to-end approach, while optimizing the extraction of temporal and spatial features. This means that the embodiment of the present application needs to design a comprehensive loss function that not only considers the accuracy of the final positioning, but also includes constraints on temporal continuity and spatial consistency. For example, the embodiment of the present application can introduce a temporal smoothing term to penalize drastic changes in prediction results between adjacent time steps; at the same time, the embodiment of the present application can add a spatial structure preservation term to ensure that the predicted spatial features are consistent with the known environmental structure.

[0257] In order to ensure that the learned features have good spatiotemporal consistency, the embodiments of the present application introduce regularization constraints. This can be achieved by adding additional loss terms. For example, the embodiments of the present application can use temporal difference regularization to encourage the features learned by the model to change smoothly in time; at the same time, the embodiments of the present application can apply spatial gradient regularization to promote the continuity of features in space. These regularization terms help improve the generalization ability of the model, so that it can maintain good performance when facing new and unseen scenes.

[0258] In addition, the embodiment of the present application may also consider introducing adversarial training strategies to further enhance the robustness of spatiotemporal features. Specifically, the embodiment of the present application may design a discriminator network whose task is to distinguish between real spatiotemporal sequences and sequences generated by the model. In this way, the generator (i.e., the main network of the embodiment of the present application) is forced to learn to produce more realistic and coherent spatiotemporal feature representations.

[0259] In practical applications, the effect of the spatiotemporal constraint learning mechanism can be verified and analyzed through visualization tools. For example, the embodiment of the present application can generate an attention heat map to intuitively show the spatial areas that the model focuses on at different time points; the embodiment of the present application can also draw feature activation maps to observe how spatiotemporal features at different levels evolve. These visualization tools not only help the embodiments of the present application to understand and improve the model, but also provide explanatory results for end users and enhance the credibility of the system.

[0260] Assume that in the embodiment of the present invention there is a continuous image sequence of a forklift moving in a warehouse, with 10 frames per second, 30 seconds in total, and 300 frames of images in total.

[0261] S3.1 Preliminary feature extraction of binary spiking network:

[0262] Input: 300 frames of 1280x720 RGB images;

[0263] Network structure: 3-layer binary spiking neural network, 100 neurons per layer;

[0264] Output: Each frame generates a 100-dimensional binary feature vector;

[0265] S3.2 Sigma Delta Neural Network Spatiotemporal Feature Learning:

[0266] Input: 300 100-dimensional feature vector sequences obtained from S3.1;

[0267] Network structure: 4 layers of SDNNs, 64 Sigma-Delta neurons in each layer;

[0268] Output: 300 64-dimensional continuous-valued feature vectors;

[0269] S3.3 Spatiotemporal Constrained Learning Mechanism:

[0270] S3.3.1 Temporal Convolution Module:

[0271] Use 1D dilated convolution, kernel size=3, dilation rates=[1,2,4,8];

[0272] Capture the temporal dependencies of 1, 2, 4, and 8 frames;

[0273] S3.3.2 Spatial Attention Module:

[0274] Divide the image into 8x8 grid;

[0275] Calculate the attention weight for each grid area;

[0276] Focus on key areas such as shelf edges and aisle lines;

[0277] S3.3.3 Spatiotemporal coupling layer:

[0278] Use ConvLSTM structure to fuse temporal features and spatial features;

[0279] Input shape: (300, 8, 8, 64), 300 is the time step, 8x8 is the spatial dimension, 64 is the feature dimension;

[0280] S3.3.4 End-to-end training:

[0281] Loss function: position estimation MSE + time smoothing loss + spatial consistency loss;

[0282] Optimizer: Adam, learning rate 0.001;

[0283] Batch size: 16 sequences, 30 frames per sequence;

[0284] S3.3.5 Regularization constraints:

[0285] L2 regularization: weight decay coefficient 0.0001;

[0286] Temporal difference regularization: penalizes drastic changes in features between adjacent time steps;

[0287] Spatial gradient regularization: encourages smooth feature distribution in space;

[0288] S3.4 Output:

[0289] Time series features: 300x64 matrix, representing the time series features of every 0.1 second within 30 seconds;

[0290] Spatial structure information: A 8x8x64 tensor representing the spatial features of each grid after the image is divided into 8x8 grids.

[0291] This process comprehensively utilizes the efficiency of binary spike networks, the temporal modeling capabilities of Sigma Delta neural networks, and the comprehensive feature extraction capabilities of spatiotemporal constraint learning mechanisms. It can effectively extract the spatiotemporal characteristics of forklift motion from image sequences, laying the foundation for subsequent precise positioning.

[0292] By carefully designing and implementing this spatiotemporal constraint learning mechanism, the system of the embodiment of the present application can more comprehensively and accurately understand the movement pattern and environment of the forklift in the warehouse. This not only improves the accuracy of positioning, but also provides a solid foundation for subsequent trajectory prediction and decision-making. In a dynamically changing warehouse environment, this in-depth understanding of spatiotemporal information is crucial to achieving safe and efficient forklift automation operations.

[0293] S3.4: outputting the time series features and the spatial structure information;

[0294] In this sub-step, the processing model will output the time series features and spatial structure information after learning the spatiotemporal constraints. These outputs will serve as the basis for subsequent positioning and decision-making.

[0295] The output features include:

[0296] 1. Time series features: representing dynamic information such as the speed, acceleration, and direction change of the forklift;

[0297] 2. Spatial structure information: including the geometry of the storage location line, the location of surrounding obstacles and other static environmental information;

[0298] 3. Joint spatiotemporal features: high-level feature representation that integrates temporal and spatial information.

[0299] These output features will provide the key basis for generating accurate position information of the forklift relative to the storage location line in the next step.

[0300] In one of the embodiments, in addition to the above-mentioned integrated model, the processing model also includes a feature fusion network based on an attention mechanism.

[0301] In this step, the embodiment of the present application introduces a feature fusion network based on the attention mechanism, such as Figure 4 As shown in the figure, it aims to more effectively integrate time series features and spatial structure information to generate accurate position information of the forklift relative to the warehouse line. The core idea of ​​this network is to improve the accuracy and robustness of position estimation by adaptively focusing on the most relevant features.

[0302] First, the feature fusion network based on the attention mechanism receives the time series features and spatial structure information extracted in the previous step. These features contain the dynamic information of the forklift movement and the static structure information of the surrounding environment. The input layer of the network is designed to be able to process these two types of features of different nature at the same time, laying the foundation for subsequent fusion.

[0303] Next, the network performs multi-scale decomposition on the received time series features and spatial structure information. This process is similar to a multi-level feature pyramid that can capture information at different scales. For example, for time series features, the embodiments of the present application focus on short-term (such as within a few seconds), medium-term (such as within a few minutes), and long-term (such as the entire task cycle) patterns; for spatial structure information, the embodiments of the present application focus on local details (such as the shape of a single storage location line), medium range (such as the layout of several storage locations), and global structure (such as the layout of the entire warehouse). This multi-scale decomposition helps the network understand the motion state and environment of the forklift at different levels.

[0304] The network then uses an attention mechanism to calculate the importance weights of features of different scales for forklift positioning. The core of this attention mechanism is a learnable function that can dynamically adjust the degree of attention to different features according to the current task context. For example, when a forklift is performing a precise alignment operation, it will pay more attention to local spatial structure information; while when the forklift moves a long distance in the warehouse, it will pay more attention to large-scale spatial features and long-term motion patterns. This dynamic adjustment capability enables the network to adapt to different positioning scenarios and improve the versatility and robustness of the system.

[0305] After calculating the importance weights, the network will perform an adaptive weighted fusion of the time series features and spatial structure information. This process can be understood as an intelligent feature selection and combination. The network will highlight the information that is most critical to the current positioning task, while suppressing irrelevant or redundant features. For example, in good light and clearly visible storage lines, the network will rely more on spatial structure information; in poor light or when the storage lines are partially blocked, the network will rely more on time series features to infer the location.

[0306] Based on the weighted fused features, the network constructs a comprehensive spatiotemporal feature representation (first spatiotemporal feature). This representation combines information from both time and space, providing a rich and comprehensive input for subsequent position estimation. This layer of the network usually uses advanced feature extraction techniques in deep learning, such as residual connections or dense connections, to ensure that information can be effectively transmitted and integrated.

[0307] Finally, the network uses this comprehensive spatiotemporal feature representation to generate the location coordinates or relative position description of the forklift through a regression or classification model. The regression model can directly output continuous coordinate values, which is suitable for scenarios that require high-precision positioning; while the classification model can discretize the warehouse space into multiple areas and output the area where the forklift is most located, which is suitable for coarse-grained positioning or scenarios that require rapid response. Regardless of which method is used, the network will output confidence or uncertainty estimates, providing important reference for subsequent decision-making.

[0308] Exemplarily, it is assumed that the embodiment of the present invention has the following input:

[0309] Time series features: 300x64 matrix (300 time steps, 64-dimensional features per step);

[0310] Spatial structure information: 8x8x64 tensor (8x8 spatial grid, 64-dimensional features per grid);

[0311] B1. Features Receiving:

[0312] The network receives both time series features and spatial structure information.

[0313] B2. Multi-scale decomposition:

[0314] Time Series Eigendecomposition:

[0315] 1. Small scale: features within 1 second (10 time steps);

[0316] 2. Mesoscale: features within 5 seconds (50 time steps);

[0317] 3. Large scale: global 30-second features (300 time steps);

[0318] Spatial structure information decomposition:

[0319] 1. Small scale: 2x2 grid;

[0320] 2. Medium scale: 4x4 grid;

[0321] 3. Large scale: original 8x8 grid;

[0322] B3. Attention mechanism calculates importance weights:

[0323] Use the multi-head attention mechanism to calculate the importance of each scale feature:

[0324] Temporal feature attention weights: [0.2, 0.3, 0.5] (small, medium, large scale);

[0325] Spatial feature attention weights: [0.1, 0.4, 0.5] (small, medium, large scale);

[0326] B4. Adaptive Weighted Fusion:

[0327] According to the attention weight, the features of different scales are weighted and summed:

[0328] Fused temporal features: 64-dimensional vector;

[0329] Fusion spatial features: 64-dimensional vector;

[0330] B5. Construct the first spatiotemporal feature representation:

[0331] The fused temporal features and spatial features are concatenated to obtain a 128-dimensional spatiotemporal joint feature representation.

[0332] B6. Generate location information:

[0333] Using the Multilayer Perceptron (MLP) regression model:

[0334] Input: 128-dimensional spatiotemporal features;

[0335] Hidden layers: 128 -> 64 -> 32;

[0336] Output: 3D vector [x, y, θ] (x, y are coordinates, θ is the orientation angle);

[0337] For example, the output is: [23.5, 45.2, 0.78], which means that the position of the forklift in the warehouse coordinate system is (23.5m, 45.2m) and the heading angle is 0.78 radians.

[0338] B7. Output location information:

[0339] Output [23.5, 45.2, 0.78] as the final position estimation result.

[0340] This process effectively integrates temporal and spatial information through multi-scale decomposition and attention mechanism, and can adaptively adjust the importance of features according to different situations, thereby obtaining a more accurate forklift position estimation.

[0341] Through this feature fusion method based on the attention mechanism, the embodiment of the present application can make full use of information from different sources and different scales, and adaptively adjust the positioning strategy, so as to achieve accurate forklift positioning in various complex environments. The flexibility and robustness of this method make it particularly suitable for application in dynamically changing warehouse environments.

[0342] In addition, the embodiment of the present invention may also use spectral diffusion MRI technology to pre-process the input image data.

[0343] In this step, the embodiment of the present application introduces an innovative preprocessing technology, spectral diffusion MRI technology, to enhance the quality and information richness of the input image. This technology is generally used in the field of medical imaging, but in the present invention it is cleverly applied to industrial visual scenes, providing more detailed and reliable visual information for forklift positioning.

[0344] like Figure 5 As shown, first, the embodiment of the present application needs to collect multispectral image data of the warehouse environment. This is different from traditional RGB images. Multispectral images contain a wider range of electromagnetic band information. The embodiment of the present application can use a special multispectral camera to capture images of multiple bands such as visible light, near infrared and short-wave infrared at the same time. The advantage of doing this is that environmental information that is difficult to detect with the naked eye, such as the reflective properties of different materials, can be obtained, which is very helpful for accurately identifying storage lines and other key structures.

[0345] Next, the embodiment of the present application uses a diffusion model to denoise and enhance the multispectral image. The core idea of ​​the diffusion model is to gradually remove the noise in the image by simulating the diffusion process in thermodynamics while retaining and enhancing important structural information. In practice, the embodiment of the present application can design a deep learning network to simulate this diffusion process, which learns how to restore the original signal under different noise levels. This method is particularly suitable for processing low-light, occluded or blurred images, which are common challenges in warehouse environments.

[0346] Then, the embodiment of the present application constructs a spectral-spatial joint representation to capture the correlation between different spectral channels. The purpose of this step is to fully utilize the advantages of multispectral data and mine the complementary information between different bands. The embodiment of the present application can design a special neural network layer, such as a spectral attention layer, to learn the interaction between different spectral channels. This joint representation can highlight the spectral features that are most valuable for forklift positioning while suppressing redundant or irrelevant information.

[0347] After the above processing, the embodiment of the present application inputs the enhanced multispectral data into the feature fusion network. The design of this network needs to take into account the characteristics of multispectral data, and the conventional convolutional layer needs to be modified to adapt to multi-channel input. The shallow layer of the network will contain a dedicated spectral processing module, while the deep layer will focus more on the extraction of spatial features.

[0348] In the feature fusion network, the embodiment of the present application uses the attention mechanism to adaptively select the spectral information that is most conducive to forklift positioning. This process can be understood as an intelligent "palette". The network will dynamically adjust the weights of different spectral channels according to the current environmental conditions and positioning requirements. For example, in the case of sufficient light, the visible light channel obtains a higher weight; in the case of insufficient light or strong reflection, the near infrared or short-wave infrared channel will become more important.

[0349] Finally, based on the selected spectral information, the embodiment of the present application generates a position feature representation. This representation combines the advantages of multispectral data, not only including rich visual information, but also including environmental features that are difficult to detect with the naked eye. This enhanced feature representation provides a more reliable and comprehensive input for subsequent positioning algorithms, which helps to improve the accuracy and robustness of positioning.

[0350] By introducing spectral diffusion MRI technology, the embodiment of the present application greatly enhances the ability of image preprocessing. This method not only improves image quality, but also expands the information dimension that can be used for positioning. In a complex warehouse environment, this technology can help the system better cope with various challenges, such as lighting changes, occlusion, reflection, etc., thereby providing more accurate and reliable positioning support for forklifts.

[0351] Next, the embodiments of the present application will continue to explore how to use the image data enhanced by the spectral diffusion MRI technology to further improve the positioning accuracy of the forklift.

[0352] After obtaining the enhanced image data, the embodiment of the present application needs to extract the time series features and spatial structure information therein, and input this information into the attention-based feature fusion network. This process involves complex feature extraction and fusion techniques, aiming to fully utilize the advantages of multispectral data.

[0353] In the feature extraction stage, the embodiment of the present application can use a specially designed multi-spectral convolutional neural network. This network is different from the traditional CNN, and it can process the information of multiple spectral channels at the same time. For example, the embodiment of the present application can design a "spectral convolution layer", which not only performs convolution operations in the spatial dimension, but also performs feature extraction in the spectral dimension. In this way, the relationship between different spectral channels can be captured, thereby extracting richer and more meaningful features.

[0354] For the extraction of time series features, the embodiments of the present application may consider using a long short-term memory network (LSTM) or a temporal convolutional network (TCN). These network structures are particularly suitable for processing sequence data and can effectively capture the temporal dynamic characteristics of forklift motion. The embodiments of the present application may input a continuous multispectral image sequence into these networks to extract the time series features reflecting the forklift motion pattern.

[0355] The extraction of spatial structural information can utilize the spatial attention mechanism. The embodiment of the present application can design a multi-scale spatial attention module, which can adaptively focus on the area in the image that is most relevant to positioning. For example, when extracting the features of the warehouse line, the attention mechanism will focus more on the ground area; while when identifying obstacles, it will pay more attention to other parts of the image.

[0356] After the extracted features are input into the attention-based feature fusion network, the embodiment of the present application focuses on the key areas enhanced by spectral diffusion through the attention mechanism. This process can be understood as a kind of "intelligent focusing", and the network will automatically adjust the attention to different features to maximize the use of useful information. For example, if a spectral channel shows a unique pattern in a specific area (such as the infrared channel more clearly shows the location of the heat source), the attention mechanism will give this area a higher weight.

[0357] Next, the embodiment of the present application fuses image information from multiple frequency bands to generate a more comprehensive and robust spatiotemporal feature representation. This fusion process is not a simple feature splicing, but an intelligent, context-aware fusion. The embodiment of the present application can use technologies such as gating mechanisms or dynamic convolution to dynamically adjust the fusion strategy according to current environmental conditions and task requirements. For example, in well-lit conditions, information in the visible light band dominates; in low-light conditions, information in the near-infrared band becomes more important.

[0358] Finally, the embodiment of the present application uses this fused spatiotemporal feature representation (second spatiotemporal feature) to generate the position coordinates or relative position description of the forklift through a regression or classification model. This model can be a deep neural network or a complex system that integrates multiple sub-models. Regardless of the method used, the output of the model should include the uncertainty of the position estimate, which is very important for subsequent decision making.

[0359] Assume that the embodiment of the present invention is in a large warehouse environment:

[0360] C1. Spectral Diffusion MRI Technology Preprocessing:

[0361] C1.1 Collect multispectral image data:

[0362] Use a multispectral camera to collect images in five bands: visible light (RGB), near infrared (NIR), and short-wave infrared (SWIR);

[0363] Each band resolution: 1920x1080;

[0364] Acquisition frequency: 10 frames / second, lasting 30 seconds, a total of 300 frames;

[0365] C1.2 Diffusion Model Denoising and Enhancement:

[0366] Diffusion model using U-Net structure;

[0367] Input: 5-channel noisy image;

[0368] Output: 5-channel denoised and enhanced image;

[0369] For example, the enhanced near-infrared channel will make shelf edges clearer;

[0370] C1.3 Constructing spectral-spatial joint representation:

[0371] Use 3D convolutional network, input size: (5, 1920, 1080);

[0372] Output: 256-channel feature map, size (256, 240, 135);

[0373] C1.4 Input feature fusion network:

[0374] Input the (256, 240, 135) feature map into the feature fusion network;

[0375] C1.5 Attention mechanism selects spectral information:

[0376] Use the channel attention mechanism to assign weights to 256 channels;

[0377] For example, in bright environments, the visible light channel gets a higher weight; in dim environments, the near-infrared channel is more important.

[0378] C1.6 Generate position feature representation:

[0379] Based on the weighted feature map, global average pooling is used to obtain a 256-dimensional position feature vector;

[0380] C2. Extracting time series and spatial structure information:

[0381] Time series features: Use 3D CNN to process 300 frames of data and output a 300x128 matrix;

[0382] Spatial structure information: Use 2D CNN to process a single frame and output a 16x16x128 feature map;

[0383] C3. Attention mechanism focuses on key areas:

[0384] Use the spatial attention mechanism to generate a 16x16 attention map;

[0385] Highlight key areas in the warehouse, such as shelf edges, aisle intersections, etc.

[0386] C4. Generate the second spatiotemporal feature representation:

[0387] Fusion of time series features (300x128) and weighted spatial features (16x16x128);

[0388] Use LSTM network to process time series information;

[0389] Output: 512-dimensional spatiotemporal joint feature vector;

[0390] C5. Generate position coordinates:

[0391] Input: 512-dimensional spatiotemporal feature vector;

[0392] Use a 3-layer fully connected network:

[0393] 512 -> 256 -> 128 -> 3;

[0394] Output: [x, y, θ] ;

[0395] For example: [34.5, 78.2, 1.57];

[0396] This means that the forklift is at x=34.5m, y=78.2m, and the heading angle is 1.57 radians (about 90 degrees).

[0397] This process greatly enhances the quality and information content of the original image by utilizing multispectral information and advanced image processing technology. At the same time, through multi-level feature extraction and fusion, as well as the application of attention mechanism, the system can more accurately estimate the position and orientation of the forklift. This method is particularly suitable for use in large warehouses with complex lighting conditions and changing environments, and can significantly improve the accuracy and reliability of forklift positioning.

[0398] Through this method, the embodiment of the present application not only fully utilizes the image data enhanced by the spectral diffusion MRI technology, but also further improves the accuracy of forklift position estimation through advanced feature extraction and fusion technology. This multi-level, multi-modal processing method enables the system to better cope with complex warehouse environments and provide more accurate and reliable positioning results.

[0399] S4: generating position information of the forklift relative to the storage location line based on the time series characteristics and the spatial structure information;

[0400] The purpose of this step is to use the time series features and spatial structure information extracted in the previous step to generate accurate position information of the forklift relative to the storage line. This process involves multiple aspects such as feature fusion, position estimation and coordinate transformation.

[0401] Specifically include:

[0402] 1. Feature Fusion:

[0403] The attention mechanism is used to perform weighted fusion of time series features and spatial structure information.

[0404] Design a multi-layer perceptron (MLP) or convolutional neural network (CNN) to learn the nonlinear mapping of fused features.

[0405] 2. Position estimation:

[0406] A regression model (such as support vector regression SVR or deep regression network) is used to predict the relative position coordinates of the forklift.

[0407] Use a classification model (such as a random forest or deep classification network) to predict the discrete area where the forklift is located, and then interpolate to get the precise location.

[0408] For example,

[0409] 1. The regression model predicts the relative position coordinates of the forklift;

[0410] In this method, the embodiment of the present invention directly predicts the continuous coordinate values ​​of the forklift. It is assumed that the embodiment of the present invention uses support vector regression (SVR) or a deep regression network to achieve this goal.

[0411] Assume that the coordinate system of the warehouse is an area of ​​100m x 100m, with the origin (0,0) located at the southwest corner of the warehouse.

[0412] The input features include:

[0413] Visual features extracted from images (e.g., direction and distance of location lines);

[0414] Distance information obtained from LiDAR;

[0415] Historical location data;

[0416] Output:

[0417] The model directly outputs the (x, y) coordinates of the forklift.

[0418] For example, the model predicts the position of the forklift to be (34.5m, 67.2m). This means that the forklift is 34.5 meters from the west boundary and 67.2 meters from the south boundary.

[0419] Advantages: This method can provide accurate and continuous position estimation and is suitable for scenarios that require high-precision positioning.

[0420] 2. The classification model predicts the discrete area where the forklift is located, and then interpolates to get the precise location;

[0421] In this method, the embodiment of the present invention first divides the warehouse space into discrete areas, then uses a classification model to predict the area where the forklift is located, and finally obtains a more accurate location through interpolation.

[0422] Assume that the embodiment of the present invention divides the same 100m x 100m warehouse into 10x10 grids, each grid being 10m x 10m.

[0423] The input features are the same as for the regression model.

[0424] Output:

[0425] The model outputs the grid where the forklift is most likely located, as well as the probabilities of the adjacent grids.

[0426] For example, the model predicts:

[0427] The probability of the grid (3,6) is 60%;

[0428] The probability of the grid (4,6) is 30%;

[0429] The probability of the grid (3,7) is 10%;

[0430] Here, the grid (3,6) means that the forklift is located in the grid at the 3rd column and the 6th row, and the coordinates of its lower left corner are (30m, 60m).

[0431] Then, the embodiment of the present invention can use these probabilities to perform weighted average interpolation:

[0432] x = (3 * 0.6 + 4 * 0.3 + 3 * 0.1) * 10 = 32 m;

[0433] y = (6 * 0.6 + 6 * 0.3 + 7 * 0.1) * 10 = 61 m;

[0434] Therefore, the estimated position of the forklift is (32m, 61m).

[0435] Advantages: This approach can strike a balance between computational efficiency and positioning accuracy. It is particularly suitable for scenarios that require fast response or when centimeter-level accuracy is not necessary.

[0436] In practical applications, the present invention combines these two methods:

[0437] 1. Use the classification model for rough positioning to quickly determine the approximate location of the forklift.

[0438] 2. Within the determined area, a regression model is used for fine positioning to provide more accurate coordinates.

[0439] This hybrid approach balances speed and accuracy, providing reliable position information for automated forklift operations.

[0440] 3. Coordinate transformation:

[0441] Convert the predicted relative position to an absolute position in the warehouse's global coordinate system.

[0442] Using the coordinate transformation matrix, the transformation is performed taking into account the orientation and initial position of the forklift.

[0443] 4. Confidence estimation:

[0444] To calculate the uncertainty of the position estimate, methods such as Monte Carlo dropout or Bayesian neural networks can be used.

[0445] Output the confidence interval of the position estimate to provide a basis for subsequent decision-making.

[0446] 5. Filtering and smoothing:

[0447] Use a Kalman filter or a particle filter to perform temporal smoothing on the position estimate to reduce the effects of jitter and outliers.

[0448] Combine historical location information to optimize current location estimates.

[0449] Through these steps, the embodiment of the present application can obtain an accurate forklift position estimate with confidence, providing a basis for subsequent auxiliary positioning and navigation.

[0450] S5: Utilize the position information to assist in positioning the forklift to improve the positioning accuracy of the forklift in the warehouse.

[0451] The goal of this step is to apply the generated position information to the actual forklift positioning to improve the overall positioning accuracy. This process involves multi-source data fusion, positioning optimization and feedback correction.

[0452] 1. Multi-source data fusion:

[0453] Combined with other positioning information sources, such as lidar, inertial measurement unit (IMU), wheel odometer, etc.

[0454] Use data fusion algorithms (such as extended Kalman filters or information filters) to integrate data from multiple sources.

[0455] 2. Map matching:

[0456] Match the estimated location with a pre-built warehouse map.

[0457] Use map matching algorithms (such as ICP algorithm or particle filter-based SLAM) to improve positioning accuracy.

[0458] 3. Dynamic correction:

[0459] Monitor positioning errors in real time and trigger the repositioning process when the error exceeds the threshold.

[0460] Visual odometry is used to estimate the relative motion of the forklift through continuous image sequences.

[0461] 4. Adaptive weighting:

[0462] Dynamically adjust the weights of different information sources in the final positioning results according to their reliability.

[0463] An adaptive weighting algorithm is designed to consider the impact of environmental factors (such as lighting conditions and obstacle density) on the reliability of each information source.

[0464] 5. Positioning result output:

[0465] Generates a complete positioning result including position coordinates, orientation angle, and uncertainty estimate.

[0466] The positioning results are output to the forklift control system in an appropriate frequency and format.

[0467] 6. Feedback optimization:

[0468] Collect actual operation data, including positioning errors, environmental changes and other information.

[0469] The collected data is used to regularly update and optimize the positioning algorithm to improve the long-term performance of the system.

[0470] Figure 8 This is a schematic diagram of auxiliary positioning through these methods, such as Figure 8 As shown in the figure, the forklift obtains the relative distance between the garage line and its own position through image analysis, and combines it with the laser radar for precise positioning, thereby improving the accuracy of the forklift's positioning.

[0471] The embodiments of the present application can make full use of the position estimation based on image data, combined with other sensor information and prior knowledge, to significantly improve the positioning accuracy of the forklift in the warehouse. This high-precision positioning provides a solid foundation for the automated operation and intelligent navigation of the forklift.

[0472] Next, the embodiments of the present application will discuss how to use the laser radar on the top of the forklift to obtain three-dimensional environmental data, and fuse these data with the previously obtained position information to further improve positioning accuracy and environmental perception capabilities.

[0473] The purpose of installing a high-precision LiDAR on top of a forklift is to obtain detailed 3D point cloud data of the surrounding environment. This type of LiDAR usually uses a rotary scanning technology, which can generate a high-density, high-precision 3D point cloud in a short time. The advantage of LiDAR is that it can directly measure distance and is not affected by lighting conditions, so it can make up for the shortcomings of the visual system in some cases.

[0474] After acquiring the three-dimensional environmental data, the embodiment of the present application needs to align these data with the position information previously obtained through image processing. This process involves point cloud registration technology, the core of which is to find the best match between the two types of data. The embodiment of the present application can use the iterative closest point (ICP) algorithm or its improved version, such as the NDT (Normal Distributions Transform) algorithm to achieve this goal. These algorithms use iterative optimization to find the best rigid body transformation to align the point cloud data with a known environmental map or a previous position estimate.

[0475] After the registration is completed, the embodiment of the present application needs to use a multimodal data fusion algorithm to fuse the three-dimensional point cloud data and the previously obtained position information. Figure 6 As shown, a probabilistic graph model or a deep learning based method can be used here. For example, an embodiment of the present application can design a graph neural network to represent point cloud data as nodes and encode location information as attributes of edges. Through the message passing mechanism, the network can learn how to optimally combine these two sources of information. Another method is to use a particle filter to dynamically update the position estimate of the forklift by taking into account observations from image processing and lidar at the same time.

[0476] Based on the fused data, the embodiment of the present application can construct an accurate three-dimensional model of the environment around the forklift. This model not only contains static structures, such as shelves and walls, but also dynamic obstacles, such as other moving forklifts or workers. The embodiment of the present application can use data structures such as voxel grids or octrees to efficiently represent and update this three-dimensional model. At the same time, the embodiment of the present application can apply semantic segmentation technology to add semantic labels to different parts of the model, such as "aisle", "shelf", "obstacle", etc., which will help with subsequent path planning and obstacle avoidance.

[0477] With this accurate three-dimensional model, the embodiment of the present application can better locate and navigate the forklift. In terms of positioning, the embodiment of the present application can use map matching technology to compare the currently observed three-dimensional structure with the pre-established environmental map to obtain a more accurate position estimate. This method is particularly suitable for use in indoor environments where GPS signals are not available.

[0478] In terms of navigation, the accurate three-dimensional model enables the embodiments of the present application to achieve more intelligent path planning and dynamic obstacle avoidance. For example, the embodiments of the present application can use the Rapid Exploration Random Tree (RRT) or its variants to plan the optimal path in three-dimensional space. Considering the motion characteristics of the forklift, the embodiments of the present application can add kinematic constraints to the path planning to ensure that the generated path is actually executable by the forklift.

[0479] In addition, the 3D model also allows the embodiments of the present application to achieve predictive obstacle avoidance. By tracking the movement trajectory of dynamic obstacles, the embodiments of the present application can predict their future locations and adjust the forklift's movement plan accordingly. This forward-looking obstacle avoidance strategy can significantly improve the operating efficiency and safety of forklifts in busy warehouses.

[0480] In general, by fusing LiDAR data and visual positioning information, the embodiment of the present application builds a comprehensive environmental perception and positioning system. This system not only improves the accuracy and reliability of positioning, but also provides a solid foundation for the intelligent navigation and safe operation of forklifts. In a complex and changing warehouse environment, this multi-sensor fusion method can significantly improve the autonomous performance of forklifts, enabling them to complete various tasks more efficiently and safely.

[0481] Finally, the embodiments of the present application will explore how to establish a motion prediction model for a forklift based on historical positioning data, and use this model to optimize the motion control strategy of the forklift to improve safety and efficiency.

[0482] like Figure 7 As shown, first of all, the embodiment of the present application needs to establish a robust data collection and storage system to record the historical motion data of the forklift. These data should include the position, speed, acceleration, direction and surrounding environment of the forklift at different time points. The embodiment of the present application can use a time series database to efficiently store and retrieve this data, such as InfluxDB or TimescaleDB. These databases are specially optimized for processing time series data and can provide fast query and aggregation functions.

[0483] Based on the collected historical data, the embodiment of the present application can build a motion prediction model for the forklift. The goal of this model is to predict the motion trajectory of the forklift in the short term (such as the next few seconds to minutes) based on the current state and historical patterns. The embodiment of the present application can consider using variants of recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs) or gated recurrent units (GRUs), which are particularly suitable for processing sequence data and capturing long-term dependencies.

[0484] In order to improve the accuracy of prediction, the embodiment of the present application can adopt an integrated learning method. For example, the embodiment of the present application can train multiple LSTM models with different structures or initializations, and then combine their prediction results through weighted averaging or more complex integration strategies. This method can effectively reduce the deviation and variance of a single model and improve the stability of the overall prediction.

[0485] In addition, the embodiments of the present application can also introduce an attention mechanism to enhance the expressiveness of the model. The attention mechanism allows the model to dynamically focus on different parts of the historical data when making predictions. For example, when predicting short-term trajectories, the model will pay more attention to the last few time steps; when predicting long-term trends, it will consider more of the earlier historical data.

[0486] During the training process, the embodiments of the present application need to take into account the particularity of the warehouse environment. For example, the movement of forklifts is usually restricted by fixed aisles and shelf layouts. The embodiments of the present application can encode this prior knowledge into the model, for example, by adding constraints to the loss function, or designing special network layers to process this structured information.

[0487] Once the motion prediction model is established, the embodiments of the present application can combine it with the current image data processing results to generate a more accurate short-term trajectory estimate. This process can be achieved through a fusion network that simultaneously receives the output of the prediction model and the data of the real-time perception system, and then outputs a comprehensive trajectory estimate.

[0488] With this short-term trajectory estimate, the embodiment of the present application can optimize the motion control strategy of the forklift. The embodiment of the present application can design a model predictive control (MPC) system, which calculates the optimal control input based on the current state and predicted trajectory at each time step. The advantage of MPC is that it can consider multiple objectives at the same time, such as path tracking accuracy, energy efficiency, and safety.

[0489] In terms of safety, the embodiments of the present application can use trajectory estimation to achieve active obstacle avoidance. The system predicts potential collision risks and adjusts the movement path of the forklift in advance. For example, if a collision with other moving objects is predicted, the system can slow down or change the path in advance. This forward-looking obstacle avoidance strategy can significantly improve the safety of forklifts in busy warehouses.

[0490] To further improve efficiency, the embodiments of the present application can implement collaborative planning. If there are multiple forklifts operating simultaneously in a warehouse, the system of the embodiments of the present application can consider the predicted trajectories of all forklifts and perform global optimization. This can reduce traffic congestion and improve overall work efficiency.

[0491] Finally, the system of the embodiment of the present application should have the ability of adaptive learning. Over time, the system can continuously learn and update its prediction model and control strategy. For example, the embodiment of the present application can use an online learning algorithm to fine-tune the model parameters based on the differences observed in actual operation. This continuous learning method enables the system to adapt to dynamic changes in the warehouse environment, such as seasonal cargo flow patterns or layout adjustments.

[0492] Through this comprehensive approach, the embodiments of the present application not only improve the accuracy of forklift positioning and navigation, but also achieve intelligent motion control and active safety management. This system can predict and avoid potential dangerous situations while optimizing motion efficiency, providing a powerful and flexible solution for modern warehouse management.

[0493] The present application also provides a computer device, the computer device comprising:

[0494] at least one processor; and,

[0495] a memory communicatively connected to the at least one processor; wherein,

[0496] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned auxiliary positioning processing method based on forklift image data acquisition.

[0497] An embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned auxiliary positioning processing method based on forklift image data acquisition.

[0498] An embodiment of the present application further provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the above-mentioned auxiliary positioning processing method based on forklift image data acquisition.

[0499] Each module in the auxiliary positioning processing system based on forklift image data acquisition can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0500] In one embodiment, a computer device is also provided, which is the auxiliary positioning processing system based on forklift image data acquisition mentioned in the above method embodiment. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface.

[0501] Among them, the processor of the computer device is used to provide computing and control capabilities, and can be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but is not limited thereto. The processor may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. The processor may also include one or more dedicated processors, and the dedicated processor may include a GPU, an FPGA, etc., for accelerated processing. The processor is used to call the program code and data in the memory and execute the steps in the above method embodiment. For details, please refer to the description in the method embodiment, which will not be repeated here.

[0502] The memory of the computer device includes, but is not limited to, non-volatile storage media and internal memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium.

[0503] The input / output interface of the computer device is used to exchange information between the processor and external devices.

[0504] The communication interface of the computer device is used to communicate with an external terminal via a network connection.

[0505] When the computer program is executed by a processor, the above method is implemented.

[0506] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the division of each unit / module is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling, direct coupling, or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0507] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0508] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.

[0509] The system of the embodiments of the present disclosure can execute the method provided by the embodiments of the present disclosure, and the implementation principles are similar. The actions performed by each module in the system of each embodiment of the present disclosure correspond to the steps in the method of each embodiment of the present disclosure. For the detailed functional description of each module of the system, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0510] The above is only an optional implementation method for some implementation scenarios of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the scheme of the present disclosure, other similar implementation methods based on the technical ideas of the present disclosure are also within the protection scope of the embodiments of the present disclosure.

[0511] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An auxiliary positioning processing method based on forklift image data acquisition, characterized in that: The following steps are involved: Use the camera to scan the storage location lines on the ground to obtain image data of the forklift's surroundings; Inputting the image data into a processing model, the processing model comprising an integrated model of binary spikes and Sigma Delta neural networks (SDNNs); The processing model utilizes a spatiotemporal constraint learning mechanism to extract time series features and spatial structure information from the image data; Based on the time series characteristics and the spatial structure information, generating the position information of the forklift relative to the storage location line; Utilizing the position information to assist in positioning the forklift, so as to improve the positioning accuracy of the forklift in the warehouse; The processing model uses a spatiotemporal constraint learning mechanism to extract time series features and spatial structure information from the image data, including: Perform preliminary feature extraction on image data using binary spiking networks; Inputting the extracted preliminary features into the Sigma Delta neural network for spatiotemporal feature learning; By means of a spatiotemporal constraint learning mechanism, the temporal continuity and spatial correlation of the image data are simultaneously considered; Outputting the time series features and the spatial structure information; The processing model further includes a feature fusion network based on an attention mechanism, which generates the position information of the forklift relative to the storage location line based on the time series features and spatial structure information, including: The feature fusion network based on the attention mechanism receives the time series features and spatial structure information; Performing multi-scale decomposition on the time series features and spatial structure information to obtain feature representations at different scales; Using the attention mechanism, the importance weights of features of different scales for forklift positioning are calculated. The importance weights are used to highlight the spatiotemporal feature information that is most critical to forklift positioning and suppress irrelevant or redundant feature information. The spatiotemporal feature information that is most critical to forklift positioning includes the geometric structure of the storage line and the temporal pattern of forklift movement, and the irrelevant or redundant feature information includes background noise or non-critical objects. According to the calculated importance weight, adaptively weighted fusion of the time series features and spatial structure information is performed; Based on the weighted fused features, construct the first spatiotemporal feature representation; Using the first spatiotemporal feature representation, generating the position coordinates or relative position description of the forklift through a regression or classification model; Output the generated location information.

2. The method according to claim 1, characterized in that Before inputting the image data into the processing model, the method further comprises the following steps: applying Taylor expansion to the feature extractor of the processing model to recast the position metric as an input-dependent squared error of the Jacobian matrix; Based on the recast positional metric, local features of the image data are calculated block by block; The Jacobian matrix is ​​approximated using random dimensionality reduction techniques to reduce computational complexity.

3. The method according to claim 2, characterized in that The block-by-block calculation of local features of the image data includes: Dividing the input image data into a plurality of overlapping local blocks; Apply a lightweight feature extraction network to each local block; Calculate the feature representation and corresponding Jacobian matrix of each local block; Use the weighted average method to fuse the features and Jacobian matrices of all local blocks; Based on the fusion results, a globally consistent forklift position estimate is generated.

4. The method according to claim 1, characterized in that: The method further comprises: Preprocessing the input image data using spectral diffusion MRI technology to enhance image contrast and details; Extracting the time series features and spatial structure information of the enhanced image data, and inputting them into the attention-based feature fusion network; Focus on key areas enhanced by spectral diffusion through the attention mechanism; Fusing image information from multiple frequency bands to generate a second spatiotemporal feature representation; The second spatiotemporal feature representation is used to generate the position coordinates or relative position description of the forklift through a regression or classification model to improve the accuracy of the forklift position estimation.

5. The method according to claim 4, characterized in that The preprocessing of the input image data using the spectral diffusion MRI technology includes: Collect multispectral image data of warehouse environment; Denoising and enhancing multispectral images using diffusion models; Construct a spectral-spatial joint representation to capture the correlation between different spectral channels; The enhanced multispectral data is input into the feature fusion network; Using the attention mechanism, the spectral information that is most beneficial to forklift positioning is adaptively selected; Based on the selected spectral information, a position feature representation is generated.

6. The method according to claim 1, characterized in that Utilizing the position information to assist in positioning the forklift includes: Use the laser radar on the top of the forklift to obtain three-dimensional environmental data; registering the three-dimensional environment data with the location information; Using a multimodal data fusion algorithm to fuse the three-dimensional point cloud data and the position information; Based on the fused data, an accurate 3D model of the forklift’s surroundings is constructed; The three-dimensional model is used to locate and navigate the forklift.

7. The method according to claim 1, characterized in that The spatiotemporal constraint learning mechanism includes: Design a temporal convolution module to capture short-term and long-term temporal dependencies in image sequences; Build a spatial attention module to identify the spatial regions in the image that are most critical for positioning; Use the spatiotemporal coupling layer to establish the association between temporal features and spatial features; Train the network in an end-to-end manner to optimize the extraction of temporal and spatial features; Regularization constraints are introduced to ensure that the learned features have good spatiotemporal consistency.

8. The method according to claim 1, characterized in that The method further comprises: Based on historical positioning data, a forklift motion prediction model is established; Combine the current image data processing results with the motion prediction to generate a short-term trajectory estimate of the forklift; Using the trajectory estimation, the motion control strategy of the forklift is optimized to avoid collision with surrounding cargo.

Images