Forklift action control method, model training method, electronic device, and storage medium

Abstract

The application discloses a forklift action control method, a model training method, an electronic device and a storage medium, and relates to the technical field of engineering vehicle control. Since the forklift action is predicted only by relying on visual images collected by multiple cameras and state data of the forklift itself, the deployment cost is reduced, the high dependence on depth true values is eliminated, the cross-modal fusion features and the visual splicing features contain global spatial correlation information of multi-view images, the accuracy of the action block prediction result is ensured, and through the increase of the rollback judgment mechanism, when a prediction error occurs, the error action can be found in time, and rollback correction can be performed in time, so that the collision of the material cage caused by the action control error is avoided. In addition, by introducing the forklift arm descending judgment mechanism, it can be ensured that the forklift has moved the transported material cage to the position directly above the target material cage below, and the corner column is aligned, and then the material cage is lowered, so that the safety of the material cage stacking is further ensured.

Description

Technical Field

[0001] This application relates to the field of engineering vehicle control technology, and in particular to a forklift motion control method, model training method, electronic equipment and storage medium. Background Technology

[0002] Forklift cage stacking is a common operation in warehousing and logistics. The task involves lifting one cage and precisely stacking it on top of another cage. This task requires high precision in horizontal alignment and collision avoidance.

[0003] In related technologies, sensors such as depth cameras and lidar are typically used to obtain the precise pose of the cage and forklift before motion planning is performed.

[0004] However, the relevant technologies are sensitive to the accuracy, calibration and environmental conditions of sensors such as depth cameras and lidar. The sensor deployment cost is high, and because the method is highly dependent on the true depth value, it is difficult to generalize in scenarios where the accuracy of the true depth value cannot be guaranteed. In fact, the cage may even collide due to motion control errors. Summary of the Invention

[0005] This application provides a forklift motion control method, a model training method, electronic equipment, and a storage medium to at least solve problems in related technologies such as high sensor deployment costs, high dependence on depth ground truth, difficulty in generalization in scenarios where the accuracy of depth ground truth cannot be guaranteed, and the possibility of material cage collisions due to motion control errors.

[0006] This application provides a forklift motion control method, including: Acquire multi-source visual images and current control status data of the forklift; wherein the multi-source visual images include at least the left visual image captured by the left-side camera of the forklift, the right-side visual image captured by the right-side camera of the forklift, and the forklift arm visual image captured by the forklift arm camera. Based on the multi-source visual images and the current control state data, the cross-modal fusion features and visual stitching features of the forklift are determined respectively. Based on the target motion prediction model, and according to the cross-modal fusion features of the forklift, the motion block prediction results for future consecutive frames are determined; wherein, the motion block prediction results include the multi-dimensional motion vectors of the forklift. Based on the target back-back judgment model, the back-back judgment result of the current frame is determined according to the visual stitching features of the forklift; Based on the target forklift boom descent judgment model, the forklift boom descent judgment result of the current frame is determined according to the visual stitching features of the forklift. Based on the rollback discrimination result, the forklift arm descent discrimination result, and the current forklift arm height, determine whether the current frame meets the trigger conditions for the rollback coverage mode and / or the trigger conditions for the forklift arm descent coverage mode. If the triggering conditions of the fallback overlay mode are met in the current frame, the forklift is controlled to perform a preset fixed backward correction action; If the triggering conditions for the forklift arm descent coverage mode are met in the current frame, control the forklift to perform a preset fixed descent placement action; If the trigger conditions for the fallback coverage mode and the fallback coverage mode are not met in the current frame, the action block prediction results of the next multiple consecutive frames are subjected to multi-dimensional temporal aggregation processing to obtain the target control action of the current frame, so as to control the forklift to execute the target control action.

[0007] Optionally, determining the cross-modal fusion features and visual stitching features of the forklift based on the multi-source visual images and the current control state data includes: Based on the multi-source visual images, left-side visual features, right-side visual features, and forklift arm visual features are determined respectively; wherein, the multi-source visual images are color images; Based on the current control state data, determine the characteristics of the current control state; The left visual features, right visual features, forklift arm visual features, and current control state features are fused to obtain the cross-modal fusion features of the forklift. The visual features of the left side, the right side, and the forklift arm are stitched together to obtain the visual stitched features of the forklift. The current control status data includes at least the forklift arm lateral displacement, forklift arm height, chassis rotation angle, and chassis linear velocity.

[0008] Optionally, the step of determining the action block prediction results for future consecutive frames based on the target action prediction model and the cross-modal fusion features of the forklift includes: The cross-modal fusion features and multiple preset latent variables are input into the action block decoder; wherein, the number of preset latent variables is the same as the number of frames in the next several consecutive frames; Based on the attention mechanism in the action block decoder, global context information is extracted from the cross-modal fusion features; Based on the global context information, determine the intermediate feature vector corresponding to each of the preset latent variables; Based on the intermediate feature vectors corresponding to each of the preset latent variables, the prediction results of action blocks for multiple consecutive frames in the future are determined. The target action prediction model includes at least the action block decoder. During the training phase of the target action prediction model, random latent variables sampled from a standard normal distribution are input into the action block decoder. During the deployment phase of the target action prediction model, the random latent variables are set to a vector of all zeros.

[0009] Optionally, the target-based back-back judgment model, based on the visual stitching features of the forklift, determines the back-back judgment result of the current frame, including: The visual stitching features are input into a first multilayer perceptron classifier to determine a first original logical value corresponding to the visual stitching features based on the first multilayer perceptron classifier; wherein, the first original logical value is used to characterize the confidence level of the backoff judgment. Based on the first original logical value corresponding to the visual stitching feature, determine the backtracking judgment result of the current frame; The target back-off judgment model includes at least the first multilayer perceptron classifier. During the training phase of the target back-off judgment model, samples with a horizontal offset between two cages that is less than a preset horizontal offset threshold and a rotational offset that is greater than a preset rotational offset threshold are marked as back-off samples.

[0010] Optionally, the target forklift boom descent judgment model, based on the visual stitching features of the forklift, determines the forklift boom descent judgment result for the current frame, including: The visual stitching features are input into a second multilayer perceptron classifier to determine a second original logical value corresponding to the visual stitching features based on the second multilayer perceptron classifier; wherein, the second original logical value is used to characterize the confidence level of the forklift arm descent judgment; Based on the second original logic value corresponding to the visual stitching feature, determine the forklift arm descent discrimination result of the current frame; The target forklift arm descent judgment model includes at least the second multilayer perceptual classifier. During the training phase of the target forklift arm descent judgment model, the descent frame and all frames after the descent frame are marked as descent samples. Starting from the frame before the descent frame, the model continuously backtracks and marks all frames where the forklift is stationary as descent samples.

[0011] Optionally, determining whether the current frame meets the trigger conditions for the fallback coverage mode and / or the forklift arm descent coverage mode based on the fallback discrimination result, the forklift arm descent discrimination result, and the current forklift arm height includes: Based on the back-off judgment result and the forklift arm descent judgment result, the corresponding back-off probability and forklift arm descent probability are determined respectively. Determine whether the current forklift boom height is above the first preset height threshold; If so, then the rollback condition is valid; Determine whether the current forklift boom height is above the second preset height threshold; If so, the forklift arm descent judgment condition is determined to be valid; wherein, the second preset height threshold is greater than the first preset height threshold; If the fallback judgment condition is valid, determine whether the fallback probability of the current frame and the previous frame both exceed the first preset probability threshold. If so, determine that the trigger condition of the fallback coverage mode is met. If the forklift arm descent judgment condition is valid, determine whether the probability of forklift arm descent in the current frame and the previous frame both exceed the second preset probability threshold. If so, determine that the trigger condition of the forklift arm descent coverage mode is met. Specifically, when both the fallback coverage mode and the drop coverage mode trigger conditions are met simultaneously, the fallback coverage mode trigger condition is responded to first.

[0012] Optionally, if the triggering conditions for the fallback coverage mode and the fallback coverage mode are not met in the current frame, the step of performing multi-dimensional temporal aggregation processing on the action block prediction results of the next multiple frames to obtain the target control action of the current frame, and controlling the forklift to execute the target control action, includes: If the triggering conditions for the fallback coverage mode and the fallback coverage mode are not met in the current frame, the prediction results of the action blocks for the next multiple consecutive frames are stored in the target queue. For the current moment, extract multiple target action prediction values ​​covering the current moment from the target queue; wherein, the multi-dimensional action vector includes multiple target action prediction values; Based on the multiple target action prediction values, calculate the exponential decay weight corresponding to each target action prediction value; Based on the exponential decay weight, the predicted values ​​of each target action are weighted and aggregated to obtain the aggregated predicted values ​​of the target actions; wherein, the predicted values ​​of the target actions in the forklift arm lateral movement dimension adopt the smallest exponential decay coefficient. The aggregated target action prediction value is constrained to the corresponding preset physical value range to obtain the target control action of the current frame.

[0013] This application also provides a forklift motion control device, including: The acquisition module is used to acquire multi-source visual images and current control status data of the forklift; wherein, the multi-source visual images include at least the left visual image acquired by the left-side camera of the forklift, the right-side visual image acquired by the right-side camera of the forklift, and the forklift arm visual image acquired by the forklift arm camera. The feature module is used to determine the cross-modal fusion features and visual stitching features of the forklift based on the multi-source visual images and the current control state data, respectively. The motion prediction module is used to determine the motion block prediction results for multiple consecutive frames in the future based on the target motion prediction model and the cross-modal fusion features of the forklift; wherein the motion block prediction results include the multi-dimensional motion vectors of the forklift. The rollback judgment module is used to determine the rollback judgment result of the current frame based on the target rollback judgment model and the visual stitching features of the forklift. The descent judgment module is used to determine the forklift arm descent judgment result of the current frame based on the target forklift arm descent judgment model and the visual stitching features of the forklift. The trigger condition judgment module is used to determine whether the current frame meets the trigger conditions of the fallback coverage mode and / or the trigger conditions of the forklift arm descent coverage mode based on the fallback judgment result, the forklift arm descent judgment result and the current forklift arm height. The rollback control module is used to control the forklift to perform a preset fixed rollback correction action when the triggering conditions of the rollback overlay mode are met in the current frame. The descent control module is used to control the forklift to perform a preset fixed descent placement action when the triggering conditions of the forklift arm descent coverage mode are met in the current frame. The motion control module is used to perform multi-dimensional temporal aggregation processing on the motion block prediction results of the future consecutive frames when the trigger conditions of the fallback coverage mode and the fall coverage mode are not met in the current frame, so as to obtain the target control action of the current frame and control the forklift to execute the target control action.

[0014] Optionally, the feature module is specifically used for: Based on the multi-source visual images, left-side visual features, right-side visual features, and forklift arm visual features are determined respectively; wherein, the multi-source visual images are color images; Based on the current control state data, determine the characteristics of the current control state; The left visual features, right visual features, forklift arm visual features, and current control state features are fused to obtain the cross-modal fusion features of the forklift. The visual features of the left side, the right side, and the forklift arm are stitched together to obtain the visual stitched features of the forklift. The current control status data includes at least the forklift arm lateral displacement, forklift arm height, chassis rotation angle, and chassis linear velocity.

[0015] Optionally, the action prediction module is specifically used for: The cross-modal fusion features and multiple preset latent variables are input into the action block decoder; wherein, the number of preset latent variables is the same as the number of frames in the next several consecutive frames; Based on the attention mechanism in the action block decoder, global context information is extracted from the cross-modal fusion features; Based on the global context information, determine the intermediate feature vector corresponding to each of the preset latent variables; Based on the intermediate feature vectors corresponding to each of the preset latent variables, the prediction results of action blocks for multiple consecutive frames in the future are determined. The target action prediction model includes at least the action block decoder. During the training phase of the target action prediction model, random latent variables sampled from a standard normal distribution are input into the action block decoder. During the deployment phase of the target action prediction model, the random latent variables are set to a vector of all zeros.

[0016] Optionally, the rollback determination module is specifically used for: The visual stitching features are input into a first multilayer perceptron classifier to determine a first original logical value corresponding to the visual stitching features based on the first multilayer perceptron classifier; wherein, the first original logical value is used to characterize the confidence level of the backoff judgment. Based on the first original logical value corresponding to the visual stitching feature, determine the backtracking judgment result of the current frame; The target back-off judgment model includes at least the first multilayer perceptron classifier. During the training phase of the target back-off judgment model, samples with a horizontal offset between two cages that is less than a preset horizontal offset threshold and a rotational offset that is greater than a preset rotational offset threshold are marked as back-off samples.

[0017] Optionally, the descent determination module is specifically used for: The visual stitching features are input into a second multilayer perceptron classifier to determine a second original logical value corresponding to the visual stitching features based on the second multilayer perceptron classifier; wherein, the second original logical value is used to characterize the confidence level of the forklift arm descent judgment; Based on the second original logic value corresponding to the visual stitching feature, determine the forklift arm descent discrimination result of the current frame; The target forklift arm descent judgment model includes at least the second multilayer perceptual classifier. During the training phase of the target forklift arm descent judgment model, the descent frame and all frames after the descent frame are marked as descent samples. Starting from the frame before the descent frame, the model continuously backtracks and marks all frames where the forklift is stationary as descent samples.

[0018] Optionally, the trigger condition judgment module is specifically used for: Based on the back-off judgment result and the forklift arm descent judgment result, the corresponding back-off probability and forklift arm descent probability are determined respectively. Determine whether the current forklift boom height is above the first preset height threshold; If so, then the rollback condition is valid; Determine whether the current forklift boom height is above the second preset height threshold; If so, the forklift arm descent judgment condition is determined to be valid; wherein, the second preset height threshold is greater than the first preset height threshold; If the fallback judgment condition is valid, determine whether the fallback probability of the current frame and the previous frame both exceed the first preset probability threshold. If so, determine that the trigger condition of the fallback coverage mode is met. If the forklift arm descent judgment condition is valid, determine whether the probability of forklift arm descent in the current frame and the previous frame both exceed the second preset probability threshold. If so, determine that the trigger condition of the forklift arm descent coverage mode is met. Specifically, when both the fallback coverage mode and the drop coverage mode trigger conditions are met simultaneously, the fallback coverage mode trigger condition is responded to first.

[0019] Optionally, the motion control module is specifically used for: If the triggering conditions for the fallback coverage mode and the fallback coverage mode are not met in the current frame, the prediction results of the action blocks for the next multiple consecutive frames are stored in the target queue. For the current moment, extract multiple target action prediction values ​​covering the current moment from the target queue; wherein, the multi-dimensional action vector includes multiple target action prediction values; Based on the multiple target action prediction values, calculate the exponential decay weight corresponding to each target action prediction value; Based on the exponential decay weight, the predicted values ​​of each target action are weighted and aggregated to obtain the aggregated predicted values ​​of the target actions; wherein, the predicted values ​​of the target actions in the forklift arm lateral movement dimension adopt the smallest exponential decay coefficient. The aggregated target action prediction value is constrained to the corresponding preset physical value range to obtain the target control action of the current frame.

[0020] This application also provides a method for training a forklift motion control model, including: During the simulation pre-training phase, script strategies are used to collect the operation trajectories of multiple experts. Based on the multiple expert operation trajectories, simulation samples for model training are determined, and backtracking simulation samples and descent simulation samples are marked in the simulation samples; Based on the simulation samples trained by the model, the original motion prediction model, the original back-off judgment model, and the original forklift arm descent judgment model are trained to obtain the intermediate motion prediction model, the intermediate back-off judgment model, and the intermediate forklift arm descent judgment model. During the real data fine-tuning stage, multiple real operation trajectories of the forklift are acquired; wherein, the real operation trajectory includes at least multi-source visual image trajectory and control state change trajectory; Based on the multiple real operation trajectories, fine-tuning samples for model training are determined, and backtracking fine-tuning samples and descent fine-tuning samples are marked in the fine-tuning samples. Based on the fine-tuned samples trained by the model, the model parameters of the intermediate action prediction model, the intermediate back-off judgment model and the intermediate forklift arm descent judgment model are fine-tuned to obtain the target action prediction model, the target back-off judgment model and the target forklift arm descent judgment model. The forklift motion control model includes the target motion prediction model, the target retraction judgment model, and the target forklift arm descent judgment model. The target motion prediction model, the target retraction judgment model, and the target forklift arm descent judgment model are applied to any of the forklift motion control methods described above.

[0021] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described forklift motion control methods or the steps of the forklift motion control model training method.

[0022] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described forklift motion control methods or the steps of the forklift motion control model training method.

[0023] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described forklift motion control methods or the steps of a forklift motion control model training method.

[0024] This application reduces deployment costs and eliminates the high dependence on depth ground truth by relying solely on visual images acquired by multiple cameras and the forklift's own state data for forklift motion prediction. Furthermore, the obtained cross-modal fusion features and visual stitching features contain global spatial correlation information of multi-view images, ensuring the accuracy of motion block prediction results. By adding a backtracking judgment mechanism, erroneous actions can be detected in time when prediction errors occur, and backtracking corrections can be performed promptly to avoid collisions of the material cages due to motion control errors. In addition, by introducing a forklift arm descent judgment mechanism, it can be ensured that the forklift only begins to lower the material cage after the material cage to be transported has been moved directly above the target material cage below and the corner posts have been aligned, further ensuring the safety of material cage stacking. Attached Figure Description

[0025] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the forklift motion control system on which the embodiments of this application are based; Figure 2 A flowchart illustrating the forklift motion control method provided in an embodiment of this application; Figure 3 A flowchart illustrating the forklift motion control model training method provided in this application embodiment; Figure 4 This is a schematic diagram of the forklift motion control device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0028] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0029] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] The specific application environment architecture or specific hardware architecture on which the forklift motion control method depends is described here.

[0031] First, the structure of the forklift motion control system on which this application is based will be described: The forklift motion control method, model training method, electronic device, and storage medium provided in this application are suitable for intelligent control of forklifts. Figure 1 The diagram shown is a structural schematic of the forklift motion control system based on an embodiment of this application. It mainly includes a left-side camera, a right-side camera, a forklift arm camera, a control status data acquisition device, and a forklift motion control device. The left-side camera acquires left-side visual images, the right-side camera acquires right-side visual images, the forklift arm camera acquires forklift arm visual images, and the control status data acquisition device acquires the current control status data of the forklift. The forklift motion control device performs motion control on the forklift based on the images acquired by the left-side camera, right-side camera, forklift arm camera, and control status data acquisition device, along with the current control status data. The left-side camera, right-side camera, forklift arm camera, and control status data acquisition device are all deployed at the forklift end, while the forklift motion control device can be deployed at the forklift end or remotely.

[0032] This application provides a forklift motion control method for intelligently controlling the movements of a forklift. The executing entity in this application is an electronic device, such as an in-vehicle computing device, server, desktop computer, laptop computer, tablet computer, or other electronic devices that can be used for intelligent control of the forklift's movements.

[0033] like Figure 2 The diagram shown is a flowchart illustrating a forklift motion control method provided in an embodiment of this application. The method includes: Step 201: Obtain multi-source visual images and current control status data of the forklift.

[0034] The multi-source visual images include at least the left-side visual image captured by the left-side camera of the forklift, the right-side visual image captured by the right-side camera of the forklift, and the forklift arm visual image captured by the forklift arm camera. It should be noted that the left-side visual image captured by the left-side camera is a color image taken by a fixed camera mounted on the left side of the forklift; the right-side visual image captured by the right-side camera is a color image taken by a fixed camera mounted on the right side of the forklift; and the forklift arm visual image captured by the forklift arm camera is a color image taken by a ToF (Time-of-Flight) camera mounted on the forklift arm. The characteristic of a ToF camera is that it can not only capture color images but also measure the distance of each pixel to the camera. However, in this embodiment, the forklift arm visual image is an RGB image captured by the ToF camera, excluding the depth map or distance map output by the ToF camera. Only the color image channel of the camera is used; its depth information is not used. The current control state data is feedback data that the forklift itself can acquire in real time, also known as current state data.

[0035] Step 202: Based on the multi-source visual images and the current control state data, determine the cross-modal fusion features and visual stitching features of the forklift, respectively.

[0036] Cross-modal fusion features refer to the deep coupling of high-dimensional visual information from an image with low-dimensional state information from the current control state data. The aim is to enable the current control state data to guide visual features to focus on key regions related to the current action. Visual stitching features are achieved by superimposing three independent image features along the channel dimension, with the aim of preserving the most original semantic information from each viewpoint.

[0037] Step 203: Based on the target action prediction model, determine the action block prediction results for future consecutive frames according to the cross-modal fusion characteristics of the forklift.

[0038] The action block prediction results include the forklift's multi-dimensional action vectors.

[0039] It should be noted that the embodiments of this application differ from traditional single-step action prediction. The model outputs action blocks for multiple consecutive frames (e.g., 8 frames) at once, which makes the action output smoother and reduces motion control jitter caused by noise in a single frame image.

[0040] Step 204: Based on the target back-back judgment model, determine the back-back judgment result of the current frame according to the visual stitching features of the forklift.

[0041] The target retraction result is used to determine whether the current situation is dangerous, where the upper and lower cages are close in horizontal position but have a large angular deviation. By adding a retraction judgment branch, collisions between the cages due to motion control errors can be avoided.

[0042] Step 205: Based on the target forklift arm descent judgment model, determine the forklift arm descent judgment result of the current frame according to the visual stitching features of the forklift.

[0043] Specifically, by adding a forklift arm descent discrimination branch, it can be ensured that the corner posts of the stacking cages can accurately fall into the grooves of the target cage below, avoiding stacking failure and cage damage caused by descending too early or too late.

[0044] Step 206: Based on the rollback judgment result, the forklift arm descent judgment result, and the current forklift arm height, determine whether the current frame meets the trigger conditions for the rollback coverage mode and / or the trigger conditions for the forklift arm descent coverage mode.

[0045] Specifically, based on the rollback determination result and the current forklift arm height, it can be determined whether the current frame meets the trigger conditions for the rollback coverage mode. Similarly, based on the forklift arm descent determination result and the current forklift arm height, it can be determined whether the current frame meets the trigger conditions for the forklift arm descent coverage mode. If either the rollback coverage mode trigger conditions or the forklift arm descent coverage mode trigger conditions are met, the forklift is further triggered to perform the corresponding action, instead of executing the previously predicted action.

[0046] Step 207: If the triggering conditions of the rollback overlay mode are met in the current frame, control the forklift to perform a preset fixed rollback correction action.

[0047] It should be noted that after the forklift has lifted the cage to be transported and raised the forklift arm to near the target height, the forklift needs to approach the target cage below. If it moves forward rashly at this point without being fully aligned, it may cause the corner posts to misalign or even collide. Therefore, in response to the backtracking overlay mode, the forklift is controlled to perform a preset fixed backtracking correction action. This preset fixed backtracking correction action may involve reversing a short distance, adjusting the direction, and then moving forward again. After entering the backtracking overlay mode, the existing action block prediction results in the target queue are cleared or paused.

[0048] Specifically, when performing the preset fixed reverse correction action, the forklift arm offset and forklift arm height remain unchanged, the steering (chassis angle) is returned to zero, and the vehicle reverses at a speed of 2.5 m / s for 40 frames. The prediction results of the action blocks of the next several consecutive frames obtained from the previous prediction are discarded, and the prediction is re-predicted after 40 frames.

[0049] Step 208: If the triggering conditions for the forklift arm descent coverage mode are met in the current frame, control the forklift to perform a preset fixed descent placement action.

[0050] It should be noted that once the forklift has moved the transported cage to directly above the target cage below and the corner posts are aligned, the cage needs to be lowered. However, the timing of lowering the cage needs to be accurately judged. Lowering it too early will cause the corner posts to collide, while lowering it too late will waste time. Therefore, this application embodiment provides a judgment on the triggering condition of the forklift arm lowering coverage mode, and when the condition is met, controls the forklift to perform a preset fixed lowering and placement action, that is, lower the cage.

[0051] Step 209: If the triggering conditions of the fallback coverage mode and the fallback coverage mode are not met in the current frame, the action block prediction results of the next multiple consecutive frames are processed by multi-dimensional temporal aggregation to obtain the target control action of the current frame, so as to control the forklift to execute the target control action.

[0052] It should be noted that, since the embodiments of this application predict the action block prediction results of multiple consecutive future frames at once, in order to ensure the smoothness of the action, the action block prediction results of adjacent frames will overlap. By performing dimensional temporal aggregation processing on the action block prediction results of multiple consecutive future frames, the action fusion between frames can be performed, and the noise of single-frame prediction can be filtered out, so as to further achieve smooth control of the action.

[0053] Specifically, if the trigger conditions for the fallback coverage mode and the fall coverage mode are not met in the current frame, it can be determined that the forklift is currently in normal operation, such as normal driving or picking up stacked cages. At this time, the target control action of the current frame can be obtained by performing multi-dimensional temporal aggregation processing based on the predicted action block results of the previously determined future consecutive frames, and then controlling the forklift to execute the target control action.

[0054] Based on the above embodiments, as an implementable approach, in one embodiment, the cross-modal fusion features and visual stitching features of the forklift are determined according to multi-source visual images and current control state data, including: Step 2021: Based on the multi-source visual images, determine the left visual features, right visual features, and forklift arm visual features respectively; wherein, the multi-source visual images are color images; Step 2022: Determine the characteristics of the current control state based on the current control state data; Step 2023: Perform feature fusion on the left visual features, right visual features, forklift arm visual features, and current control state features to obtain the cross-modal fusion features of the forklift. Step 2024: Perform feature stitching on the left visual features, right visual features, and forklift arm visual features to obtain the visual stitching features of the forklift.

[0055] The current control status data includes at least the forklift arm lateral displacement (fork_y), forklift arm height (fork_z), chassis steering angle (steering), and chassis linear velocity (velocity).

[0056] Specifically, for visual features, each RGB image is preprocessed into a raw color image [B, 3, 224, 224], and then input into an independent ResNet18, outputting a 512-dimensional feature vector. This is then passed through an Adapter layer (a single-layer linear mapping, 512→256) to compress the features from 512 dimensions to 256 dimensions (left visual features, right visual features, and forklift arm visual features). 512 dimensions are the standard output dimension of ResNet18, while 256 dimensions are the token dimension requirement for the subsequent Transformer module. The Adapter performs format conversion, mapping the image features to the same dimensional space as the state features, allowing them to be processed side-by-side. Each of the three cameras has its own independent ResNet18+Adapter, and their parameters are not shared.

[0057] Specifically, for the current control state features, the current control state features are projected onto the state encoder (containing the multilayer perceptron dimension change trajectory from input to output) (3-layer MLP, 4→128→256) as state features of [B,256], so that they can be used as tokens of the same dimension alongside image features in subsequent Transformer processing.

[0058] Furthermore, the left-side visual features, right-side visual features, forklift arm visual features, and current control state features are upgraded to state tokens of the same dimension as the high-dimensional visual tokens through MLP, and then fed together with the multiple visual tokens into a unified Transformer for cross-modal fusion.

[0059] Specifically, three visual features (left visual feature, right visual feature, and forklift arm visual feature, each [B, 256]) can be reused and concatenated into a visual feature of [B, 768]. That is, three 256-dimensional feature vectors are directly concatenated into a single 768-dimensional vector.

[0060] Based on the above embodiments, as an implementable approach, in one embodiment, based on the target action prediction model and according to the cross-modal fusion characteristics of the forklift, the action block prediction results for future consecutive frames are determined, including: Step 2031: Input the cross-modal fusion features and multiple preset latent variables into the action block decoder; wherein, the number of preset latent variables is the same as the number of frames in the next consecutive frames, and each preset latent variable corresponds to the action block prediction result of the next frame; Step 2032: Extract global contextual information from cross-modal fusion features based on the attention mechanism in the action block decoder; Step 2033: Determine the intermediate feature vectors corresponding to each preset latent variable based on the global context information; Step 2034: Determine the prediction results of action blocks for future consecutive frames based on the intermediate feature vectors corresponding to each preset latent variable.

[0061] The target action prediction model includes at least an action block decoder. During the training phase of the target action prediction model, random latent variables sampled from the standard normal distribution are input into the action block decoder. During the deployment phase of the target action prediction model, the random latent variables are set to a vector of all zeros.

[0062] It should be noted that the target action prediction model includes the ACT policy branch (Action Chunking Transformer). ACT is based on the CVAE architecture, using variational inference (VAE latent variable z) during training and deterministic patterns (latent variable z=0) during inference. Its data flow is as follows: ObsFusion (Observation Fusion, Transformer Encoder, 4 layers, 8 heads): Fuses four observation tokens (left / right / ToF / state) across modalities using self-attention, outputting [B, 4, 256]. Fusion allows each modal token to carry a global context, such as establishing a regional correspondence between the side of the cage seen by the left camera and the top of the cage seen by the ToF camera.

[0063] Latent variable branch: z([B,32], set to 0 during inference) is mapped to a z_token([B,1,256]) via z_proj(32→256).

[0064] ChunkDecoder (Action Block Decoder, Transformer Decoder, 4 layers, 8 heads): Query: 8 learnable vectors chunk_query[8,256], each corresponding to an action at a future time, superimposed with a sinusoidal positional code to distinguish which step in the future; Memory: The fusion token output by ObsFusion + z_token; Information is extracted from memory through cross-attention and the relationship between the 8 queries is coordinated through self-attention (ensuring that the action sequence is smooth and does not change abruptly), and the output is [B,8,256].

[0065] ActionHead: The structure is LayerNorm→Linear(256,256)→GELU→Linear(256,4), which decodes 8 256-dimensional tokens into 8-step × 4-dimensional action chunk pred_actions[B,8,4].

[0066] Predicting eight action blocks (chunks) in one go, rather than single-step actions, provides a foundation for subsequent temporal aggregation; and provides latent variables for training VAEs and deterministic patterns with z=0 during inference.

[0067] Specifically, the cross-modal fused features can be fed into an action block decoder, TransformerEncoder (4 layers, 8 attention heads). The core mechanism of the action block decoder is self-attention: in each layer, each token can see all other tokens and pair them according to their correlation, thus forming a more complete global understanding. After 4 layers of fusion, each token no longer only contains information from its own modality, but also contains the global context of all modalities, thus obtaining global context information.

[0068] Specifically, the input to the ChunkDecoder (a Transformer Decoder with 4 layers and 8 attention heads) consists of two parts: first, the 4 fused observation tokens output by ObsFusion; and second, 8 learnable vectors (called chunk queries), also known as pre-defined latent variables, with each query corresponding to an action at a future time step. At the output, the 8 256-dimensional tokens processed by the Decoder (intermediate feature vectors corresponding to each pre-defined latent variable) are each mapped to 8 4-dimensional action vectors (predicted action blocks for multiple consecutive frames) through an ActionHead (a linear mapping from 256 to 4). The four components of each 4-dimensional action vector directly correspond to the four control dimensions of the forklift: forklift boom lateral displacement (fork_y), forklift boom height (fork_z), chassis steering angle (steering), and chassis velocity (velocity).

[0069] It should be noted that if the model only predicts the action in the current frame, then instantaneous noise in a single frame (such as camera shake or lighting flicker) can cause drastic fluctuations in the action. When the model predicts 8 steps at a time, the prediction includes the temporal context. During deployment, there is a large overlap in the prediction chunks of adjacent frames (frame 1 predicts steps 1-8, frame 2 predicts steps 2-9, and so on). Subsequent temporal aggregation can weighted average the results of multiple predictions at the same moment, thereby significantly suppressing the impact of single-frame noise.

[0070] In training the target action prediction model, ACT is essentially a model based on a Conditional Variational Autoencoder (CVAE). During training, in addition to receiving image and state information, the CVAE-style model also receives a 32-dimensional random vector z (sampled from a standard normal distribution). z is transformed into a token through a projection layer (32→256) and fed into the Decoder along with the observation token. During deployment and inference, z is directly set to an all-zero vector (z=0).

[0071] Based on the above embodiments, as an implementable approach, in one embodiment, based on the target back-back judgment model and according to the visual stitching features of the forklift, the back-back judgment result of the current frame is determined, including: Step 2041: Input the visual stitching features into the first multilayer perceptron classifier to determine the first original logical value corresponding to the visual stitching features based on the first multilayer perceptron classifier; wherein, the first original logical value is used to characterize the confidence level of the backoff judgment. Step 2042: Determine the backtracking judgment result of the current frame based on the first original logical value corresponding to the visual stitching features.

[0072] The target back-off judgment model includes at least a first multilayer perceptron classifier. During the training phase of the target back-off judgment model, samples where the horizontal offset between two cages is less than a preset horizontal offset threshold and the rotational offset is greater than a preset rotational offset threshold are marked as back-off samples. The back-off label is dynamically calculated during the training phase, based on the horizontal and rotational offsets between the two cages. Specifically, when the horizontal offset is less than 0.2 meters and the rotational offset is greater than 0.02 radians, the frame is marked as "should back-off" (back=1). The physical meaning of this condition is that the forklift arm has basically aligned with the horizontal position of the pallet (small offset), but there is still an angular deviation (large rotational offset). At this point, further forward movement cannot correct the angle, and it is necessary to back off a certain distance to realign.

[0073] It's important to note that in a real-world environment, there are no true pose values ​​available—no global coordinates provide the precise distance and angle between the two cages. Therefore, the backoff labels must be derived from the forklift control state data. The labeling rules are based on two fields in the robot's state (pose): velocity (chassis linear velocity) and fork_z (forklift boom height).

[0074] The annotation logic consists of two rules, executed in order of priority, with the latter overriding the former: Rule 1: Speed ​​Reversal Detection and Backtracking Mark. Traverse each frame, and when the speed is found to have reversed from negative to positive, mark that frame and all previous frames in which the forklift was stationary as backtracking.

[0075] Rule 2: Forklift boom height coverage. Force all frames with a forklift boom height below 1.62 meters to be marked as back=0, regardless of the result of Rule 1, because backing up is only meaningful when the forklift boom has already risen (i.e., the transported cage has been lifted and is approaching the target cage below).

[0076] It should be further explained that the first multilayer perceptron classifier is actually an MLP classifier (768→512→512→1, 4 layers, dropout 0.3) which outputs the first raw logit pred_back [B,1] (the raw logit without sigmoid, not constrained to 0 and 1, and can be any real number).

[0077] Specifically, the backoff decision branch only uses three visual features and does not use state tokens; the output is a raw logit instead of a probability. Backoff (back=1) is a rare event (approximately 7% to 93% in the natural distribution). Extreme imbalance can be addressed using only a two-layer mechanism: Data layer: The weighted random sampler WeightedRandomSampler ensures that back=0 and back=1 each account for approximately 50% of each batch; Loss layer: Focal Loss, in the form of:

[0078] in, This represents the Focal Loss value calculated for the current sample. This value decreases continuously during training and is used for backpropagation to update network parameters. This represents the predicted probability of the model. It is a repressor (to focus training on hard-to-distinguish samples). To focus parameters, The category balancing weights are used to adjust the relative importance of positive and negative samples in the loss calculation.

[0079] In this embodiment of the application, Focal Loss is taken , Among them, the selected Under uniform sampling, the back=1 sample is repeated approximately 12 times. Equal weighting makes the effective weight of back=1 approximately 12 times that of back=0, leading to a surge in the false positive rate; The effective gradient of each original back=1 sample is approximately twice that of back=0, balancing recall and precision. Two reasons for outputting raw logit (instead of probability): (1) numerical stability, as Focal Loss requires logit calculation internally; (2) threshold flexibility, as the caller can select the threshold (currently 0.0, i.e., sigmoid(0)=0.5, which can be adjusted to a more conservative logit>1.0, i.e., confidence>73%, in the future).

[0080] Based on the above embodiments, as an implementable approach, in one embodiment, the forklift arm descent judgment result of the current frame is determined according to the visual stitching features of the forklift, based on the target forklift arm descent judgment model, including: Step 2051: Input the visual stitching features into the second multilayer perceptron classifier to determine the second original logical value corresponding to the visual stitching features based on the second multilayer perceptron classifier; wherein, the second original logical value is used to characterize the confidence level of the forklift arm descent judgment. Step 2052: Determine the forklift arm descent discrimination result of the current frame based on the second original logical value corresponding to the visual stitching features.

[0081] The target forklift arm descent judgment model includes at least a second multilayer perceptual classifier. During the training phase of the target forklift arm descent judgment model, the descent frame and all frames after the descent frame are marked as descent samples. Starting from the frame before the descent frame, the model continuously backtracks and marks all frames where the forklift is stationary as descent samples.

[0082] It should be noted that the descent labeling is also based on the forklift status data's fork_z and velocity fields, but the logic is different. Each complete operation trajectory will have at most one descent frame (i.e., the moment the transported cage begins to be lowered). The labeling algorithm is as follows: 1. Traverse from the beginning of the trajectory backward. When fork_z is detected to exceed 1.64 meters for the first time, it is determined that the forklift arm has been raised (up=1).

[0083] 2. Continue traversing thereafter. When fork_z is first detected to be below 1.60 meters, determine that this is the moment when the descent begins (down frame), record the frame index, and stop the search.

[0084] Once the falling frame is found, the labeling rule is as follows: mark the falling frame and all subsequent frames as down=1. Starting from the frame preceding the falling frame, continuously backtrack and mark all frames where the forklift is stationary as down=1 as well.

[0085] The target forklift arm descent judgment model and the target retreat judgment model use the same type of perception classifier and have the same training principle, which will not be elaborated here.

[0086] Based on the above embodiments, as an implementable approach, in one embodiment, determining whether the current frame meets the trigger conditions for the fallback coverage mode and / or the trigger conditions for the forklift arm descent coverage mode, based on the fallback discrimination result, the forklift arm descent discrimination result, and the current forklift arm height, includes: Step 2061: Based on the retraction judgment result and the forklift arm descent judgment result, determine the corresponding retraction probability and forklift arm descent probability respectively; Step 2062: Determine whether the current forklift boom height is above the first preset height threshold. Step 2063: If yes, then the rollback condition is confirmed to be valid. Step 2064: Determine whether the current forklift boom height is above the second preset height threshold. Step 2065: If yes, then determine that the forklift arm descent judgment condition is valid; wherein, the second preset height threshold is greater than the first preset height threshold. Step 2066: If the fallback judgment condition is valid, determine whether the fallback probability of the current frame and the previous frame both exceed the first preset probability threshold. If so, determine that the trigger condition of the fallback coverage mode is met. Step 2067: If the forklift arm descent judgment condition is valid, determine whether the probability of forklift arm descent in the current frame and the previous frame both exceed the second preset probability threshold. If so, determine that the trigger condition of the forklift arm descent coverage mode is met.

[0087] Specifically, when both the fallback coverage mode and the fall coverage mode trigger conditions are met simultaneously, the fallback coverage mode trigger condition is responded to first. If both the fallback coverage mode trigger conditions and the forklift arm fall coverage mode trigger conditions are met in the current frame, the forklift is controlled to execute the preset fixed backward correction action first.

[0088] Specifically, the back-off judgment result and the forklift arm descent judgment result can be normalized to obtain the corresponding back-off probability and forklift arm descent probability. The first preset height threshold can be set to 1.58 meters, because the classifier's back-off judgment is only meaningful when the cage has been raised to a high position. If the current forklift arm height is above the first preset height threshold, and the back-off probability in two consecutive frames exceeds the first preset probability threshold (0.8), then back-off override is triggered, i.e., the triggering condition for back-off override mode is met. The second preset height threshold can be set to 1.64 meters. If the current forklift arm height is above the second preset height threshold, then forklift arm descent judgment will be performed; otherwise, the forklift arm descent judgment process will be omitted. If the current forklift arm height is above the second preset height threshold, and the back-off probability in two consecutive frames exceeds the second preset probability threshold, then the triggering condition for forklift arm descent override mode is met. When the descent coverage mode is triggered, the output of the motion prediction model is ignored. Instead, the chassis speed is controlled to be zero or at a preset low speed, the chassis angle is controlled to be zero or at the current angle, the forklift arm lateral movement is maintained, and the forklift arm is lowered at a preset descent speed until the forklift arm height reaches the placement height or continues for a preset number of frames.

[0089] The logic for forklift boom descent override is similar, including discriminator judgment, height condition, continuous confirmation, and fixed descent action coverage, but it has a lower priority than rollback override.

[0090] Based on the above embodiments, as an implementable approach, in one embodiment, when the current frame does not meet the triggering conditions for the fallback coverage mode and the fallback coverage mode, the action block prediction results of multiple future consecutive frames are subjected to multi-dimensional temporal aggregation processing to obtain the target control action of the current frame, so as to control the forklift to execute the target control action, including: Step 2091: If the triggering conditions for the fallback coverage mode and the fallback coverage mode are not met in the current frame, store the action block prediction results for the next several consecutive frames into the target queue. Step 2092: For the current moment, extract multiple target action prediction values ​​covering the current moment from the target queue; wherein, the multi-dimensional action vector includes multiple target action prediction values; Step 2093: Calculate the exponential decay weight corresponding to each target action prediction value based on multiple target action prediction values; Step 2094: Based on the exponential decay weight, the predicted values ​​of each target action are weighted and aggregated to obtain the aggregated predicted values ​​of the target actions; among them, the predicted values ​​of the target actions in the forklift arm lateral movement dimension adopt the smallest exponential decay coefficient. Step 2095: Constrain the aggregated target action prediction values ​​to the corresponding preset physical value range to obtain the target control action of the current frame.

[0091] It should be noted that the ACT branch predicts action chunks of length 8. Due to frame-by-frame inference, there is significant prediction overlap between adjacent frames. For example, if frame 1 predicts actions for steps 1-8, and frame 2 predicts actions for steps 2-9, then step 3, a future moment, will be predicted once by frames 1, 2, and 3. The same future moment can be predicted simultaneously across multiple frames, and these predictions may differ slightly. By serializing actions into fixed-length chunks in time, the motion jitter problem inherent in traditional single-step control methods in long-cycle, high-precision tasks is resolved. This design not only improves control continuity but also enhances the system's robustness to sensor noise, providing core algorithmic support for achieving precise cage stacking.

[0092] Specifically, in order to achieve temporal aggregation, this embodiment maintains a queue of the most recent 8 frame prediction chunks (i.e., a sliding window). For the current time, it identifies all predictions that cover this time and performs an exponentially decaying weighted average on them. The weight calculation formula is as follows: ,in It is the "age" of this prediction (how many frames ago this prediction was generated). It's an exponential decay coefficient. Older predictions have smaller weights in the weighting. Different action dimensions use different coefficients. Values: Use smaller values ​​for slow variables (such as forklift boom lateral movement fork_y). (The predicted target motion values ​​for the forklift boom lateral movement dimension use a relatively small exponential decay coefficient), meaning that historical predictions are retained longer, resulting in a smoother output; fast variables (such as steering and speed) use larger... This means that historical predictions are quickly forgotten, making the model more sensitive to new scenario changes.

[0093] The expression for weighted aggregation of the predicted values ​​of each target action is as follows:

[0094] in, Indicates the predicted value of the target action. Indicates the first [item] in the target queue The specific dimension corresponding to the current moment in the four-dimensional motion vector predicted by the frame. The predicted value of the target action.

[0095] Finally, based on the physical range of each action dimension (forklift arm lateral displacement, forklift arm height, chassis angle, and chassis linear velocity, such as chassis angle ∈ [-0.1, 0.1], forklift arm height ∈ [0, 1.6]), the aggregated action vector is numerically clamped to prevent outliers from driving the forklift to perform actions beyond the safe range. The final output after temporal aggregation and clamping is a 4-dimensional action vector (target control action), which is directly used as the real-time control signal for the forklift.

[0096] The forklift motion control method provided in this application reduces deployment costs and eliminates the high dependence on depth ground truth by relying solely on visual images acquired by multiple cameras and the forklift's own state data for forklift motion prediction. Furthermore, the obtained cross-modal fusion features and visual stitching features contain global spatial correlation information of multi-view images, ensuring the accuracy of motion block prediction results. By adding a backtracking judgment mechanism, erroneous actions can be detected and corrected promptly when prediction errors occur, avoiding collisions of the material cages due to motion control errors. Additionally, by introducing a forklift arm descent judgment mechanism, it can be ensured that the forklift only begins to lower the material cage after it has been moved directly above the target material cage below and the corner posts are aligned, further ensuring the safety of material cage stacking. Moreover, through motion segmentation, multi-dimensional temporal aggregation, and multi-dimensional value domain pruning, a smooth and reasonably responsive motion control signal is ensured, further guaranteeing the safety and reliability of forklift motion control.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0098] The embodiments of this application also provide a forklift motion control model training method, wherein the forklift motion control model trained by the method is applied to the forklift motion control method provided in the above embodiments.

[0099] like Figure 3 The diagram shown is a flowchart illustrating the forklift motion control model training method provided in this embodiment of the application. The method includes: Step 301: In the simulation pre-training stage, multiple expert operation trajectories are collected using script strategies; Step 302: Based on the operation trajectories of multiple experts, determine the simulation samples for model training, and mark the backtracking simulation samples and the descent simulation samples in the simulation samples; Step 303: Based on the simulation samples trained by the model, train the original motion prediction model, the original back-off judgment model and the original forklift arm descent judgment model to obtain the intermediate motion prediction model, the intermediate back-off judgment model and the intermediate forklift arm descent judgment model. Step 304: In the real data fine-tuning stage, acquire multiple real operation trajectories of the forklift; wherein, the real operation trajectory includes at least the multi-source visual image trajectory and the control state change trajectory. Step 305: Based on multiple real operation trajectories, determine the fine-tuning samples for model training, and mark the backtracking fine-tuning samples and the descent fine-tuning samples in the fine-tuning samples. Step 306: Based on the fine-tuning samples from model training, fine-tune the model parameters of the intermediate action prediction model, the intermediate back-off judgment model, and the intermediate forklift arm descent judgment model to obtain the target action prediction model, the target back-off judgment model, and the target forklift arm descent judgment model.

[0100] The forklift motion control model includes a target motion prediction model, a target retraction judgment model, and a target forklift arm descent judgment model. The target motion prediction model, the target retraction judgment model, and the target forklift arm descent judgment model are applied to the forklift motion control method provided in the above embodiments.

[0101] The labeling rules for the rollback simulation samples and descent simulation samples, as well as the rollback fine-tuning samples and descent fine-tuning samples, are as described in the above embodiments and will not be repeated here.

[0102] It should be noted that the target action prediction model, target retreat judgment model, and target forklift arm descent judgment model involved in the embodiments of this application are all obtained through model training. The embodiments of this application provide the following model training process: Phase 1: Simulation Pre-training Phase (Isaac Sim) Using a scripted policy, 1000–2000 expert operation trajectories (each approximately 400–700 frames) are collected to obtain simulation data (simulation samples).

[0103] The simulation data provides complete task phase annotations and true values ​​of cage and forklift poses, and is free of sensor noise.

[0104] The model learns the complete stacking process: approach the cage → align → raise the forklift arm → insert → reverse.

[0105] The ACT policy network and the discriminator are trained separately (independently).

[0106] While the simulation can obtain the absolute pose ground truth values ​​of the forklift and the two cages, these poses are only used by the script strategy and are never input into the model; the model only receives images and the robot (forklift) state. This ensures that the strategy does not depend on simulation-specific ground truth values, thus allowing it to be transferred to real-world scenarios without pose ground truth values.

[0107] Phase Two: Fine-tuning with Real Data Based on the first-stage checkpoint, training continues using 100–300 real operation trajectories (approximately 10,000–20,000 frames) collected from real vehicles (forklifts in actual scenarios).

[0108] The real data comes from the recording of an actual operation of an automated strategy with RGB + depth input, including real sensor noise and dynamic characteristics.

[0109] The aim is to adapt the model to the visual and dynamic characteristics of the real world. While automated strategies for acquiring real-world data may utilize depth information, the model to be trained / deployed in this embodiment still only uses RGB+ states.

[0110] The trained PyTorch checkpoint is exported via ONNX and compiled with TensorRT to generate an inference engine optimized for a specific GPU, which is then deployed on the Jetson Orin GPU platform.

[0111] Robot (forklift) end: After each frame of image and state is input into the engine, the ACT branch outputs 8-step action prediction and the Discriminator branch outputs backtracking judgment. The results of the two branches are checked for backtracking and time-series aggregation to generate the final action to be executed in the current frame. The entire inference is completed on the GPU.

[0112] Regarding the training data source and preprocessing, the simulation data (Isaac Sim) format is as follows: left / right / ToF image [T,224,224,3]uint8; action sequence [T,4]float32 (fork_y,steering,fork_z,velocity); pose state [T,4]float32.

[0113] Preprocessing: The image is already 224×224uint8. When loading, it is directly normalized to float32[0,1] by dividing by 255, without the need for padding or rotation.

[0114] Acquisition of back_label in simulation: The simulation data is not pre-labeled with back_labels. Instead, back_labels are calculated in real time during training based on the relative poses of the two cages. The rule is: when the horizontal offset between the two cages is <0.2m and the rotational offset is >0.02rad, it is marked as a retraction state. The physical meaning is that the horizontal alignment is basically complete, but there is still an angular deviation, requiring retraction for correction.

[0115] The real data (actual vehicle ROS2 MCAP bag) first undergoes conversion preprocessing, as follows: Time alignment: Based on the left camera timestamp, the remaining topics are matched using nearest neighbor matching; Image processing: Original frame → center padding to form a square → resize to 224×224; Rotation: The left and right cameras rotate 90 degrees (k=3) (270° clockwise), while the ToF sensor does not rotate. Discard the first two frames: The frame rate is unstable during the camera initialization phase.

[0116] Real data back_label annotation (still under testing): When the forklift speed changes from negative to positive (i.e., starts to move backward), mark the 5 frames before and after that moment as backward (label=1).

[0117] The back_label of the simulation and the real data use two different annotation / calculation rules (the simulation uses the relative pose of the cage, and the real data uses 5 frames before and after the velocity crosses zero).

[0118] The forklift motion control model training method provided in this application, through model training in a simulation pre-training stage and a real data fine-tuning stage, enables the final target motion prediction model, target back-off judgment model, and target forklift arm descent judgment model to have high accuracy, further improving the accuracy and reliability of forklift motion control.

[0119] The embodiments of this application also provide a forklift motion control device for executing the forklift motion control method provided in the above embodiments.

[0120] like Figure 4 The diagram shown is a structural schematic of the forklift motion control device provided in an embodiment of this application. The forklift motion control device 40 includes: an acquisition module 401, a feature module 402, a motion prediction module 403, a back-off judgment module 404, a descent judgment module 405, a trigger condition judgment module 406, a back-off control module 407, a descent control module 408, and a motion control module 409.

[0121] The system includes the following modules: an acquisition module for acquiring multi-source visual images and current control state data of the forklift; the multi-source visual images include at least the left-side visual image captured by the left-side camera, the right-side visual image captured by the right-side camera, and the forklift arm visual image captured by the forklift arm camera; a feature module for determining the forklift's cross-modal fusion features and visual stitching features based on the multi-source visual images and current control state data; an action prediction module for determining the action block prediction results for future consecutive frames based on the target action prediction model and the forklift's cross-modal fusion features; the action block prediction results include the forklift's multi-dimensional action vectors; a backtracking judgment module for determining the backtracking judgment result for the current frame based on the target backtracking judgment model and the forklift's visual stitching features; and a descent judgment module for determining the descent judgment result for the target forklift arm based on the target forklift arm descent judgment model and the forklift's visual stitching features. The system comprises the following modules: a forklift boom descent detection module, a trigger condition judgment module, and an action control module. The forklift boom descent detection module determines whether the current frame meets the trigger conditions for the back-up coverage mode and / or the forklift boom descent coverage mode based on the back-up detection result, the forklift boom descent detection result, and the current forklift boom height. The back-up control module controls the forklift to execute a preset fixed back-up correction action if the current frame meets the trigger conditions for the back-up coverage mode. The descent control module controls the forklift to execute a preset fixed descent placement action if the current frame meets the trigger conditions for the forklift boom descent coverage mode. The action control module performs multi-dimensional temporal aggregation processing on the action block prediction results for multiple future consecutive frames to obtain the target control action for the current frame, in order to control the forklift to execute the target control action if the current frame does not meet the trigger conditions for the back-up coverage mode and the descent coverage mode.

[0122] Specifically, in one embodiment, the feature module is specifically used for: Based on the multi-source visual images, the left visual features, right visual features, and forklift arm visual features were determined respectively; among them, the multi-source visual images were color images; Based on the current control status data, determine the characteristics of the current control status; The left visual features, right visual features, forklift arm visual features, and current control state features are fused to obtain the cross-modal fusion features of the forklift. The visual features of the left side, the right side, and the forklift arm are stitched together to obtain the visual stitched features of the forklift. The current control status data includes at least the forklift arm lateral displacement, forklift arm height, chassis rotation angle, and chassis linear velocity.

[0123] Specifically, in one embodiment, the action prediction module is specifically used for: Cross-modal fusion features and multiple preset latent variables are input into the action block decoder; wherein the number of preset latent variables is the same as the number of frames in the next several consecutive frames; Based on the attention mechanism in the action block decoder, global contextual information is extracted from cross-modal fusion features; Based on the global context information, determine the intermediate feature vectors corresponding to each preset latent variable; Based on the intermediate feature vectors corresponding to each preset latent variable, the prediction results of action blocks for multiple consecutive frames in the future are determined. The target action prediction model includes at least an action block decoder. During the training phase of the target action prediction model, random latent variables sampled from the standard normal distribution are input into the action block decoder. During the deployment phase of the target action prediction model, the random latent variables are set to a vector of all zeros.

[0124] Specifically, in one embodiment, the rollback determination module is specifically used for: The visual stitching features are input into a first multilayer perceptron classifier to determine the first original logical value corresponding to the visual stitching features based on the first multilayer perceptron classifier; wherein, the first original logical value is used to characterize the confidence level of the backoff judgment; Based on the first original logical value corresponding to the visual stitching features, determine the backtracking judgment result of the current frame; The target back-off judgment model includes at least a first multilayer perceptron classifier. During the training phase of the target back-off judgment model, samples with a horizontal offset between two cages that is less than a preset horizontal offset threshold and a rotational offset that is greater than a preset rotational offset threshold are marked as back-off samples.

[0125] Specifically, in one embodiment, the descent determination module is specifically used for: The visual stitching features are input into the second multilayer perceptron classifier to determine the second original logical value corresponding to the visual stitching features based on the second multilayer perceptron classifier; wherein, the second original logical value is used to characterize the confidence level of the forklift arm descent judgment; Based on the second original logical value corresponding to the visual stitching features, determine the forklift arm descent discrimination result of the current frame; The target forklift arm descent judgment model includes at least a second multilayer perceptual classifier. During the training phase of the target forklift arm descent judgment model, the descent frame and all frames after the descent frame are marked as descent samples. Starting from the frame before the descent frame, the model continuously backtracks and marks all frames where the forklift is stationary as descent samples.

[0126] Specifically, in one embodiment, the triggering condition judgment module is specifically used for: Based on the back-off judgment result and the forklift arm descent judgment result, determine the corresponding back-off probability and forklift arm descent probability respectively; Determine whether the current forklift boom height is above the first preset height threshold; If so, then the rollback condition is valid; Determine whether the current forklift boom height is above the second preset height threshold; If so, the forklift arm descent judgment condition is determined to be valid; wherein, the second preset height threshold is greater than the first preset height threshold; If the fallback judgment condition is valid, determine whether the fallback probability of the current frame and the previous frame both exceed the first preset probability threshold. If so, determine that the trigger condition of the fallback coverage mode is met. If the forklift arm descent judgment condition is valid, determine whether the probability of forklift arm descent in the current frame and the previous frame both exceed the second preset probability threshold. If so, determine that the trigger condition of forklift arm descent coverage mode is met. When both the fallback coverage mode and the fallback coverage mode trigger conditions are met simultaneously, the fallback coverage mode trigger condition will be responded to first.

[0127] Specifically, in one embodiment, the motion control module is specifically used for: If the triggering conditions for fallback coverage mode and fallback coverage mode are not met in the current frame, the action block prediction results for multiple consecutive future frames will be stored in the target queue. For the current moment, extract multiple target action prediction values ​​that cover the current moment from the target queue; where the multi-dimensional action vector includes multiple target action prediction values; Calculate the exponentially decaying weight corresponding to each target action prediction value based on multiple target action prediction values. Based on the exponential decay weight, the predicted values ​​of each target action are weighted and aggregated to obtain the aggregated predicted values ​​of the target actions; among them, the predicted values ​​of the target actions in the forklift arm lateral movement dimension are based on the smallest exponential decay coefficient. The aggregated target action prediction values ​​are constrained to the corresponding preset physical value range to obtain the target control action of the current frame.

[0128] For a description of the features in the embodiment corresponding to the forklift motion control device, please refer to the relevant description of the embodiment corresponding to the forklift motion control method, which will not be repeated here.

[0129] Embodiments of this application also provide an electronic device, such as... Figure 5 The diagram shown is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, including a processor 10 and a memory 20. The memory 20 stores a computer program, and the processor 10 is configured to run the computer program to execute the steps in any of the above-described forklift motion control method embodiments.

[0130] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the forklift motion control method embodiments described above or the steps in any of the forklift motion control model training method embodiments described above when it is run.

[0131] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0132] The embodiments of this application also provide a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements any of the above-described forklift motion control methods or executes the steps in the embodiments of the above-described forklift motion control model training method.

[0133] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements any of the forklift motion control methods described above or executes the steps in any of the forklift motion control model training method embodiments described above.

[0134] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0135] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] The foregoing has provided a detailed description of the forklift motion control method, model training method, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A forklift motion control method, characterized in that, include: Acquire multi-source visual images and current control status data of the forklift; wherein the multi-source visual images include at least the left visual image captured by the left-side camera of the forklift, the right-side visual image captured by the right-side camera of the forklift, and the forklift arm visual image captured by the forklift arm camera. Based on the multi-source visual images and the current control state data, the cross-modal fusion features and visual stitching features of the forklift are determined respectively. Based on the target motion prediction model, and according to the cross-modal fusion features of the forklift, the motion block prediction results for future consecutive frames are determined; wherein, the motion block prediction results include the multi-dimensional motion vectors of the forklift. Based on the target back-back judgment model, the back-back judgment result of the current frame is determined according to the visual stitching features of the forklift; Based on the target forklift boom descent judgment model, the forklift boom descent judgment result of the current frame is determined according to the visual stitching features of the forklift. Based on the rollback discrimination result, the forklift arm descent discrimination result, and the current forklift arm height, determine whether the current frame meets the trigger conditions for the rollback coverage mode and / or the trigger conditions for the forklift arm descent coverage mode. If the triggering conditions of the fallback overlay mode are met in the current frame, the forklift is controlled to perform a preset fixed backward correction action; If the triggering conditions for the forklift arm descent coverage mode are met in the current frame, control the forklift to perform a preset fixed descent placement action; If the trigger conditions for the fallback coverage mode and the fallback coverage mode are not met in the current frame, the action block prediction results of the next multiple consecutive frames are subjected to multi-dimensional temporal aggregation processing to obtain the target control action of the current frame, so as to control the forklift to execute the target control action.

2. The forklift motion control method according to claim 1, characterized in that, The step of determining the cross-modal fusion features and visual stitching features of the forklift based on the multi-source visual images and the current control state data includes: Based on the multi-source visual images, left-side visual features, right-side visual features, and forklift arm visual features are determined respectively; wherein, the multi-source visual images are color images; Based on the current control state data, determine the characteristics of the current control state; The left visual features, right visual features, forklift arm visual features, and current control state features are fused to obtain the cross-modal fusion features of the forklift. The visual features of the left side, the right side, and the forklift arm are stitched together to obtain the visual stitched features of the forklift. The current control status data includes at least the forklift arm lateral displacement, forklift arm height, chassis rotation angle, and chassis linear velocity.

3. The forklift motion control method according to claim 1, characterized in that, The target action prediction model, based on the cross-modal fusion features of the forklift, determines the action block prediction results for multiple consecutive frames in the future, including: The cross-modal fusion features and multiple preset latent variables are input into the action block decoder; wherein, the number of preset latent variables is the same as the number of frames in the next several consecutive frames; Based on the attention mechanism in the action block decoder, global context information is extracted from the cross-modal fusion features; Based on the global context information, determine the intermediate feature vector corresponding to each of the preset latent variables; Based on the intermediate feature vectors corresponding to each of the preset latent variables, the prediction results of action blocks for multiple consecutive frames in the future are determined. The target action prediction model includes at least the action block decoder. During the training phase of the target action prediction model, random latent variables sampled from a standard normal distribution are input into the action block decoder. During the deployment phase of the target action prediction model, the random latent variables are set to a vector of all zeros.

4. The forklift motion control method according to claim 1, characterized in that, The target-based back-back judgment model determines the back-back judgment result of the current frame based on the visual stitching features of the forklift, including: The visual stitching features are input into a first multilayer perceptron classifier to determine a first original logical value corresponding to the visual stitching features based on the first multilayer perceptron classifier; wherein, the first original logical value is used to characterize the confidence level of the backoff judgment. Based on the first original logical value corresponding to the visual stitching feature, determine the backtracking judgment result of the current frame; The target back-off judgment model includes at least the first multilayer perceptron classifier. During the training phase of the target back-off judgment model, samples with a horizontal offset between two cages that is less than a preset horizontal offset threshold and a rotational offset that is greater than a preset rotational offset threshold are marked as back-off samples.

5. The forklift motion control method according to claim 1, characterized in that, The target forklift boom descent judgment model determines the forklift boom descent judgment result of the current frame based on the visual stitching features of the forklift, including: The visual stitching features are input into a second multilayer perceptron classifier to determine a second original logical value corresponding to the visual stitching features based on the second multilayer perceptron classifier; wherein, the second original logical value is used to characterize the confidence level of the forklift arm descent judgment; Based on the second original logic value corresponding to the visual stitching feature, determine the forklift arm descent discrimination result of the current frame; The target forklift arm descent judgment model includes at least the second multilayer perceptual classifier. During the training phase of the target forklift arm descent judgment model, the descent frame and all frames after the descent frame are marked as descent samples. Starting from the frame before the descent frame, the model continuously backtracks and marks all frames where the forklift is stationary as descent samples.

6. The forklift motion control method according to claim 1, characterized in that, The step of determining whether the current frame meets the trigger conditions for the fallback coverage mode and / or the forklift arm descent coverage mode based on the fallback discrimination result, the forklift arm descent discrimination result, and the current forklift arm height includes: Based on the back-off judgment result and the forklift arm descent judgment result, the corresponding back-off probability and forklift arm descent probability are determined respectively. Determine whether the current forklift boom height is above the first preset height threshold; If so, then the rollback condition is valid; Determine whether the current forklift boom height is above the second preset height threshold; If so, the forklift arm descent judgment condition is determined to be valid; wherein, the second preset height threshold is greater than the first preset height threshold; If the fallback judgment condition is valid, determine whether the fallback probability of the current frame and the previous frame both exceed the first preset probability threshold. If so, determine that the trigger condition of the fallback coverage mode is met. If the forklift arm descent judgment condition is valid, determine whether the probability of forklift arm descent in the current frame and the previous frame both exceed the second preset probability threshold. If so, determine that the trigger condition of the forklift arm descent coverage mode is met. Specifically, when both the fallback coverage mode and the drop coverage mode trigger conditions are met simultaneously, the fallback coverage mode trigger condition is responded to first.

7. The forklift motion control method according to claim 1, characterized in that, When the triggering conditions for the fallback coverage mode and the fallback coverage mode are not met in the current frame, the action block prediction results of the next multiple consecutive frames are subjected to multi-dimensional temporal aggregation processing to obtain the target control action of the current frame, so as to control the forklift to execute the target control action, including: If the triggering conditions for the fallback coverage mode and the fallback coverage mode are not met in the current frame, the prediction results of the action blocks for the next multiple consecutive frames are stored in the target queue. For the current moment, extract multiple target action prediction values ​​covering the current moment from the target queue; wherein, the multi-dimensional action vector includes multiple target action prediction values; Based on the multiple target action prediction values, calculate the exponential decay weight corresponding to each target action prediction value; Based on the exponential decay weight, the predicted values ​​of each target action are weighted and aggregated to obtain the aggregated predicted values ​​of the target actions; wherein, the predicted values ​​of the target actions in the forklift arm lateral movement dimension adopt the smallest exponential decay coefficient. The aggregated target action prediction value is constrained to the corresponding preset physical value range to obtain the target control action of the current frame.

8. A method for training a forklift motion control model, characterized in that, include: During the simulation pre-training phase, script strategies are used to collect the operation trajectories of multiple experts. Based on the multiple expert operation trajectories, simulation samples for model training are determined, and backtracking simulation samples and descent simulation samples are marked in the simulation samples; Based on the simulation samples trained by the model, the original motion prediction model, the original back-off judgment model, and the original forklift arm descent judgment model are trained to obtain the intermediate motion prediction model, the intermediate back-off judgment model, and the intermediate forklift arm descent judgment model. During the real data fine-tuning stage, multiple real operation trajectories of the forklift are acquired; wherein, the real operation trajectory includes at least multi-source visual image trajectory and control state change trajectory; Based on the multiple real operation trajectories, fine-tuning samples for model training are determined, and backtracking fine-tuning samples and descent fine-tuning samples are marked in the fine-tuning samples. Based on the fine-tuned samples trained by the model, the model parameters of the intermediate action prediction model, the intermediate back-off judgment model and the intermediate forklift arm descent judgment model are fine-tuned to obtain the target action prediction model, the target back-off judgment model and the target forklift arm descent judgment model. The forklift motion control model includes the target motion prediction model, the target retraction judgment model, and the target forklift arm descent judgment model. The target motion prediction model, the target retraction judgment model, and the target forklift arm descent judgment model are applied to the forklift motion control method according to any one of claims 1 to 7.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the forklift motion control method as described in any one of claims 1 to 7 or the steps of the forklift motion control model training method as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the forklift motion control method as described in any one of claims 1 to 7 or the steps of the forklift motion control model training method as described in claim 8.

Images