A forklift truck goods pallet positioning method and system based on reinforcement learning

Abstract

The application discloses a forklift goods pallet positioning method and system based on reinforcement learning, which comprises the following steps: collecting forklift running state data and insertion contact event data, and preprocessing to generate state-event joint sequences; generating insertion success identifiers or insertion failure identifiers for insertion trajectory samples; generating a relationship state set; dividing an insertion relationship success domain and an insertion relationship collapse domain; performing a relationship state reachability determination process to generate an executability determination result; constructing an improved IQL to generate an action strategy; and positioning and inserting a forklift goods pallet according to the action strategy. The application combines reinforcement learning and relationship state analysis, optimizes forklift pallet insertion operation decision-making, and has the advantages of high efficiency, stability and strong self-adaptability.

Description

Technical Field

[0001] This invention relates to the field of forklift pallet positioning, and more particularly to a forklift pallet positioning method and system based on reinforcement learning. Background Technology

[0002] In industrial logistics and warehousing operations, the positioning and insertion of forklift pallets is a crucial step in achieving automated handling. Existing forklift pallet positioning methods mostly rely on pose estimation based on geometric models, visual or laser sensors to obtain pallet position and attitude information, and plan insertion paths or generate control commands accordingly. Some methods also introduce rule control or traditional reinforcement learning techniques to optimize the insertion process through manually set reward functions. These methods typically assume that the geometric models of the forklift and pallet are accurate, the environment is stable, or rely on a large amount of human experience to debug the control rules.

[0003] In practical applications, due to pallet manufacturing tolerances, uneven ground, elastic deformation of forklift tires, and uncertainty in the contact sequence during insertion, insertion failure may still occur even if alignment is achieved geometrically. Existing technologies struggle to structurally characterize the physical accessibility and failure risk during insertion without relying on high-precision modeling and complex rule configuration. Traditional reinforcement learning methods also struggle to effectively distinguish between success and collapse states, resulting in low policy learning efficiency and insufficient stability, making it difficult to form a reliable forklift pallet insertion decision mechanism. Summary of the Invention

[0004] One objective of this invention is to propose a forklift pallet positioning method and system based on reinforcement learning. This invention combines reinforcement learning and relational state analysis to optimize forklift pallet insertion operation decisions, and has the advantages of high efficiency, stability and strong adaptability.

[0005] A forklift pallet positioning method based on reinforcement learning according to an embodiment of the present invention includes the following steps: During the pallet approach, alignment and insertion operations performed by the forklift, forklift operation status data and insertion contact event data are collected, and preprocessed to generate a state-event joint sequence. Insertion trajectory samples are constructed based on state-event joint sequences. Insertion success or failure indicators are generated for the insertion trajectory samples according to whether the insertion job is completed or interrupted. Generate a set of relation states based on the insertion trajectory samples with insertion success or insertion failure indicators; Based on the relation state set, the insertion relation success domain and the insertion relation collapse domain are divided; Based on the current relationship state and action constraints, perform the relationship state reachability determination process and generate the executability determination result; An improved IQL is constructed based on a set of relation states, taking whether the relation state enters the insertion relation collapse domain during the time unfolding process as the learning objective to generate action policies; In actual pallet insertion operations, based on the current relationship status and the set of executable actions obtained by filtering the executableness judgment results, the forklift control actions are output according to the action strategy to complete the forklift pallet positioning and insertion.

[0006] Optionally, the preprocessing specifically includes time synchronization, noise filtering, outlier removal, state variable normalization, contact event alignment, and state-event timing splicing.

[0007] Optionally, the generation of the inserted trajectory sample specifically includes: Based on the state-event joint sequence, the start and end events of the forklift pallet insertion operation are identified; Using the identified start event and corresponding termination event as boundaries, the state-event joint sequence is segmented to generate multiple continuous insertion trajectory samples. Each insertion trajectory sample contains forklift running status data and insertion contact event data from the start of insertion to the end of insertion. Based on each insertion trajectory sample, the corresponding insertion termination status information is extracted, including the fork arm insertion depth, the change of contact force during insertion, the forklift movement status, and the duration of the insertion operation. Based on the insertion termination status information, and according to the insertion completion conditions, an insertion completion determination is performed on each insertion trajectory sample. When the insertion trajectory sample meets the following conditions, the fork arm reaches the target insertion depth, the forklift movement state enters a stable state, no abnormal contact event is triggered during the insertion process, and the duration of the insertion operation does not exceed the allowable range, an insertion success identifier is generated. Based on the insertion termination status information, when the insertion trajectory sample triggers an exit event or an interruption event during the insertion process, or the fork arm does not reach the target insertion depth, or an abnormal contact event occurs during the insertion process, or the duration of the insertion operation exceeds the allowable range, an insertion failure flag is generated. The inserted trajectory samples are associated with the insertion success flag or the insertion failure flag and stored together to form a set of inserted trajectory samples with the insertion success flag or the insertion failure flag.

[0008] Optionally, the generation of the relation state set specifically includes: Based on the insertion trajectory samples with insertion success or insertion failure indicators, the local state-event joint sequence corresponding to each insertion trajectory sample is analyzed in chronological order; Each local state-event joint sequence is segmented according to the time window length and sliding step size to form multiple trajectory segments; For each trajectory segment, extract relational state elements, including statistical characteristics of forklift operation status, trend characteristics of operation status changes, and characteristics of the occurrence of insertion contact events; Perform discretization mapping on relation state elements to convert continuous relation state elements into a finite number of relation state identifiers; Based on the insertion success or insertion failure flag corresponding to the insertion trajectory sample, the relationship status flags are classified. Relationship status flags from successfully inserted trajectory samples are classified into the successful relationship status subset, and relationship status flags from failed inserted trajectory samples are classified into the failed relationship status subset. The relationship state identifiers contained in the subsets of successful relationship states and the subsets of failed relationship states are deduplicated and integrated to generate a set of relationship states.

[0009] Optionally, the generation of the insertion relation success field and the insertion relation collapse field specifically includes: Based on the relation state set, obtain the occurrence record of each relation state identifier in the insertion trajectory sample set with insertion success identifier or insertion failure identifier, and count the number of times the relation state identifier appears in the insertion success trajectory sample and the number of times it appears in the insertion failure trajectory sample respectively. For each relation status identifier in the relation status set, count the number of times the relation status identifier appears in the successful insertion trajectory sample and the number of times it appears in the failed insertion trajectory sample. Based on the two types of occurrences, calculate the success rate corresponding to the relation status identifier. Set success domain determination thresholds and collapse domain determination thresholds for the relation state set, and perform domain attribute determination for each relation state identifier based on the comparison result between the success rate and the determination thresholds. When the success rate corresponding to the relation status identifier meets the success field determination threshold, the relation status identifier is classified into the insertion relation success field. When the success rate corresponding to the relation status identifier meets the collapse domain determination threshold, the relation status identifier is classified into the insertion relation collapse domain.

[0010] Optionally, the generation of the executability determination result specifically includes: Based on real-time collected forklift operating status data and inserted contact event data, update the state-event joint sequence and map the current relationship state according to the relationship state set; Construct a set of motion constraints, including forklift travel speed constraints, steering amplitude constraints, fork tooth lifting range constraints, fork tooth pitch range constraints, braking intensity constraints, and safety distance constraints during insertion. The forklift control motion space is parsed based on the set of motion constraints to generate multiple candidate motions, and motion constraint verification rules are configured for each candidate motion. Perform action constraint verification for each candidate action to ensure that the candidate action meets all constraints in the action constraint set. For each candidate action, perform an executability determination. If the candidate action satisfies all action constraints, generate an executability determination result of "executable"; otherwise, generate an executability determination result of "non-executable". Based on the executability determination results of all candidate actions, a set of executable actions is generated.

[0011] Optionally, the generation of the action strategy specifically includes: Based on the set of relation states, we analyze the proportion of each relation state in the successful and failed insertion trajectory samples, and determine whether each relation state is located in the successful insertion domain, the collapsed insertion domain, or the boundary region based on the threshold. The relation status identifiers are classified. If it is in the successful insertion domain, it is marked as successful; if it is in the collapsed insertion domain, it is marked as collapsed; if it is in the boundary region, it is marked as boundary. During the training process, "whether or not to enter the insertion relation collapse domain" is taken as the core learning objective for improving IQL; For relation states marked as successful, their value is updated based on the temporal evolution of the relation states to optimize action selection; For relation states marked as collapsed, value update suppression is implemented; During training, the action strategy is optimized by using the value update rule of reinforcement learning to ensure that the generated strategy can effectively avoid the entry of the insertion relation collapse domain. The strategy is dynamically adjusted by using environmental feedback to generate the final action strategy.

[0012] Optionally, the completion of the forklift pallet positioning and insertion specifically includes: In each control cycle, collect the current operating status data of the forklift and insert contact event data, update the state-event joint sequence, and generate the current relationship state based on the updated state-event joint sequence. Based on the current relationship state, the relationship state reachability determination process is invoked to determine the candidate control actions and filter to generate a set of executable actions corresponding to the current relationship state. Invoke the action strategy, limit the action strategy to the set of executable actions, and obtain the result of the selection of executable actions in the current relationship state; Based on the results of the selectable actions, determine the actual actions to be executed within the current control cycle; The actual actions are converted into a set of forklift control commands, including driving control commands, steering control commands, mast lifting control commands, and fork pitch control commands. Output a set of forklift control commands to the forklift actuator to drive the forklift to perform corresponding approach, alignment or insertion actions, and complete the positioning and insertion of the forklift pallet.

[0013] A forklift pallet positioning system based on reinforcement learning according to an embodiment of the present invention includes: The status event acquisition module is used to collect forklift operating status data and insert contact event data to generate a joint status-event sequence. The insertion trajectory construction module is used to construct insertion trajectory samples based on the joint sequence of state and event, and generate insertion success or insertion failure indicators; The relation state generation module is used to generate a set of relation states based on the inserted trajectory samples; The insertion relation domain partitioning module is used to partition the insertion relation success domain and the insertion relation collapse domain; The relation reachability determination module is used to generate an executability determination result based on the current relation state and action constraints. The action policy learning module is used to build an improved IQL based on a set of relation states and generate action policies. The control decision execution module is used to output forklift control actions based on the action strategy to complete pallet positioning and insertion.

[0014] The beneficial effects of this invention are: This invention proposes a reinforcement learning-based method for forklift pallet positioning, which significantly improves the positioning and insertion efficiency of forklifts in dynamic environments. Unlike traditional positioning methods based on geometric models and rule control, this invention utilizes the core advantages of reinforcement learning by constructing a set of relational states and performing dynamic learning, enabling the forklift to adaptively adjust according to environmental changes during the actual insertion process. By introducing the concepts of insertion success domain and insertion collapse domain, this invention effectively avoids the shortcomings of traditional methods that rely on high-precision modeling and overcomes insertion failure problems caused by factors such as manufacturing tolerances, uneven ground, and forklift tire deformation.

[0015] This invention constructs insertion trajectory samples based on a joint state-event sequence and generates insertion success or failure flags by combining whether the insertion operation is successfully completed or interrupted. It further constructs a set of relational states. This method avoids the dependence on geometric models in traditional methods and can more accurately reflect the real interaction between the forklift and the pallet. By dynamically dividing the relational states and combining the learning objectives of reinforcement learning, the forklift can optimize its insertion strategy during continuous training, ensuring that the forklift can avoid entering the insertion relation collapse domain when performing insertion tasks, thereby improving the success rate of insertion operations.

[0016] The innovation of this invention lies in combining the relational state during the insertion process with reinforcement learning and implementing a value update suppression mechanism to avoid the negative impact of unstable samples on the learning strategy. By suppressing collapsed state samples, this invention can ensure the stability of the reinforcement learning process, thereby providing more reliable decision support in actual operations. This mechanism not only optimizes the control strategy of forklifts, but also significantly improves operational efficiency and safety.

[0017] In summary, this invention has significant advantages in improving the automation level of forklift pallet positioning and insertion operations. Through optimized reinforcement learning algorithms combined with real-time environmental feedback, this invention can not only effectively avoid the failure problems in traditional methods, but also adapt to complex and ever-changing actual operating environments, providing reliable technical support for the intelligent application of forklifts in the warehousing and logistics fields. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a forklift pallet positioning method and system based on reinforcement learning proposed in this invention. Figure 2 This is a schematic diagram illustrating the partitioning of the insertion relationship success domain and insertion relationship collapse domain in a forklift cargo pallet positioning method and system based on reinforcement learning proposed in this invention. Figure 3 This is a schematic diagram illustrating the action strategy learning based on improved IQL for a forklift pallet positioning method and system based on reinforcement learning proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figures 1-3 A forklift pallet positioning method based on reinforcement learning includes the following steps: During the pallet approach, alignment and insertion operations performed by the forklift, forklift operation status data and insertion contact event data are collected, and preprocessed to generate a state-event joint sequence. Insertion trajectory samples are constructed based on state-event joint sequences. Insertion success or failure indicators are generated for the insertion trajectory samples according to whether the insertion job is completed or interrupted. Generate a set of relation states based on the insertion trajectory samples with insertion success or insertion failure indicators; Based on the relation state set, the insertion relation success domain and the insertion relation collapse domain are divided; Based on the current relationship state and action constraints, perform the relationship state reachability determination process and generate the executability determination result; An improved IQL is constructed based on a set of relation states, taking whether the relation state enters the insertion relation collapse domain during the time unfolding process as the learning objective to generate action policies; In actual pallet insertion operations, based on the current relationship status and the set of executable actions obtained by filtering the executableness judgment results, the forklift control actions are output according to the action strategy to complete the forklift pallet positioning and insertion.

[0021] In this embodiment, the preprocessing specifically includes time synchronization, noise filtering, outlier removal, state variable normalization, contact event alignment, and state-event timing splicing.

[0022] In this embodiment, the generation of the inserted trajectory sample specifically includes: Based on the state-event joint sequence, the start and end events of the forklift pallet insertion operation are identified; Using the identified start event and corresponding termination event as boundaries, the state-event joint sequence is segmented to generate multiple continuous insertion trajectory samples. Each insertion trajectory sample contains forklift running status data and insertion contact event data from the start of insertion to the end of insertion. Based on each insertion trajectory sample, the corresponding insertion termination status information is extracted, including the fork arm insertion depth, the change of contact force during insertion, the forklift movement status, and the duration of the insertion operation. Based on the insertion termination status information, and according to the insertion completion conditions, an insertion completion determination is performed on each insertion trajectory sample. When the insertion trajectory sample meets the following conditions, the fork arm reaches the target insertion depth, the forklift movement state enters a stable state, no abnormal contact event is triggered during the insertion process, and the duration of the insertion operation does not exceed the allowable range, an insertion success identifier is generated. Based on the insertion termination status information, when the insertion trajectory sample triggers an exit event or an interruption event during the insertion process, or the fork arm does not reach the target insertion depth, or an abnormal contact event occurs during the insertion process, or the duration of the insertion operation exceeds the allowable range, an insertion failure flag is generated. The inserted trajectory samples are associated with the insertion success flag or the insertion failure flag and stored together to form a set of inserted trajectory samples with the insertion success flag or the insertion failure flag.

[0023] In this embodiment, the generation of the relation state set specifically includes: Based on the insertion trajectory samples with insertion success or insertion failure indicators, the local state-event joint sequence corresponding to each insertion trajectory sample is analyzed in chronological order; Each local state-event joint sequence is segmented according to the time window length and sliding step size to form multiple trajectory segments; For each trajectory segment, extract relational state elements, including statistical characteristics of forklift operation status, trend characteristics of operation status changes, and characteristics of the occurrence of insertion contact events; The generation of relational state elements specifically includes: Based on the forklift's speed, acceleration, rotation angle, and rate of change, the relative positional change rate between the forklift and the pallet, and the state data of the forklift's power system, including battery charge, power output, and the forklift operator's control input, including the intensity and frequency of throttle and brake operation, statistical characteristics of the forklift's operating state are generated. Based on the acceleration and deceleration trends of the forklift's motion, the error trend between the forklift and the pallet, and the forklift's motion patterns in different operating stages, including acceleration, constant speed, and deceleration, characteristics of the changing operating state are generated. Based on the contact time, contact pressure, and contact sequence between the forklift and the pallet, characteristics of the occurrence of insertion contact events are generated. Through comprehensive processing of these three types of characteristics, relational state elements are generated. Perform discretization mapping on relation state elements to convert continuous relation state elements into a finite number of relation state identifiers; Based on the insertion success or insertion failure flag corresponding to the insertion trajectory sample, the relationship status flags are classified. Relationship status flags from successfully inserted trajectory samples are classified into the successful relationship status subset, and relationship status flags from failed inserted trajectory samples are classified into the failed relationship status subset. The relationship state identifiers contained in the subsets of successful relationship states and the subsets of failed relationship states are deduplicated and integrated to generate a set of relationship states.

[0024] In this embodiment, the generation of the insertion relation success domain and the insertion relation collapse domain specifically includes: Based on the relation state set, obtain the occurrence record of each relation state identifier in the insertion trajectory sample set with insertion success identifier or insertion failure identifier, and count the number of times the relation state identifier appears in the insertion success trajectory sample and the number of times it appears in the insertion failure trajectory sample respectively. For each relation status identifier in the relation status set, count the number of times the relation status identifier appears in the successful insertion trajectory sample and the number of times it appears in the failed insertion trajectory sample. Based on the two types of occurrences, calculate the success rate corresponding to the relation status identifier. Set success domain determination thresholds and collapse domain determination thresholds for the relation state set, and perform domain attribute determination for each relation state identifier based on the comparison result between the success rate and the determination thresholds. When the success rate corresponding to the relation status identifier meets the success field determination threshold, the relation status identifier is classified into the insertion relation success field. When the success rate corresponding to the relation status identifier meets the collapse domain determination threshold, the relation status identifier is classified into the insertion relation collapse domain.

[0025] In this embodiment, the generation of the executability determination result specifically includes: Based on real-time collected forklift operating status data and inserted contact event data, update the state-event joint sequence and map the current relationship state according to the relationship state set; Construct a set of motion constraints, including forklift travel speed constraints, steering amplitude constraints, fork tooth lifting range constraints, fork tooth pitch range constraints, braking intensity constraints, and safety distance constraints during insertion. The forklift control motion space is parsed based on the set of motion constraints to generate multiple candidate motions, and motion constraint verification rules are configured for each candidate motion. Perform action constraint verification for each candidate action to ensure that the candidate action meets all constraints in the action constraint set. For each candidate action, perform an executability determination. If the candidate action satisfies all action constraints, generate an executability determination result of "executable"; otherwise, generate an executability determination result of "non-executable". Based on the executability determination results of all candidate actions, a set of executable actions is generated.

[0026] In this embodiment, the generation of the action strategy specifically includes: Based on the set of relation states, we analyze the proportion of each relation state in the successful and failed insertion trajectory samples, and determine whether each relation state is located in the successful insertion domain, the collapsed insertion domain, or the boundary region based on the threshold. The relation status identifiers are classified. If it is in the successful insertion domain, it is marked as successful; if it is in the collapsed insertion domain, it is marked as collapsed; if it is in the boundary region, it is marked as boundary. During the training process, "whether or not to enter the insertion relation collapse domain" is taken as the core learning objective for improving IQL; The specific improvements to IQL include: At the semantic layer, by introducing a set of relation states, the interaction process between the forklift and the pallet is divided into the insertion relation success domain, the insertion relation collapse domain, and the boundary region, thus achieving a structured classification of states. At the learning objective layer, "whether to enter the insertion relation collapse domain" is taken as the core learning objective, focusing on avoiding failure states. At the update mechanism layer, value update suppression is implemented on samples of collapse states to avoid the negative impact of unstable samples on policy learning. Combined with environmental feedback, it is ensured that the generated action policy can adapt to the dynamically changing environment and effectively avoid the insertion relation collapse domain. For relation states marked as successful, their value is updated based on the temporal evolution of the relation states to optimize action selection; Optimizing action selection specifically includes: Based on the relationships marked as successful states, we analyze their evolution in the time series, extracting changes in the physical characteristics of the forklift's position, speed, and acceleration. Combining the forklift's operating state characteristics and the occurrence characteristics of insertion contact events, we construct the temporal evolution relationship of successful insertion trajectory samples. Based on this relationship, we evaluate executable candidate actions in the current state. According to the temporal evolution relationship, we calculate the probability that each candidate action can achieve the target success rate in the current state, sort the actions according to the probability distribution, and select actions with higher success rates. Based on the optimized candidate action priorities and the forklift's motion constraints, we generate the final control actions, ensuring that the generated actions are executable under physical constraints and optimizing the task success rate. Through repeated optimization of the action selection process, we adjust the selection strategy. For relation states marked as collapsed, value update suppression is implemented; Value renewal inhibition specifically includes: Based on the relational state marked as a collapse state, identify the state-action pair corresponding to the relational state and generate a collapse state identifier record; for the state-action pair in the collapse state identifier record, block the value propagation path corresponding to the state-action pair in the improved IQL value update process and generate a value update masking identifier; based on the value update masking identifier, skip the value backpropagation calculation of the corresponding state-action pair during the training phase and generate a restricted value update result. During training, the action strategy is optimized by the value update rule of reinforcement learning to ensure that the generated strategy can effectively avoid the entry of the insertion relation collapse domain. The strategy is dynamically adjusted by environmental feedback to generate the final action strategy. The generation of the final action strategy specifically includes: Based on the set of relation states and their corresponding relation state identifiers, and combined with the value update results of each state-action pair, a relation state-action mapping is generated. During training, the priority of actions is dynamically adjusted by collecting and analyzing environmental feedback to ensure that the generated actions can effectively avoid entering the relation collapse domain. Based on the environmental feedback and the current state-action mapping results, the optimal action for each relation state is selected, and an action selection set for each relation state is generated based on the selection of the optimal action. The action selection sets of all relation states are summarized to obtain a set of action selection rules covering all relation states. Based on the set of action selection rules, the final action strategy is generated.

[0027] In this embodiment, the completion of forklift pallet positioning and insertion specifically includes: In each control cycle, collect the current operating status data of the forklift and insert contact event data, update the state-event joint sequence, and generate the current relationship state based on the updated state-event joint sequence. Based on the current relationship state, the relationship state reachability determination process is invoked to determine the candidate control actions and filter to generate a set of executable actions corresponding to the current relationship state. Invoke the action strategy, limit the action strategy to the set of executable actions, and obtain the result of the selection of executable actions in the current relationship state; Based on the results of the selectable actions, determine the actual actions to be executed within the current control cycle; The actual actions are converted into a set of forklift control commands, including driving control commands, steering control commands, mast lifting control commands, and fork pitch control commands. Output a set of forklift control commands to the forklift actuator to drive the forklift to perform corresponding approach, alignment or insertion actions, and complete the positioning and insertion of the forklift pallet.

[0028] A forklift pallet positioning system based on reinforcement learning includes: The status event acquisition module is used to collect forklift operating status data and insert contact event data to generate a joint status-event sequence. The insertion trajectory construction module is used to construct insertion trajectory samples based on the joint sequence of state and event, and generate insertion success or insertion failure indicators; The relation state generation module is used to generate a set of relation states based on the inserted trajectory samples; The insertion relation domain partitioning module is used to partition the insertion relation success domain and the insertion relation collapse domain; The relation reachability determination module is used to generate an executability determination result based on the current relation state and action constraints. The action policy learning module is used to build an improved IQL based on a set of relation states and generate action policies. The control decision execution module is used to output forklift control actions based on the action strategy to complete pallet positioning and insertion.

[0029] Example 1: To verify the feasibility of the present invention in practice, it was applied to an automated pallet handling operation using forklifts in a large warehousing and logistics park. This logistics park handles frequent, multi-batch inbound and outbound goods operations. Pallet specifications vary, and the ground conditions are uneven due to long-term use. In actual operation, factors such as pallet manufacturing tolerances, pallet placement angle deviations, and forklift tire deformation under load are unavoidable. In this scenario, the traditional forklift pallet insertion method, which relies on geometric alignment and manual configuration, can complete the operation under ideal conditions, but in actual operation, it often encounters problems such as insertion obstruction, mid-operation withdrawal, or failure to complete insertion after multiple attempts, which seriously affects operational efficiency and equipment stability.

[0030] In this application scenario, the forklift is configured with the ability to collect operational status and sense insertion contact events. Throughout the entire process of approaching, aligning, and inserting the pallet, the system continuously collects the forklift's travel status, steering status, mast lifting status, and fork pitch status. Simultaneously, it collects contact event information generated during the insertion process. The system first preprocesses the collected data, forming a continuous and consistent state-event joint sequence through time synchronization, noise filtering, and state normalization. Each time the forklift attempts to perform a pallet insertion operation, the system automatically identifies the start and end of the insertion operation, divides the complete insertion process into independent insertion trajectory samples, and assigns corresponding success or failure flags to the insertion trajectory samples based on whether the insertion operation is successfully completed or forcibly interrupted.

[0031] During continuous system operation, the system analyzes the interaction process between the forklift and the pallet based on a large number of insertion trajectory samples with success or failure indicators, and gradually constructs a set of relational states. The relational states do not depend on the precise geometric model of the pallet or forklift, but rather reflect the relational characteristics presented at different stages of the insertion process through a comprehensive characterization of the forklift's operating state change characteristics and the characteristics of insertion contact events. The system further divides the relational states into the insertion relationship success domain, the insertion relationship collapse domain, and the boundary region between the two based on the distribution of relational states in the success and failure trajectories, thereby establishing a risk structure for the insertion process at the state level.

[0032] In actual operation, when the forklift performs a new pallet insertion task, the system updates the state-event joint sequence in real time and maps the current relational state accordingly. Based on this, the system combines the forklift's own action constraints to determine the relational reachability of the optional control actions, retaining only actions that are physically feasible in the current relational state, forming a set of executable actions. At the same time, the system continuously trains the action strategy based on an improved reinforcement learning method, taking whether the relational state enters the insertion relation collapse domain during the time unfolding process as the core learning objective. For relational states that frequently cause insertion failures during training, the system suppresses their corresponding value update process to avoid such unstable states misleading the action strategy, thereby guiding the strategy to gradually converge toward a safer and more controllable state evolution path.

[0033] In the actual operation of the logistics park, the method provided by this invention has been continuously applied to pallet insertion tasks in multiple work areas and at different times. Through long-term observation of the operation process, it can be found that compared with the traditional method that relies on geometric alignment and manual rules, this invention can still maintain relatively stable insertion decision behavior even when there are slight pallet offsets, changes in ground conditions, and uncertain insertion contact order. When facing complex insertion situations, the system no longer relies on the result of a single alignment, but avoids insertion risks in advance through the dynamic evolution of relational states, thereby reducing the situation of mid-way withdrawal and repeated attempts.

[0034] Feedback from practical applications shows that this invention effectively reduces the frequency of insertion failures during operation, reduces ineffective forklift movements and unnecessary posture adjustments during insertion, and lessens the operator's reliance on repeated adjustments to equipment parameters. The forklift maintains good adaptability to different pallet sizes and operating environments, the insertion process is more seamless, and the equipment operates more stably. Comprehensive analysis of operation records and logs confirms that this invention demonstrates significant technical effects in improving pallet insertion stability, enhancing the system's adaptability to environmental uncertainties, and reducing overall operational risks, thus verifying the feasibility and practical value of this invention in practical applications.

[0035] Table 1. Comparison of experimental performance between the method of this invention and the traditional method.

[0036] As shown in Table 1, under the same operating environment and pallet specifications, the insertion success rate of the traditional method is 85.2%, while the insertion success rate of the method of this invention reaches 92.8%, an improvement of 7.6 percentage points. In terms of the insertion failure trigger rate, the failure rate of the traditional method due to alignment deviation or abnormal contact events during the insertion process is 11.6%, while that of the method of this invention is reduced to 4.3%. This result is directly related to the explicit modeling of the insertion relationship collapse domain in the relation state set, which enables the strategy to avoid high-risk relation states in advance during the evolution process, thereby reducing the occurrence of interruption or exit events.

[0037] In terms of operational efficiency, the traditional method takes an average of 39.6 seconds to complete one tray insertion, while the method of this invention reduces this time to 30.8 seconds, a reduction of 8.8 seconds. At the same time, the standard deviation of the time fluctuation during the insertion process is reduced from 5.1 seconds to 2.9 seconds, indicating that the method of this invention has a more stable action execution rhythm in continuous insertion operations. This result is mainly due to the fact that the action strategy takes "whether to enter the insertion relationship collapse domain" as the core learning objective during the training phase, which makes the strategy tend to choose more conservative but more continuous control actions when approaching the insertion critical stage, reducing repeated corrections and backtracking operations.

[0038] From the perspective of real-time control performance, the average response delay of control commands using traditional methods is 480 milliseconds, while the method of this invention reduces it to 310 milliseconds, a reduction of 170 milliseconds. This improvement is related to the relational reachability determination mechanism, which performs executability screening on the action space in each control cycle, ensuring that the strategy makes decisions only within the set of executable actions. This reduces the evaluation of invalid actions and the generation of conflicting control commands, thereby improving the overall control response efficiency.

[0039] Regarding energy consumption, the average energy consumption of a single insertion operation using the traditional method is 52.4 kJ, while the method of this invention reduces it to 45.1 kJ, a reduction of 7.3 kJ. This change does not stem from a reduction in the energy consumption of a single action, but rather from a reduction in the number of invalid adjustment actions during the insertion process and a significant decrease in the number of repetitive operations after insertion failure. This effectively controls the cumulative energy consumption per unit insertion task. Combined with the simultaneous decrease in the insertion failure trigger rate, it can be seen that the method of this invention achieves synergistic optimization between operational efficiency and energy consumption while improving the success rate.

[0040] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A forklift pallet positioning method based on reinforcement learning, characterized in that, Includes the following steps: During the pallet approach, alignment and insertion operations performed by the forklift, forklift operation status data and insertion contact event data are collected, and preprocessed to generate a state-event joint sequence. Insertion trajectory samples are constructed based on state-event joint sequences. Insertion success or failure indicators are generated for the insertion trajectory samples according to whether the insertion job is completed or interrupted. Generate a set of relation states based on the insertion trajectory samples with insertion success or insertion failure indicators; Based on the relation state set, the insertion relation success domain and the insertion relation collapse domain are divided; Based on the current relationship state and action constraints, perform the relationship state reachability determination process and generate the executability determination result; An improved IQL is constructed based on a set of relation states, taking whether the relation state enters the insertion relation collapse domain during the time unfolding process as the learning objective to generate action policies; In actual pallet insertion operations, based on the current relationship status and the set of executable actions obtained by filtering the executableness judgment results, the forklift control actions are output according to the action strategy to complete the forklift pallet positioning and insertion.

2. The forklift pallet positioning method based on reinforcement learning according to claim 1, characterized in that, The preprocessing specifically includes time synchronization, noise filtering, outlier removal, state normalization, contact event alignment, and state-event timing splicing.

3. The forklift pallet positioning method based on reinforcement learning according to claim 1, characterized in that, The generation of the inserted trajectory sample specifically includes: Based on the state-event joint sequence, the start and end events of the forklift pallet insertion operation are identified; Using the identified start event and corresponding termination event as boundaries, the state-event joint sequence is segmented to generate multiple continuous insertion trajectory samples. Each insertion trajectory sample contains forklift running status data and insertion contact event data from the start of insertion to the end of insertion. Based on each insertion trajectory sample, the corresponding insertion termination status information is extracted, including the fork arm insertion depth, the change of contact force during insertion, the forklift movement status, and the duration of the insertion operation. Based on the insertion termination status information, and according to the insertion completion conditions, an insertion completion determination is performed on each insertion trajectory sample. When the insertion trajectory sample meets the following conditions, the fork arm reaches the target insertion depth, the forklift movement state enters a stable state, no abnormal contact event is triggered during the insertion process, and the duration of the insertion operation does not exceed the allowable range, an insertion success identifier is generated. Based on the insertion termination status information, when the insertion trajectory sample triggers an exit event or an interruption event during the insertion process, or the fork arm does not reach the target insertion depth, or an abnormal contact event occurs during the insertion process, or the duration of the insertion operation exceeds the allowable range, an insertion failure flag is generated. The inserted trajectory samples are associated with the insertion success flag or the insertion failure flag and stored together to form a set of inserted trajectory samples with the insertion success flag or the insertion failure flag.

4. The forklift pallet positioning method based on reinforcement learning according to claim 1, characterized in that, The generation of the relation state set specifically includes: Based on the insertion trajectory samples with insertion success or insertion failure indicators, the local state-event joint sequence corresponding to each insertion trajectory sample is analyzed in chronological order; Each local state-event joint sequence is segmented according to the time window length and sliding step size to form multiple trajectory segments; For each trajectory segment, extract relational state elements, including statistical characteristics of forklift operation status, trend characteristics of operation status changes, and characteristics of the occurrence of insertion contact events; Perform discretization mapping on relation state elements to convert continuous relation state elements into a finite number of relation state identifiers; Based on the insertion success or insertion failure flag corresponding to the insertion trajectory sample, the relationship status flags are classified. Relationship status flags from successfully inserted trajectory samples are classified into the successful relationship status subset, and relationship status flags from failed inserted trajectory samples are classified into the failed relationship status subset. The relationship state identifiers contained in the subsets of successful relationship states and the subsets of failed relationship states are deduplicated and integrated to generate a set of relationship states.

5. The forklift pallet positioning method based on reinforcement learning according to claim 1, characterized in that, The generation of the insertion relation success field and the insertion relation collapse field specifically includes: Based on the relation state set, obtain the occurrence record of each relation state identifier in the insertion trajectory sample set with insertion success identifier or insertion failure identifier, and count the number of times the relation state identifier appears in the insertion success trajectory sample and the number of times it appears in the insertion failure trajectory sample respectively. For each relation status identifier in the relation status set, count the number of times the relation status identifier appears in the successful insertion trajectory sample and the number of times it appears in the failed insertion trajectory sample. Based on the two types of occurrences, calculate the success rate corresponding to the relation status identifier. Set success domain determination thresholds and collapse domain determination thresholds for the relation state set, and perform domain attribute determination for each relation state identifier based on the comparison result between the success rate and the determination thresholds. When the success rate corresponding to the relation status identifier meets the success field determination threshold, the relation status identifier is classified into the insertion relation success field. When the success rate corresponding to the relation status identifier meets the collapse domain determination threshold, the relation status identifier is classified into the insertion relation collapse domain.

6. The forklift pallet positioning method based on reinforcement learning according to claim 1, characterized in that, The generation of the executability determination result specifically includes: Based on real-time collected forklift operating status data and inserted contact event data, update the state-event joint sequence and map the current relationship state according to the relationship state set; Construct a set of motion constraints, including forklift travel speed constraints, steering amplitude constraints, fork tooth lifting range constraints, fork tooth pitch range constraints, braking intensity constraints, and safety distance constraints during insertion. The forklift control motion space is parsed based on the set of motion constraints to generate multiple candidate motions, and motion constraint verification rules are configured for each candidate motion. Perform action constraint verification for each candidate action to ensure that the candidate action meets all constraints in the action constraint set. For each candidate action, perform an executability determination. If the candidate action satisfies all action constraints, generate an executability determination result of "executable"; otherwise, generate an executability determination result of "non-executable". Based on the executability determination results of all candidate actions, a set of executable actions is generated.

7. The forklift pallet positioning method based on reinforcement learning according to claim 1, characterized in that, The generation of the action strategy specifically includes: Based on the set of relation states, we analyze the proportion of each relation state in the successful and failed insertion trajectory samples, and determine whether each relation state is located in the successful insertion domain, the collapsed insertion domain, or the boundary region based on the threshold. The relation status identifiers are classified. If it is in the successful insertion domain, it is marked as successful; if it is in the collapsed insertion domain, it is marked as collapsed; if it is in the boundary region, it is marked as boundary. During the training process, "whether or not to enter the insertion relation collapse domain" is taken as the core learning objective for improving IQL; For relation states marked as successful, their value is updated based on the temporal evolution of the relation states to optimize action selection; For relation states marked as collapsed, value update suppression is implemented; During training, the action strategy is optimized by using the value update rule of reinforcement learning to ensure that the generated strategy can effectively avoid the entry of the insertion relation collapse domain. The strategy is dynamically adjusted by using environmental feedback to generate the final action strategy.

8. A forklift pallet positioning method based on reinforcement learning according to claim 1, characterized in that, The completion of the forklift pallet positioning and insertion specifically includes: In each control cycle, collect the current operating status data of the forklift and insert contact event data, update the state-event joint sequence, and generate the current relationship state based on the updated state-event joint sequence. Based on the current relationship state, the relationship state reachability determination process is invoked to determine the candidate control actions and filter to generate a set of executable actions corresponding to the current relationship state. Invoke the action strategy, limit the action strategy to the set of executable actions, and obtain the result of the selection of executable actions in the current relationship state; Based on the results of the selectable actions, determine the actual actions to be executed within the current control cycle; The actual actions are converted into a set of forklift control commands, including driving control commands, steering control commands, mast lifting control commands, and fork pitch control commands. Output a set of forklift control commands to the forklift actuator to drive the forklift to perform corresponding approach, alignment or insertion actions, and complete the positioning and insertion of the forklift pallet.

9. A forklift pallet positioning system based on reinforcement learning, comprising the forklift pallet positioning method based on reinforcement learning as described in any one of claims 1 to 8, characterized in that, include: The status event acquisition module is used to collect forklift operating status data and insert contact event data to generate a joint status-event sequence. The insertion trajectory construction module is used to construct insertion trajectory samples based on the joint sequence of state and event, and generate insertion success or insertion failure indicators; The relation state generation module is used to generate a set of relation states based on the inserted trajectory samples; The insertion relation domain partitioning module is used to partition the insertion relation success domain and the insertion relation collapse domain; The relation reachability determination module is used to generate an executability determination result based on the current relation state and action constraints. The action policy learning module is used to build an improved IQL based on a set of relation states and generate action policies. The control decision execution module is used to output forklift control actions based on the action strategy to complete pallet positioning and insertion.

Images