Intelligent anti-sway control method for crane based on reinforcement learning

Abstract

The present invention relates to the field of crane control technology and discloses a method for intelligent crane anti-sway control based on reinforcement learning. The method comprises: collecting crane motion state data (including position and speed) and load sway data; judging whether control actions effectively reduce sway based on this data combined with a reinforcement learning model; and executing anti-sway operations if effective. The method defines a state space (including crane position, load sway angle, and speed), an action space (including control instruction direction and amplitude), and a reward function based on sway amplitude changes and energy consumption to train the model. Data is collected in real time via sensors and processed through filtering and smoothing. During model training, the model is updated using convergence criteria, and control actions and data points are accurately associated and secondary verified through feature matching, group statistics, and other methods. Direction and amplitude control verification is performed during operation. This method improves the accuracy, robustness, and energy efficiency of crane anti-sway control and is suitable for complex industrial scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of crane control, and in particular to an intelligent anti-sway control method for a crane based on reinforcement learning. Background Art

[0002] Cranes are critical equipment in industrial production, logistics, and transportation. Load sway during operation has always been a significant challenge, impacting operational efficiency, safety, and accuracy. Traditional crane anti-sway control methods rely primarily on classical control theories, such as PID control. These methods typically require the establishment of precise mathematical models. However, crane systems have complex characteristics such as nonlinearity, strong coupling, and time-varying behavior. Accurate models are difficult to obtain in practical applications, resulting in limited anti-sway control effectiveness. Control accuracy and robustness are particularly limited when faced with complex operating conditions or load variations.

[0003] With the advancement of industrial automation and intelligentization, the control requirements for cranes are increasing, and the limitations of traditional control methods are becoming increasingly apparent. For example, in scenarios such as port loading and unloading and large-scale equipment installation, cranes must be able to quickly and accurately transport and position loads while effectively suppressing sway to avoid collisions and improve operational efficiency. Traditional methods often struggle to achieve ideal anti-sway effects in complex environments with multiple variables and strong interference. Manual adjustments require extensive operator experience, increasing labor costs and operational risks.

[0004] In recent years, the rise of intelligent control technology has provided new ideas for crane anti-sway control. Reinforcement learning, a trial-and-error-based machine learning method, can learn optimal control strategies through interaction with the environment without a precise mathematical model, making it suitable for solving control problems of complex systems. However, the application of reinforcement learning to the field of intelligent crane anti-sway control still faces many challenges. For example, how to reasonably define the state space, action space, and reward function to accurately describe the operating state of the crane system and the effect of control actions; how to efficiently collect and process the crane's motion state data and load sway data to ensure data accuracy and real-time performance; how to design an effective model training and update mechanism to improve the convergence speed and control performance of the reinforcement learning model; and how to achieve accurate matching and verification of control actions and state space to ensure the reliability and effectiveness of anti-sway control operations.

[0005] While some existing research has explored the application of intelligent control in crane anti-sway control, most suffer from issues such as incomplete state space definition, poor reward function design, low model training efficiency, and imperfect control action verification mechanisms. These issues result in inconsistent anti-sway performance in practical applications, making it difficult to meet the demands of industrial sites for intelligent, high-precision crane control. Therefore, there is an urgent need for an intelligent crane anti-sway control method based on reinforcement learning. This method leverages the advantages of reinforcement learning and incorporates the characteristics of crane systems to address the shortcomings of traditional control methods and improve crane anti-sway control performance and operational efficiency under complex operating conditions. Summary of the Invention

[0006] The purpose of the present invention is to provide an intelligent anti-sway control method for a crane based on reinforcement learning to solve the problems raised in the above background technology.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an intelligent anti-sway control method for a crane based on reinforcement learning, the method comprising:

[0008] Collecting crane motion state data and load sway data, wherein the motion state data includes position and speed information;

[0009] Based on the motion state data and the sway data, combined with a preset reinforcement learning model, determining whether each control action effectively reduces the load sway;

[0010] If it is determined to be valid, performing an anti-sway control operation based on the control action;

[0011] Among them, judging whether each control action is effective includes defining the state space, action space and reward function, and training the reinforcement learning model;

[0012] The state space includes crane position, load sway angle, and velocity;

[0013] The action space includes the direction and magnitude of the control command;

[0014] The reward function is calculated based on the swing amplitude change and energy consumption.

[0015] Preferably, the motion state data is real-time data collected by sensors, including data collected by an inertial measurement unit set on the crane boom and an angle sensor on the load; the position information is extracted based on global positioning system data, and the extraction method is filtering and smoothing.

[0016] Preferably, the method for defining the state space is as follows: calculating the dimension of the state space in data acquisition; setting a feature subset for the state space in data acquisition, and uniformly selecting k feature points in the feature subset as associated feature points of the state space, where k is a positive integer; wherein the feature subset of the state space is composed of all feature points whose distance from the boundary of the state space is not less than r and not greater than s, where r is a preset first boundary threshold and s is a preset second boundary threshold.

[0017] Preferably, the training of the reinforcement learning model includes taking the associated feature points as input to update the model, the steps being as follows:

[0018] S401: Check whether each input feature point meets the convergence criterion. If so, mark the feature point as a training sample and add it to the current model;

[0019] The convergence criterion is specifically as follows: if the Euclidean distance between the feature point and the model prediction in the error space is less than a preset error threshold, the feature point meets the convergence criterion; otherwise, the feature point does not meet the convergence criterion;

[0020] S402: Marking the input that has completed the convergence criterion check of all feature points as processed;

[0021] S403: Repeat S401 to S402, and the input marked as processed will no longer be included in the check until all inputs are marked as processed, and the model update is stopped; the parameter range of the current model is recorded as the updated reinforcement learning model.

[0022] Preferably, the associated feature points are data points corresponding to the state space; the method for determining the associated data points of each feature point is as follows:

[0023] Extract m data points updated based on j associated feature points in the state space; deduplicate the m data points and delete completely overlapping data points; if only one data point remains after deduplication, the remaining data point is the associated data point of the feature point; if n data points remain after deduplication, where n is a positive integer greater than 1, count the number of associated feature points contained in each data point, and the data point containing the most associated feature points is the associated data point of the feature point.

[0024] Preferably, the load swing data is extracted based on real-time data collected by the angle sensor; the method for marking the position of each control action in the action space is as follows: detect and identify each control action in the action space, and extract the identification information of the control action; find the state point corresponding to each control action in the action space in the state space based on the identification information; the corresponding control actions in the action space and the state space have the same identification information; obtain the number of each control action and mark the position of each control action in the action space.

[0025] Preferably, after determining the associated data point of each control action in the action space, a secondary verification is performed on each control action and the corresponding associated data point, including the following steps:

[0026] Segment the associated data points of the control action from the state space and mark them as the first associated point set; segment the associated data points of the control action from the action space and mark them as the second associated point set;

[0027] Based on feature matching, determine whether the first set of associated points and the second set of associated points belong to the same group of control sequences; if so, pass secondary verification; if not, re-determine the associated data points of the control action; belonging to the same group of control sequences indicates that the first set of associated points and the second set of associated points are representations of the same control action in different spaces.

[0028] Preferably, the method for re-determining the associated data points of the control action is as follows:

[0029] The t data points after segmentation and removal from the state space are all marked as state data points; the u data points after segmentation and removal from the action space are all marked as action data points; t and u are both positive integers;

[0030] Based on feature matching, the state data points and action data points are grouped; any group contains a state data point and corresponding action data point belonging to the same group of control sequences; state data points and action data points that are not grouped are deleted;

[0031] If there is only one group, the state data points and action data points in the group are all associated data points of the control action; if there is more than one group, the total number of associated feature points contained in each group is counted, and the state data points and action data points in the group containing the most associated feature points are all associated data points of the control action; the state data points in the associated data points are marked as the first associated point set, and the action data points in the associated data points are marked as the second associated point set.

[0032] Preferably, the control action includes direction adjustment and amplitude adjustment; the execution of the anti-sway control operation includes direction control verification and amplitude control verification;

[0033] The method for direction control verification is as follows: comparing and matching the identification information of the control action with the action template in the preset action database to identify the control direction; if the identified control direction is consistent with the control direction obtained by defining the action space, the direction control verification is passed;

[0034] The amplitude control verification method is as follows: extracting the amplitude information of all control actions and calculating the total amplitude of all actions, which is recorded as the instruction total amplitude; calculating the total amplitude actually executed through sensor data, which is recorded as the actual total amplitude; and calculating the amplitude difference between the actual total amplitude and the instruction total amplitude.

[0035] Preferably, the method for generating identification information of the control action is as follows: assigning a unique action code to each control action; binding the action code to the position coordinates in the state space; generating an identification string containing the direction type and amplitude level based on the action code; the identification string is stored in the index field of the action database for matching the state point with the action position.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] At the data acquisition and processing level, the inertial measurement unit installed on the crane boom, the angle sensor on the load, and the global positioning system enable the precise collection of multi-source real-time data such as the crane's position, speed, and load swing angle. This position information is processed through methods such as filtering and smoothing, ensuring the accuracy and reliability of the input data and providing a solid data foundation for subsequent control decisions. This multi-sensor fusion data acquisition method can comprehensively and real-timely reflect the operating status of the crane system, overcoming the limitations of single-sensor data and improving the system's ability to perceive environmental changes.

[0038] In terms of reinforcement learning model construction, by rationally defining the state space, action space, and reward function, the model accurately describes the dynamic characteristics of the crane system and the effectiveness of control actions. The state space covers key parameters such as crane position, load sway angle, and speed, comprehensively reflecting the system's state; the action space clarifies the direction and amplitude of control instructions, making the description of control actions more refined; the reward function is based on sway amplitude changes and energy consumption calculations, considering both the core goal of anti-sway control—reducing load sway—and optimizing the system's energy consumption, achieving a balance between control performance and economic efficiency. In addition, by calculating the state space dimension, setting feature subsets, and selecting associated feature points, the state space structure is optimized, improving the model's training efficiency and generalization ability.

[0039] During model training, a feature point processing mechanism based on convergence criteria is employed. By determining whether the Euclidean distance between a feature point and the model prediction in the error space is less than a preset threshold, only feature points that meet the required accuracy are included in the training sample, thereby improving the quality of model training. By repeatedly checking and marking processed inputs, convergence verification of all feature points is gradually completed, ensuring the comprehensiveness and accuracy of model updates, enabling the model to continuously approach the optimal control strategy, and improving control accuracy and robustness.

[0040] In terms of associating and verifying control actions with data points, the system extracts associated feature points, removes duplicates, and utilizes feature-matching-based grouping and statistical methods to ensure precise association between control actions and data points in the state space, avoiding data confusion and mismatches. A secondary verification mechanism further enhances the consistency and reliability of control actions and states by comparing sets of associated points in the state and action spaces to ensure they belong to the same control sequence, ensuring the effectiveness of anti-sway control operations. This multi-level data verification and association mechanism effectively reduces the risk of system misoperation and improves the accuracy of control decisions.

[0041] During the execution and verification of control actions, the correct execution of control instructions is ensured through direction control verification and amplitude control verification mechanisms. Direction control verification ensures the accuracy of the control direction by comparing the identification information of the control action with the preset action template. Amplitude control verification achieves a quantitative assessment of the execution effect of the control action by calculating the difference between the actual total amplitude and the commanded total amplitude. This facilitates the timely detection and adjustment of control deviations, improving the stability and controllability of the control process. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a working principle diagram of the intelligent anti-sway control method for cranes based on reinforcement learning according to the present invention;

[0043] Figure 2 Flowchart for motion status data collection;

[0044] Figure 3 Flowchart for defining the state-space method;

[0045] Figure 4 Flowchart for marking the locations of control actions in action space. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] See also Figures 1-4 The present invention relates to a crane intelligent anti-sway control method based on reinforcement learning, and the specific implementation steps are as follows:

[0048] Sensors are used to collect data on the crane's motion state and load sway. Motion state data, including position and velocity information, is collected in real time by an inertial measurement unit installed on the crane's boom. Position information is extracted from GPS data and processed using filtering and smoothing methods. Load sway data is collected in real time by an angle sensor installed on the load.

[0049] Based on the collected motion and sway data, combined with a pre-set reinforcement learning model, the effectiveness of each control action in reducing load sway is determined. This is achieved by defining the state space, action space, and reward function, and then training the reinforcement learning model.

[0050] State space: includes crane position, load sway angle, and velocity.

[0051] Action space: includes the direction and amplitude of control instructions.

[0052] Reward function: This function is used to evaluate the effectiveness of control actions based on the swing amplitude change and energy consumption calculation.

[0053] If the control action is determined to be valid, anti-sway control operations are performed based on the control action, including direction control verification and amplitude control verification. Direction control verification compares and matches the control action's identification information with the action templates in the preset action database to identify the control direction and verify consistency. Amplitude control verification extracts the control action's amplitude information to calculate the command total amplitude, combines it with sensor data to calculate the actual total amplitude, and verifies the difference between the two amplitudes.

[0054] The present invention will be further described below in conjunction with Examples 1 to 5:

[0055] Example 1: In the data collection process, the collection of motion state data and load swing data needs to rely on specific sensors to ensure the real-time and accuracy of the data. Among them, the motion state data includes position and speed information, which is specifically collected in real time by an inertial measurement unit installed on the crane boom. The inertial measurement unit can sensitively capture parameters such as acceleration and angular velocity of the crane boom during movement, and then obtain speed information through data processing. The position information is extracted based on the global positioning system (GPS) data. Since the GPS signal may be affected by factors such as multipath effects and atmospheric interference in the actual environment, resulting in noise in the positioning data, it is necessary to use a filtering and smoothing method to process the original GPS data to remove noise interference and improve the accuracy and stability of the position information. The load swing data is collected in real time by an angle sensor installed on the load. The angle sensor can directly measure the swing angle of the load relative to the crane, providing key data support for subsequent anti-sway control.

[0056] The definition of the state space requires a series of rigorous steps. First, the dimension of the state space in data acquisition is calculated. The dimension of the state space is determined by the physical quantities it contains. In this solution, the state space includes the crane position, the load swing angle, and the speed. Therefore, during data acquisition, the corresponding data must be obtained for each of these three physical quantities to determine that the state space is a three-dimensional space. Next, a feature subset is set for the state space during data acquisition. The selection of the feature subset must meet specific conditions, that is, it is composed of all feature points whose distance from the boundary of the state space is not less than the preset first boundary threshold r and not greater than the preset second boundary threshold s. The boundary thresholds r and s here are pre-set based on factors such as the actual working range of the crane, the physical characteristics of the load, and the control accuracy requirements. By setting the feature subset, the data acquisition range can be limited to a reasonable interval to avoid adverse effects on subsequent model training and control effects due to the instability of the data near the boundary.

[0057] Within the feature subset, k feature points are uniformly selected as associated feature points in the state space, where k is a positive integer. This uniform selection of feature points ensures that all regions of the state space are adequately covered, avoiding data sampling bias. For example, in a three-dimensional state space, sampling can be performed at regular intervals within the range of crane position, load sway angle, and speed, ensuring that the k selected feature points are evenly distributed within the feature subset. These associated feature points will serve as input data for subsequent reinforcement learning model training, and the rationality of their selection directly impacts the model's ability to characterize the crane's motion and load sway.

[0058] In practical applications, cranes operate in complex and ever-changing environments. Factors such as the load's weight, shape, and lifting height can affect the load's sway characteristics. Therefore, when setting the feature subset's boundary thresholds, r and s, it's necessary to comprehensively consider factors such as the crane's maximum operating range and the load's maximum potential sway angle and speed. For example, if the crane's maximum operating radius is R, then when setting the boundary thresholds for the crane's position, r can be set to 0.1R and s to 0.9R. This limits the feature subset to between 10% and 90% of the crane's operating radius, avoiding potential anomalous data near the crane's base and the maximum operating radius boundary while still covering the majority of the normal operating range. The boundary thresholds for the load's sway angle and speed can be determined based on statistical analysis of the load's physical characteristics and historical operating data, ensuring that the data within the feature subset reflects the load's sway under normal operating conditions.

[0059] When uniformly selecting k feature points, the constraints of computing resources and model training efficiency must also be considered. Excessively large values ​​of k may lead to a surge in data volume, increasing the computational cost and time of model training. However, too small a value of k may fail to fully characterize the state space, affecting model accuracy. Therefore, it is important to choose a reasonable value of k based on the actual situation. For example, in a three-dimensional state space, each dimension can be divided into m equally spaced intervals. Then, one feature point can be selected from each interval, resulting in k = m × m × m feature points. By adjusting the value of m, a balance can be achieved between data volume and model performance.

[0060] Furthermore, during data collection, attention must be paid to the sensor's installation location and calibration accuracy. The inertial measurement unit should be installed near the crane boom's center of gravity to ensure accurate measurement of the boom's motion. The angle sensor should be securely mounted on the load, and its orientation should align with the load's sway to ensure accurate measurement results. Furthermore, sensors should be regularly calibrated and maintained to ensure their measurement accuracy meets requirements. This prevents inaccurate state-space data due to sensor errors, which could affect reinforcement learning model training and anti-sway control effectiveness.

[0061] The above data collection and state space definition methods provide high-quality input data for intelligent crane anti-sway control based on reinforcement learning. This enables the reinforcement learning model to accurately perceive the crane's motion state and load sway, laying a solid foundation for judging the effectiveness of control actions and executing anti-sway control operations. During subsequent model training and control, based on the data from these associated feature points, the model continuously learns and optimizes control strategies, effectively suppressing crane load sway and improving the safety and efficiency of crane operations.

[0062] Example 2: When training a reinforcement learning model, it is necessary to use associated feature points as input to update the model. The specific implementation method is as follows:

[0063] The core process of model updating revolves around the convergence criterion. First, the definition and judgment logic of the convergence criterion must be clarified. The convergence criterion measures the consistency between the feature point and the model prediction through the Euclidean distance in the error space. Specifically, if the Euclidean distance between the feature point and the model prediction in the error space is less than the preset error threshold, then the feature point meets the convergence criterion, otherwise it does not meet the convergence criterion. Among them, the error space refers to the space composed of the difference between the actual value of the feature point and the model prediction value as the coordinate component, and the Euclidean distance is used to quantify the overall deviation between the two. Let the actual value vector of the feature point be , the model prediction value vector is , then the Euclidean distance calculation formula is:

[0064]

[0065] Where, represents the Euclidean distance, is the dimension of the feature point (i.e., the number of parameters of crane position, load swing angle, and speed in the state space, where ), For the The actual value of the characteristic parameter, For the The model prediction value of each feature parameter. The preset error threshold is a constant pre-set according to the accuracy requirements of the crane anti-sway control. It is used to determine whether the feature point is close enough to the model prediction, thereby deciding whether to include it in the training sample.

[0066] The specific steps for model updating are as follows:

[0067] Step S401: Perform convergence criterion check on each input feature point. First, the actual value vector of the feature point is Input the current reinforcement learning model, and the model outputs the predicted value vector through its internal neural network structure or algorithm logic Then, according to the above Euclidean distance formula, , and and the preset error threshold Compare. , indicating that the deviation between the feature point and the model prediction is within the allowable range and meets the convergence criterion, then the feature point is marked as a training sample and added to the current model to update the model parameters (such as the weight and bias of the neural network); if , then the feature point does not meet the convergence criterion and is not included in the training sample for the time being, and needs to wait for subsequent processing.

[0068] In actual operation, the order of inputting feature points may affect the efficiency and stability of model updating. Therefore, batch processing can be used to group a certain number of feature points into a batch, and then perform convergence criterion checks and processing on each feature point in the batch. For example, each time you input Feature points ( The batch size (where _ is the batch size, which can be set based on computing resources and model training requirements) is calculated for each feature point and the Euclidean distance is determined to determine whether it meets the convergence criteria. For feature points that meet the criteria, they are immediately used as training samples to update the model parameters. For feature points that do not meet the criteria, their status is recorded and the next step is performed after all feature points in the batch have been processed.

[0069] Step S402: After all feature points in a batch have completed the convergence criterion check, the batch of inputs is marked as "processed." The purpose of marking "processed" is to avoid repeated processing of feature points in the same batch, ensuring the orderliness and efficiency of the model update process. Processed inputs are no longer subject to convergence criterion checks in subsequent model update iterations, thereby reducing the waste of computing resources.

[0070] Step S403: Repeat steps S401 to S402 until all inputs are marked as "processed." During the iterative process, each time a new batch of unlabeled feature points is processed, the model parameters are updated using the training samples from the previous batch, so the model's predictive ability for subsequent feature points gradually improves. As the number of iterations increases, more and more feature points will meet the convergence criteria and be included in the training samples until all input feature points have been processed. At this point, the model update is stopped and the parameter range of the current model is recorded as the updated reinforcement learning model.

[0071] During model training, pay attention to the following points:

[0072] Error threshold setting: Error threshold The size of directly affects the convergence speed and accuracy of the model. If the value is set too small, a large number of feature points will not meet the convergence criteria, and the model will require more iterations to complete training, and may even fall into overfitting. If the value is set too high, the model will not be able to accurately fit the feature points, which will lead to inaccurate judgment of the effectiveness of the control action and affect the anti-sway control effect. Therefore, it is necessary to reasonably set the value based on the actual control requirements and historical data of the crane through experiments or simulations. For example, for the crane position parameter, you can set is 0.1 m to ensure that the model can accurately capture the position changes of the crane at meter-level accuracy; for the load swing angle parameter, you can set 1 degree to meet the angle accuracy requirements of anti-sway control.

[0073] Feature point preprocessing: Before inputting feature points, they need to be normalized to convert feature parameters of different dimensions into a unified numerical range to avoid model training deviations caused by dimensional differences in feature parameters. For example, the unit of crane position is meters, the unit of load swing angle is degrees, and the unit of speed is meters per second. The feature parameters of each dimension can be converted into values ​​in the range [0, 1] through normalization. The specific formula is:

[0074]

[0075] Where, is the standardized eigenvalue, and Respectively The historical minimum and maximum values ​​of each feature parameter. Standardization allows the model to more efficiently learn the relationship between different feature parameters, improving training efficiency and prediction accuracy.

[0076] Model parameter update strategy: When adding feature points that meet the convergence criteria to the current model, an appropriate parameter update strategy should be adopted, such as stochastic gradient descent (SGD) and adaptive moment estimation (Adam). These strategies can automatically adjust the update step size of the model parameters based on the gradient information of the training samples, ensuring that the model parameters are optimized in the direction of reducing prediction error. Taking stochastic gradient descent as an example, each time the model parameters are updated, the parameter update amount is calculated based on the gradient of the current batch of training samples. The formula is:

[0077]

[0078] Where, For the The model parameter vector at the iteration, is the learning rate (a preset constant used to control the update step size), is the loss function exist Through continuous iterative updates, the model parameters gradually converge to the optimal solution, enabling the model to accurately judge the effectiveness of the control action based on the input feature points.

[0079] Prevention of overfitting: To prevent the model from overfitting during training (i.e., the model overfits the training data, resulting in a decrease in generalization ability on new data), regularization methods such as L1 regularization and L2 regularization can be used to add regularization terms to the loss function to penalize the complexity of the model parameters. For example, the loss function expression of L2 regularization is:

[0080]

[0081] Where, is the number of training samples, For the The loss function of samples (such as mean square error), is the regularization parameter (a preset constant used to control the regularization strength), is the number of model parameters. By introducing the regularization term, the absolute value of the model parameters can be forced to be as small as possible, thereby reducing the complexity of the model and improving its generalization ability.

[0082] Through the above implementation methods, the reinforcement learning model can be iteratively trained based on the associated feature point data, and gradually optimize the model parameters, so that it can accurately determine whether each control action effectively reduces the load swing.

[0083] Example 3: The implementation method of determining the associated data points of each feature point and marking the control action position in the action space is as follows:

[0084] When determining the associated data points for a feature point, the first step is to extract the data points associated with the associated feature points from the state space. Specifically, m data points are extracted based on the update of j associated feature points in the state space, where j is the pre-set number of associated feature points involved in the data point update and m is the total number of data points generated after the update. Because repeated sampling or noise interference may occur during the data acquisition and processing process, resulting in complete overlap among the m data points, these data points need to be deduplicated and completely overlapping data points removed to ensure the uniqueness and validity of the data points subsequently processed.

[0085] After deduplication is complete, the method for determining associated data points must be determined based on the number of remaining data points. If only one data point remains after deduplication, it indicates that this data point is unique and valid and can be directly determined as the associated data point of the feature point. For example, if the update process of j associated feature points is relatively stable and the data collection accuracy is high, only one unique data point may be generated. In this case, this data point can accurately reflect the associated information of the feature point.

[0086] If n data points remain after deduplication (n is a positive integer greater than 1), the associated data points need to be determined by counting the number of associated feature points contained in each data point. The specific method is: each data point is analyzed, and the number of feature points actually involved in data generation among the j associated feature points it contains is counted. The data point with the most associated feature points is determined to be the associated data point of the feature point. This is because data points with more associated feature points can more comprehensively reflect the combined influence of each characteristic parameter in the state space, have a stronger correlation with the feature point, and are more suitable as a basis for model training and control action judgment. For example, in a three-dimensional state space, if a data point contains complete information on three associated feature points: crane position, load swing angle, and speed, while other data points only contain information on two or one of these feature points, then this data point will be preferentially selected as the associated data point.

[0087] To mark the location of each control action in the motion space, the control actions must first be detected and identified. Specifically, sensors or control systems monitor the crane's control commands in real time, identifying the type and parameters of each control action, such as its direction (e.g., lifting, lowering, translation), and amplitude (e.g., velocity, acceleration), and extracting the identification information for each control action. This identification information is the characteristic data used to uniquely identify each control action, and it must be generated in a way that ensures a one-to-one correspondence between the control actions in the motion space and the state space.

[0088] Based on the extracted identification information, the corresponding state point in the state space must be found for each control action in the action space. Since the corresponding control actions in the action space and state space share the same identification information, a mapping between the two can be achieved by matching the identification information. For example, each control action is assigned a unique code as identification information. This code is also associated with the state parameters in the state space, such as the crane's position, load sway angle, and speed, when executing the control action. This establishes a one-to-one correspondence between the action space and the state space.

[0089] After completing the mapping of control actions and state points, it is necessary to obtain the number of each control action and mark its position in the action space. The number of the control action is a unique identifier assigned to each control action in a certain logical order. For example, it is sorted and numbered according to the type (direction) and amplitude level of the control action to facilitate subsequent management and calling. When marking the position of the control action in the action space, the number of the control action is used as the index, and its corresponding direction and amplitude parameters are used as coordinate values ​​to determine the specific position in the coordinate system of the action space. For example, in a two-dimensional action space, the control direction is the horizontal coordinate and the control amplitude is the vertical coordinate. Each control action corresponds to a point in the coordinate system, and the coordinates of the point are the position of the control action in the action space.

[0090] In real-world applications, crane control actions can be affected by a variety of factors, such as load weight, crane boom length, and operating environment. Therefore, the dynamic changes of these factors must be fully considered when determining associated data points and marking control action locations. For example, when the load weight changes, the same control action may result in different load swing angles and speeds. In this case, it is necessary to re-extract data points in the state space and update the associated data points and position markers in the action space based on the new data points to ensure that the model can adapt to changes in the crane's operating conditions in real time.

[0091] Furthermore, the accuracy of data acquisition and processing is crucial for determining associated data points and marking the locations of control actions. Sensor measurement errors can lead to inaccurate data points in the state space, which in turn affects the selection of associated data points and the accuracy of the mapping between the action space and the state space. Therefore, regular sensor calibration and maintenance are necessary to ensure the accuracy of their measurement results. Furthermore, during data processing, algorithms such as filtering and smoothing can be used to pre-process the raw data to remove noise and improve data reliability.

[0092] Example 4: After determining the associated data points of each control action in the action space, it is necessary to perform a secondary verification on the control action and the corresponding associated data points to ensure that the mapping relationship between the action space and the state space is accurate. The following describes the implementation method in detail with reference to specific examples:

[0093] Suppose a crane performs a translational control action in a certain operating scenario. The data points associated with this action collected in the state space include the crane's position, load swing angle, and speed. The corresponding control action in the action space includes direction (horizontally to the right) and amplitude (speed of 0.5 m / s). In this case, the data points associated with this control action are segmented from the state space and labeled as the first associated point set (for example, containing three data points: A (position 1, angle 5°, speed 0.2 m / s), B (position 2, angle 3°, speed 0.1 m / s), and C (position 3, angle 4°, speed 0.15 m / s)). The data points associated with this control action are segmented from the action space and labeled as the second associated point set (for example, containing two data points: X (direction right, amplitude 0.5 m / s) and Y (direction right, amplitude 0.4 m / s)).

[0094] Based on feature matching, it is determined whether two sets of associated points belong to the same control sequence. In other words, it is determined whether the data points in the state space and the data points in the action space represent the same control action in different spaces. The basis for feature matching includes the logical association between the direction type and amplitude level of the control action and the position, angle, and speed in the state space. For example, in a translational control action, if the control amplitude in the action space is 0.5m / s, the speed parameter in the state space should be close to this amplitude value, and the load swing angle should have a reasonable dynamic relationship with the change in the crane position (for example, the angle may increase and then decrease due to inertia as the position moves further).

[0095] If the first and second associated point sets belong to the same control sequence after feature matching (for example, the velocity parameters of data point B and X are close, and the angle change conforms to the physical laws of translation), then the associated data points of the control action are confirmed to be valid through secondary verification. If the judgment is not valid (for example, the Y amplitude in the action space is 0.4m / s, which is significantly different from the velocity parameters of all data points in the state space and has no reasonable physical correlation), the associated data points of the control action need to be re-determined.

[0096] When re-determining the associated data points, first remove the duplicated t data points from the state space (for example, t = 5, including data points D, E, F, G, and H) and mark them as state data points; then remove the duplicated u data points from the action space (for example, u = 4, including data points M, N, O, and P) and mark them as action data points. Next, group the state and action data points based on feature matching. Each group should contain a state data point and its corresponding action data point belonging to the same control sequence.

[0097] Specific methods of feature matching include:

[0098] Direction consistency matching: The direction type of the action data point must be consistent with the crane movement direction in the status data point. For example, if the direction of the action data point is "horizontally right," the crane position change in the status data point should be reflected as movement to the right (e.g., increasing coordinate values).

[0099] Amplitude correlation matching: The amplitude level (e.g., speed) of the motion data point must have a reasonable numerical correlation with the speed parameter in the status data point. For example, when the motion amplitude is 0.6m / s, the speed of the status data point should fluctuate within the range of 0.6m / s±0.1m / s. The specific fluctuation range can be preset based on the mechanical characteristics and control accuracy of the crane.

[0100] Dynamic logic matching: The load sway angle changes in the state data points must conform to the physical laws of the control action corresponding to the action data points. For example, in a lifting control action, the load sway angle typically fluctuates slightly at the beginning due to inertia and then gradually stabilizes. In a translation control action, the angle change may initially increase and then decrease, with the peak value related to the translation speed and distance.

[0101] Take the example of redetermining the associated data points of the above translation control action:

[0102] State data point D (position 4, angle 6°, speed 0.6 m / s) and motion data point M (direction right, amplitude 0.6 m / s): The directions are consistent and the speed values ​​match. The angle of 6° may be the inertial swing during the translation start-up, which conforms to dynamic logic and is divided into a group.

[0103] State data point E (position 5, angle 2°, speed 0.55 m / s) and motion data point N (direction right, amplitude 0.55 m / s): They have the same direction and similar speed values. The angle of 2° may indicate a stable state during translation, so they are grouped together.

[0104] State data point F (position 3, angle 8°, speed 0.7 m / s) and action data point O (direction left, amplitude 0.7 m / s): The directions are inconsistent and are excluded from the group.

[0105] State data point G (position 6, angle 3°, speed 0.58 m / s) and action data point P (direction right, amplitude 0.58 m / s): The directions are consistent, the speed values ​​match, and the angle of 3° meets the characteristics of the stable stage. They are divided into one group.

[0106] State data point H (position 2, angle 5°, speed 0.4 m / s): No matching action data point, not grouped.

[0107] After grouping is complete, delete the state data points H and action data points O that are not included in the group. At this point, there are three groups (DM, EN, GP). Count the total number of associated feature points contained in each group (each state data point contains 3 associated feature points: position, angle, and speed; each action data point contains 2 associated feature points: direction and amplitude). The total number of associated feature points in each group is 5 (3 + 2). If there are multiple groups with the same total number, any one of them can be selected as the associated data points, or further filtered based on additional conditions such as time series continuity. For example, according to the time order of data collection, select the earliest collected group DM as the associated data point, mark the state data point D as the first associated point set, and mark the action data point M as the second associated point set.

[0108] In another scenario, if there is only one group after re-segmentation (e.g., the state data point Q and the action data point R completely match), the state data points and action data points in the group are directly determined as the associated data points of the control action without further statistics.

[0109] The secondary verification and re-determination of associated data points ensures that the mapping between the action space and the state space conforms to the crane's actual motion patterns and control logic. For example, in a hoisting control action, if the action space amplitude is "low-speed hoisting," the speed parameter in the state space should be in the low-speed range (e.g., 0.1-0.3 m / s), and the load sway angle should be small. If a high speed parameter (e.g., 0.8 m / s) matches the "low-speed hoisting" action, feature matching will identify this as an anomaly, triggering a re-determination process.

[0110] In real-world applications, crane control actions may involve complex operations (such as simultaneous lifting and translation). In this case, the associated data points for each independent control action must be verified and determined separately. For example, a complex action can be decomposed into lifting and translation, and the state and motion data points for each are extracted and verified and matched one by one according to the above steps.

[0111] Example 5: The generation of identification information of the control action and the execution process of the anti-sway control operation are as follows, which are described in detail with reference to specific examples:

[0112] 1. Identification Information Generation and Mapping

[0113] When assigning a unique action code to each control action, a "type-amplitude-timestamp" combination is used to ensure uniqueness. For example, if a hoist control action is type 01 (defined as the hoist direction), amplitude level 03 (preset low speed, corresponding to a speed of 0.2m / s), and the timestamp is 20250604143001, the action code is "01-03-20250604143001." This code is associated with the position coordinates in state space. Assuming the initial position coordinates of the crane when executing this action are (X=10m, Y=5m, Z=3m), the identification information includes these coordinates as the spatial positioning reference.

[0114] When generating an identification string based on an action code, the direction type and amplitude level must be parsed. For example, in the action code "01-03-20250604143001," "01" corresponds to the direction type "lift," and "03" corresponds to the amplitude level "low gear." Therefore, the identification string is "QS-D3" (QS stands for lift, and D3 stands for low gear, third gear). This string is stored in the index field of the action database for subsequent matching with state points. When the crane performs a translation control action (right direction, 0.5 m / s amplitude), the action code is "02-05-20250604143510" (02 stands for translation direction, 05 stands for medium gear), the identification string is "PY-Z5," and the initial position of the bound state space is (X=12 m, Y=5 m, Z=3 m).

[0115] 2. Direction Control Verification Example

[0116] Assume that the hoisting action template preset in the action database contains the direction type "hoisting" and its corresponding identification string prefix "QS-." When the crane performs a control action, the identification string "QS-D3" is extracted and matched with the "QS-" prefix in the action template, identifying the control direction as "hoisting." Furthermore, the control direction obtained when defining the action space is also "hoisting." These two are consistent, thus passing the direction control verification.

[0117] If an abnormal situation occurs, such as the identification string of a control action is "PY-D3" (prefixed with translation direction PY), but the crane moves in the descending direction during actual execution, by comparing the identification string with the action template, it is found that the direction type is "translation" and the actual movement direction is "descending". The two are inconsistent, and the direction control verification fails. The system will trigger an alarm and stop the execution of the action to avoid aggravated load sway due to direction misjudgment.

[0118] 3. Amplitude Control Verification Example

[0119] For example, consider a set of translation control actions consisting of three consecutive movements: Action 1 (amplitude 0.3 m / s), Action 2 (amplitude 0.4 m / s), and Action 3 (amplitude 0.5 m / s). The commanded total amplitude is 0.3 + 0.4 + 0.5 = 1.2 m / s. Using real-time data collected by speed sensors installed on the crane's drive motors, the actual total amplitude is calculated: Action 1's actual speed is 0.29 m / s, Action 2's actual speed is 0.41 m / s, and Action 3's actual speed is 0.50 m / s. The total actual amplitude is 0.29 + 0.41 + 0.50 = 1.20 m / s. The calculated amplitude difference is 1.20 - 1.2 = 0 m / s, indicating that the actual amplitude is consistent with the command, and amplitude control verification has passed.

[0120] In another example, if the commanded total amplitude is 0.8m / s (composed of two movements with amplitudes of 0.4m / s), and the sensor data shows the actual total amplitude is 0.75m / s, with an amplitude difference of 0.05m / s. At this point, it is necessary to determine whether this difference is within the preset allowable range (e.g., ±0.1m / s). If it is, verification passes. If it is outside the range, the compensation mechanism is triggered, adjusting the amplitude of subsequent movements to correct the deviation. For example, if the allowable range is ±0.1m / s, a difference of 0.05m / s is within the allowable range, and verification passes. If the difference is 0.15m / s, verification fails, and the system automatically increases the amplitude of the next movement by 0.05m / s to balance the accumulated error.

[0121] 4. Verification of Multi-Action Collaboration Scenario

[0122] When a crane performs a complex control action (such as lifting and translation at the same time), the direction and amplitude of each independent action must be verified separately. For example, a lifting action (identifier string QS-D3, amplitude 0.2m / s) and a translation action (identifier string PY-Z5, amplitude 0.5m / s) are performed simultaneously:

[0123] Direction verification: The "QS-" prefix of the lifting action identification string matches the lifting direction in the action template, and the "PY-" prefix of the translation action matches the translation direction. Both directions are consistent with the directions obtained in the defined action space, and the verification passes.

[0124] Amplitude Verification: The command amplitude for the lifting action was 0.2m / s, and the sensor measured 0.19m / s. The command amplitude for the translation action was 0.5m / s, and the sensor measured 0.51m / s. The lifting amplitude difference was -0.01m / s, and the translation amplitude difference was +0.01m / s, both within the allowable range, and verification passed.

[0125] If a particular action fails verification—for example, if the measured amplitude for a horizontal movement is 0.6m / s and the amplitude deviation is 0.1m / s (exactly at the upper limit of the allowable range)—the system will record the anomaly, even though it passes verification, and subsequently monitor the execution parameters for that action. If deviations approaching the upper limit of the allowable range occur repeatedly, the system will automatically adjust the amplitude control parameters for that action, such as lowering the command amplitude preset value, to reduce actual execution deviations.

[0126] 5. Application of Identification Information and Database Indexes

[0127] The action database's index fields utilize identification strings for rapid retrieval. For example, to query the corresponding state point in the state space for a specific lifting action, simply enter the identification string "QS-D3," and the database will directly locate the bound position coordinates (X=10m, Y=5m, Z=3m) and the corresponding historical state data (such as the load sway angle curve and speed fluctuation data during the action). This mapping ensures real-time linkage between control actions and state data, enabling the system to quickly access relevant information during anti-sway control and optimize control strategies.

[0128] In actual operation, cranes may experience changes in the effectiveness of the same control action due to factors such as load fluctuations and mechanical wear. For example, the actual speed of a PY-Z5 translation action (0.5 m / s) may decrease due to increased motor load after the load increases, causing the amplitude difference to exceed the allowable range. In this case, the system updates the associated state points of the action in the action database using real-time collected state data, regenerating the binding relationship between identification information and state coordinates. This ensures that the model can adapt to changes in the crane's dynamic characteristics and maintain the effectiveness of anti-sway control.

[0129] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0130] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. The intelligent anti-sway control method for crane based on reinforcement learning is characterized by: The following steps are involved: Collecting crane motion state data and load sway data, wherein the motion state data includes position and speed information; Based on the motion state data and the sway data, combined with a preset reinforcement learning model, determining whether each control action effectively reduces the load sway; If it is determined to be valid, performing an anti-sway control operation based on the control action; Among them, judging whether each control action is effective includes defining the state space, action space and reward function, and training the reinforcement learning model; The state space includes crane position, load sway angle, and velocity; The action space includes the direction and magnitude of the control command; The reward function is calculated based on the swing amplitude change and energy consumption.

2. The intelligent anti-sway control method for cranes based on reinforcement learning according to claim 1, characterized in that: The motion state data is real-time data collected by sensors, including data collected by an inertial measurement unit installed on the crane boom and an angle sensor on the load; the position information is extracted based on global positioning system data, and the extraction method is filtering and smoothing.

3. The intelligent anti-sway control method for crane based on reinforcement learning according to claim 2, characterized in that: The method for defining the state space is as follows: calculating the dimension of the state space in data acquisition; setting a feature subset for the state space in data acquisition, and uniformly selecting k feature points in the feature subset as associated feature points of the state space, where k is a positive integer; wherein the feature subset of the state space is composed of all feature points whose distance from the boundary of the state space is not less than r and not greater than s, where r is a preset first boundary threshold and s is a preset second boundary threshold.

4. The intelligent anti-sway control method for crane based on reinforcement learning according to claim 3 is characterized in that: The training of the reinforcement learning model includes taking the associated feature points as input to update the model, and the steps are as follows: S401: Check whether each input feature point meets the convergence criterion. If so, mark the feature point as a training sample and add it to the current model; The convergence criterion is specifically as follows: if the Euclidean distance between the feature point and the model prediction in the error space is less than a preset error threshold, the feature point meets the convergence criterion; otherwise, the feature point does not meet the convergence criterion; S402: Marking the input that has completed the convergence criterion check of all feature points as processed; S403: Repeat S401 to S402, and the input marked as processed will no longer be included in the check until all inputs are marked as processed, and the model update is stopped; the parameter range of the current model is recorded as the updated reinforcement learning model.

5. The intelligent anti-sway control method for cranes based on reinforcement learning according to claim 4, characterized in that: The associated feature points are data points corresponding to the state space; the method for determining the associated data points of each feature point is as follows: Extract m data points updated based on j associated feature points in the state space; deduplicate the m data points and delete completely overlapping data points; if only one data point remains after deduplication, the remaining data point is the associated data point of the feature point; if n data points remain after deduplication, where n is a positive integer greater than 1, count the number of associated feature points contained in each data point, and the data point containing the most associated feature points is the associated data point of the feature point.

6. The intelligent anti-sway control method for cranes based on reinforcement learning according to claim 5, characterized in that: The load sway data is extracted based on real-time data collected by the angle sensor. The method for marking the position of each control action in the action space is as follows: detecting and identifying each control action in the action space and extracting identification information of the control action; and finding the state point corresponding to each control action in the action space in the state space based on the identification information; The corresponding control actions in the action space and the state space have the same identification information; the number of each control action is obtained and the position of each control action is marked in the action space.

7. The intelligent anti-sway control method for cranes based on reinforcement learning according to claim 6, characterized in that: After determining the associated data points of each control action in the action space, a secondary verification is performed on each control action and the corresponding associated data points, including the following steps: Segment the associated data points of the control action from the state space and mark them as the first associated point set; segment the associated data points of the control action from the action space and mark them as the second associated point set; Based on feature matching, determine whether the first set of associated points and the second set of associated points belong to the same group of control sequences; if so, pass secondary verification; if not, re-determine the associated data points of the control action; belonging to the same group of control sequences indicates that the first set of associated points and the second set of associated points are representations of the same control action in different spaces.

8. The intelligent anti-sway control method for cranes based on reinforcement learning according to claim 7, characterized in that: The method for re-determining the associated data points of the control action is as follows: The t data points after segmentation and removal from the state space are all marked as state data points; the u data points after segmentation and removal from the action space are all marked as action data points; t and u are both positive integers; Based on feature matching, the state data points and action data points are grouped; any group contains a state data point and corresponding action data point belonging to the same group of control sequences; state data points and action data points that are not grouped are deleted; If there is only one group, the state data points and action data points in the group are all associated data points of the control action; if there is more than one group, the total number of associated feature points contained in each group is counted, and the state data points and action data points in the group containing the most associated feature points are all associated data points of the control action; The state data points in the associated data points are marked as a first associated point set, and the action data points in the associated data points are marked as a second associated point set.

9. The intelligent anti-sway control method for cranes based on reinforcement learning according to claim 8, characterized in that: The control action includes direction adjustment and amplitude adjustment; the execution of the anti-sway control operation includes direction control verification and amplitude control verification; The method for direction control verification is as follows: comparing and matching the identification information of the control action with the action template in the preset action database to identify the control direction; if the identified control direction is consistent with the control direction obtained by defining the action space, the direction control verification is passed; The amplitude control verification method is as follows: extracting the amplitude information of all control actions and calculating the total amplitude of all actions, which is recorded as the command total amplitude; calculating the total amplitude actually executed through sensor data, which is recorded as the actual total amplitude; The amplitude difference between the actual total amplitude and the command total amplitude is calculated.

10. The intelligent anti-sway control method for crane based on reinforcement learning according to claim 6, characterized in that: The method for generating identification information of the control action is as follows: assigning a unique action code to each control action; binding the action code to the position coordinates in the state space; generating an identification string containing the direction type and amplitude level based on the action code; and storing the identification string in the index field of the action database for matching the state point with the action position.

Images