Intelligent control method and system for dredging robot arm based on DPPO reinforcement learning

Abstract

This invention discloses an intelligent control method and system for a dredging robotic arm based on DPPO reinforcement learning, relating to the field of intelligent technology for dredging engineering equipment. The method includes real-time acquisition of hydraulic pressure, vibration spectrum, and temperature data of the dredging and silt removal robotic arm of a dredging vessel, followed by preprocessing. The preprocessed data is then input into an LSTM-Attention model, where features are extracted based on a long short-term memory network and attention mechanism, and the remaining service life of the robotic arm is predicted. A degradation-efficiency coupling model is established, and the robotic arm health index and operational efficiency are calculated based on the predicted remaining service life. A DPPO reinforcement learning controller is established, using the robotic arm health index, operational efficiency, energy consumption, and robotic arm joint angles as the state space, and outputting a dynamic load adjustment command action space. The control strategy is optimized through a reward function, and the load adjustment command is processed by dynamic constraints to drive the hydraulic actuator of the robotic arm.

Description

Technical Field

[0001] This invention relates to the field of intelligent technology for dredging engineering equipment, and more specifically to an intelligent control method and system for a dredging robotic arm based on DPPO reinforcement learning. Background Technology

[0002] Dredging robotic arms are used for dredging operations in ports, rivers, and other areas. As heavy machinery, they operate in harsh environments and are difficult to maintain. Traditional maintenance methods include periodic maintenance or repair after a breakdown, both of which are inefficient and costly. Periodic maintenance can lead to over-maintenance or under-maintenance, while repair after a breakdown results in downtime losses and safety risks.

[0003] Existing technologies lack effective real-time monitoring methods, making it impossible to accurately predict the remaining life of equipment and dynamically adjust maintenance strategies. Furthermore, traditional prediction models may not consider the interaction between equipment degradation and operational efficiency, leading to inaccurate predictions or suboptimal maintenance strategies. Therefore, traditional dredging vessel robotic arm maintenance has the following limitations: 1. Lagging nature of traditional maintenance methods: Dredging robotic arms typically employ periodic maintenance or post-failure repair (passive maintenance), failing to predict equipment degradation trends and easily leading to sudden downtime, operational interruptions, or high repair costs. 2. Separate analysis of equipment degradation and operational efficiency: Existing methods only focus on a single dimension of equipment degradation (e.g., component wear) or operational efficiency (e.g., task completion rate), failing to quantify the dynamic coupling relationship between the two, making it difficult to formulate maintenance strategies that balance efficiency and reliability. 3. Insufficient prediction accuracy under complex operating conditions: In dredging operations, multiple parameters such as hydraulic pressure, vibration, and temperature are nonlinearly coupled, making it difficult for traditional statistical models (e.g., linear regression) or single LSTM networks to capture key degradation characteristics, resulting in large errors in remaining life (RUL) prediction.

[0004] Therefore, how to provide an intelligent control method for dredging robotic arms based on DPPO reinforcement learning to solve the above-mentioned drawbacks, improve the efficiency of dredging operations, reduce costs, and extend the service life of equipment is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides an intelligent control method and system for dredging robotic arms based on DPPO reinforcement learning. The present invention provides an efficient intelligent control solution for heavy dredging equipment through hardware-algorithm collaborative architecture (perception-decision-execution closed loop), dynamic constraint processing (action safety mapping), and industrial-grade deployment (TensorRT acceleration, OPC UA protocol).

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The intelligent control method for dredging robotic arms based on DPPO reinforcement learning includes:

[0008] Real-time acquisition and preprocessing of hydraulic pressure, vibration spectrum, and temperature data of the dredging robotic arm;

[0009] The preprocessed data is input into the LSTM-Attention model, which performs feature extraction based on the long short-term memory network and attention mechanism, and outputs the predicted value of the remaining service life of the robotic arm.

[0010] A degradation-efficiency coupling model was established, and the robot arm health index and operational efficiency were calculated based on the predicted value of the remaining service life of the robot arm.

[0011] A DPPO reinforcement learning controller is established, with the robotic arm health index, work efficiency, energy consumption and robotic arm joint angle as the state space, and the action space of output dynamic load adjustment commands. The control strategy is optimized through the reward function, and the load adjustment commands are driven by the hydraulic actuator of the robotic arm after being processed by dynamic constraints.

[0012] Preferably, the preprocessing includes:

[0013] Missing value handling: Lagrange interpolation is used to fill in missing data;

[0014] Outlier handling: Outliers are detected using Tukey's Test and error values ​​are removed;

[0015] Normalization: Min-Max normalization was used to normalize the collected multidimensional parameters;

[0016] Substitute the vibration spectrum after the above preprocessing into the formula:

[0017]

[0018] Among them, V rms Let N be the root mean square value of the vibration velocity at time t, and N be the sample value selected for calculation.

[0019] Number of points, v i The instantaneous vibration velocity measurement value at the i-th sampling point;

[0020]

[0021] Among them, F peak (t) represents the maximum amplitude of the vibration signal at time t in the 1-5kHz frequency band. The spectrum obtained by performing a Fourier transform on the vibration velocity signal v(t) is max(...). 1-5kHz This takes the maximum amplitude value within the frequency range of 1-5kHz.

[0022] Preferably, the preprocessed data is input into the LSTM-Attention model, which extracts features based on a long short-term memory network and attention mechanism, and outputs a predicted value for the remaining lifespan of the robotic arm, including:

[0023] The input layer receives preprocessed multi-source time-series data, including hydraulic pressure and root mean square value of vibration velocity V. rms Maximum vibration amplitude F peak and temperature parameters;

[0024] The system sequentially extracts local temporal features through convolutional layers, performs max pooling through pooling layers, and flattens the three-dimensional tensor output by the pooling layers into one-dimensional vector data.

[0025] The flattened one-dimensional vector data is input into the LSTM layer. Long-term temporal dependencies are captured through forget gate, input gate, output gate and cell state, and the hidden state sequence is output. Combined with the multi-head parallel attention mechanism, the weights are dynamically allocated to focus on the key degradation stage, and finally the remaining service life prediction value of the robotic arm is generated.

[0026] The preferred degradation-performance coupling model is as follows:

[0027]

[0028] E = E0·e -0.25HI

[0029] Where HI represents the robotic arm health index, E represents operational efficiency, and S represents the robotic arm health index. vib The vibration entropy is expressed as F. peak (t) is obtained by extracting high-frequency features and then calculating them through wavelet packet decomposition or spectral entropy. ΔP represents the change in hydraulic pressure, E0 represents the initial value of efficiency, and RUL represents the predicted value of the remaining service life of the robotic arm.

[0030] Preferably, the reward function is:

[0031] R = -0.3 × energy consumption + 0.5 × E + 0.2 × HI

[0032] Where R is the reward function.

[0033] The preferred training process for the DPPO reinforcement learning controller:

[0034] The policy network and value network employ a dual-delay update mechanism;

[0035] In the experience replay pool, priority is given to sampling critical state data where the HI decreases by more than 10%;

[0036] The policy gradient update amount is calculated using generalized advantage estimation.

[0037] Preferably, the load adjustment command is processed by dynamic constraints to drive the hydraulic actuator of the robotic arm, including:

[0038] Abrupt signals are eliminated by using a first-order low-pass filter, limiting the single adjustment amplitude to no more than 5% to prevent the robotic arm from being impacted by sudden changes in load adjustment commands; at the same time, the load adjustment commands are mapped to the joint torque safety range based on the Jacobian matrix to ensure that the movement conforms to mechanical dynamics constraints.

[0039] The processed load adjustment commands are inferred in milliseconds through the TensorRT acceleration engine and converted into standard industrial signals. These signals are then transmitted to the PLC controller via the OPC UA protocol, driving the hydraulic proportional valve of the robotic arm to dynamically adjust the oil pressure and flow, thereby achieving precise load control.

[0040] Preferably, it also includes dynamic health monitoring and early warning:

[0041] When HI≥0.8, a Level 1 warning is triggered, the current operating condition snapshot is automatically recorded, and manual intervention is prompted for inspection;

[0042] When HI≥0.9, the system is forced to switch to low-load protection mode to reduce operational efficiency and slow down equipment degradation. At the same time, maintenance work orders are pushed through the cloud, which are associated with fault types and spare parts inventory information.

[0043] The intelligent control system for a dredging robotic arm based on DPPO reinforcement learning includes:

[0044] Data acquisition and preprocessing module: Real-time acquisition of hydraulic pressure, vibration spectrum and temperature data of the dredging robotic arm, and preprocessing of the data;

[0045] Model building module: Input the preprocessed data into the LSTM-Attention model, extract features based on the long short-term memory network and attention mechanism, and output the predicted value of the remaining service life of the robotic arm;

[0046] Degradation-Efficiency Coupled Model Module: Establishes a degradation-efficiency coupled model and calculates the robot arm health index and efficiency decay coefficient based on the predicted value of the remaining service life of the robot arm;

[0047] Predictive target module: Establish a DPPO reinforcement learning controller, with the robotic arm health index, work efficiency, energy consumption and robotic arm joint angle as the state space, output dynamic load adjustment command action space, optimize the control strategy through reward function, and drive the robotic arm hydraulic actuator after the load adjustment command is processed by dynamic constraints.

[0048] Preferred options also include:

[0049] Maintenance strategy module: used for dynamic health monitoring and early warning, specifically:

[0050] When HI≥0.8, a Level 1 warning is triggered, the current operating condition snapshot is automatically recorded, and manual intervention is prompted for inspection;

[0051] When HI≥0.9, the system is forced to switch to low-load protection mode to reduce operational efficiency and slow down equipment degradation. At the same time, maintenance work orders are pushed through the cloud, which are associated with fault types and spare parts inventory information.

[0052] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses an intelligent control method and system for dredging robotic arms based on DPPO reinforcement learning. The present invention solves the three major pain points of traditional methods, namely maintenance lag, energy efficiency imbalance and command infeasibility, through the full-link design of "state perception-strategy optimization-safe execution-closed-loop early warning", and realizes the engineering implementation of intelligent control of heavy equipment. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 A flowchart of the intelligent control method for a dredging robotic arm based on DPPO reinforcement learning provided by the present invention;

[0055] Figure 2 This is an internal structure diagram of the LSTM-Attention model provided by the present invention;

[0056] Figure 3 This is a schematic diagram of the internal structure of the LSTM network provided by the present invention;

[0057] Figure 4 A flowchart for multi-head parallel attention;

[0058] Figure 5 This is an internal structure diagram of the DPPO reinforcement learning algorithm application provided by the present invention;

[0059] Figure 6 The structural block diagram of the intelligent control system for the dredging robotic arm based on DPPO reinforcement learning provided by the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] This invention discloses an intelligent control method for a dredging robotic arm based on DPPO reinforcement learning, such as... Figure 1 As shown, it includes:

[0062] Real-time data collection of hydraulic pressure, vibration spectrum, and temperature of the dredging and cleaning robotic arm of the dredging vessel, followed by preprocessing.

[0063] The preprocessed data is input into the LSTM-Attention model, which extracts features based on the long short-term memory network and attention mechanism, and outputs a prediction of the remaining lifespan.

[0064] A degradation-efficiency coupling model was established, and the health index and operational efficiency of the robotic arm were calculated based on the predicted remaining service life.

[0065] A DPPO reinforcement learning controller is established, with the robotic arm health index, work efficiency, energy consumption and robotic arm joint angle as the state space, and the action space of output dynamic load adjustment commands. The control strategy is optimized through the reward function, and the load adjustment commands are driven by the hydraulic actuator of the robotic arm after being processed by dynamic constraints.

[0066] The following is a further explanation of each of the above steps.

[0067] S1. Multimodal time-series data of the robotic arm's operating status are collected in real time using hydraulic pressure sensors (range 0-40MPa, accuracy ±0.5%FS), triaxial vibration accelerometers (bandwidth 5-10kHz), and infrared temperature measurement modules (accuracy ±1℃) mounted on the dredging vessel. This includes hydraulic pressure, vibration spectrum, and temperature data of the dredging robotic arm, and preprocessing is performed. Specifically:

[0068] S101, Data Format Definition

[0069] Each training sample is in the following format:<t,X_m,Y> ,in:

[0070] Time window identifier t: indicates the time window to which the data belongs (e.g., 60 seconds is a window, with a step size of 10 seconds); Input feature X_m: contains m multi-source sensor and operation parameter features; Label Y: a classification label for the health status of the robotic arm, using one-hot encoding to define the label for each category, that is, using data 0 and 1 to distinguish the three states of the robotic arm.

[0071] Normal state → [1,0,0]

[0072] Slight degradation → [0,1,0]

[0073] Severe degradation → [0,0,1]

[0074] S102, Data Acquisition

[0075] This example collected 518,400 data points on the normal operation of the robotic arm of a dredging vessel over 30 consecutive days of operation (60 seconds / data point × 24 hours × 30 days × 60 minutes / hour ÷ 60 seconds). Using a Gardner-Child Adversarial Network (CGAN), a simulated robotic arm degradation scenario was generated, producing 1,036,800 synthetic data points including faults such as hydraulic leakage, bearing wear, and high-temperature overload. This ensured a 1:2 ratio of normal to abnormal data, mitigating class imbalance. The total sample size for this example is 1,555,200 data points.

[0076] S103, Data Augmentation

[0077] All data will be processed and normalized, specifically:

[0078] Missing value handling: Lagrange interpolation is used to fill in missing sensor data;

[0079] Outlier handling: Outliers are detected using Tukey's Test and error values ​​are removed;

[0080] Normalization: Min-Max normalization was used to normalize the collected multidimensional parameters. The preprocessed dataset was then randomly divided into training, validation, and test sets in a 7:2:1 ratio for model training, hyperparameter tuning, and generalization ability evaluation, respectively. The format is as follows:

[0081] Training samples:

[0082]

[0083] Validation sample:

[0084]

[0085] Test sample:

[0086]

[0087] Where n represents the nth data item.

[0088] Substitute the vibration frequency after the above preprocessing into the formula:

[0089]

[0090] Among them, V rms The root mean square value of the vibration velocity at time t represents the effective energy level of the vibration (units such as m / s); N is the number of sample points selected in the calculation (corresponding to the amount of data within a certain time window); v i The instantaneous vibration velocity measurement value at the i-th sampling point;

[0091]

[0092] Among them, F peak (t) represents the maximum amplitude of the vibration signal at time t in the 1-5kHz frequency band (which may correspond to the amplitude of the fault characteristic frequency); The spectrum is obtained by performing a Fourier transform on the vibration velocity signal v(t); max(...) 1-5kHz This takes the maximum amplitude value within the frequency range of 1-5kHz.

[0093] Substituting the vibration spectrum into formula V rms (t) and F peak The core function of (t) is as follows:

[0094] (1) Feature extraction: V rms The root mean square value of vibration velocity is used to quantify the overall energy level of the vibration signal in the time domain, reflecting the vibration intensity of the robotic arm. peak (t)(maximum amplitude in the 1-5kHz band) is used to capture high-frequency vibration characteristics and identify specific fault modes (such as bearing wear, abnormal gear meshing, etc.).

[0095] (2) Data dimensionality reduction and standardization: The original vibration spectrum data is high-dimensional and complex. By calculating V rms and F peak (t) compresses the high-dimensional spectral signal into key feature values, which facilitates subsequent model processing.

[0096] S2, such as Figures 2-4 As shown, an LSTM-Attention prediction model is constructed. Its network architecture design includes an LSTM layer that captures the temporal dependencies of parameters such as hydraulic pressure and vibration through forget gate, input gate, output gate and cell state, and an Attention layer that dynamically allocates weights to focus on key degradation stages such as high-frequency vibration surges. The model input is preprocessed multi-source sensor time-series data (hydraulic pressure, vibration spectrum characteristics, and temperature at time t), and the output is the predicted value of the remaining service life (RUL) of the robotic arm.

[0097] A multimodal degradation feature adaptive attention mechanism is introduced into the LSTM-Attention model. This mechanism optimizes the traditional single-head attention structure to address the coupling characteristics of multi-source sensor data (hydraulic, vibration, and temperature) from the dredging robot arm. The final result is used to predict the robot arm's remaining service life (RUL). Specific improvements are as follows:

[0098] S201, Multi-head Parallel Attention:

[0099] To address the differences in physical characteristics of multimodal sensor data such as hydraulic pressure, vibration, and temperature from dredging robotic arms, an independent attention head modal analysis mechanism is designed. The three-branch attention network of this invention comprises three specialized feature extraction modules: a hydraulic pressure fluctuation head, which constructs a pressure time-series analysis model to capture abnormal fluctuations in hydraulic system pressure, such as sudden increases and decreases (e.g., a sudden pressure increase can accurately identify valve blockage faults), enabling quantitative analysis of the frequency and amplitude of pressure fluctuations; a vibration frequency domain energy head, employing wavelet packet energy entropy analysis technology, which monitors energy mutation patterns in specific frequency bands (e.g., 10-15kHz) (typically, a sudden increase in frequency band energy caused by bearing wear) to quantitatively assess the modal anomaly degree of vibration signals; and a temperature trend head, based on a sliding window algorithm, captures gradual temperature change trends (e.g., changes in the temperature rise slope caused by motor overload), effectively identifying the long-term cumulative effect of heat load.

[0100] S202, Adaptive weighting of degradation features:

[0101] Based on a historical fault database of robotic arms (containing six typical fault types such as hydraulic failure, bearing wear, and motor overheating), a dynamic weight allocation strategy for attention heads was constructed. This strategy achieves intelligent fusion of multi-source signals through two-stage optimization: First, in the offline training stage, a fault type label supervised learning method is used to determine the initial weight allocation of each attention head (e.g., 40% for hydraulic pressure fluctuation head, 35% for vibration frequency energy head, and 25% for temperature trend head); second, in the online inference stage, by real-time monitoring of sensor data characteristics (e.g., vibration energy entropy value, pressure fluctuation amplitude, etc.), when a specific fault mode symptom is detected (e.g., a sudden increase in vibration entropy value indicating bearing wear), the dynamic weight adjustment mechanism is automatically triggered.

[0102] S203, Spatiotemporal Feature Fusion:

[0103] The spatiotemporal feature fusion architecture achieves complementary enhancement of multi-dimensional degradation information. Its core fusion mechanism includes three key steps: First, it uses an LSTM network to capture the long-term evolution trend of parameters such as hydraulic pressure and temperature (e.g., the pressure drop curve caused by hydraulic leakage); second, it extracts the wavelet packet energy entropy features of vibration signals based on the attention mechanism (e.g., a sudden increase in the energy proportion in the 2000-4000Hz frequency band can accurately identify gearbox pitting faults); finally, it concatenates the temporal feature vector output by the LSTM with the frequency domain feature vector generated by the attention module at the feature level, and achieves deep fusion of multimodal features through a fully connected network to generate the final RUL prediction value.

[0104] The trained model can be used to predict the RUL of the dredging vessel's robotic arm.

[0105] The process of generating the predictive robotic arm's RUL includes:

[0106] (1) Input layer: Receives preprocessed multi-source time-series data, including hydraulic pressure and vibration spectrum characteristics (V). rms F peak Temperature, etc.

[0107] (2) Convolutional layer (CNN) processing: A one-dimensional convolutional kernel (kernel_size=3, filters=32) is used to slide along the time axis to extract local temporal features and generate feature maps; Activation function: ReLU, to enhance non-linear expressive ability, and the output feature map dimension is [batch_size, sequence_length=20, filters=32].

[0108] Here, (kernel_size=3, filters=32) are the convolutional layer parameters. kernel_size=3 indicates that the time span (time window size) of the one-dimensional convolutional kernel is 3 time steps, and filters=32 indicates that 32 different convolutional kernels are used, each responsible for extracting a specific local feature. [batch_size, sequence_length=20, filters=32] are the output dimensions of the convolutional layer. batch_size represents the number of samples input to the model in a single batch (i.e., batch size), and sequence_length=20 represents the time series length (number of time steps) of each sample. After the convolution operation, the time steps of the original time series data may be compressed or kept at 20 due to padding or stride adjustment. filters=32 represents the number of feature channels corresponding to each time step (i.e., the number of convolutional kernels). Each channel corresponds to a feature map extracted by one convolutional kernel.

[0109] (3) Pooling layer: Max pooling is performed along the time axis (pool_size=2, stride=2), compressing the feature dimension to [batch_size, sequence_length=10, filters=32], selecting the maximum value within each window, retaining significant features and reducing computational complexity.

[0110] (pool_size=2, stride=2) are the pooling layer parameters, where pool_size=2 indicates that the pooling window spans two time steps. The pooling operation slides along the time axis, covering two consecutive time steps each time. stride=2 indicates that the sliding step size of the pooling window is 2. Each time it moves two time steps, and the windows do not overlap. [batch_size, sequence_length=10, filters=32] are the output dimensions of the pooling layer, where sequence_length=10 compresses the time steps from 20 to 10, while batch_size and filters=32 remain unchanged; only the time axis is compressed.

[0111] (4) Flatten the three-dimensional tensor ([batch_size,10,32]) output by the pooling layer into a one-dimensional vector with the shape [batch_size,320] to adapt to the input of the subsequent time series modeling module.

[0112] Expand the 3D tensor [batch_size, 10, 32] by its last dimension to obtain a 1D vector. The calculation formula is: 10 (time steps) × 32 (feature channels) = 320.

[0113] (5) Input to the LSTM-Attention model:

[0114] The flattened one-dimensional vector data is input into an LSTM layer. Long-term temporal dependencies are captured through forgetting gates, input gates, output gates, and cell states, outputting a hidden state sequence. Combined with a multi-head parallel attention mechanism (hydraulic, vibration, and temperature modal processing), weights are dynamically allocated to focus on key degradation stages (such as periods of sudden increases in high-frequency vibration), ultimately generating a predicted remaining service life (RUL) value for the robotic arm. The specific steps are as follows:

[0115] Sa, temporal feature extraction:

[0116] The vibration signal is processed in real time using Fourier transform to calculate 15 features, including the root mean square (RMS), peak factor, kurtosis, impulse factor, waveform factor, margin factor, band energy (low frequency), band energy (mid frequency), band energy (high frequency), dominant frequency component, spectral centroid, spectral entropy, zero-crossing rate, temporal variance, and envelope spectrum peak value. The output shape is [batch_size, sequence_length = 20, feature_dim = 15], where batch_size is the batch size (number of samples processed at one time); sequence_length = 20: each sample contains data from 20 consecutive time points; and feature_dim = 15: 15 features are extracted from each time point.

[0117] Sb, multi-head parallel attention mechanism:

[0118] Based on multimodal sensing data including hydraulics, vibration, and temperature, three independent attention heads were designed for hydraulic pressure fluctuation, vibration frequency energy, and temperature trend. These heads perform modal processing to dynamically focus on key degradation features. Specifically, the hydraulic pressure fluctuation head extracts the hydraulic-related temporal dimension from the LSTM output, then uses time-series analysis to capture fluctuations such as sudden pressure rises and falls, generating a pressure fluctuation feature vector to identify abnormal conditions in the hydraulic system, such as valve blockage. The vibration frequency energy head extracts the vibration-related temporal dimension from the LSTM output to detect bearing wear or gearbox pitting faults. The temperature trend head extracts the temperature-related temporal dimension from the LSTM output and uses a sliding window smoothing algorithm to extract gradual trends such as changes in the temperature rise slope, generating temperature trend parameters to ultimately capture the effects of motor overload or long-term thermal load accumulation.

[0119] Sc, Dynamic allocation of attention weights:

[0120] The initial weights are allocated based on historical fault data statistics, with hydraulic pressure fluctuation head, vibration frequency energy head, and temperature trend head accounting for 40%, 35%, and 25%, respectively. During dynamic adjustment, the system monitors the characteristics of each modality in real time (such as a sudden increase in vibration entropy) and automatically increases the weight of the corresponding attention head (e.g., the weight of the vibration head can be increased to 50%). The weight calculation generates an attention score through SoftMax-like normalization, using the following formula:

[0121]

[0122] Among them, F(Q) i ,K i ) is a similarity function between the query and the key.

[0123] Sd, multimodal feature fusion:

[0124] Feature fusion and prediction are accomplished through the following steps: First, a feature concatenation method is used to concatenate the outputs of the three parallel attention heads (with a shape of [batch_size, 64]) along the feature dimension to form a fused feature vector (with a shape of [batch_size, 192]). Then, a mapping network consisting of two fully connected layers (using ELU activation function) is used to non-linearly map the fused features to the target space, finally outputting the RUL prediction value (with a shape of [batch_size, 1]). The attention weight matrix calculation is accelerated using a GPU (complexity O(n^2)). 2 Output processing: The output of the fully connected layer is activated by an exponentially modified linear unit (ELU) to obtain the RUL prediction value.

[0125] It may also include:

[0126] Se, establish a three-dimensional evaluation framework that includes statistical indicators, trend consistency, and fault warning rate:

[0127] Statistical indicators:

[0128]

[0129] Trend fit: The similarity between the predicted curve and the actual degradation curve is calculated using the Dynamic Time Warping (DTW) algorithm, and a trend deviation threshold is set (such as ±15% prediction range).

[0130] Fault warning rate: Early warning time: Warning is triggered when the predicted RUL is less than T0 (T0 = 24 hours).

[0131]

[0132] In this embodiment, the model iterative optimization constructs a closed-loop optimization mechanism, including:

[0133] Online learning: 1000 newly collected samples (including 50 faulty samples) will be added to the training set each week.

[0134] Hyperparameter tuning: Using the Bayesian optimization algorithm to search for the optimal parameter combination:

[0135] Set the number of LSTM layers to [1,3]; the number of attention heads to [2,8]; and the learning rate to [1e-4,1e-3].

[0136] This invention predicts the remaining lifespan of the robotic arm using an LSTM-Attention neural network, and combines the predicted remaining lifespan of the robotic arm (RUL, 40%), the dynamically monitored rate of change of hydraulic pressure (ΔP, 30%), and the vibration entropy (S) vibA composite health index (HI) was constructed (30%). A decrease in RUL (Reduced Low Usage Level) indirectly indicates accelerated equipment aging; abnormal fluctuations in ΔP reflect hydraulic system failures; and S (Surveillance Level)... vib Entropy increase indicates mechanical structural abnormalities, and a dual-threshold early warning mechanism is used to achieve risk-based management and control of equipment.

[0137] S3. Establish a degradation-efficiency coupling model, and calculate the robot arm health index and operational efficiency based on the predicted value of the remaining service life of the robot arm;

[0138] The operational efficiency of a robotic arm (such as dredging efficiency) is not constant but dynamically declines with the equipment's health status. The degradation-efficiency coupling model maps the HI value to operational efficiency using an exponential function:

[0139]

[0140] E = E0·e -0.25HI

[0141] Where HI represents the robotic arm health index, E represents operational efficiency, and S represents the robotic arm health index. vib Let the vibration entropy be the vibration spectrum, and then substitute it into the formula. High-frequency features are extracted, and S is further obtained through wavelet packet decomposition or spectral entropy calculation. vib ΔP represents the change in hydraulic pressure, which is obtained by calculating the pressure difference between adjacent time windows (or the rate of pressure change within the sliding window) during preprocessing.

[0142] When HI=0 (optimal health), the performance remains at the initial value E0; when HI=1 (worst health), the performance drops to about 78% of E0, forcing the system to trade off between efficiency and lifespan.

[0143] S4. The DPPO-based reinforcement learning controller achieves dynamic optimization by fusing the robotic arm health index (HI) with a degradation-efficiency coupling model: it senses 10-dimensional state-space data in real time, including the robotic arm health index HI, operational efficiency, energy consumption, and robotic arm joint angles, and dynamically adjusts hydraulic load and vibration gain to match the health state; it balances multiple objectives with operational efficiency (50%), energy consumption (30%), and robotic arm health index (20%) as objective functions, and combines kinematic constraints and low-pass filtering to limit the single adjustment amplitude to ≤5%; it adopts a dual-delay strategy for training, combined with priority playback and TensorRT acceleration, and achieves real-time control through OPCUA.

[0144] a. Robotic Arm Health Index (HI): A core indicator of overall health status;

[0145] b. Job Performance (E): The percentage of current job performance relative to the initial value;

[0146] c. Energy consumption indicators: energy consumption per unit time (such as power of hydraulic system);

[0147] d. Robotic arm joint angles: Real-time angle values ​​of 6 joints, reflecting the motion state.

[0148] The state space data is updated every 5 seconds, the data is input into the DPPO controller, and an adaptive control strategy is generated.

[0149] like Figure 5 As shown, specifically:

[0150] a. State awareness:

[0151] The controller receives a 10-dimensional state vector in real time, including the robotic arm health index (HI), current work efficiency (E(t)), energy consumption index, and the angles of the six joints of the robotic arm, to comprehensively perceive the health, efficiency, and motion status of the equipment.

[0152] Input to the state space: Joint angles are used as one of the parameters of the state space to characterize the real-time motion state of the robotic arm (such as load distribution and motion trajectory).

[0153] b. Action decision-making:

[0154] Output continuous load adjustment commands to dynamically adjust the hydraulic system load (±20% of rated pressure) and vibration suppression gain, ensuring dynamic matching between the load and the health status;

[0155] c. Strategy optimization:

[0156] The controller is driven by a reward function (energy consumption percentage -30%, work efficiency percentage +50%, and robotic arm health index percentage +20%) to balance three objectives: prioritizing efficiency while suppressing excessive energy consumption growth and slowing down equipment degradation.

[0157] The reward function is:

[0158] R = -0.3 × energy consumption + 0.5 × E + 0.2 × HI

[0159] The joint angles of the robotic arm are not directly reflected in the reward function, but they indirectly affect the control strategy in the following ways:

[0160] Implicit effects of strategy optimization: Joint angles are transmitted to the DPPO controller through the state space, affecting the generation of motion space (such as load adjustment commands).

[0161] d. Safety constraints:

[0162] The safety constraint mechanism, through the synergy of the robotic arm joint angles and the Jacobian matrix, maps load adjustment commands to the safe operating range of joint torque, effectively preventing damage to the mechanical structure due to excessive stress. Specifically, the robotic arm joint angles characterize the robotic arm's geometric configuration in real time—such as the degree of bending of each joint and other physical states. This parameter directly determines the torque requirements of each joint. Since different joint angle configurations alter the mechanical properties of the robotic arm (e.g., changes in the torque arm length), the same load adjustment command may trigger significantly different joint torque responses under different configurations. The Jacobian matrix, as a mathematical tool for establishing the mapping relationship between the motion velocity of the robotic arm's end effector and the joint angular velocity, can accurately convert the end-load force vector into the torque vector of each joint through its transpose. By combining real-time acquired joint angle data with the Jacobian matrix model, the system can dynamically calculate the torque safety threshold under the current configuration, ensuring that the execution of load adjustment commands always remains within the safe operating range of the mechanical structure.

[0163] To avoid mechanical damage due to excessive limits, and to smooth commands through a low-pass filter (cutoff frequency 5Hz), the single-step adjustment range is limited to ≤5%.

[0164] e. Training and Deployment:

[0165] The policy network and value network employ a dual-delay update mechanism (50 steps for the policy network and 20 steps for the value network) and prioritize experience replay (focusing on HI drop conditions). The policy gradient update is calculated through generalized advantage estimation, and millisecond-level inference is achieved using the TensorRT acceleration engine. Finally, instructions are sent to the industrial PLC via the OPC UA protocol to complete the control loop. This design overcomes the static limitations of traditional rule-based control. The policy network is responsible for generating dynamic load adjustment instructions (motion space), i.e., selecting the optimal control strategy based on the current state (robotic arm health index, operational efficiency, energy consumption, and joint angles); the value network is responsible for evaluating the value of the current state (i.e., the expected cumulative reward) to optimize the policy network's decisions.

[0166] The S5 and DPPO controllers generate maintenance strategies and performance calculations through multi-dimensional state perception (health index HI, real-time performance E(t), energy consumption, and joint angle). The maintenance measurement and performance calculation in S501 and S502 are performed simultaneously, without any temporal order.

[0167] S501, Maintenance Strategy:

[0168] a. Level 1 warning (HI≥0.8):

[0169] When the robotic arm's health index (HI) reaches or exceeds 0.8, the system automatically triggers the following operations:

[0170] Data logging: Saves snapshots of the current operating conditions, including vibration spectrum, hydraulic pressure curve and temperature trend.

[0171] Manual intervention prompts: Alarms are issued through the human-machine interface (HMI) to prompt operators to conduct on-site inspections or remote diagnostics.

[0172] b. Secondary protection mode (HI≥0.9):

[0173] When HI rises further to 0.9, the system will implement mandatory protection measures:

[0174] Load Limitation: Switch to low load mode to reduce operational efficiency to 60% of the initial value E0 to slow down equipment degradation. Simultaneously, maintenance work orders are automatically generated, associated with fault types (such as bearing wear, hydraulic leakage) and spare parts inventory information, and pushed to the management platform via the cloud.

[0175] c. Intelligent management of maintenance work orders;

[0176] Fault location and spare parts matching: Based on the historical fault database and combined with real-time sensor data, the faulty components (such as pitting in the gearbox) are accurately located, and suitable spare parts are recommended.

[0177] Dynamic priority adjustment: The execution priority of maintenance work orders is dynamically adjusted based on the rate of change of the HI value and the urgency of the task.

[0178] d. Security constraints and command execution guarantees

[0179] The Jacobian matrix maps the command to the joint torque safety range, and a first-order low-pass filter (cutoff frequency 5Hz) is used to smooth the command, limiting the single adjustment amplitude to ≤5% to avoid mechanical shock.

[0180] S502, Performance Calculation:

[0181] a. State awareness and multi-objective optimization

[0182] Input parameters:

[0183] Health Index (HI): A comprehensive reflection of the equipment's degradation status.

[0184] Operational efficiency (E): The ratio of real-time dredging efficiency to initial efficiency.

[0185] Energy consumption index: Power of hydraulic system per unit time (kWh / m) 3 ).

[0186] Joint angles: Real-time configuration parameters of 6 joints.

[0187] After inputting the parameters, the reward function is designed. The reward function R is based on multi-objective optimization principles, quantitatively coupling energy consumption, operational efficiency, and the robotic arm's health index. Specifically, it is defined as:

[0188] This function reflects a priority strategy through weight allocation: the highest weight of 50% is used to ensure operational efficiency and ensure that the system completes the dredging task efficiently; at the same time, the weights of 30% and 20% are used to balance energy consumption suppression and maintaining the health index of the robotic arm, respectively, so as to control energy consumption and delay equipment degradation while improving operational efficiency, and achieve a balanced optimization of performance, energy efficiency and reliability.

[0189] b. Dynamic load regulation and energy consumption optimization

[0190] In the hydraulic load command generation stage, the system relies on the output of the DPPO reinforcement learning controller to dynamically generate load adjustment commands, thereby achieving precise adjustment of the hydraulic system load within ±20% of the rated pressure range. Simultaneously, it optimizes the vibration suppression gain parameters to balance operational efficiency and system stability.

[0191] During industrial-grade deployment, the following technical solutions ensure the efficient transmission and accurate execution of control commands:

[0192] S503 Technical Effect Comparison: End-to-end control delay <100ms, supports 20Hz high-frequency regulation, improving response speed by 2 times compared to traditional PID control (delay >200ms); sudden failure rate reduced by 67% (average 12 failures per year under traditional rule control → reduced to 4 times with this invention), maintenance response time shortened from 2 hours to 15 minutes; through dynamic load adjustment, robotic arm lifespan extended by 41.6% (MTBF 1200h → 1700h), unit operation energy consumption reduced by 23% (2.8 → 2.15kWh / m²). 3 ).

[0193] Compared with existing technologies, this invention solves the three major pain points of traditional methods—maintenance lag, energy efficiency imbalance, and command infeasibility—through a full-link design of "state perception-strategy optimization-safe execution-closed-loop early warning," thereby realizing the engineering implementation of intelligent control for heavy equipment.

[0194] This invention discloses an intelligent control system for a dredging robotic arm based on DPPO reinforcement learning, such as... Figure 6 As shown, it includes:

[0195] Data acquisition and preprocessing module: Real-time acquisition of hydraulic pressure, vibration spectrum and temperature data of the dredging and cleaning robotic arm of the dredging vessel, and preprocessing of the data.

[0196] Model building module: Input the preprocessed data into the LSTM-Attention model, extract features based on the long short-term memory network and attention mechanism, and output the predicted remaining lifespan;

[0197] Degradation-Efficiency Coupled Model Module: Establishes a degradation-efficiency coupled model and calculates the robotic arm health index and efficiency decay coefficient based on the predicted remaining service life;

[0198] Predictive target module: Establish a DPPO reinforcement learning controller, with the robotic arm health index, work efficiency, energy consumption and robotic arm joint angle as the state space, output dynamic load adjustment command action space, optimize the control strategy through reward function, and drive the robotic arm hydraulic actuator after the load adjustment command is processed by dynamic constraints.

[0199] Preferred options also include:

[0200] Maintenance strategy module: used for dynamic health monitoring and early warning, specifically:

[0201] When HI≥0.8, a Level 1 warning is triggered, the current operating condition snapshot is automatically recorded, and manual intervention is prompted for inspection;

[0202] When HI≥0.9, the system is forced to switch to low-load protection mode to reduce operational efficiency and slow down equipment degradation. At the same time, maintenance work orders are pushed through the cloud, which are associated with fault types and spare parts inventory information.

[0203] The implementation process and method of each part of the system of this invention are the same, and will not be repeated here.

[0204] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0205] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for intelligent control of a dredging robotic arm based on DPPO reinforcement learning, characterized in that, include: Real-time acquisition and preprocessing of hydraulic pressure, vibration spectrum, and temperature data of the dredging robotic arm; The preprocessed data is input into the LSTM-Attention model, which performs feature extraction based on the long short-term memory network and attention mechanism, and outputs the predicted value of the remaining service life of the robotic arm. A degradation-efficiency coupling model was established, and the robot arm health index and operational efficiency were calculated based on the predicted value of the remaining service life of the robot arm. A DPPO reinforcement learning controller is established, with the robotic arm health index, work efficiency, energy consumption and robotic arm joint angle as the state space, and the action space of output dynamic load adjustment commands. The control strategy is optimized through the reward function, and the load adjustment commands are driven by the hydraulic actuator of the robotic arm after being processed by dynamic constraints. Preprocessing includes: Missing value handling: Lagrange interpolation is used to fill in missing data; Outlier handling: Outliers are detected using Tukey's Test and error values ​​are removed; Normalization: Min-Max normalization was used to normalize the collected multidimensional parameters; Substitute the preprocessed vibration spectrum into the formula: in, Let be the root mean square value of the vibration velocity at time t, and N be the number of sample points selected for calculation. The instantaneous vibration velocity measurement value at the i-th sampling point; in, The maximum amplitude of the vibration signal at time t in the 1-5 kHz frequency band is given. The spectrum obtained by performing a Fourier transform on the vibration velocity signal v(t) To take the maximum amplitude within the frequency range of 1-5 kHz; The degradation-efficiency coupling model is as follows: in, This indicates the health index of the robotic arm, where E represents operational efficiency. The vibration entropy is obtained through... After extracting high-frequency features, the results are obtained through wavelet packet decomposition or spectral entropy calculation. Indicates changes in hydraulic pressure. RUL represents the initial value of efficiency, and RUL represents the predicted remaining service life of the robotic arm.

2. The intelligent control method for dredging robotic arms based on DPPO reinforcement learning according to claim 1, characterized in that, The preprocessed data is input into the LSTM-Attention model, which extracts features based on a long short-term memory network and attention mechanism, and outputs a predicted value for the remaining lifespan of the robotic arm, including: The input layer receives preprocessed multi-source time-series data, including hydraulic pressure and root mean square values ​​of vibration velocity. Maximum amplitude of vibration and temperature parameters; The system sequentially extracts local temporal features through convolutional layers, performs max pooling through pooling layers, and flattens the three-dimensional tensor output by the pooling layers into one-dimensional vector data. The flattened one-dimensional vector data is input into the LSTM layer. Long-term temporal dependencies are captured through forget gate, input gate, output gate and cell state, and the hidden state sequence is output. Combined with the multi-head parallel attention mechanism, the weights are dynamically allocated to focus on the key degradation stage, and finally the remaining service life prediction value of the robotic arm is generated.

3. The intelligent control method for dredging robotic arms based on DPPO reinforcement learning according to claim 1, characterized in that, The reward function is: Where R is the reward function, The value represents the health index of the robotic arm, and E represents operational efficiency.

4. The intelligent control method for dredging robotic arms based on DPPO reinforcement learning according to claim 1, characterized in that, Training process of the DPPO reinforcement learning controller: The policy network and value network employ a dual-delay update mechanism; In the experience replay pool, priority is given to sampling critical state data where the HI decreases by more than 10%; The policy gradient update amount is calculated using generalized advantage estimation.

5. The intelligent control method for dredging robotic arms based on DPPO reinforcement learning according to claim 1, characterized in that, The load adjustment command, after being processed by dynamic constraints, drives the hydraulic actuator of the robotic arm, including: Abrupt signals are eliminated by using a first-order low-pass filter, limiting the single adjustment amplitude to no more than 5% to prevent the robotic arm from being impacted by sudden changes in load adjustment commands; at the same time, the load adjustment commands are mapped to the joint torque safety range based on the Jacobian matrix to ensure that the movement conforms to mechanical dynamics constraints. The processed load adjustment commands are inferred in milliseconds through the TensorRT acceleration engine and converted into standard industrial signals. These signals are then transmitted to the PLC controller via the OPC UA protocol, driving the hydraulic proportional valve of the robotic arm to dynamically adjust the oil pressure and flow, thereby achieving precise load control.

6. The intelligent control method for a dredging robotic arm based on DPPO reinforcement learning according to claim 1, characterized in that, It also includes dynamic health monitoring and early warning: When HI≥0.8, a Level 1 warning is triggered, the current operating condition snapshot is automatically recorded, and manual intervention is prompted for inspection; When HI≥0.9, the system is forced to switch to low-load protection mode to reduce operational efficiency and slow down equipment degradation. At the same time, maintenance work orders are pushed through the cloud, which are associated with fault types and spare parts inventory information.

7. A dredging robotic arm intelligent control system based on DPPO reinforcement learning, used to implement the dredging robotic arm intelligent control method based on DPPO reinforcement learning as described in any one of claims 1-6, characterized in that, include: Data acquisition and preprocessing module: Real-time acquisition of hydraulic pressure, vibration spectrum and temperature data of the dredging robotic arm, and preprocessing of the data; Model building module: Input the preprocessed data into the LSTM-Attention model, extract features based on the long short-term memory network and attention mechanism, and output the predicted value of the remaining service life of the robotic arm; Degradation-Efficiency Coupled Model Module: Establishes a degradation-efficiency coupled model and calculates the robot arm health index and efficiency decay coefficient based on the predicted value of the remaining service life of the robot arm; Predictive target module: Establish a DPPO reinforcement learning controller, with the robotic arm health index, work efficiency, energy consumption and robotic arm joint angle as the state space, output dynamic load adjustment command action space, optimize the control strategy through reward function, and drive the robotic arm hydraulic actuator after the load adjustment command is processed by dynamic constraints.

8. The intelligent control system for the dredging robotic arm based on DPPO reinforcement learning according to claim 7, characterized in that, Also includes: Maintenance strategy module: used for dynamic health monitoring and early warning, specifically: When HI≥0.8, a Level 1 warning is triggered, the current operating condition snapshot is automatically recorded, and manual intervention is prompted for inspection; When HI≥0.9, the system is forced to switch to low-load protection mode to reduce operational efficiency and slow down equipment degradation. At the same time, maintenance work orders are pushed through the cloud, which are associated with fault types and spare parts inventory information.

Images