Control methods and cleaning robots for can cleaning robots

Abstract

This application provides a control method for a can-cleaning robot and the can-cleaning robot itself. The method includes: collecting multi-source temporal state data during the can-cleaning robot's operation via a sensing and detection module; inputting the multi-source temporal state data into a temporal feature extraction unit, which outputs a temporal feature vector; inputting the temporal feature vector into a reinforcement learning unit, which generates an action vector; a control execution drive module receiving the action vector and driving the mechanical body module to complete the can-cleaning operation; collecting new temporal state data after the can-cleaning operation is executed via the sensing and detection module, calculating the reward value of this action based on a preset reward function, and optimizing the parameters of the reinforcement learning unit using an experience playback mechanism. This control method for the can-cleaning robot can accurately capture changes in temporal operating conditions and autonomously optimize operating parameters, thereby reducing manual intervention and ensuring the safety and reliability of the operation.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a control method for a can cleaning robot and the can cleaning robot itself. Background Technology

[0002] As the core facility for aviation fuel storage, the sludge, impurities, and other contaminants that accumulate inside aviation fuel tanks over a long period of time can directly affect fuel quality. Therefore, regular tank cleaning is a key step in ensuring aviation safety.

[0003] Currently, the mainstream tank cleaning methods in the industry are mainly divided into two categories: manual tank cleaning and traditional mechanical tank cleaning. Manual tank cleaning requires workers to enter the enclosed, flammable, and explosive interior of the tank, which is not only extremely labor-intensive and exposes them to multiple safety risks, but also inefficient, time-consuming, and costly. While traditional mechanical tank cleaning equipment has replaced manual operation to some extent, it lacks autonomous sensing and decision-making capabilities and cannot adapt to the complex changes in the tank's internal conditions in real time. For example, it cannot react promptly to uneven sludge thickness, terrain undulations, and hydraulic system pressure fluctuations. Furthermore, the control precision of traditional mechanical tank cleaning equipment relies on human experience and does not consider the temporal correlation of state parameters during operation, easily leading to incomplete cleaning and equipment jamming, making it difficult to meet the high-efficiency, safe, and intelligent requirements of tank cleaning. Summary of the Invention

[0004] In view of this, this application provides a control method for a tank cleaning robot, which can accurately capture changes in the time sequence of working conditions, autonomously optimize operating parameters, thereby reducing manual intervention and shortening the operation cycle, and ensuring the safety and reliability of the operation.

[0005] In a first aspect, this application provides a control method for a can-cleaning robot, applied to a can-cleaning robot comprising a mechanical body module, a sensing and detection module, an intelligent control module, and an execution drive module. The intelligent control module integrates a temporal feature extraction unit and a reinforcement learning unit. The method includes:

[0006] Multi-source time-series status data during the operation of the cleaning robot is collected by the sensing and detection module. The multi-source time-series status data includes the power system parameters of the cleaning robot.

[0007] The multi-source time-series state data is input into the time-series feature extraction unit, and the time-series feature extraction unit outputs a time-series feature vector characterizing the dynamic operation characteristics of the tank cleaning robot;

[0008] The temporal feature vector is input into the reinforcement learning unit, and the reinforcement learning unit generates an action vector that adapts to the current working condition.

[0009] The execution drive module receives the action vector, converts the action vector into a drive control signal adapted to the mechanical body module, and drives the mechanical body module to complete the tank cleaning operation.

[0010] The sensor detection module collects new time-series state data after the tank cleaning operation is performed, calculates the reward value of this operation based on a preset reward function, and optimizes the parameters of the reinforcement learning unit using an experience playback mechanism.

[0011] Optionally, the temporal feature extraction unit is a long short-term memory network, and the method further includes:

[0012] The power system parameters, motion parameters, and working environment parameters of the cleaning robot are collected by the sensing and detection module as the multi-source time-series state data. The power system parameters include the pressure value and / or flow rate value of the hydraulic system. The motion parameters include at least one of the moving speed of the cleaning robot, the joint angle of the robotic arm, the rotation speed of the end effector, and the rate of change of rotation speed. The working environment parameters include the thickness of the sludge in the cleaning area and / or the cleaning resistance.

[0013] The multi-source temporal state data at the current moment and the hidden state at the previous moment are input into the long short-term memory network. The current moment is the sampling moment corresponding to the current multi-source temporal state data acquisition by the sensing and detection module, and the previous moment is the previous sensing sampling moment immediately adjacent to the current moment.

[0014] The Long Short-Term Memory (LSTM) network is used to filter temporal features, update the memory state, and output the hidden state at the current moment as the temporal feature vector.

[0015] Optionally, the Long Short-Term Memory (LSTM) network has a forgetting gate module, an input gate module, and an output gate module. The step of filtering temporal features through the LTM network, updating the memory state, and outputting the current hidden state as the temporal feature vector includes:

[0016] The forgetting gate module is used to calculate the forgetting gate vector in order to filter effective features from historical memory states.

[0017] The input gate module calculates the input gating vector and candidate memory states to filter the effective features of the currently input temporal state vector and generate candidate update content.

[0018] The memory state at the current moment is calculated by fusing historical valid features and current candidate features through the update module of the Long Short-Term Memory Network.

[0019] The output gate vector is calculated by the output gate module, and the hidden state at the current time is generated based on the memory state at the current time as the temporal feature vector.

[0020] Optionally, calculating the reward value for this action based on a preset reward function includes:

[0021] Parameters are extracted from the new time-series state vector, including the pressure value of the hydraulic system, the flow value of the hydraulic system, and the speed of the end effector.

[0022] Evaluation indicators are calculated based on the extracted parameters. These evaluation indicators include the deviation of the hydraulic system pressure value from the preset optimal pressure, the deviation of the hydraulic system real-time flow rate from the preset optimal flow rate, and the rate of change of the end effector speed.

[0023] The reward value is calculated using a reward function based on the evaluation indicators.

[0024] Optionally, the reinforcement learning unit is implemented based on a deep deterministic policy gradient algorithm, which includes a decision network, a value assessment network, a target decision network, and a target value assessment network. The step of inputting the feature vector into the reinforcement learning unit of the deep learning model to generate control action instructions adapted to the current working condition includes:

[0025] The time-series feature vector is input into the decision network, which performs policy mapping calculation on the feature vector and outputs the action vector used to drive the cleaning robot's operation.

[0026] The temporal feature vector and the action vector are input into the value evaluation network. The value evaluation network evaluates the rationality of the current action vector in adapting to the current working condition and outputs the action value evaluation value. The action value evaluation value is used to assist in the optimization of the parameters of the strategy network and the value evaluation network.

[0027] Through a preset soft update mechanism, the parameters of the decision network and the value assessment network are updated to the target decision network and the target value assessment network at a preset ratio according to a preset time period.

[0028] Optionally, the step of calculating the policy mapping of the feature vector by the decision network and outputting the action vector for driving the tank cleaning robot operation includes:

[0029] The decision network generates an action vector through forward propagation calculation. The action vector includes the end effector speed adjustment and the hydraulic system flow adjustment, and the end effector speed adjustment is limited to a preset safety threshold range.

[0030] Optionally, optimizing the parameters of the reinforcement learning unit using the experience replay mechanism includes:

[0031] The experience data set is stored in the experience replay pool of the reinforcement learning unit, wherein the experience data set includes the multi-source temporal state data, the action vector, the reward value and the new temporal state data;

[0032] When the amount of data in the experience replay pool reaches a preset sampling threshold, a batch of experience data is randomly sampled to update the network parameters of the reinforcement learning unit.

[0033] Optionally, updating the network parameters of the reinforcement learning unit further includes:

[0034] Calculate the target value and the action value output by the value assessment network for each sample;

[0035] The loss value of the value assessment network is calculated using a preset loss function. The parameters of the value assessment network are updated using a backpropagation algorithm until all loss values ​​converge to a preset threshold. The expression for the loss function is: ;

[0036] Where L is the loss value, N is the number of samples in the batch training, t is the ordinal index of the sample in the batch training, and its value is a positive integer from 1 to N, y i Let i be the target value of the i-th sample. Let be the action value assessment value of the i-th sample by the value assessment network.

[0037] Optionally, the step of inputting the temporal feature vector into the reinforcement learning unit, and the reinforcement learning unit generating an action vector adapted to the current working condition, further includes:

[0038] The preset multi-field coupled dynamics model of the cleaning robot is invoked. The multi-field coupled dynamics model integrates the coupling relationship between hydraulic drive characteristics, mechanical transmission law and cleaning load in advance.

[0039] The time-series feature vector is input into the multi-field coupled dynamics model. Through the physical law verification logic built into the model, abnormal dimensions in the time-series feature vector are removed, and an effective time-series feature vector that conforms to the physical operation law is output.

[0040] The reinforcement learning unit is controlled to receive the effective temporal feature vector and generate a preliminary action vector.

[0041] The preliminary motion vector is input into the multi-field coupled dynamics model to verify whether each motion parameter in the preliminary motion vector exceeds the preset operating range of the equipment. The motion parameters that do not meet the constraints are corrected by gradient to obtain a compliant motion vector.

[0042] The action vector that passes the constraint verification will be output as the action vector adapted to the current working condition.

[0043] Secondly, this application provides a tank cleaning robot, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the control method of the tank cleaning robot provided in the embodiments of the present invention.

[0044] The control method for the tank cleaning robot provided in this application collects multi-source time-series state data through a sensor detection module and extracts dynamic operating characteristics from the data through a time-series feature extraction unit. This method can accurately capture real-time changes in working conditions such as the thickness of sludge in the tank, cleaning resistance, and hydraulic system status, providing accurate decision-making basis for the reinforcement learning unit. Furthermore, the reinforcement learning unit generates action vectors adapted to the current working conditions, ensuring that the tank cleaning operation maintains an appropriate operating state under different sludge distributions and working areas. For example, when the sludge is thick, the cleaning torque of the tank cleaning robot is increased and its moving speed is reduced; when the sludge is thin, its traveling efficiency is increased. This effectively improves the operational accuracy and efficiency of the tank cleaning robot. Moreover, by precisely controlling the hydraulic flow and motor speed to reduce energy consumption, energy-saving operations can be effectively achieved, saving on the cleaning costs of aviation fuel storage tanks. Furthermore, this implementation method continuously optimizes the network parameters of the reinforcement learning unit through an experience playback mechanism, making the generation of action vectors more stable and accurate. This avoids safety hazards such as equipment collisions and hydraulic system instability caused by sudden changes in actions, and is particularly suitable for high-risk operating environments such as enclosed oil tanks and flammable and explosive environments, thereby improving operational safety. This method enables autonomous control of cleaning aviation fuel storage tanks, effectively reducing the intensity of manual labor, shortening operation time, and thus effectively preventing operational safety accidents. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the steps of the control method for the tank cleaning robot provided in the embodiments of this application;

[0047] Figure 2 This is a schematic diagram illustrating the working principle of the tank cleaning robot provided in the embodiments of this application;

[0048] Figure 3 This is a schematic diagram illustrating the working principle of a tank cleaning robot according to another embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the logic flow of the control method for the tank cleaning robot provided in the embodiments of this application;

[0050] Figure 5 This is a schematic diagram illustrating the algorithm principle of the control method for the tank cleaning robot provided in this application embodiment;

[0051] Figure 6 This is a schematic diagram of the structure of the tank cleaning robot provided in the embodiments of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0054] In this document, references to "embodiment" or "implementation" mean that a particular feature, structure, or characteristic described in connection with an embodiment or implementation may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0055] Please see Figure 1 , Figure 2 , Figure 3 and Figure 4This application provides a control method for a can-cleaning robot, applied to a can-cleaning robot 100 comprising a mechanical body module, a sensing and detection module, an intelligent control module, and an execution drive module. The intelligent control module integrates a temporal feature extraction unit and a reinforcement learning unit. The method includes the following steps:

[0056] S101. Collect multi-source time-series status data during the operation of the cleaning robot through the sensing and detection module. The multi-source time-series status data includes the power system parameters of the cleaning robot.

[0057] Optionally, the tank cleaning robot 100 includes a mechanical body module, a sensing and detection module, an intelligent control module, an execution and drive module, and a power system module. The mechanical body module includes a walking mechanism, a cleaning execution mechanism, and a posture adjustment mechanism. The walking mechanism adapts to the complex terrain inside the storage tank, and the cleaning execution mechanism can adaptively adjust the cleaning angle according to the thickness of the sludge. The power system module provides power output for the operation of each mechanism of the robot. The sensing and detection modules are distributed between the mechanical body and the power system module to achieve real-time acquisition of multi-dimensional data. The intelligent control module has a built-in processor and a deep learning model. The deep learning model includes the temporal feature extraction unit and the reinforcement learning unit. The intelligent control module can receive sensor data and generate control commands to drive the coordinated operation of each mechanism. In this embodiment, the tank cleaning robot 100 is applicable to tank cleaning operations in industries such as aerospace. The types of sediments it can handle include high-viscosity sludge, asphalt deposits, etc. It is especially suitable for large closed tanks that are difficult for humans to enter, high-risk tanks containing flammable and explosive gases, and tanks with serious high-viscosity sludge deposits. It can effectively solve the problems of low efficiency and poor safety of traditional manual tank cleaning operations.

[0058] Optionally, the power system module of the tank cleaning robot 100 includes a hydraulic system and a drive motor. The hydraulic system consists of a hydraulic pump, a hydraulic motor, a hydraulic valve group, an oil tank, and pipelines. The hydraulic pump is the power source, used to convert mechanical energy into hydraulic energy. The hydraulic motor is the actuator, used to drive the walking mechanism and the cleaning actuator. The hydraulic valve group is used to control the pressure, flow rate, and direction of the hydraulic oil. The drive motor can adjust its output speed and torque in real time according to the load requirements of the hydraulic system, providing a stable and suitable mechanical energy input to the hydraulic pump. In this embodiment, the hydraulic system has advantages such as high output power, strong load adaptability, and stable operation, which can meet the high torque requirements when cleaning high-viscosity sludge. Simultaneously, it has strong impact resistance and can adapt to complex load changes inside the storage tank.

[0059] Optionally, the tank cleaning robot 100 has an autonomous cleaning function, which can autonomously plan a cleaning path according to the distribution of oil sludge inside the tank, and complete the stripping, collection and transportation of oil sludge through the cleaning actuator. The tank cleaning robot 100 also has a state perception function, which uses a sensor detection module to monitor its own operating status and the working environment status in real time, such as power system parameters, mechanism posture, oil sludge thickness and gas concentration inside the tank. The tank cleaning robot 100 also has an intelligent control function, which generates motion vectors adapted to the working conditions based on the collected multi-source time-series state data, and realizes dynamic adjustment of parameters such as motor speed, hydraulic stability, cleaning force and walking speed through the control method described in this application. It also has a fault self-diagnosis function, which automatically stops and alarms when an abnormal state is detected.

[0060] Optionally, the sensing and detection module is used to collect various status data during the operation of the tank cleaning robot 100 in real time. The sensing and detection module consists of various types of sensors, such as power system sensors. The power system sensors may include pressure sensors, flow sensors, and speed sensors, etc.

[0061] Optionally, the pressure sensor is used to monitor key pressure parameters of the hydraulic system in real time, including the output pressure of the hydraulic pump and the inlet and outlet pressures of the hydraulic motor. In this embodiment, by continuously collecting pressure data from the hydraulic system through the pressure sensor, it is possible to determine in real time whether the hydraulic system is within the normal operating pressure range, avoiding excessive pressure that could lead to pipe rupture or damage to hydraulic components, or insufficient cleaning power due to excessive pressure. Furthermore, the pressure data collected by the pressure sensor will serve as the basis for calculating the pressure deviation index in the reward function of the deep learning model, providing data support for the deep learning model to optimize the motor speed control strategy and ensuring stable hydraulic system pressure.

[0062] Optionally, the flow sensor is used to collect real-time flow data of the hydraulic oil in the hydraulic system. In this embodiment, the flow sensor monitors the flow changes in the hydraulic system to determine whether the hydraulic valve group control is effective and whether there is any blockage in the pipeline. The collected flow data serves as the basis for calculating the flow deviation index in the reward function of the deep learning model, assisting the deep learning model in adjusting the motor speed and the state of the hydraulic valve group to ensure that the flow is stable within a preset optimal range. This avoids excessive flow leading to energy waste and increased oil temperature, or insufficient flow causing sluggish movement and cleaning actions of the cleaning robot 100.

[0063] Optionally, the speed sensor is used to monitor the real-time speed of the drive motor and the output speed of the hydraulic motor. The speed sensor can provide feedback on the execution effect of the motor speed control strategy, ensuring that the drive motor speed remains stable within the optimal range, providing a smooth mechanical energy input to the hydraulic pump. Furthermore, the speed data collected by the speed sensor is used to calculate the speed stability index in the reward function, guiding the deep learning model to output smooth speed adjustment commands, avoiding sudden increases or decreases in hydraulic system pressure and flow caused by sudden speed changes, and reducing wear on the mechanical transmission mechanism.

[0064] Optionally, the multi-source time-series status data can be understood as tank cleaning operation status data collected by the sensing and detection module from multiple different data sources and continuously recorded in time sequence.

[0065] It should be noted that the smooth output of the hydraulic system is crucial for the operation of the tank cleaning robot 100. Specifically, a stable power output from the hydraulic system can prevent sludge splashing caused by vibration of the cleaning mechanism, preventing contamination of uncleaned areas and aviation fuel residue inside the tank. It also prevents sudden pressure spikes and drops in the hydraulic system from impacting the tank wall. Furthermore, the stable operation of the hydraulic system reduces wear on hydraulic components and extends the continuous operating time of the equipment. Therefore, in this embodiment, the power system parameters are collected in real time through a sensing module. The core parameters collected include the real-time pressure and flow rate of the hydraulic system. After preprocessing, the collected parameters will serve as the core input data for the subsequent deep learning model, providing data support for the deep learning model to analyze the hydraulic system's operating patterns, predict load change trends, and generate precise power adjustment commands.

[0066] Optionally, the preprocessing includes denoising, normalization, and outlier removal to form standardized multi-source time-series state data, avoiding interference from noise in the original data on subsequent model processing.

[0067] S102. Input the multi-source time-series state data into the time-series feature extraction unit, and output a time-series feature vector characterizing the dynamic operation characteristics of the tank cleaning robot through the time-series feature extraction unit.

[0068] Optionally, the temporal feature extraction unit is used to extract key feature vectors characterizing the dynamic operating characteristics of the cleaning system from multi-source temporal state data, capable of capturing the patterns and correlations of data changes over time. Specifically, the temporal feature extraction unit preferably employs a Long Short-Term Memory (LSTM) network. The LSTM network has the advantage of capturing long-term temporal dependencies and can accurately mine dynamic correlation features in multi-source temporal state data. In specific implementation, the input to the LSTM is the preprocessed multi-source temporal state data sequence and the hidden state of the previous time step. Through the gating control mechanism of the forget gate module, input gate module, and output gate module, the cell states within the network are updated, and finally, a temporal feature vector is output. This temporal feature vector can characterize the dynamic operating characteristics of the cleaning robot 100 at the current time step.

[0069] S103. Input the temporal feature vector into the reinforcement learning unit, and generate an action vector that adapts to the current working condition through the reinforcement learning unit.

[0070] Optionally, the reinforcement learning unit receives the temporal feature vector output by the temporal feature extraction unit, learns the optimal control strategy through a reinforcement learning algorithm, and generates control action instructions adapted to the current working conditions.

[0071] Optionally, the reinforcement learning unit preferably employs the Deep Deterministic Policy Gradient (DDPG) algorithm, which is suitable for control scenarios in continuous action spaces and can generate smooth and precise control action commands. After receiving the temporal feature vector output by the temporal feature extraction unit, the reinforcement learning unit performs forward propagation calculation through the decision network to output the action vector.

[0072] S104. Control the execution drive module to receive the action vector, convert the action vector into a drive control signal adapted to the mechanical body module, and drive the mechanical body module to complete the tank cleaning operation.

[0073] Optionally, the execution drive module can be understood as a functional module in the cleaning robot 100 used for outputting power drive. It integrates a signal receiving unit, a signal conversion unit, and a power output unit, and can be adapted to various drive methods such as hydraulic drive and motor drive. It establishes a control link with both the intelligent control module and the mechanical body module.

[0074] Optionally, the motion vector can be understood as a set of standardized digital control parameters output by the reinforcement learning unit, representing an abstract control command adapted to the current working condition. Its components include, but are not limited to, motion parameters such as motor speed adjustment, hydraulic flow adjustment, robotic arm joint angle adjustment, and end effector motion parameters. All motion parameters are within a preset safety threshold range to ensure the stable operation of the tank cleaning robot 100.

[0075] Optionally, the mechanical body module can be understood as the physical working carrier of the tank cleaning robot 100, integrating various execution components required for hydraulic tank cleaning operations, including a magnetic or negative pressure wall-climbing mobile chassis, a multi-degree-of-freedom robotic arm, an end-effector high-pressure hydraulic cleaning component, and auxiliary operating components. It possesses the ability to perform wall-climbing movement and hydraulic cleaning operations in complex environments on the inner wall of the storage tank. The magnetic or negative pressure wall-climbing mobile chassis is adapted to the adsorption requirements of the storage tank wall, enabling stable movement within the tank. The multi-degree-of-freedom robotic arm is equipped with an end-effector high-pressure hydraulic cleaning component, which includes a high-pressure hydraulic cleaning spray gun and matching scraping components. It completes the cleaning of oil sludge from the inner wall of the storage tank through a combination of high-pressure water jets and mechanical scraping. The auxiliary operating components include wall adsorption aids, high-pressure water transmission components, and sealing and protective components for the working area, respectively used to enhance wall adsorption stability, ensure the transmission of high-pressure cleaning water, and reduce irregular splashing of cleaning waste liquid.

[0076] Optionally, the drive control signal can be understood as a concrete control signal generated by the execution drive module after converting the motion vector, which is adapted to the hardware characteristics of each execution component of the mechanical body module. It includes analog voltage signals, pulse signals, or servo control signals, etc., and can directly drive the execution components to run.

[0077] In one possible embodiment of this application, the drive control signal is transmitted to the drive motor of the mobile chassis. By adjusting the motor speed and direction, the robot can perform actions such as forward movement, backward movement, turning, and speed adjustment within the tank, adapting to the position switching requirements of different cleaning areas. In another possible embodiment of this application, the drive control signal is transmitted to the drive motor or hydraulic motor of the end effector, adjusting the speed or torque of the actuator to achieve targeted cleaning of sludge of different thicknesses. In yet another possible embodiment of this application, the drive control signal is transmitted to the joint drive units of the multi-degree-of-freedom robotic arm, adjusting the joint angles and postures to drive the end effector to perform actions such as lifting, pitching, and swinging, adapting to the cleaning needs of different locations within the tank and ensuring thorough cleaning. In yet another possible embodiment of this application, for special operating scenarios, the drive control signal drives the actions of auxiliary execution components, such as controlling the valve switch of the waste liquid recovery device and adjusting the angle of the explosion-proof lighting equipment, ensuring the continuity and safety of the operation process.

[0078] S105. Collect new time-series state data after the tank cleaning operation is executed through the sensor detection module, calculate the reward value of this operation based on the preset reward function, and optimize the parameters of the reinforcement learning unit using the experience playback mechanism.

[0079] Optionally, the new time-series state data can be understood as a set of time-series data collected by the sensing and detection module after the cleaning robot 100 performs the cleaning operation, which represents the latest state of the robot and the working environment. It is feedback data reflecting the effect of this action execution. Its parameter type is consistent with the multi-source time-series state data, but the data value is the real-time state value after the action is executed.

[0080] Optionally, the reward function is a quantitative evaluation function preset in the intelligent control module, which can evaluate the degree of adaptation of the current action vector to the current working condition based on the new time-series state data, output a quantitative reward value, and provide directional guidance for the parameter optimization of the reinforcement learning unit. Its function form and weight coefficient can be dynamically adjusted according to the tank cleaning operation target.

[0081] Optionally, the reward value can be understood as an evaluation index calculated based on the new time-series state data, used to measure the adaptability and operational effectiveness of the control action, and to guide the reinforcement learning unit to learn the optimal strategy.

[0082] In one possible embodiment of this application, the reward value is calculated using a reward function. This reward function calculates a comprehensive reward value by weighting reward items from three core dimensions: pressure deviation, flow deviation, and speed stability. Specifically, the pressure deviation refers to the difference between the real-time pressure of the hydraulic system and the preset optimal pressure during the operation of the cleaning robot 100. The flow deviation refers to the difference between the real-time flow rate of the hydraulic system and the preset optimal flow rate. Speed ​​stability is evaluated using the rate of change of the drive motor speed as an indicator; that is, the ratio of the difference in drive motor speed between two adjacent data collection points to the time interval, reflecting the smoothness of the drive motor speed.

[0083] Optionally, the experience replay mechanism can be understood as a mechanism for storing state, action, reward and new state data generated during the job, and optimizing the parameters of the deep learning model by randomly sampling batch data, which can break the temporal correlation of data and improve the stability of model training.

[0084] Optionally, the parameters of the reinforcement learning unit can be understood as the core optimizable parameters of the reinforcement learning unit, including the weight matrix and bias vector of each network layer.

[0085] Please refer to it again. Figure 4 . Figure 4This illustration shows a logical flow diagram of the control method for the tank cleaning robot provided in this application embodiment. The process starts with the upper-level robot operation execution link, where the operator manually triggers commands. These commands sequentially pass through motor speed judgment, hydraulic pump speed adjustment, robot drive wheel control, and robot path and posture adjustment stages, transforming the control commands into the actual operation actions of the tank cleaning robot. Simultaneously, a pressure and flow measurement system in the end-effector hydraulic pipeline collects state data during the operation, storing the experience data set, including the current state, executed action, new state after the action, and corresponding reward, in an experience playback pool. The core of the process is the network module of the intermediate reinforcement learning unit, employing a dual-network architecture of a policy network and a value network using a deep deterministic policy gradient algorithm, comprising a current network and a target network. The decision network receives the current state and outputs an adapted action, while the value evaluation network receives the state and action and outputs a value evaluation value. The target decision network and target value evaluation network process the new state and new action and output associated evaluation values, while a soft update mechanism synchronizes the parameters of the current network and the target network. The lower layer of the process is the network parameter optimization link. It optimizes the action generation strategy based on the value assessment results by using the policy gradient. It calculates the deviation through the loss function and backpropagates to update the value assessment network parameters. The optimized parameters are fed back to the motor speed judgment link of the operation execution link in the form of weight coefficients. Finally, a closed loop of operation execution, data collection, experience storage, network optimization and instruction correction is formed, realizing the autonomous updating and optimization of the cleaning robot's operation strategy.

[0086] In one specific embodiment of this application, the work scenario is an oil tank to be cleaned. The cleaning robot 100 is required to efficiently complete the oil sludge cleaning operation while ensuring safety. The cleaning robot 100 is equipped with sensing modules such as pressure sensors, flow sensors, speed sensors, oil sludge thickness sensors, and combustible gas sensors. The specific implementation process of the cleaning robot 100 is as follows: First, the sensing module collects multi-source time-series state data, including real-time hydraulic system pressure, hydraulic flow, end effector speed, oil sludge thickness, and combustible gas concentration. After noise reduction and normalization preprocessing, a standardized data sequence is formed. Then, the standardized data sequence is input into a time-series feature extraction unit, which extracts features from the data sequence and outputs a time-series feature vector. This time-series feature vector represents the dynamic operating state of the current oil tank cleaning operation. Second, a reinforcement learning unit receives the time-series feature vector and, through its decision network, calculates and outputs an action vector. This action vector includes motor speed adjustment and hydraulic flow adjustment, and is within a safe threshold range. Subsequently, the motion vector is transmitted to the execution drive module, which drives the end-effector cleaning device to accelerate and the hydraulic system to increase flow. The cleaning robot 100 executes the sludge cleaning operation according to the instructions, while the sensing and detection module synchronously collects new time-series state data. Finally, the reward value is calculated, and the state, motion, reward, and new state data generated during the operation are stored in the experience playback buffer. When the amount of data in the buffer reaches a preset threshold, the sampled data updates the parameters of the deep learning model, completing one model optimization. After multiple iterations of optimization, the deep learning model can accurately generate control motion instructions adapted to different sludge thicknesses and different working environments, thereby improving the cleaning efficiency of the cleaning robot 100 and reducing the fluctuation range of hydraulic system pressure and flow, thus improving the safety and stability of the operation.

[0087] In summary, the control method for the tank cleaning robot 100 provided in this embodiment collects multi-source time-series state data through a sensing and detection module, and combines this with a time-series feature extraction unit to mine the dynamic operating characteristics in the data. This allows for the accurate capture of real-time changes in working conditions such as the thickness of sludge in the oil tank, cleaning resistance, and hydraulic system status, providing precise decision-making basis for the reinforcement learning unit. Furthermore, by generating action vectors adapted to the current working conditions through the reinforcement learning unit, the cleaning operation can maintain an appropriate operating state under different sludge distributions and different operating areas. For example, when the sludge is thick, the cleaning torque of the tank cleaning robot 100 can be increased and its moving speed reduced; when the sludge is thin, its travel efficiency can be improved. This effectively improves the operating accuracy and efficiency of the tank cleaning robot 100. Moreover, by precisely controlling the hydraulic flow and motor speed to reduce energy consumption, energy-saving operations can be effectively achieved, saving on the cleaning costs of aviation fuel storage tanks. Furthermore, this implementation method continuously optimizes the network parameters of the reinforcement learning unit through an experience playback mechanism, making the generation of action vectors more stable and accurate. This avoids safety hazards such as equipment collisions and hydraulic system instability caused by sudden changes in actions, and is particularly suitable for high-risk operating environments such as enclosed oil tanks and flammable and explosive environments, thereby improving operational safety. This method enables autonomous control of cleaning aviation fuel storage tanks, effectively reducing the intensity of manual labor, shortening operation time, and thus effectively preventing operational safety accidents.

[0088] Optionally, the step of collecting multi-source time-series state data during the operation of the cleaning robot 100 through the sensing and detection module includes collecting power system parameters, motion parameters, and operating environment parameters of the cleaning robot 100 through the sensing and detection module as the multi-source time-series state data. The power system parameters include the pressure value and / or flow rate value of the hydraulic system. The motion parameters include at least one of the moving speed of the cleaning robot 100, the joint angle of the robotic arm, the rotation speed of the end effector, and the rate of change of rotation speed. The operating environment parameters include the thickness of the sludge in the cleaning area and / or the cleaning resistance.

[0089] Optionally, the power system parameters are used to characterize the operating status of the power drive unit of the tank cleaning robot 100. The core parameters include the pressure and flow rates of the hydraulic system, used to determine whether the power system is operating stably. Specifically, the hydraulic system pressure reflects the hydraulic drive load, and the hydraulic system flow rate reflects the hydraulic power output efficiency.

[0090] Optionally, the motion parameters can be understood as a set of parameters characterizing the motion state of the cleaning robot 100, including but not limited to the overall movement of the cleaning robot 100, the posture of the robotic arm, and the operation of the end effector. Specifically, they can include movement speed, robotic arm joint angles, end effector rotation speed and rotation speed change rate, which can be correlated with the accuracy and effectiveness of the operation. Among them, the movement speed can reflect the travel state of the cleaning robot 100, the robotic arm joint angles can reflect the adaptability of the cleaning posture, and the end effector rotation speed and rotation speed change rate can reflect the cleaning intensity and dynamic adjustment characteristics.

[0091] Optionally, the operating environment parameters can be understood as parameters characterizing the on-site environment and operating load of the tank cleaning operation, including the sludge thickness and cleaning resistance of the cleaning area, which can be used to determine the current operating conditions and as a basis for adjusting the operating strategy. Specifically, the sludge thickness of the cleaning area reflects the difficulty of the operation and the degree of completion of the cleaning target, while the cleaning resistance reflects the load intensity of the end effector.

[0092] In this embodiment, by collecting parameters from multiple sources, covering core dimensions such as power, motion, and environment, the dynamic working conditions of the cleaning robot 100 can be characterized more completely, avoiding misjudgment of working conditions caused by collecting single parameters, providing comprehensive data support for subsequent intelligent decision-making, and improving the accuracy of decision-making.

[0093] Optionally, the temporal feature extraction unit is a long short-term memory network. The step of inputting the multi-source temporal state data into the temporal feature extraction unit and outputting a temporal feature vector characterizing the dynamic operating characteristics of the tank cleaning robot 100 through the temporal feature extraction unit includes:

[0094] S1021. Input the multi-source temporal state data at the current moment and the hidden state at the previous moment into the long short-term memory network, wherein the current moment is the sampling moment corresponding to the current multi-source temporal state data acquisition by the sensing and detection module, and the previous moment is the previous sensing sampling moment immediately adjacent to the current moment.

[0095] Optionally, the Long Short-Term Memory Network (LSTM) can be understood as a recurrent neural network with long short-term memory capabilities. By regulating the retention and updating of historical information through the gating mechanism of the forget gate module, input gate module, and output gate module, it can effectively solve the gradient vanishing problem of traditional recurrent neural networks and is suitable for long-term dependency feature mining of multi-source time-series state data in tank cleaning operations.

[0096] Optionally, the current moment can be understood as the time node when the sensing and detection module completes this multi-source temporal state data acquisition, denoted as t, corresponding to the acquired single frame of multi-source temporal state data. The timestamp of the current moment is bound to the acquired data one-to-one, serving as the time reference for temporal feature extraction.

[0097] Optionally, the "previous time" can be understood as the time node immediately preceding the current time of the previous sensing sampling, denoted as t-1, and the time interval between it and the current time is equal to the preset sensing sampling period. The hidden state of the previous time can be understood as a carrier of historical feature information retained by the Long Short-Term Memory network.

[0098] Optionally, the hidden state can be understood as a vector, denoted as h, used in a Long Short-Term Memory (LSTM) network to store historical feature information of time-series data, and its dimension is preset by the network structure. Also, the hidden state at the previous time step is denoted as h. t-1 This is used to carry historical temporal features. The hidden state h at the current moment. t The output result after fusing historical features with current data features is the final time-series feature vector.

[0099] S1022. The temporal features are filtered through the Long Short-Term Memory network, the memory state is updated, and the hidden state at the current moment is output as the temporal feature vector.

[0100] Optionally, the memory state can be understood as the core vector inside the Long Short-Term Memory (LSTM) network used for long-term storage of temporal information, denoted as c. It achieves information filtering and updating through a gating mechanism, and is not directly output, but only provides data support for the generation of hidden states. It is the key to the LSM network realizing long-term memory function.

[0101] Optionally, the temporal feature vector is composed of the hidden state h at the current time. t The feature set serves as a high-dimensional feature set characterizing the dynamic operation characteristics of the cleaning robot 100. It integrates the temporal correlation characteristics of multi-source time-series state data, such as the temporal pattern of hydraulic pressure change with sludge thickness and the dynamic coupling relationship between end effector speed and cleaning resistance, and can be directly input into the reinforcement learning unit.

[0102] In one optional embodiment of this application, for the dynamic working conditions of the tank cleaning robot 100, the Long Short-Term Memory (LSTM) network achieves adaptive feature selection through dynamic adjustment of the gating coefficient. For example, when the working conditions are stable, the forgetting gating coefficient approaches 1, retaining more historical features to reduce interference from current small fluctuations in data; when the working conditions change abruptly, the forgetting gating coefficient approaches 0, and the input gating coefficient approaches 1, thereby quickly forgetting old historical features and focusing on retaining the features of the current abruptly changed working conditions, ensuring that the temporal feature vector can accurately match the real-time working conditions.

[0103] In one optional embodiment of this application, the temporal feature extraction unit further includes an anomaly handling mechanism. For example, if the data missing rate at the current moment is greater than or equal to a preset threshold, the current feature extraction is paused, the temporal feature vector from the previous moment is output as a temporary input, and a fault signal is fed back to the sensor detection module. As another example, if the gating coefficient calculation overflows, the network parameters are reset, and the calculation is recalculated using preset initial parameters. Furthermore, if the transmission with the reinforcement learning unit is interrupted, the temporal feature vector is cached locally and transmitted preferentially after communication is restored to avoid data loss in the decision-making process. In this embodiment, the anomaly handling mechanism can effectively cope with sudden problems such as data loss, computational overflow, and communication interruption, avoiding work stoppage caused by feature extraction interruption, improving the operational stability of the intelligent control module, adapting to the long-term continuous operation requirements in the high-risk environment of sealed oil tanks, and ensuring stable operation.

[0104] In this embodiment, the gating mechanism of the Long Short-Term Memory (LSTM) network can effectively capture the long-term dependencies of multi-source temporal state data, such as the temporal coupling law between hydraulic pressure changes and sludge thickness changes. This overcomes the shortcomings of traditional feature extraction methods in handling temporally correlated data, effectively improving the accuracy of the output temporal feature vector in representing the dynamic operating characteristics of the tank cleaning robot 100, and laying the foundation for accurate decision-making by the subsequent reinforcement learning unit. Furthermore, the LSM network, through dynamic adjustment of the gating coefficient, can adapt to both stable and abrupt changes in tank cleaning operations. Under stable conditions, it retains historical features to reduce interference; under abrupt changes, it quickly updates features to adapt to new conditions, avoiding feature distortion caused by changes in conditions, and ensuring that temporal feature extraction continuously matches real-time operational requirements. In addition, the LSM network, through its gating mechanism, can filter out invalid and redundant data. Compared to directly inputting the original multi-source temporal data into the reinforcement learning unit, this effectively reduces the computational load of the reinforcement learning unit, thereby effectively improving the decision-making efficiency of action vector generation.

[0105] Please see Figure 5 The Long Short-Term Memory (LSTM) network has a forgetting gate module, an input gate module, and an output gate module. The process of filtering temporal features through the LTM network, updating the memory state, and outputting the current hidden state as the temporal feature vector includes the following steps:

[0106] S102a, Calculate the forgetting gate vector δ using the forgetting gate module. f To filter effective features in historical memory states, where δ f =σ(W f ×[h t-1 ,s t ]+b f ).

[0107] Optionally, the forgetting gate module can be understood as a functional module in a long short-term memory network responsible for filtering historical memory information, by calculating the forgetting gate vector δ. f It enables the filtering of historical memory states, retaining effective historical features related to the current tank cleaning operation, and filtering out redundant or interfering historical information.

[0108] S102b, Calculate the input gate vector δ using the input gate module. i and candidate memory state c' t To filter the effective features of the currently input temporal state vector and generate candidate update content, where δ i =σ(W i ×[h t-1 ,s t ]+b i ), c' t =tanh(W c ×[h t-1 s t ]+b c ).

[0109] Optionally, the input gate module can be understood as a functional module in a long short-term memory network responsible for filtering the current input information and generating updated candidate content, and outputting an input gate vector δ. i and candidate memory state c' t This enables accurate feature extraction from current multi-source time-series state data. The input gating vector δ... i The candidate memory state c' is used to filter currently valid features. t It can be understood as a candidate update carrier that carries the currently valid features.

[0110] S102c: The update module of the Long Short-Term Memory network fuses historical valid features with current candidate features to calculate the current memory state c. t , where c t =δ f ×c t-1 +δ i ×c' t .

[0111] Optionally, the update module can be understood as a functional module in a long short-term memory network that integrates historical features with current features. Based on the historical memory states filtered by the forgetting gate module and the candidate memory states generated by the input gate module, the current memory state c is obtained by updating it element-wise. t This completes the inheritance and updating of time sequence information.

[0112] S102d, Calculate the output gating vector δ through the output gate module. oAnd generate the hidden state h at the current time based on the memory state at the current time. t As the time-series feature vector, where δ o =σ(W o ×[h t-1 , s t ]+b o ), h t =δ o ×tanh(c t ).

[0113] Optionally, the output gate module can be understood as a functional module in a long short-term memory network responsible for controlling the output from the memory state to the hidden state, by calculating the output gate vector δ. o By filtering the core effective features in the current memory state, a hidden state h representing the dynamic operating characteristics of the tank cleaning robot 100 is finally generated. t That is, to generate the time-series feature vector.

[0114] Among them, s t h represents the time-series state data at the current moment. t-1 and c t-1 These are the hidden state and the memory state from the previous time step, respectively, h t and c t δ represents the output hidden state and memory state at the current moment. f δ i δ o Let be the gate vectors for the forget gate, input gate, and output gate, respectively; σ be the activation function; tanh be the hyperbolic tangent activation function; and W be the gate vector for the output gate. f W i W c W o b is the weight matrix for the corresponding network layer. f b i b c b o This is the bias vector for the corresponding network layer.

[0115] Optionally, the gate vector δ f δ i δ o It can be understood as a standardized vector output by the corresponding gating module, with element values ​​ranging from 0 to 1. Its core function is to act as an information filtering switch. That is, the closer the element value is to 1, the more information in the corresponding dimension needs to be retained, input, or output. The closer the element value is to 0, the more information in the corresponding dimension needs to be forgotten or filtered.

[0116] Optionally, the activation function σ is formulated as σ(x) = 1 / (1 + e^x). -The core function of the gated module is to map the calculation results of the gated module to the 0-1 interval, generate a standardized gated vector, and realize the quantitative control of information filtering.

[0117] Optionally, the formula for the hyperbolic tangent activation function tanh is tanh(x) = (eˣ - eˣ). - ˣ) / (eˣ+e - ˣ), with a value range of -1 to 1, its core function is to evaluate candidate memory states c' t and the current memory state c t Normalization is performed to enhance the stability of network training, while highlighting the positive and negative correlations of features. For example, features that are positively correlated with cleaning efficiency are given positive values, and those that are negatively correlated are given negative values.

[0118] Optionally, the weight matrix W f W i W c W o The learnable parameter matrix corresponding to the gating module or update module is used to hide the previous state h of the input. t-1 and the current time-series state data s t The fusion vector undergoes a linear transformation to adjust the weight proportions of different input information; its dimension is determined by h. t-1 and s t The dimensions together determine this.

[0119] Optionally, the bias vector b f b i b c b o The learnable bias parameter vector corresponding to the gating module or update module is used to compensate for the offset after linear transformation, thereby improving the network's ability to adapt to complex tank cleaning operations. The dimension is consistent with the output dimension of the corresponding weight matrix.

[0120] Please refer to it again. Figure 5 . Figure 5 This paper illustrates the algorithm principle of the control method for the cleaning robot provided in the embodiments of this application, and presents a schematic diagram of the core network architecture for temporal feature extraction and reinforcement learning decision-making of the cleaning robot 100. The diagram shows the core process of the temporal feature extraction unit processing multi-source temporal state data and the reinforcement learning unit generating action vectors in the control method: The left side of the diagram shows the long short-term memory network temporal feature extraction unit, which receives multi-source temporal state data during the operation of the cleaning robot 100 and combines it with the hidden state h from the previous time step. t-1 With memory state c t-1 The system utilizes the built-in gating module and tanh activation function of the Long Short-Term Memory (LSTM) network to filter historical features and fuse them with current features, thereby updating the memory state c at the current time step. tAt the same time, output the current hidden state h. t , this h t This refers to the temporal feature vector characterizing the robot's dynamic operational characteristics. Also, the right side of the figure shows a reinforcement learning unit based on deep deterministic policy gradients, comprising a dual architecture of a value network and a policy network. The policy network receives the temporal feature vector h output by the long short-term memory network. t The neural network generates an action vector a that is adapted to the current working condition. t Simultaneously, a target policy network is set up to support subsequent soft parameter updates, while the value network receives the temporal feature vector h. t The action vector a output by the policy network t The algorithm calculates and outputs an action value assessment value Q to determine the rationality of the action's adaptation to the working conditions, and also includes a target value network. The overall process realizes a closed loop of multi-source temporal state data of the cleaning robot 100, temporal feature extraction through the long short-term memory network, input of temporal feature vectors into the reinforcement learning unit, generation of action vectors by the policy network, and completion of value assessment by the value network, providing support for the operation action decision-making and strategy optimization of the cleaning robot 100.

[0121] In one optional embodiment of this application, the gating vectors of the forget gate module, the input gate module, and the output gate module are all calculated and generated based on the fusion information of historical hidden states and current time-series data, enabling real-time response to dynamic changes in the tank cleaning operation conditions. Specifically, when the operating conditions are stable, δ f Approaching 1, δ i Approaching 0, historical stable characteristics are preferentially preserved to reduce small fluctuations; when operating conditions change abruptly, δ f Approaching 0, δ i Approaching 1, thus quickly forgetting old historical features and focusing on capturing current mutation features; when in a security warning scenario, δ o The screening weights of safety-related features can be strengthened in a targeted manner, so that the extraction of time-series features can continuously match the real-time operation requirements and dynamically adapt to changes in operation conditions.

[0122] In this embodiment, the forgetting gate module, input gate module, and output gate module of the Long Short-Term Memory (LSTM) network enable precise filtering of temporal features. The forgetting gate module accurately retains effective features from historical memory, such as the timing pattern of hydraulic pressure under stable operating conditions. The input gate module accurately extracts core features from the current temporal state data, such as abrupt changes in sludge thickness. The output gate further filters key features from the memory state, ensuring that the final output temporal feature vector accurately represents the dynamic operating characteristics of the tank cleaning robot 100. Furthermore, through the operation of the update module, precise fusion of historical effective features and current candidate features is achieved. Historical features ensure the continuity of temporal features, enabling feature extraction to connect with the operating conditions of previous operations. Current features ensure the real-time nature of features, allowing features to quickly adapt to new operating conditions. The LSM network effectively mines long-term dependencies in multi-source temporal data, such as the temporal coupling pattern between hydraulic pressure changes and sludge thickness changes, allowing the temporal feature vector to more comprehensively reflect the dynamic operation process of the tank cleaning robot 100.

[0123] Optionally, calculating the reward value for this action based on a preset reward function includes the following steps:

[0124] S1051. Extract parameters from the new timing state vector, the parameters including the pressure value of the hydraulic system, the flow value of the hydraulic system, and the rotational speed of the end effector.

[0125] Optionally, the preset reward function can be understood as a quantitative evaluation function integrated into the intelligent control module of the tank cleaning robot 100. Its core function is to quantitatively judge the adaptability of the output action vector of the reinforcement learning unit by calculating the deviation of key parameters of the tank cleaning operation, and to provide directional guidance for the optimization of the parameters of the reinforcement learning unit. The weight coefficient of the preset reward function can be dynamically adjusted according to the tank cleaning operation objective, such as hydraulic stability and high operational efficiency.

[0126] Optionally, the new temporal state vector is a temporal state data vector collected and preprocessed by the sensing and detection module after the cleaning robot 100 performs the cleaning operation. It includes multi-dimensional parameters such as hydraulic system, end effector, and working environment. It is feedback data reflecting the effect of this action and provides the original parameter source for reward value calculation.

[0127] S1052. Calculate evaluation indicators based on the extracted parameters. The evaluation indicators include the deviation ΔP between the pressure value of the hydraulic system and the preset optimal pressure, the deviation ΔQ between the real-time flow rate of the hydraulic system and the preset optimal flow rate, and the rate of change of the end effector speed Δn / Δt.

[0128] Optionally, the evaluation index can be understood as a quantitative deviation index designed for the operating status of the core execution components of the tank cleaning robot 100, used to accurately characterize the impact of motion execution on the robot's operational stability and job adaptability. For example, ΔP is used to reflect the degree to which the hydraulic system pressure deviates from the optimal working condition, ΔQ is used to reflect the adaptability of the hydraulic flow, and Δn / Δt is used to reflect the dynamic change range of the end effector's rotational speed.

[0129] S1053. Based on the evaluation index, the reward value is calculated using a reward function, which is R = ω1 × |ΔP| + ω2 × |ΔQ| + ω3 × |Δn / Δt|. Where R is the reward value, ω1 to ω3 are weighting coefficients, and ω1 + ω2 + ω3 = 1.

[0130] Optionally, the weighting coefficients can be understood as coefficients in a preset reward function used to adjust the importance of each evaluation indicator, and their values ​​are set based on the priority of tank cleaning operations. For example, since the hydraulic system is the power core of the tank cleaning robot 100, the weights of ω1 and ω2 can be higher than ω3 to ensure that the stability of the hydraulic system is prioritized and to meet the requirements of equipment operation stability in the high-risk working environment of oil tanks.

[0131] Optionally, the reward value is a quantitative result calculated by the reward function and is positively correlated with the evaluation index deviation value. In other words, the smaller the deviation, the smaller the reward value, indicating better action adaptability; the larger the deviation, the larger the reward value, indicating that the action needs optimization.

[0132] In this embodiment, by focusing on the key power output and execution units of the tank cleaning robot 100 during operation—namely, the hydraulic system and the end effector—core parameters such as hydraulic pressure, hydraulic flow rate, and end effector speed are extracted as the basis for evaluation. The hydraulic pressure, hydraulic flow rate, and end effector speed reflect the power stability, operational effectiveness, and equipment safety of the tank cleaning robot 100. Simultaneously, by calculating quantitative evaluation indicators such as pressure deviation ΔP, flow rate deviation ΔQ, and speed change rate Δn / Δt, the quality of the robot's dynamic operating state is transformed into directly calculable numerical deviations, ensuring a unified evaluation standard for action effects under different operating conditions and providing reliable data for optimizing reinforcement learning unit parameters. Furthermore, by setting weight coefficients ω1, ω2, and ω3, the importance of each evaluation indicator can be flexibly adjusted according to the priority requirements of the tank cleaning operation. For example, in high-risk tank operation environments, the stability of the hydraulic system directly affects operational safety; therefore, the weight ratio of ω1 and ω2 can be increased to prioritize the stability of hydraulic pressure and flow rate. For example, in the scenario of cleaning thick sludge, the stability of the end effector's rotational speed affects the cleaning efficiency and equipment lifespan. The weight of ω3 can be increased to focus on constraining situations of sudden changes in rotational speed, so that the reward function can fit the core needs of different working conditions.

[0133] Please refer to it again. Figure 4 The reinforcement learning unit is implemented based on a deep deterministic policy gradient algorithm, which includes a decision network, a value evaluation network, a target decision network, and a target value evaluation network. The step of inputting the feature vector into the reinforcement learning unit of the deep learning model to generate control action commands adapted to the current working condition includes the following steps:

[0134] S1031. Input the time-series feature vector into the decision network, and the decision network performs policy mapping calculation on the feature vector to output the action vector used to drive the cleaning robot 100 to operate.

[0135] Optionally, the reinforcement learning unit is the core decision-making unit of the intelligent control module of the cleaning robot 100, and is implemented based on the Deep Deterministic Policy Gradient (DDPG) algorithm.

[0136] Optionally, the decision network can be understood as the action generation network of the DDPG algorithm, integrated into the reinforcement learning unit of the cleaning robot 100. It is used to receive the temporal feature vector representing the current working condition, and calculate and output continuous action vectors through nonlinear mapping, such as hydraulic flow adjustment amount, end effector speed adjustment amount, etc., to provide decision basis for the execution drive module. Its parameter optimization direction can be to improve the working condition adaptation accuracy of the action vector.

[0137] In this embodiment, the reinforcement learning unit is based on a deep deterministic policy gradient algorithm, which can handle continuous motion space decision-making and directly output motion vectors composed of continuous control parameters such as hydraulic flow adjustment, end effector speed adjustment, and robotic arm joint angle fine-tuning. This adapts to working conditions such as gradual changes in sludge thickness and dynamic changes in cleaning resistance during tank cleaning operations, making the motion adjustment of the tank cleaning robot 100 smoother and more precise, and enabling it to continuously adapt to the operational needs of different areas.

[0138] S1032. Input the temporal feature vector and the action vector into the value evaluation network. The value evaluation network evaluates the rationality of the current action vector in adapting to the current working condition and outputs the action value evaluation value. The action value evaluation value is used to assist in the parameter optimization of the strategy network and the value evaluation network.

[0139] Optionally, the value assessment network can be understood as an action effect evaluation network of the DDPG algorithm, used to quantify the rationality of the action vector output by the decision network. By inputting temporal feature vectors and action vectors, it outputs action value assessment values, providing gradient guidance for parameter optimization of the decision network and itself, and avoiding the generation of invalid or dangerous actions. The action value assessment value can be understood as quantifying the contribution of the action to the subsequent tank cleaning operation objective.

[0140] In this embodiment, the value assessment network, by inputting fused information of temporal feature vectors and action vectors, outputs a quantified action value assessment value. This provides clear guidance for the decision network and its own parameter optimization, making the parameter optimization direction more precise and enabling rapid iterative learning of the optimal control strategy under different working conditions. As iterative optimization progresses, the accuracy of the action vectors in adapting to real-time working conditions can be continuously improved, thereby effectively improving the efficiency of tank cleaning operations, while reducing the wear and tear on the hydraulic system and actuators, and extending the service life of the tank cleaning robot 100.

[0141] S1033. Through a preset soft update mechanism, the parameters of the decision network and the value assessment network are updated to the target decision network and the target value assessment network at a preset ratio according to a preset time period.

[0142] Optionally, the target decision network and the target value evaluation network can be understood as the target network used in the DDPG algorithm to ensure the stability of parameter optimization. Their network structures are consistent with the decision network and the value evaluation network, respectively. However, the parameter updates are not real-time. Instead, the main network parameters are periodically synchronized through a soft update mechanism to avoid sudden changes in action decisions caused by parameter mutations, thus ensuring the continuity and stability of the cleaning robot 100's operation.

[0143] Optionally, the soft update mechanism can be understood as the core mechanism in the DDPG algorithm for synchronizing the parameters of the decision network and value assessment network with those of the target network. By weighting the parameters of the decision network and value assessment network with those of the target network according to a preset ratio, the smooth update of the target network parameters is achieved, avoiding the impact of sudden parameter changes on the tank cleaning operation decision, thereby adapting to the stable operation requirements under the complex working conditions of the oil tank.

[0144] In this embodiment, the deep deterministic policy gradient algorithm quantifies the rationality of action vectors through a value evaluation network, effectively avoiding the output of invalid or dangerous actions. Simultaneously, a soft update mechanism ensures smooth parameter synchronization, preventing fluctuations in network parameter updates from affecting action decisions. This guarantees that the tank cleaning robot 100 can maintain a continuous and stable operating state in the high-risk environment of enclosed oil tanks and flammable / explosive conditions, reducing safety hazards caused by sudden action changes. Furthermore, the soft update mechanism is adjustable and can be flexibly adapted to the complexity of the operating conditions. For example, under complex conditions, the update coefficient of the soft update mechanism can be reduced to slow down the update pace and ensure stability; under stable conditions, the update coefficient can be increased to accelerate the model's learning iteration, improving the algorithm's flexibility in adapting to different operating conditions. Furthermore, the deep deterministic strategy gradient algorithm enables the tank cleaning robot 100 to autonomously adjust its control strategy based on real-time operating conditions without the need for manual intervention to adjust parameters in real time. This is particularly suitable for operation scenarios where the inside of the oil tank is small, high-risk, or difficult for humans to access, reducing manual maintenance and operation costs. Moreover, the algorithm's autonomous learning capability can be adapted to tank cleaning operations of different volumes and different levels of sludge contamination, improving the practicality and applicability of the control method.

[0145] Optionally, the step of performing strategy mapping calculation on the feature vector by the decision network and outputting the action vector for driving the cleaning robot 100 to operate includes: the decision network generating the action vector through forward propagation calculation, the action vector including the end effector speed adjustment amount and the hydraulic system flow adjustment amount, and the end effector speed adjustment amount being limited to a preset safety threshold range.

[0146] Optionally, the forward propagation calculation can be understood as the process in which the temporal feature vector enters from the network input layer, passes through the linear transformation and activation mapping of the hidden layer in sequence, and finally outputs the action vector from the output layer. This process has no parameter reverse update, and only completes the forward mapping from features to actions, ensuring the real-time generation of actions and adapting to the rapid response requirements of the dynamic working conditions of tank cleaning operations.

[0147] Optionally, the end effector speed adjustment can be understood as an incremental value output by the decision network for adjusting the speed of the drive motor of the cleaning robot 100. The magnitude of the incremental value is related to the cleaning efficiency and equipment load of the cleaning robot 100. The incremental value can be positive or negative. When the incremental value is positive, the drive motor is in an accelerated state; when the incremental value is negative, the drive motor is in a decelerated state.

[0148] Optionally, the hydraulic system is the power source for the tank cleaning robot 100. The flow adjustment of the hydraulic system can be understood as the incremental value used to adjust the output flow of hydraulic oil. By changing the flow, the driving force of the end effector can be indirectly controlled. The incremental value can be positive or negative.

[0149] In this embodiment, the decision network completes the mapping from feature vectors to action vectors through a forward propagation calculation process, avoiding the delay problem caused by complex calculations. It can accurately match the working conditions of gradual changes in sludge thickness and dynamic fluctuations in cleaning resistance during tank cleaning operations, ensuring that the robot can quickly respond to real-time changes in working conditions and output appropriate speed and flow adjustment commands in a timely manner. This avoids incomplete cleaning or equipment idling waste caused by decision lag, thereby improving the continuity and efficiency of the tank cleaning robot 100.

[0150] Optionally, the preset safety threshold can be understood as a physical safety constraint range set for the adjustment amount of the end effector speed, which is determined by the hardware limits of the cleaning robot 100 and the operational safety requirements, thereby preventing the decision network from outputting dangerous actions and ensuring the equipment safety and operational safety of the cleaning robot 100.

[0151] In this embodiment, a preset safety threshold is set for the speed adjustment of the end effector, thereby limiting excessive speed output and effectively avoiding problems such as equipment overload damage caused by excessive drive motor speed, oil sludge splashing and contamination of the tank environment, or safety hazards caused by mechanical vibration, thus improving the operational safety of the tank cleaning robot 100. Furthermore, the decision network synchronously regulates the end effector speed and hydraulic flow, ensuring that the end effector speed and hydraulic flow are well-matched, avoiding insufficient power due to high speed and low flow, and also avoiding power waste and component wear due to high flow and low speed. This effectively extends the service life of the tank cleaning robot 100 and reduces its operation and maintenance costs.

[0152] Optionally, optimizing the parameters of the reinforcement learning unit using the experience replay mechanism includes the following steps:

[0153] S1054. Store the experience data set into the experience replay pool of the reinforcement learning unit, wherein the experience data set includes the multi-source temporal state data, the action vector, the reward value, and the new temporal state data.

[0154] Optionally, the experience playback mechanism can be understood as the reinforcement learning unit of the tank cleaning robot 100 storing the state, action, reward and new state experience data in the operation process, and then randomly sampling batch data for network parameter updates. This can break the temporal correlation of data, improve the stability of parameter optimization, and enable the reinforcement learning unit to adapt to complex tank cleaning conditions.

[0155] Optionally, the experience data set can be understood as a complete data set of the entire process of decision-making, execution and feedback of a single operation of the cleaning robot 100, including multi-source time-series state data, action vectors, reward values ​​and new time-series state data.

[0156] S1055. When the amount of data in the experience replay pool reaches the preset sampling threshold, a batch of experience data is randomly sampled to update the network parameters of the reinforcement learning unit.

[0157] Optionally, the experience playback pool can be understood as a data cache unit in the reinforcement learning unit for storing experience data groups, used to accumulate sufficient operational experience data to provide data support for subsequent random sampling, and its storage capacity can be preset according to the complexity of the tank cleaning operation.

[0158] Optionally, the preset sampling threshold can be understood as the critical value of the amount of empirical data that triggers parameter updates. It is jointly determined by the convergence requirements of the reinforcement learning unit and the real-time requirements of the tank cleaning operation. When the amount of data in the experience playback pool reaches this threshold, sampling and parameter updates are initiated, thereby avoiding optimization deviations caused by insufficient data.

[0159] Optionally, the batch of empirical data can be understood as a set of empirical data randomly selected from the experience replay pool. By calculating the network loss function and updating the parameters through batch data, the impact of noise from a single set of data on optimization can be reduced, the stability of parameter updates can be improved, and optimization efficiency can be improved through batch computation.

[0160] In this embodiment, the experience data set encompasses multi-source temporal state data, motion vectors, reward values, and new temporal state data of the tank cleaning robot 100 during operation. This data accurately captures the correlation logic between the pre-action conditions, the executed action, the action's effect, and the post-action conditions. Parameter optimization is performed based on this experience data set, enabling the reinforcement learning unit to clearly perceive the actual impact of different actions on the tank cleaning process. This avoids optimization biases caused by missing data, ensures that parameter update directions align with the tank cleaning operation objectives, and improves the adaptation accuracy of the decision network and value assessment network parameters. Furthermore, the temporal state data of tank cleaning operations exhibits strong correlations, such as the gradual changing trends of continuously sampled sludge thickness and hydraulic pressure data. The experience playback mechanism, by storing experience data in a playback pool and then randomly sampling batch data for optimization, can break the temporal correlation of the data, enabling the batch data to cover diverse working conditions, such as working scenarios with different sludge thicknesses and different cleaning resistances. This avoids the network overfitting to local working conditions, ensuring that the reinforcement learning unit can adapt to the complex and ever-changing working environment inside the tank. Moreover, the reinforcement learning unit can learn more comprehensive tank cleaning operation control rules, rather than being limited to the current real-time working conditions, thereby improving the adaptability of the reinforcement learning unit.

[0161] Please refer to it again. Figure 4 The updating of the network parameters of the reinforcement learning unit further includes the following steps:

[0162] S105a. Calculate the target value and the action value assessment value output by the value assessment network for each sample.

[0163] Optionally, the target value can be understood as a reference value used in the reinforcement learning unit to characterize the value of the sample action and working condition association. It is calculated by combining the immediate reward of the tank cleaning operation with the potential value of future working conditions and serves as a benchmark for judging the deviation of the value assessment network output.

[0164] Optionally, the action value assessment value can be understood as the quantitative output result of the value assessment network on the adaptability of 100 sample actions of the tank cleaning robot.

[0165] S105b. Calculate the loss value of the value assessment network using a preset loss function, and update the parameters of the value assessment network using a backpropagation algorithm until the loss value of the value assessment network converges to a preset threshold. The expression of the loss function is: ;

[0166] Where L is the loss value, N is the number of samples in the batch training, t is the ordinal index of the sample in the batch training, and its value is a positive integer from 1 to N, y i Let i be the target value of the i-th sample. Let be the action value assessment value of the i-th sample by the value assessment network.

[0167] Among them, s i For the tank cleaning operation time sequence status data corresponding to the sample, θ Q This is the current parameter of the value assessment network. This value is used to compare with the target value and measure the accuracy of the value assessment network in judging the value of the tank cleaning action.

[0168] Optionally, the preset loss function can be understood as a mathematical function used to quantify the deviation between the target value and the action value assessment value. The mean deviation of the batch samples is calculated in the form of mean square error to ensure that the loss value can intuitively reflect the network assessment accuracy and provide clear gradient guidance for backpropagation.

[0169] Optionally, the backpropagation algorithm can be understood as an algorithm for updating the parameters of the value assessment network of the cleaning robot 100. By calculating the gradient of the loss function with respect to the parameters of each layer of the network, the parameters are adjusted in the reverse direction of the output layer, hidden layer, and input layer to gradually reduce the loss value and improve the accuracy of the network in assessing the value of the cleaning action.

[0170] Optionally, the preset threshold can be understood as the termination criterion for updating the value assessment network parameters. It is set in combination with the real-time requirements of tank cleaning operations and the requirements for assessment accuracy. When the loss value converges to or below the threshold, the parameter update is stopped to ensure that the network parameters are in a stable state.

[0171] Optionally, the number of samples for batch training can be understood as the number of tank cleaning operation experience sample groups randomly sampled from the experience replay pool. By calculating the loss value through batch samples, the impact of noise from a single sample group on parameter updates can be reduced, thereby improving update stability and efficiency.

[0172] In this embodiment, a preset loss function in the form of mean squared error is used. By calculating the squared mean of the difference between the target value and the action value assessment value, the judgment bias of the value assessment network on the tank cleaning operation action can be accurately quantified. This quantification method can effectively amplify key biases, such as the assessment bias of action adaptability under thick sludge conditions, providing gradient guidance for parameter updates. It solves the problem of unclear optimization direction caused by the fuzziness of bias in traditional assessment networks, enabling the value assessment network to more accurately match the assessment needs of the tank cleaning robot 100 under different operating conditions.

[0173] Optionally, the step of inputting the temporal feature vector into the reinforcement learning unit, and the reinforcement learning unit generating an action vector adapted to the current working condition, further includes the following steps:

[0174] S103a. Call the preset multi-field coupled dynamics model of the cleaning robot. The multi-field coupled dynamics model pre-integrates the coupling relationship between hydraulic drive characteristics, mechanical transmission law and cleaning load.

[0175] Optionally, the multi-field coupled dynamics model can be understood as a physical simulation model for the cleaning robot 100. It pre-integrates the coupling relationship between the hydraulic drive characteristics, mechanical transmission law and cleaning load during the operation of the cleaning robot 100, and replicates the physical operation law of the cleaning robot 100. This provides a physical basis for the anomaly removal of time-series feature vectors and compliance verification of action parameters, ensuring the safety and rationality of action generation.

[0176] S103b. Input the time-series feature vector into the multi-field coupled dynamics model, and through the physical law verification logic built into the model, remove abnormal dimensions in the time-series feature vector and output an effective time-series feature vector that conforms to the physical operation law.

[0177] Optionally, the effective temporal feature vector can be understood as the compliant feature vector with abnormal dimensions removed after the temporal feature vector is verified by the physical laws of the multi-field coupled dynamics model, so as to ensure that the data input to the reinforcement learning unit conforms to the actual operation logic of the cleaning robot 100 and avoids erroneous decisions caused by abnormal data.

[0178] S103c: Control the reinforcement learning unit to receive the effective temporal feature vector and generate a preliminary action vector.

[0179] Optionally, the preliminary motion vector can be understood as a set of motion control parameters initially generated by the reinforcement learning unit based on the effective temporal feature vector, including hydraulic flow adjustment and end effector speed adjustment, etc.

[0180] S103d. Input the preliminary action vector into the multi-field coupled dynamics model, verify whether each action parameter in the preliminary action vector exceeds the preset operating range of the equipment, perform gradient correction on the action parameters that do not meet the constraints, and obtain a compliant action vector.

[0181] Optionally, the preset operating range of the device can be understood as a range of action parameter constraints set by combining the hardware parameters of the cleaning robot 100 with the operational safety requirements, used to determine whether the action parameters are compliant, and to avoid damage to the device or operational hazards caused by parameters exceeding the standard.

[0182] Optionally, the gradient correction can be understood as a correction method that smoothly adjusts action parameters that exceed the preset operating range based on the physical laws of the multi-field coupled dynamics model. The abnormal parameters are gradually corrected to the compliant range through gradient descent logic, so as to avoid the instability of the cleaning robot 100 operation caused by sudden parameter changes and ensure the stability of the cleaning robot 100 action execution process.

[0183] S103e. The action vector that has passed the constraint verification is output as the action vector adapted to the current working condition.

[0184] In this embodiment, by invoking a multi-field coupled dynamics model that pre-integrates the coupling relationship between hydraulic drive characteristics, mechanical transmission laws, and cleaning load, the verification of time-series feature vectors is provided with reliable physical theoretical support. Compared to traditional simple anomaly removal methods based on thresholds, the multi-field coupled dynamics model can accurately identify hidden anomalies in tank cleaning operations that violate physical operating laws, such as mismatch between hydraulic pressure and flow rate, and imbalance between rotational speed and load. By outputting effective time-series feature vectors that conform to physical logic, it ensures that the data input to the reinforcement learning unit truly reflects the robot's actual operating state, avoids erroneous decisions caused by abnormal data, improves the reliability of the reinforcement learning unit's action generation, and can adapt to the precise decision-making requirements under complex working conditions inside the oil tank, preventing the reinforcement learning unit from generating dangerous actions based on erroneous working conditions.

[0185] Please see Figure 6This application also provides a can cleaning robot, including: a memory 202, a processor 201, and a computer program stored in the memory 202 and executable on the processor 201. When the processor 201 executes the computer program, it implements the steps in the control method of the can cleaning robot provided in the embodiments of the present invention.

[0186] Optionally, the hardware structure of the tank cleaning robot 100 further includes a sensing and detection unit, an execution and drive unit, a communication unit, and a power management unit, used to realize functions such as tank cleaning operation status data acquisition, motion drive control, data interaction transmission, and stable power supply. The sensing and detection unit includes a pressure sensor, a flow sensor, a speed sensor, a sludge thickness sensor, and an inertial measurement unit, used to perform the acquisition function of multi-source time-series status data and new time-series status data in the control method of this invention. The execution and drive unit includes a hydraulic drive module, a motor drive module, and a servo control module, used to receive the motion vector output by the processor 201, convert it into drive signals to control the mechanical body to complete the tank cleaning operation.

[0187] When processor 201 runs the computer program for the control method of the tank cleaning robot 100 stored in memory 202, it specifically performs the following steps:

[0188] Multi-source time-series status data during the operation of the cleaning robot 100 is collected by the sensing and detection module. The multi-source time-series status data includes the power system parameters of the cleaning robot 100.

[0189] The multi-source time-series state data is input into the time-series feature extraction unit, and the time-series feature extraction unit outputs a time-series feature vector characterizing the dynamic operation characteristics of the tank cleaning robot 100.

[0190] The temporal feature vector is input into the reinforcement learning unit, and the reinforcement learning unit generates an action vector that adapts to the current working condition.

[0191] The execution drive module receives the action vector, converts the action vector into a drive control signal adapted to the mechanical body module, and drives the mechanical body module to complete the tank cleaning operation.

[0192] The sensor detection module collects new time-series state data after the tank cleaning operation is performed, calculates the reward value of this operation based on a preset reward function, and optimizes the parameters of the reinforcement learning unit using an experience playback mechanism.

[0193] Optionally, the temporal feature extraction unit is a long short-term memory network, and the method executed by the processor 201 further includes:

[0194] The power system parameters, motion parameters, and working environment parameters of the cleaning robot 100 are collected by the sensing and detection module as the multi-source time-series state data. The power system parameters include the pressure value and / or flow rate value of the hydraulic system. The motion parameters include at least one of the moving speed of the cleaning robot 100, the joint angle of the robotic arm, the rotation speed of the end effector, and the rotation speed change rate. The working environment parameters include the sludge thickness and / or cleaning resistance of the cleaning area.

[0195] The multi-source temporal state data at the current moment and the hidden state at the previous moment are input into the long short-term memory network. The current moment is the sampling moment corresponding to the current multi-source temporal state data acquisition by the sensing and detection module, and the previous moment is the previous sensing sampling moment immediately adjacent to the current moment.

[0196] The Long Short-Term Memory (LSTM) network is used to filter temporal features, update the memory state, and output the hidden state at the current moment as the temporal feature vector.

[0197] Optionally, the Long Short-Term Memory (LSTM) network has a forgetting gate module, an input gate module, and an output gate module. The process executed by the processor 201, which involves filtering temporal features through the LTM network, updating the memory state, and outputting the current hidden state as the temporal feature vector, includes:

[0198] The forgetting gate module is used to calculate the forgetting gate vector in order to filter effective features from historical memory states.

[0199] The input gate module calculates the input gating vector and candidate memory states to filter the effective features of the currently input temporal state vector and generate candidate update content.

[0200] The memory state at the current moment is calculated by fusing historical valid features and current candidate features through the update module of the Long Short-Term Memory Network.

[0201] The output gate vector is calculated by the output gate module, and the hidden state at the current time is generated based on the memory state at the current time as the temporal feature vector.

[0202] Optionally, the processor 201 executes the calculation of the reward value for this action based on a preset reward function, including:

[0203] Parameters are extracted from the new time-series state vector, including the pressure value of the hydraulic system, the flow value of the hydraulic system, and the speed of the end effector.

[0204] Evaluation indicators are calculated based on the extracted parameters. These evaluation indicators include the deviation of the hydraulic system pressure value from the preset optimal pressure, the deviation of the hydraulic system real-time flow rate from the preset optimal flow rate, and the rate of change of the end effector speed.

[0205] The reward value is calculated using a reward function based on the evaluation indicators.

[0206] Optionally, the reinforcement learning unit is implemented based on a deep deterministic policy gradient algorithm, which includes a decision network, a value evaluation network, a target decision network, and a target value evaluation network. The reinforcement learning unit, which inputs the feature vector into the deep learning model and is executed by the processor 201, generates control action instructions adapted to the current working condition, including:

[0207] The time-series feature vector is input into the decision network, which performs policy mapping calculation on the feature vector and outputs the action vector used to drive the cleaning robot 100 to operate.

[0208] The temporal feature vector and the action vector are input into the value evaluation network. The value evaluation network evaluates the rationality of the current action vector in adapting to the current working condition and outputs the action value evaluation value. The action value evaluation value is used to assist in the optimization of the parameters of the strategy network and the value evaluation network.

[0209] Through a preset soft update mechanism, the parameters of the decision network and the value assessment network are updated to the target decision network and the target value assessment network at a preset ratio according to a preset time period.

[0210] Optionally, the process executed by the processor 201, which calculates the policy mapping of the feature vectors by the decision network and outputs the action vectors for driving the cleaning robot 100, includes:

[0211] The decision network generates an action vector through forward propagation calculation. The action vector includes the end effector speed adjustment and the hydraulic system flow adjustment, and the end effector speed adjustment is limited to a preset safety threshold range.

[0212] Optionally, the processor 201 optimizes the parameters of the reinforcement learning unit using an experience replay mechanism, including:

[0213] The experience data set is stored in the experience replay pool of the reinforcement learning unit, wherein the experience data set includes the multi-source temporal state data, the action vector, the reward value and the new temporal state data;

[0214] When the amount of data in the experience replay pool reaches a preset sampling threshold, a batch of experience data is randomly sampled to update the network parameters of the reinforcement learning unit.

[0215] Optionally, the updating of the network parameters of the reinforcement learning unit performed by the processor 201 further includes:

[0216] Calculate the target value and the action value output by the value assessment network for each sample;

[0217] The loss value of the value assessment network is calculated using a preset loss function. The parameters of the value assessment network are updated using a backpropagation algorithm until all loss values ​​converge to a preset threshold. The expression for the loss function is: ;

[0218] Where L is the loss value, N is the number of samples in the batch training, t is the ordinal index of the sample in the batch training, and its value is a positive integer from 1 to N, y i Let i be the target value of the i-th sample. Let be the action value assessment value of the i-th sample by the value assessment network.

[0219] Optionally, the processor 201 further includes inputting the temporal feature vector into the reinforcement learning unit, wherein the reinforcement learning unit generates an action vector adapted to the current job condition.

[0220] The system invokes a pre-defined multi-field coupled dynamics model of the cleaning robot, which integrates the coupling relationship between hydraulic drive characteristics, mechanical transmission laws, and cleaning load.

[0221] The time-series feature vector is input into the multi-field coupled dynamics model. Through the physical law verification logic built into the model, abnormal dimensions in the time-series feature vector are removed, and an effective time-series feature vector that conforms to the physical operation law is output.

[0222] The reinforcement learning unit is controlled to receive the effective temporal feature vector and generate a preliminary action vector.

[0223] The preliminary motion vector is input into the multi-field coupled dynamics model to verify whether each motion parameter in the preliminary motion vector exceeds the preset operating range of the equipment. The motion parameters that do not meet the constraints are corrected by gradient to obtain a compliant motion vector.

[0224] The action vector that passes the constraint verification will be output as the action vector adapted to the current working condition.

[0225] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by the processor 201, it implements the various processes of the control method for the cleaning robot 100 provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0226] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0227] In this application, the terms "embodiment" and "implementation" mean that a specific feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of this application. The appearance of these phrases in various locations throughout the specification does not necessarily refer to the same embodiment, nor are they independent or alternative embodiments mutually exclusive with other embodiments. Those skilled in the art will understand, explicitly and implicitly, that the embodiments described in this application can be combined with other embodiments. Furthermore, it should be understood that the features, structures, or characteristics described in the various embodiments of this application can be arbitrarily combined to form another embodiment that does not depart from the spirit and scope of the technical solution of this application, provided there is no contradiction between them.

[0228] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the above preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application should not depart from the spirit and scope of the technical solutions of this application.

Claims

1. A control method for a tank cleaning robot, characterized in that, A method applicable to a tank cleaning robot comprising a mechanical body module, a sensing and detection module, an intelligent control module, and an execution and drive module, wherein the intelligent control module integrates a temporal feature extraction unit and a reinforcement learning unit, the method comprising: Multi-source time-series status data during the operation of the cleaning robot is collected by the sensing and detection module. The multi-source time-series status data includes the power system parameters of the cleaning robot. The multi-source time-series state data is input into the time-series feature extraction unit, and the time-series feature extraction unit outputs a time-series feature vector characterizing the dynamic operation characteristics of the tank cleaning robot. The temporal feature vector is input into the reinforcement learning unit, and the reinforcement learning unit generates an action vector that adapts to the current working condition. The execution drive module receives the action vector, converts the action vector into a drive control signal adapted to the mechanical body module, and drives the mechanical body module to complete the tank cleaning operation. The sensor detection module collects new time-series state data after the tank cleaning operation is executed, calculates the reward value of this operation based on the preset reward function, and optimizes the parameters of the reinforcement learning unit using the experience playback mechanism. The temporal feature extraction unit is a long short-term memory network. The process of inputting the multi-source temporal state data into the temporal feature extraction unit and outputting a temporal feature vector characterizing the dynamic operation of the cleaning robot through the temporal feature extraction unit includes: inputting the multi-source temporal state data at the current moment and the hidden state at the previous moment into the long short-term memory network; the current moment being the sampling moment corresponding to the current multi-source temporal state data acquisition by the sensing module, and the previous moment being the immediate preceding sensing sampling moment; the long short-term memory network then filters the temporal features, updates the memory state, and outputs the hidden state at the current moment as the temporal feature vector. The process of inputting the temporal feature vector into the reinforcement learning unit and generating an action vector adapted to the current working condition by the reinforcement learning unit includes: calling a preset multi-field coupled dynamics model of the cleaning robot, wherein the multi-field coupled dynamics model pre-integrates the coupling relationship between hydraulic drive characteristics, mechanical transmission laws and cleaning load; inputting the temporal feature vector into the multi-field coupled dynamics model, and using the physical law verification logic built into the model to remove abnormal dimensions in the temporal feature vector and output a valid temporal feature vector that conforms to the physical operating laws; controlling the reinforcement learning unit to receive the valid temporal feature vector and generate a preliminary action vector; inputting the preliminary action vector into the multi-field coupled dynamics model, verifying whether each action parameter in the preliminary action vector exceeds the preset operating range of the equipment, performing gradient correction on action parameters that do not meet the constraints, and obtaining a compliant action vector, wherein the gradient correction is a correction method based on the physical laws of the multi-field coupled dynamics model to smoothly adjust action parameters that exceed the preset operating range, and gradually correcting abnormal parameters to the compliant range through gradient descent logic; and outputting the action vector that passes the constraint verification as the action vector adapted to the current working condition.

2. The control method for the tank cleaning robot as described in claim 1, characterized in that, The process of collecting multi-source time-series state data during the operation of the cleaning robot through the sensing and detection module includes collecting power system parameters, motion parameters, and operating environment parameters of the cleaning robot through the sensing and detection module as the multi-source time-series state data. The power system parameters include the pressure value and / or flow rate value of the hydraulic system. The motion parameters include at least one of the moving speed of the cleaning robot, the joint angle of the robotic arm, the rotation speed of the end effector, and the rate of change of rotation speed. The operating environment parameters include the thickness of the sludge in the cleaning area and / or the cleaning resistance.

3. The control method for the tank cleaning robot as described in claim 2, characterized in that, The Long Short-Term Memory (LSTM) network has a forgetting gate module, an input gate module, and an output gate module. The process of filtering temporal features through the LTM network, updating the memory state, and outputting the current hidden state as the temporal feature vector includes: The forgetting gate module is used to calculate the forgetting gate vector in order to filter effective features from historical memory states. The input gate module calculates the input gating vector and candidate memory states to filter the effective features of the currently input temporal state vector and generate candidate update content. The memory state at the current moment is calculated by fusing historical valid features and current candidate features through the update module of the Long Short-Term Memory Network. The output gate vector is calculated by the output gate module, and the hidden state at the current time is generated based on the memory state at the current time as the temporal feature vector.

4. The control method for the tank cleaning robot as described in claim 1, characterized in that, The calculation of the reward value for this action based on a preset reward function includes: Parameters are extracted from the new time-series state vector, including the pressure value of the hydraulic system, the flow value of the hydraulic system, and the speed of the end effector. Evaluation indicators are calculated based on the extracted parameters. These evaluation indicators include the deviation of the hydraulic system pressure value from the preset optimal pressure, the deviation of the hydraulic system real-time flow rate from the preset optimal flow rate, and the rate of change of the end effector speed. The reward value is calculated using a reward function based on the evaluation indicators.

5. The control method for the tank cleaning robot as described in claim 1, characterized in that, The reinforcement learning unit is implemented based on a deep deterministic policy gradient algorithm, which includes a decision network, a value assessment network, a target decision network, and a target value assessment network. The reinforcement learning unit, which inputs the feature vector into the deep learning model, generates control action instructions adapted to the current working condition, including: The time-series feature vector is input into the decision network, which performs policy mapping calculation on the feature vector and outputs the action vector used to drive the cleaning robot's operation. The temporal feature vector and the action vector are input into the value evaluation network. The value evaluation network evaluates the rationality of the current action vector in adapting to the current working condition and outputs the action value evaluation value. The action value evaluation value is used to assist in the optimization of the parameters of the strategy network and the value evaluation network. Through a preset soft update mechanism, the parameters of the decision network and the value assessment network are updated to the target decision network and the target value assessment network at a preset ratio according to a preset time period.

6. The control method for the tank cleaning robot as described in claim 5, characterized in that, The step of calculating the policy mapping of the feature vector by the decision network and outputting the action vector used to drive the cleaning robot operation includes: The decision network generates an action vector through forward propagation calculation. The action vector includes the end effector speed adjustment and the hydraulic system flow adjustment, and the end effector speed adjustment is limited to a preset safety threshold range.

7. The control method for the tank cleaning robot as described in claim 5, characterized in that, The optimization of the parameters of the reinforcement learning unit using the experience replay mechanism includes: The experience data set is stored in the experience replay pool of the reinforcement learning unit, wherein the experience data set includes the multi-source temporal state data, the action vector, the reward value and the new temporal state data; When the amount of data in the experience replay pool reaches a preset sampling threshold, a batch of experience data is randomly sampled to update the network parameters of the reinforcement learning unit.

8. The control method for the tank cleaning robot as described in claim 7, characterized in that, The update of the network parameters of the reinforcement learning unit further includes: Calculate the target value and the action value output by the value assessment network for each sample; The loss value of the value assessment network is calculated using a loss function, and the parameters of the value assessment network are updated using a backpropagation algorithm until the loss value of the value assessment network converges to a preset threshold. The expression of the loss function is as follows: ； Where L is the loss value, N is the number of samples in the batch training, t is the ordinal index of the sample in the batch training, and its value is a positive integer from 1 to N, y i Let i be the target value of the i-th sample. Let be the action value assessment value of the i-th sample by the value assessment network.

9. A tank cleaning robot, characterized in that, include: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the control method for the cleaning robot as described in any one of claims 1 to 8.

Images